openvibers/openvibe.publishing README.md
README.md
| 1 | # OpenVibe.Publishing |
| 2 | |
| 3 | > Shared publishing packages for the OpenVibe publication products: Wiki, Blog, News, Reviews, |
| 4 | > Deals, Coupons and Trade. |
| 5 | |
| 6 | **Docs:** https://openvibe.services · **For AI:** https://openvibe.network/llms.txt |
| 7 | **Install and test:** `npm ci && npm test` (Node 20 or newer) |
| 8 | |
| 9 | **Status:** alpha, **1.7.1**: the stores run on PostgreSQL through the `openvibe-sdk/db` async data |
| 10 | layer (ADR-035: PostgreSQL 18 behind PgBouncer in production, PGlite in tests). Every store method is |
| 11 | async, takes the product's `openvibe-sdk/db` handle, and accepts the caller's transaction handle; each |
| 12 | store gives its DDL as `schema(prefix)` for the product's migrations. v1.1.0 adds the shared ingest |
| 13 | chassis (`openvibe-publishing/ingest`) and publication glue (`openvibe-publishing/publication`) — |
| 14 | additive, no existing export changed. v1.2.0 adds `openvibe-publishing/layout`, the page document the |
| 15 | seven sites render through `openvibe-shared/shell` (additive). v1.3.0 forwards the AI summary, facts, |
| 16 | `updated` and `url` through `layout.renderDocument` to the shell (additive). v1.4.0 adds `openvibe-publishing/search-feed`, |
| 17 | which puts a record product's pages in OpenVibe.Search (additive). v1.4.1 keeps a request path that starts with |
| 18 | several slashes (`//feed/`) on the site in `seo.canonicalUrl`, where it threw (a 500 on News, Deals and Blog). v1.5.0 adds |
| 19 | `openvibe-publishing/search-relay`, which delivers the Search documents a product queued in its own outbox table |
| 20 | through Search's owner API (Meme, Exchange and Fun queue them; additive). The exit proof (`examples/two-products`) runs in `npm test`. |
| 21 | v1.6.0 adds `openvibe-publishing/blocks`, the rich block model Blog and Wiki share: a body is Markdown plus |
| 22 | blocks, parsed, validated, rendered server-side and flattened to text (additive). Releases v0.1.0 to v1.7.1 are tagged (see [CHANGELOG.md](CHANGELOG.md)). Fifteen products pin a release. The seven |
| 23 | publication products run on PostgreSQL and are public: openvibe.wiki, openvibe.blog, openvibe.news, openvibe.reviews, |
| 24 | openvibe.deals, openvibe.coupons and openvibe.trade (on 2026-10-22 Blog, News and Deals pinned v1.4.1, the other four |
| 25 | v1.3.0). Seven more use `search-feed` to put their pages in OpenVibe.Search: Work, Rent, Help, Inventory, Quest, Host and |
| 26 | OpenVibe.Games (its creator worlds); Food uses the AI draft client. Each adopts a release by pinning its tag, and |
| 27 | OpenVibe.Contracts' `docs/ESTATE.md` lists every pin. |
| 28 | **Package:** `openvibe-publishing` (CommonJS, Node ≥ 20, production runs Node 22). |
| 29 | **License:** MIT, like OpenVibe.Shared. |
| 30 | |
| 31 | ## Purpose |
| 32 | |
| 33 | One implementation of how the publication products publish: revisions, scheduling, taxonomy, |
| 34 | citations, media references, discussion references, the indexability gate, feeds and sitemaps, |
| 35 | authorship and AI disclosure, Search documents and server-rendered HTML. Each product keeps its own |
| 36 | state and rules; this repository keeps the code they share. |
| 37 | |
| 38 | ## Owns |
| 39 | |
| 40 | - the modules below and their table layouts (the DDL each gives as `schema(prefix)`), with the |
| 41 | product's prefix, inside the product's own database |
| 42 | - the indexability gate's rules and its stable reason codes (`seo.REASONS`) |
| 43 | - the shape of the documents and envelopes the products send to OpenVibe.Search and OpenVibe.Events |
| 44 | |
| 45 | It owns no state: no runtime, port, domain or database of its own (next section). |
| 46 | |
| 47 | ## The rule: packages, not an authority |
| 48 | |
| 49 | This repository is **code, never state**. It has no runtime, no port, no domain, no database and |
| 50 | no API of its own, and it must never get one. Each product (Wiki, Blog, News, …) owns its |
| 51 | publication state in its own database; these modules take the product's `openvibe-sdk/db` handle |
| 52 | and a table prefix, and use `<prefix>_…` tables *inside that product's database* (the product's |
| 53 | migrations create them from `schema(prefix)`). Two products share the code, not the data: the wiki's |
| 54 | database holds `wiki_*` tables, the blog's holds `blog_*` tables, and nothing is shared between them |
| 55 | (proved by [test/two-products.test.js](test/two-products.test.js)). |
| 56 | |
| 57 | Following the plan (§12.4, §31.1, §31.5): share *how* something is done (revisions, the gate, |
| 58 | feeds, indexing); never share *what is true* (which revision is live, who may read it, editorial |
| 59 | rules). A generic `OpenVibe.Content` authority is explicitly not what this is. |
| 60 | |
| 61 | ## Module map |
| 62 | |
| 63 | Every module is its own entry point and can be used alone. |
| 64 | |
| 65 | | Entry point | What it does | Tables (in the consumer's DB) | |
| 66 | |---|---|---| |
| 67 | | `openvibe-publishing/revisions` | Drafts, immutable revisions with parent pointers, optimistic concurrency (`expectedRevision` → 412 `revision.conflict`), line/word diff, revert as a new revision, audited purge | `<prefix>_revisions`, `<prefix>_drafts`, `<prefix>_revision_purges` | |
| 68 | | `openvibe-publishing/schedule` | Scheduled publish/unpublish jobs: idempotent `schedule()`, lease-based `claim()`, re-run safe across worker restarts, retries with backoff, injectable clock | `<prefix>_schedule_jobs` | |
| 69 | | `openvibe-publishing/taxonomy` | Tags and terms with slugs, hierarchical categories (ancestors for breadcrumbs, descendants, trees, cycle checks) | `<prefix>_terms`, `<prefix>_term_links` | |
| 70 | | `openvibe-publishing/citations` | Source references on one revision: Sources item id and/or URL, `retrieved_at`, quote span, license note; append-only, never dropped by later revisions; `carryForward` to reuse | `<prefix>_citations`, `<prefix>_citation_purges` | |
| 71 | | `openvibe-publishing/media` | Attachments by Media object id (`med_…` / `legacy:…`), explicit `broken` state when the object is gone, `verify()` that never flips state on an outage, `<figure>` rendering with an honest placeholder | `<prefix>_attachments` | |
| 72 | | `openvibe-publishing/discussion` | Community comment-thread references: resolves via `POST /api/v1/comments/threads/resolve`, stores the thread id only (no comment content) | `<prefix>_discussion_refs` | |
| 73 | | `openvibe-publishing/seo` | The deterministic **indexability gate**; canonical URLs; history-aware redirects (old slug → 301, gone → 410); meta/robots tags; sitemaps; RSS, Atom and JSON Feed; JSON-LD built only from provided fields | `<prefix>_redirects` | |
| 74 | | `openvibe-publishing/authorship` | human / ai / hybrid / imported records with the OpenVibe.AI workflow + run id, disclosure labels, AI content held as draft + noindex until a person's review | `<prefix>_reviews` | |
| 75 | | `openvibe-publishing/index-hooks` | `search.index-document@1` documents and tombstones, the `<owner>.index_document.upserted\|deleted` events OpenVibe.Search consumes, a monotonic index-revision sequencer, and the product's own `<product>.<type>.published\|updated\|unpublished\|deleted` events | `<prefix>_index_revisions` | |
| 76 | | `openvibe-publishing/ai` | Ask OpenVibe.AI for a draft: run a registered workflow with the product's service token (polls a slow run), read its citations, get the `{ id, version, runId, model }` an AI authorship record needs; coded `AiRunError`s, never a partial draft | — | |
| 77 | | `openvibe-publishing/ssr` | auto-escaping `html` tagged templates with `raw()`, a safe Markdown subset, plain-text extraction, word count, server pagination, breadcrumbs, diff markup, honest `<time>` | — | |
| 78 | | `openvibe-publishing/blocks` | The rich block model Blog and Wiki share: a body is Markdown plus blocks (`:::name{attr="value"}` … `:::`); `parse`, `validate`, `render` (server-side HTML, every block a plain-HTML form, images as `<img>` with width/height/alt and lazy loading after the first, the credit line of the image) and `toText` for feeds, search and AI readers | — | |
| 79 | | `openvibe-publishing/ingest` | The shared ingest chassis: the OpenVibe.Sources client, a named change cursor and the `pullChanges` loop (per-item savepoint isolation, `{applied\|hold\|removed}`), the signed event consumer with an exactly-once inbox, the shared normalisers, and the generic PSL/registrable-host and freshness helpers | `<prefix>_ingest_cursor` | |
| 80 | | `openvibe-publishing/publication` | The shared publication glue: gate → document → `sequencer.stamp` → events → `outbox.enqueue` on the caller's transaction handle, tombstones on unpublish/merge, and the IndexNow ping; it emits exactly `search.index-document@1` and the product's events | — | |
| 81 | | `openvibe-publishing/search-feed` | Record pages (listings, job ads, help articles, catalog items) in OpenVibe.Search: `sync`/`remove` on the write's transaction, and a `sweep()` that re-sends only what changed and tombstones vanished rows | — | |
| 82 | | `openvibe-publishing/search-relay` | Deliver a product's queued Search documents (its `<product>_outbox` rows: one `search.index-document@1` or tombstone each) through Search's owner API with the product's service token: oldest first, older rows of a document superseded, a stale revision done, a conflict sent again one revision up, 5xx/network backoff, 429 pauses until Retry-After, other refusals logged and set aside; at most 100 a tick | — (the product's outbox) | |
| 83 | | `openvibe-publishing/layout` | `renderDocument(o)`: the whole page through `openvibe-shared/shell` `page()` (≥ 2.6.0), with robots and canonical from the gate's decision (throws without one), article times, prev/next, feeds, the app icon, stylesheets, the boost marker, the body frame and the footer init | — | |
| 84 | |
| 85 | `require('openvibe-publishing')` exposes all of them lazily (`.revisions`, `.seo`, `.indexHooks`, |
| 86 | `.ingest`, `.publication`, `.searchFeed`, `.searchRelay`, `.layout`, …), plus `schema({ … })`, the DDL of several stores at once (below). |
| 87 | |
| 88 | ## Install |
| 89 | |
| 90 | Pin the release tarball, like every OpenVibe package (never a `file:` link or a vendored copy): |
| 91 | |
| 92 | ```json |
| 93 | "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.4.0", |
| 94 | "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.60.0", |
| 95 | "pg": "^8.23.0" |
| 96 | ``` |
| 97 | |
| 98 | and `@electric-sql/pglite` (`^0.5.8`) as a devDependency for tests. `openvibe-sdk` (≥ 0.15.0) and |
| 99 | `openvibe-shared` (≥ 1.5.0) are peer dependencies, resolved from the product's own install: the |
| 100 | product brings its own database handle, so there is one copy of the data layer (and of its pool) per |
| 101 | service, and one copy of openvibe-shared. openvibe-shared's `seo` module is the network's single |
| 102 | implementation of head tags, sitemap XML, robots.txt and JSON-LD escaping; this package adds the |
| 103 | publication rules on top. There are no other runtime dependencies; `lib/` never loads a driver. |
| 104 | |
| 105 | ## Using the stores on PostgreSQL |
| 106 | |
| 107 | **The handle.** Every store takes the product's `openvibe-sdk/db` handle and a prefix. Creating a |
| 108 | store runs no query; every method that touches data is `async`. |
| 109 | |
| 110 | ```js |
| 111 | const { createDb, sql } = require('openvibe-sdk/db'); |
| 112 | const { createRevisionStore } = require('openvibe-publishing/revisions'); |
| 113 | const { createCitationStore } = require('openvibe-publishing/citations'); |
| 114 | |
| 115 | const db = createDb({ service: 'wiki' }); // DATABASE_URL, through PgBouncer |
| 116 | const revisions = createRevisionStore(db, { prefix: 'wiki_page' }); |
| 117 | const citations = createCitationStore(db, { prefix: 'wiki', revisions }); |
| 118 | const { revision } = await revisions.create({ entityId: 'pg_1', expectedRevision: 0, content: '# Rye', fields: { title: 'Rye' } }); |
| 119 | ``` |
| 120 | |
| 121 | **The schema goes in the product's migrations.** The runtime role behind PgBouncer cannot create |
| 122 | tables, so the stores never do it on their own. Each storing module exports its DDL for a prefix, and |
| 123 | the root export joins several; write the text into the product's first migration once: |
| 124 | |
| 125 | ```js |
| 126 | const publishing = require('openvibe-publishing'); |
| 127 | const ddl = publishing.schema({ revisions: 'wiki_page', citations: 'wiki', seo: 'wiki_page', authorship: 'wiki_page', indexHooks: 'wiki' }); |
| 128 | // migrations/0001_initial.sql = '-- phase: expand |
| 129 | ' + the product's own tables + ddl |
| 130 | // then at boot, as the owner: await createDb({ url: process.env.DATABASE_DIRECT_URL }).migrate({ dir }) |
| 131 | ``` |
| 132 | |
| 133 | | Module | DDL function | Tables | |
| 134 | |---|---|---| |
| 135 | | revisions | `schema(prefix)` | `<prefix>_revisions`, `<prefix>_drafts`, `<prefix>_revision_purges` + guard trigger | |
| 136 | | citations | `schema(prefix)` | `<prefix>_citations`, `<prefix>_citation_purges` + guard trigger | |
| 137 | | media | `schema(prefix)` | `<prefix>_attachments` | |
| 138 | | discussion | `schema(prefix)` | `<prefix>_discussion_refs` | |
| 139 | | schedule | `schema(prefix)` | `<prefix>_schedule_jobs` | |
| 140 | | authorship | `schema(prefix)` (the review log) | `<prefix>_reviews` + guard trigger | |
| 141 | | taxonomy | `schema(prefix)` | `<prefix>_terms`, `<prefix>_term_links` | |
| 142 | | seo | `redirectsSchema(prefix)` | `<prefix>_redirects` | |
| 143 | | index-hooks | `sequencerSchema(prefix)` | `<prefix>_index_revisions` | |
| 144 | | ingest | `schema(prefix)` | `<prefix>_ingest_cursor` | |
| 145 | |
| 146 | Every store also has `store.schema()` (the same text) and `await store.ensureSchema()`, which runs it |
| 147 | on the handle: for tests and PGlite, where the handle may create tables. The text is idempotent |
| 148 | (`IF NOT EXISTS`, `CREATE OR REPLACE`), uses PostgreSQL types (`bigint` identity keys, `jsonb` for |
| 149 | JSON, `bigint` epoch milliseconds where 0.4 stored milliseconds, `timestamptz` for citations' |
| 150 | `retrieved_at`, `COLLATE "C"` on identifier columns so they sort in byte order as SQLite did), keeps |
| 151 | 0.4's CHECK constraints, and has an index for every query a store sends (`test/schema.test.js` |
| 152 | checks each one with EXPLAIN). When a later release changes a store's DDL, its CHANGELOG entry gives |
| 153 | the migration a product adds; an applied migration is never edited. |
| 154 | |
| 155 | **Transaction handles.** Every data method takes an optional transaction handle as its **first** |
| 156 | argument. Without one it runs on the store's `db`; with one it runs inside the caller's |
| 157 | transaction, so a product's own write and the stores' writes commit together or not at all: |
| 158 | |
| 159 | ```js |
| 160 | await db.tx(async (t) => { |
| 161 | await t.exec(sql`INSERT INTO wiki_pages (id, slug, owner, updated_at) VALUES (${id}, ${slug}, ${owner}, ${Date.now()})`); |
| 162 | const { revision } = await revisions.create(t, { entityId: id, expectedRevision: 0, content, fields: { title } }); |
| 163 | await citations.attachMany(t, id, revision.number, sources); |
| 164 | }); |
| 165 | ``` |
| 166 | |
| 167 | Inside `db.tx`, pass `t` to every store call, reads included: a read on `db` would not see the |
| 168 | transaction's writes (and on PGlite, one connection, it waits for the transaction). A store write |
| 169 | that needs several statements runs them in its own transaction, or in a savepoint of the caller's |
| 170 | (`t.tx`); a store refusal (a 412 conflict, a 404, a 410) is raised before or after a clean savepoint, |
| 171 | so the caller can catch it and still commit the rest. `indexSequencer.stamp(t, document)` takes the |
| 172 | handle **always**, like the SDK outbox's `enqueue(t, …)`: the stamped revision must commit if and |
| 173 | only if the event carrying it does. |
| 174 | |
| 175 | **Concurrency.** Writers of one entity take a transaction-scoped advisory lock |
| 176 | (`pg_advisory_xact_lock`, allowed behind PgBouncer), so revisions' and citations' check-then-write, |
| 177 | purges and `setTerms` behave as they did under SQLite's single writer: racing writers of one base |
| 178 | revision get exactly one success and 412s, and nothing written during a purge survives it. The |
| 179 | scheduler claims with `FOR UPDATE SKIP LOCKED` in one statement, so any number of workers on any |
| 180 | number of hosts share one jobs table without taking a job twice. Get-or-create paths use |
| 181 | `ON CONFLICT`. `test/postgres.test.js` races all of these through PgBouncer. |
| 182 | |
| 183 | **Lists are bounded; several reads are one query.** Every list method has a `limit` (defaults keep |
| 184 | 0.4's answers for normal sizes), and lists that grow across entities page by keyset (`after`): |
| 185 | `citations.bySourceItem`, `media.entitiesUsing`, `taxonomy.entitiesFor` and `taxonomy.terms` |
| 186 | (revisions' `list` keeps its `before` cursor). For a feed or sitemap, read in batches instead of per |
| 187 | item: `revisions.getMany(refs)`, `citations.forRevisions(refs)`, `reviews.latestMany(refs)` and |
| 188 | `taxonomy.termsForMany(entityIds)` (refs are `{ entityId, revision }`; answers come back in the same |
| 189 | order). |
| 190 | |
| 191 | **Tests.** `createDb({ pglite: true })`, then `await store.ensureSchema()` (or `db.migrate({ dir })` |
| 192 | with the product's migrations): real PostgreSQL in-process. |
| 193 | |
| 194 | ## Moving a product between releases |
| 195 | |
| 196 | See [docs/MIGRATION.md](docs/MIGRATION.md): 0.4 → 1.0, 1.0 → 1.1 and 1.1 → 1.2, step by step. |
| 197 | |
| 198 | ## The indexability gate |
| 199 | |
| 200 | ```js |
| 201 | const seo = require('openvibe-publishing/seo'); |
| 202 | const decision = seo.evaluate( |
| 203 | { state: 'published', visibility: 'public', canonicalUrl, text, citationCount: 2, authorship: { mode: 'ai', reviewed: false } }, |
| 204 | { policy: { minWords: 150, requireSources: true }, now: Date.now() }, |
| 205 | ); |
| 206 | // → { indexable: false, listable: false, robots: 'noindex, nofollow', codes: ['ai_generated_unreviewed'], reasons: [...], canonical, gate } |
| 207 | ``` |
| 208 | |
| 209 | A pure function of (facts, policy, now): the same input always gives the same decision. Every "no" |
| 210 | carries stable reason codes, in a fixed order (`seo.REASONS`): |
| 211 | |
| 212 | | Effect | Codes | |
| 213 | |---|---| |
| 214 | | **hidden** — never indexed, never in sitemaps, feeds or public search | `deleted`, `takedown`, `private`, `gated`, `unlisted`, `draft` (draft or scheduled), `unpublished`, `ai_generated_unreviewed`, `stub_provider`, `unreviewed_sensitive` | |
| 215 | | **noindex** — may be served and listed in feeds, never indexed or put in a sitemap | `retracted`, `expired`, `stale_price`, `duplicate_of` (canonical points at the original), `thin`, `unsourced`, `unsupported_claims`, `missing_canonical`, `noindex_requested` | |
| 216 | |
| 217 | There is no default that makes content indexable: unknown fact names throw, a rule that needs a fact |
| 218 | the caller did not give throws, and time rules need an explicit `now` (no hidden clock). Each product |
| 219 | sets its own policy (`minWords`, `requireSources`, `minSources`, `priceMaxAgeMs`, `sensitiveCategories`). |
| 220 | `metaTags`, `robotsMeta`, `sitemap`, the three feed builders and `buildIndexDocument` (for a |
| 221 | published document) all require the decision, so an object without one cannot reach a crawler by omission. |
| 222 | |
| 223 | ## Honesty guarantees (tested) |
| 224 | |
| 225 | - **Structured data** is generated only from provided fields. A missing rating, price, currency, |
| 226 | date or author is omitted, never defaulted (no `bestRating: 5`, no assumed `InStock`, no |
| 227 | "now" as a date, no placeholder author). An `AggregateRating` needs a real count; an `Offer` needs a |
| 228 | price and a currency. [test/seo.test.js](test/seo.test.js) checks every output leaf against the input. |
| 229 | - **Feeds and sitemaps** never invent `pubDate`, `lastmod` or authors; an Atom entry without any date |
| 230 | is left out rather than dated "now". |
| 231 | - **Citations** never get a retrieval time they did not have, and cannot be edited or deleted |
| 232 | (a PL/pgSQL trigger, SQLSTATE 23001) except by an audited purge. Revisions are guarded the same way; |
| 233 | review-log rows cannot be updated. |
| 234 | - **Media** outages are `check_failed`, not "gone" and not "fine". |
| 235 | - **Discussion** failures are errors, never a fabricated thread. |
| 236 | - **AI content** starts as draft + noindex: `authorship.canPublish()` refuses it and the gate hides it |
| 237 | (`ai_generated_unreviewed`) until a person (a `usr_` subject, never a service) records an approving |
| 238 | review. Products must call these; the package cannot stop a product that bypasses its own checks. |
| 239 | |
| 240 | ## Events and Search |
| 241 | |
| 242 | `index-hooks.buildIndexDocument()` produces exactly the released `search.index-document@1` |
| 243 | (openvibe-contracts v0.12.0, owned by OpenVibe.Search): visibility `public | unlisted | members | |
| 244 | private`, ACL `subjects | groups | entitlements`, authorship `human | ai_assisted | ai_generated | |
| 245 | imported`, provenance as typed references (Sources items, the product's citation records, the AI |
| 246 | run with `stub: true` for stub output), and `indexability { decision: index|noindex, reasons }` with |
| 247 | the gate's codes mapped onto Search's known reasons (`hooks.SEARCH_REASONS`). Anything not |
| 248 | published, and unlisted content unless `includeUnlisted`, becomes the tombstone |
| 249 | `{ owner, type, id, revision, deleted: true }`. |
| 250 | |
| 251 | Search orders documents by `revision`, refuses a different document at an equal revision and lets a |
| 252 | tombstone win ties, so the revision must rise with every indexed change, not only content edits. |
| 253 | `createIndexSequencer(db, { prefix })` keeps that counter in the product's database. |
| 254 | |
| 255 | ```js |
| 256 | const seq = hooks.createIndexSequencer(db, { prefix: 'wiki' }); |
| 257 | await db.tx(async (t) => { |
| 258 | const doc = await seq.stamp(t, hooks.buildIndexDocument({ owner: 'wiki', type: 'page', id, revision: 0, state, visibility, … , decision })); |
| 259 | await outbox.enqueue(t, hooks.indexEvent({ document: doc })); // wiki.index_document.upserted | .deleted |
| 260 | }); |
| 261 | ``` |
| 262 | |
| 263 | `indexEvent()` builds the envelope Search's webhook consumes (payload = the document, or |
| 264 | `{ type, id, revision }` for a deletion; subject `{ type, id, revision }`; visibility internal). |
| 265 | `publicationEvent()` builds the product's own `<product>.<type>.<action>` event (payload: canonical |
| 266 | URL, publication state, indexability; public only for public, listable content). Neither sets |
| 267 | `event_id`: the OpenVibe.Events outbox assigns it. Enqueue both in the same transaction as the |
| 268 | product's state change. Tests validate every document and envelope with openvibe-contracts |
| 269 | v0.76.0 and mirror Search's webhook checks. |
| 270 | |
| 271 | ## Ingesting from Sources (the ingest chassis) |
| 272 | |
| 273 | The five content products (News, Reviews, Deals, Coupons, Trade) each pull OpenVibe.Sources items, track a |
| 274 | change cursor, consume signed OpenVibe.Events deliveries and normalise item fields. They share that code |
| 275 | here; each keeps its own domain tables, gate facts, facets and item mapping. |
| 276 | |
| 277 | ```js |
| 278 | const { createSourcesClient, createChangeCursor, pullChanges, createEventConsumer } |
| 279 | = require('openvibe-publishing/ingest'); |
| 280 | |
| 281 | const sources = createSourcesClient({ config }); // { sources, oauth }, from the product |
| 282 | const cursor = createChangeCursor(db, { prefix: 'deals' }); // <prefix>_ingest_cursor in a migration |
| 283 | const summary = await pullChanges({ |
| 284 | db, cursor, source: sources, name: 'sources.deals.after', |
| 285 | apply: async (item, t) => { // one savepoint per item |
| 286 | const row = await offers.upsertFromItem(t, item); |
| 287 | return row.held ? 'hold' : 'applied'; // 'removed' for a deleted item |
| 288 | }, |
| 289 | }); |
| 290 | // → { pages, applied, hold, removed, failed, after } |
| 291 | ``` |
| 292 | |
| 293 | `pullChanges` commits one transaction per page — the page's writes **and** the cursor advance — and runs |
| 294 | `apply(item, t)` in a savepoint, so one unreadable item is rolled back alone (`failed`) and never stalls |
| 295 | the page or the cursor. `apply` returns `'applied' | 'hold' | 'removed'`; `hold` is counted and the cursor |
| 296 | moves past it (the product records the hold itself). |
| 297 | |
| 298 | `createEventConsumer({ db, secrets, consumer }).apply(raw, headers, handler)` verifies the |
| 299 | `X-OpenVibe-Signature` v2 HMAC (±300 s; a v1-only or stale delivery is refused) and runs the handler |
| 300 | exactly once per `event_id` through the SDK's pg inbox, so a redelivery changes nothing and a failed |
| 301 | handler rolls back its receipt. |
| 302 | |
| 303 | `ingest.normalize` holds the normalisers the products share (folding, title/URL keys, hosts, amounts, |
| 304 | currencies, instants, enum/text parsing, the Reviews key/alias normalisers), `ingest.hosts` the PSL and |
| 305 | registrable-host helpers, and `ingest.freshness` the generic staleness rule. Nothing defaults a date to |
| 306 | "now", and no policy threshold is a constant: a product passes its own. |
| 307 | |
| 308 | ## The publication glue (createPublication) |
| 309 | |
| 310 | `createPublication` is the half the five content products each re-implement: it takes the product's gate |
| 311 | (`decide`), document builder (`document`), canonical path (`page`), the index sequencer and the outbox, and |
| 312 | runs gate → document → `sequencer.stamp(t, doc)` → event → `outbox.enqueue(t, …)` on the **caller's** |
| 313 | transaction handle, so the index document, its event and the product's own change commit or roll back |
| 314 | together. |
| 315 | |
| 316 | ```js |
| 317 | const { createPublication } = require('openvibe-publishing/publication'); |
| 318 | const publication = createPublication({ |
| 319 | owner: 'deals', sequencer, outbox, baseUrl: config.baseUrl, indexnow, |
| 320 | decide: (offer, now) => publication.decideOffer(offer), // the product's gate facts |
| 321 | document: (offer, decision) => offerDocument(offer), // hooks.buildIndexDocument/tombstone |
| 322 | page: (offer) => `/d/${offer.slug}`, |
| 323 | }); |
| 324 | await db.tx(async (t) => { |
| 325 | await t.exec(sql`UPDATE deal_offers SET … WHERE id = ${id}`); |
| 326 | await publication.sync(t, offer); // stamps + enqueues in the SAME tx |
| 327 | }); |
| 328 | ``` |
| 329 | |
| 330 | It emits exactly `search.index-document@1` (carried by `<owner>.index_document.upserted|deleted`) and the |
| 331 | product's own `<owner>.<type>.<action>` events; an unchanged document is a no-op, a resource that was |
| 332 | never indexed gets no tombstone, and unpublish/merge call `tombstone(t, …)`. It owns the IndexNow ping |
| 333 | (config stays per product): an indexable page that appeared or changed, or a page Search already had that |
| 334 | went away, is announced; a draft, private or noindex page never is. |
| 335 | |
| 336 | ## Exit proof: two products |
| 337 | |
| 338 | [examples/two-products](examples/two-products) holds a mini wiki and a mini blog, each with its own |
| 339 | database (an `openvibe-sdk/db` handle) and its own publication tables, both using revisions, |
| 340 | citations, the gate and feeds with different editorial policies. Each applies its |
| 341 | `migrations/0001_initial.sql` (its own tables plus `publishing.schema({ … })`, generated by |
| 342 | `examples/two-products/migrations.js`) with `db.migrate()`, as a service does. |
| 343 | `node examples/two-products/wiki/app.js` (port 4801) and `node examples/two-products/blog/app.js` |
| 344 | (port 4811) serve them on in-memory PGlite. [test/two-products.test.js](test/two-products.test.js) |
| 345 | shows they load the same modules, that each database holds only its own product's tables, that |
| 346 | writes never cross, that a product's write and the stores' writes share one transaction, that a feed |
| 347 | of many pages costs the same queries as a feed of one, and that pages are useful without JavaScript, |
| 348 | old slugs 301, private/VIP/deleted content leaves feeds and sitemaps, and scheduled publication |
| 349 | survives a worker restart. |
| 350 | |
| 351 | ## Capabilities |
| 352 | |
| 353 | The package implements no capability and holds no grant. Four modules call a service with the |
| 354 | **product's** token client, so the product needs the grant: `discussion` calls OpenVibe.Community |
| 355 | (`community.comment.write` to resolve a thread), `ai` calls OpenVibe.AI (`ai.run.create`, |
| 356 | `ai.run.read`), `media`'s `verify()` reads OpenVibe.Media objects (`media.object.read`), and `ingest`'s |
| 357 | Sources client reads OpenVibe.Sources items and registry records (`sources.item.read`, |
| 358 | `sources.source.read`). |
| 359 | |
| 360 | ## Acceptance |
| 361 | |
| 362 | `npm test` runs every `test/*.test.js` on real PostgreSQL in-process (PGlite, one fresh schema per |
| 363 | test) with no network. What it proves: immutable revisions, diff and revert, keyset lists, drafts in |
| 364 | one query (`revisions.test.js`); idempotent scheduling across worker restarts, concurrent claims |
| 365 | (`schedule.test.js`); append-only citations, batched inserts (`citations.test.js`); honest media |
| 366 | states (`media.test.js`); discussion references without content (`discussion.test.js`); the gate, |
| 367 | redirects, structured data and feeds built only from given fields (`seo.test.js`); AI content held |
| 368 | until a person's review (`authorship.test.js`, `ai.test.js`); Search documents and envelopes valid |
| 369 | against openvibe-contracts, and `stamp(t, …)` committing with its event (`index-hooks.test.js`); |
| 370 | escaping and the Markdown subset, including the ReDoS fix (`ssr.test.js`); the ingest chassis — the |
| 371 | normalisers and host helpers byte-for-byte against 119 cases captured from the five current product files, |
| 372 | the Sources client against a stub server, cursor advancement with per-item savepoint isolation and hold, |
| 373 | and the signed event consumer's window and exactly-once inbox (`ingest.test.js`); the publication glue — |
| 374 | contracts-valid documents and events, `stamp` + `enqueue` in one transaction, tombstones and the IndexNow |
| 375 | ping (`publication.test.js`); every subpath export loading |
| 376 | alone without a driver (`package.test.js`); the DDL: idempotent, 63-byte names, PostgreSQL types, |
| 377 | CHECKs, PL/pgSQL guards, applied by `db.migrate()`, and an index under every query the stores send |
| 378 | (`schema.test.js`); callers' transactions spanning store writes, rollbacks leaving nothing, savepoints |
| 379 | (`transactions.test.js`); and the two-product exit proof (`two-products.test.js`). |
| 380 | `test/postgres.test.js` runs the stores on PostgreSQL 18 through PgBouncer (transaction mode) when |
| 381 | `OV_TEST_PG_URL` is set: racing writers, a purge racing writes, racing workers, racing get-or-create and |
| 382 | `setTerms`, and callers' transactions behind the pooler; otherwise it prints |
| 383 | `postgresql+pgbouncer: skipped (…)`. CI starts the containers, so it always runs there. |
| 384 | |
| 385 | ## Security |
| 386 | |
| 387 | Reporting a vulnerability: [SECURITY.md](SECURITY.md). The `ssr` module auto-escapes every value in |
| 388 | its `html` templates and renders only a safe Markdown subset (linear-time since v0.2.1's ReDoS fix); |
| 389 | JSON-LD is escaped by openvibe-shared. The package holds no secrets, opens no database it was not |
| 390 | handed, and makes network calls only through the token client and base URL a product passes in. Every |
| 391 | value reaches PostgreSQL as a bind parameter (the `sql` tag); the only identifiers in statements are |
| 392 | the validated prefix's table names. A product that bypasses its own checks is not stopped by the |
| 393 | package (see Honesty guarantees). |
| 394 | |
| 395 | ## Deploy |
| 396 | |
| 397 | Nothing is deployed from this repository. A release is a git tag (`vX.Y.Z`, recorded in |
| 398 | [CHANGELOG.md](CHANGELOG.md)); products pin the tag's tarball and ship it with their own deploys |
| 399 | (`sudo ovhost deploy <product>`). A bad release is undone by the products pinning the previous tag. |
| 400 | |
| 401 | ## Development |
| 402 | |
| 403 | ```bash |
| 404 | fnm exec --using=22.22.1 npm install |
| 405 | fnm exec --using=22.22.1 npm test # every test/*.test.js on PGlite, no network |
| 406 | # with PostgreSQL 18 + PgBouncer containers (what CI runs): |
| 407 | eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && fnm exec --using=22.22.1 npm test |
| 408 | ``` |
| 409 | |
| 410 | ## Does not own |
| 411 | |
| 412 | - It does not own publication state, run workers, listen on a port, or talk to a database it was not handed. |
| 413 | - It does not decide editorial rules: each product passes its own gate policy and its own permission checks. |
| 414 | - It does not store or render comment content (Community owns it), media bytes (Media owns them) or |
| 415 | source items (Sources owns them); it stores references. |
| 416 | - It is not a notification, search or event runtime: it builds the documents and envelopes those services consume. |
| 417 | |
| 418 | ## Depends on |
| 419 | |
| 420 | - `openvibe-sdk` ≥ 0.15.0 (`openvibe-sdk/db`), a peer dependency supplied by the consumer: its |
| 421 | `createDb` handle (and `pg` in production, `@electric-sql/pglite` in tests) reach the stores through |
| 422 | the product. Tests pin v0.60.0. |
| 423 | - PostgreSQL 18 (through PgBouncer in transaction mode), the product's own database (ADR-035). |
| 424 | - `openvibe-shared` ≥ 1.5.0 (`seo`), a peer dependency supplied by the consumer (tests use v4.4.0). |
| 425 | - Contracts it produces for: `common.entity-ref@1`, `media.media-ref@1`, `events.event-envelope@1` |
| 426 | and `search.index-document@1` (validated in tests against `openvibe-contracts` v0.419.0, a |
| 427 | devDependency: nothing in `lib/` needs it at runtime). |
| 428 | |
| 429 | <!-- versions:start --> |
| 430 | - openvibe-contracts: v0.419.0 |
| 431 | - openvibe-sdk: v0.161.0 |
| 432 | - openvibe-shared: v4.241.0 |
| 433 | <!-- versions:end --> |
| 434 |