openvibers/openvibe.news Commits
News on the Publishing 1.1.0 chassis (plan T9 J2) with IndexNow and a CI performance budget (plan T11 decision 4): the Sources client, change cursor and pull loop, signed event consumer and normalisers come from openvibe-publishing/ingest; News keeps its domain half (domain/source-items.js, http/events.js). Migration 0002 seeds news_ingest_cursor once from news_state (continuity, never a reset to zero). The replaced clients/sources.js, domain/ingest.js, domain/text.js and http/webhook.js are deleted; test/chassis.test.js proves nothing requires them and replays the Sources fixture against a golden captured from the pre-change code. IndexNow (openvibe-shared/indexnow): the key file is served when INDEXNOW_KEY is set, and published or changed stories are pinged. test/perf-budget.test.js holds the home page to its measured sizes. npm test and test:pg green (verified).
Changes
@@@ -50,8 +50,9 @@ Other tables in the same database:an author).- `news_cluster_audit`: every merge, split and reversal with exactly which items moved.- `news_ingest_runs`: what every webhook delivery, pull and upstream fetch failure did.--- `news_state` (the Sources cursor), `news_source_status` (display cache of Sources' registry-- names and health), `event_outbox` and `idempotency_receipts` (openvibe-sdk),++- `news_ingest_cursor` (the OpenVibe.Sources change cursor, `openvibe-publishing/ingest`; `news_state`++ keeps the pre-chassis key it was carried from), `news_source_status` (display cache of Sources'++ registry names and health), `event_outbox` and `idempotency_receipts` (openvibe-sdk),`subject_projections` (Network names). None of them is authority for anything outside News.## Does not own@@@ -176,6 +177,10 @@ Other tables in the same database:- JSON-LD `NewsArticle` from real fields only: headline, dates, the editors named in therevision's authorship (only when the Network name is known), the section (topic), the citedsource URLs, and OpenVibe.News as publisher. Missing fields are omitted.++- IndexNow (`openvibe-shared/indexnow`): with `INDEXNOW_KEY` set, the key file is served at++ `/<key>.txt` and publishing, updating or unpublishing an indexable story pings the engines with++ the story's URL and `/sitemap.xml`. Unset, the feature is off — nothing is mounted and nothing is++ sent. Drafts, private and noindex pages are never pinged (`test/indexnow.test.js`).- Drafts, unpublished, retracted and noindex stories never appear in sitemaps or Search; feedslist published and (labelled) retracted stories, never a draft.@@@ -224,8 +229,10 @@ under [Grants the Network must hold](#grants-the-network-must-hold).- **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through`openvibe-sdk/db`; Valkey holds the per-actor limit counters (optional: without `VALKEY_URL` they count per process).--- **Packages** (pinned by release tarball): `openvibe-publishing` v1.0.0 (async PostgreSQL stores: revisions,-- citations, authorship, seo, index-hooks, ssr, taxonomy slugify), `openvibe-contracts` v0.79.0,++- **Packages** (pinned by release tarball): `openvibe-publishing` v1.1.0 (async PostgreSQL stores: revisions,++ citations, authorship, seo, index-hooks, ssr, taxonomy slugify, and the shared ingest/publication++ chassis: the Sources client, the change-cursor pull, the signed events consumer and the++ gate → document → outbox glue), `openvibe-contracts` v0.79.0,`openvibe-shared` v2.2.0 (Frame, app icon, footer, legal, release, metrics, ready, seo),`openvibe-sdk` v0.25.0 (db with after-commit hooks, PostgreSQL events outbox and inbox, webhook signatures v2,service tokens, Network sign-in (`openvibe-sdk/sso`) and the service outbox (`openvibe-sdk/events`), per-actor limits, testing).@@@ -389,7 +396,7 @@ First install (done once; kept for a rebuild):`NEWS_EDITORS`, `NEWS_FORM_SECRET`, `NEWS_EVENTS_SECRET` (`openssl rand -hex 32`),`EVENTS_URL=http://127.0.0.1:4300`, `OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720`,`BASE_URL=https://openvibe.news`, `OV_COMMUNITY_INTERNAL_URL=http://127.0.0.1:4200`, and-- optionally `OV_AI_INTERNAL_URL`.++ optionally `OV_AI_INTERNAL_URL` and `INDEXNOW_KEY`.2. **Network:** create (or give a secret to) the OAuth client `news` with redirect`https://openvibe.news/auth/callback`, and add the grants listed above.3. **Search:** make sure `news` is in `SEARCH_EVENT_OWNERS`.@@@ -9,7 +9,7 @@"domain": "openvibe.news","contracts": "openvibe-contracts v0.79.0","packages": [-- "openvibe-publishing v1.0.0",++ "openvibe-publishing v1.1.0","openvibe-contracts v0.79.0","openvibe-shared v2.2.0","openvibe-sdk v0.25.0"nnew file mode 100644@@@ -0,0 +1,20 @@++-- phase: expand++-- The ingest cursor moves from news_state ('sources_cursor') to the chassis' own table++-- (openvibe-publishing/ingest, prefix news → news_ingest_cursor). Additive: the old key is read++-- once here so a deployment continues from the cursor it had, never from zero. Generated from++-- require('openvibe-publishing').schema({ ingest: 'news' }).++++CREATE TABLE IF NOT EXISTS news_ingest_cursor (++ name text COLLATE "C" NOT NULL,++ cursor bigint NOT NULL,++ updated_at bigint NOT NULL,++ PRIMARY KEY (name)++);++++-- Continuity: carry the cursor already stored under the old key, once. news_state stores JSON, so a++-- numeric value is plain digits; anything else is ignored (the chassis cursor then starts empty).++INSERT INTO news_ingest_cursor (name, cursor, updated_at)++SELECT 'sources', value::bigint, (EXTRACT(EPOCH FROM now()) * 1000)::bigint++FROM news_state++WHERE key = 'sources_cursor' AND value ~ '^[0-9]+$'++ON CONFLICT (name) DO NOTHING;@@@ -16,7 +16,7 @@"helmet": "^8.1.0","iovalkey": "^0.4.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.79.0",-- "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.0.0",++ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.1.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.25.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.2.0","pg": "^8.23.0"@@@ -1163,9 +1163,9 @@}},"node_modules/openvibe-publishing": {-- "version": "1.0.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.0.0",-- "integrity": "sha512-g4mU0PPtI4y38NcsdppEbrXTrCYp0rPmIYWyxLxh4GpEYhn48jMWq2+cMQA2uwZCeeZlB3UiXVrQNQyQLiliwg==",++ "version": "1.1.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.1.0",++ "integrity": "sha512-bJsVl82hDivazoiM/np9ckKJE4V5F4OxyIiDwB6xuhK8M6P42sGU+V88rFCkyiI4BVQiKv9xs4GVx8Fj3pyjsg==","license": "MIT","engines": {"node": ">=20"@@@ -30,7 +30,7 @@"helmet": "^8.1.0","iovalkey": "^0.4.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.79.0",-- "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.0.0",++ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.1.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.25.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.2.0","pg": "^8.23.0"@@@ -17,18 +17,19 @@ const contracts = require('openvibe-contracts');const { createSsoClient } = require('openvibe-sdk/sso');const { createServiceOutbox } = require('openvibe-sdk/events');++const { createSourcesClient } = require('openvibe-publishing/ingest');++const { createIndexNow } = require('openvibe-shared/indexnow');const configLib = require('./config');const { openStore } = require('./db');const { createViewerResolver } = require('./auth/viewer');const { createPeople } = require('./clients/network');--const { createSourcesClient } = require('./clients/sources');const { createAi } = require('./clients/ai');const { createCommunity } = require('./clients/community');const { createPublication } = require('./domain/publication');const { createDiscussion } = require('./domain/discussion');const { createClusters } = require('./domain/clusters');--const { createIngest } = require('./domain/ingest');++const { createIngest } = require('./domain/source-items');const { createStories } = require('./domain/stories');const { createTopics } = require('./domain/topics');const { createReading } = require('./domain/reading');@@@ -36,7 +37,7 @@ const { createPublicRoutes } = require('./http/public');const { createEditorRoutes } = require('./http/editor');const { createApi } = require('./http/api');const { createDiscoveryRoutes } = require('./http/discovery');--const { createWebhook } = require('./http/webhook');++const { createEvents } = require('./http/events');const { createActorLimits } = require('./http/actor-limits');const { createNewsReadiness } = require('./observability');const { createWorker } = require('./worker');@@@ -62,7 +63,11 @@ async function createApp(opts = {}) {intervalMs: config.events.intervalMs, log, now: store.now,...(fetchImpl ? { fetch: fetchImpl } : {}),});-- const publication = createPublication({ store, config, outbox });++ // IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY; unset → off++ // (nothing mounted, nothing sent). The key file is served at /<key>.txt and publication.js++ // pings the engines when an indexable page appears, changes or goes away.++ const indexnow = opts.indexnow || createIndexNow({ host: config.baseUrl, key: config.indexnow.key, ...(fetchImpl ? { fetch: fetchImpl } : {}), log });++ const publication = createPublication({ store, config, outbox, indexnow });const clusters = createClusters({ store, config, outbox });const sources = createSourcesClient({ config, fetchImpl });const ai = createAi({ config, fetchImpl });@@@ -91,7 +96,7 @@ async function createApp(opts = {}) {const viewers = createViewerResolver({ auth, config, people });const worker = createWorker({ config, ingest, outbox, log });-- const ctx = { config, store, outbox, publication, clusters, sources, ai, community, discussion, ingest, stories, topics, people, reading, auth, viewers, worker };++ const ctx = { config, store, outbox, publication, clusters, sources, ai, community, discussion, ingest, stories, topics, people, reading, auth, viewers, worker, indexnow };const app = express();app.disable('x-powered-by');@@@ -138,9 +143,11 @@ async function createApp(opts = {}) {release.mount(app, { registry: metrics.registry });const readiness = createNewsReadiness({ store, outbox, ingest, config, release: release.release, valkey: ctx.valkey });app.get('/api/ready', readiness.handler);++ // GET /<key>.txt — the IndexNow key file (only when a key is configured; it serves itself).++ if (indexnow.enabled) app.use(indexnow.keyFile);// ── Events webhook (raw body; before any other body parser) ─-- app.use(createWebhook({ config, store, ingest, sources, log }));++ app.use(createEvents({ config, store, ingest, sources, log }));app.use(cookieParser());ddeleted file mode 100644@@@ -1,77 +0,0 @@--'use strict';----/**-- * OpenVibe.Sources client (host-local API, service tokens for audience openvibe.sources).-- *-- * sources.item.read GET /api/v1/items?category=news&after=&limit=&include_removed=1 (change order)-- * GET /api/v1/items/:id-- * sources.source.read GET /api/v1/sources/:key (optional: outlet names and source health)-- *-- * Failures are thrown as errors with a stable `code` (sources.unavailable, sources.http_<status>,-- * sources.bad_response); the caller records them. Nothing here invents an item.-- */--const { serviceAuth } = require('openvibe-contracts');----class SourcesError extends Error {-- constructor(code, message, status = null) { super(message); this.code = code; this.status = status; }--}----function createSourcesClient({ config, fetchImpl = globalThis.fetch }) {-- const enabled = Boolean(config.oauth.clientSecret && config.sources.internalUrl);-- const token = (scope) => (enabled ? serviceAuth.createTokenClient({-- tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,-- audience: 'openvibe.sources', scope, fetchImpl,-- }) : null);-- const itemTokens = token('sources.item.read');-- const sourceTokens = token('sources.source.read');---- async function call(tokens, path) {-- if (!tokens) throw new SourcesError('sources.not_configured', 'OV_OAUTH_CLIENT_SECRET or OV_SOURCES_INTERNAL_URL is not set');-- let auth;-- try { auth = await tokens.authHeaders(); } catch (err) {-- throw new SourcesError('sources.token_unavailable', `No service token for OpenVibe.Sources: ${err.message}`);-- }-- let res;-- try {-- res = await fetchImpl(`${config.sources.internalUrl}${path}`, {-- headers: { Accept: 'application/json', ...auth },-- signal: AbortSignal.timeout(config.sources.timeoutMs),-- });-- } catch (err) {-- throw new SourcesError('sources.unavailable', `OpenVibe.Sources is unreachable: ${err.message}`);-- }-- if (res.status === 401 && tokens.invalidate) tokens.invalidate();-- const body = await res.json().catch(() => null);-- if (!res.ok) throw new SourcesError(`sources.http_${res.status}`, `OpenVibe.Sources answered ${res.status}${body && (body.code || body.detail) ? `: ${body.code || ''} ${body.detail || ''}`.trimEnd() : ''}`, res.status);-- if (!body || typeof body !== 'object') throw new SourcesError('sources.bad_response', 'OpenVibe.Sources sent no JSON');-- return body;-- }---- return {-- enabled,-- SourcesError,---- /** One page of news items in change order. → { items, next_after, more, sources } */-- async listItems({ after = 0, limit = 100 } = {}) {-- const qs = new URLSearchParams({ category: config.sources.category, after: String(after), limit: String(limit), include_removed: '1' });-- const body = await call(itemTokens, `/api/v1/items?${qs}`);-- if (!Array.isArray(body.items) || !Number.isInteger(body.next_after)) throw new SourcesError('sources.bad_response', 'items page without items/next_after');-- return body;-- },---- /** One item (with its source's health). → { item, source } */-- async getItem(id) {-- const body = await call(itemTokens, `/api/v1/items/${encodeURIComponent(id)}`);-- if (!body.item || typeof body.item.id !== 'string') throw new SourcesError('sources.bad_response', 'item response without item');-- return body;-- },---- /** A registry record (name, homepage, health). Needs sources.source.read; callers treat failure as "unknown". */-- async getSource(key) {-- const body = await call(sourceTokens, `/api/v1/sources/${encodeURIComponent(key)}`);-- return body.source || body;-- },-- };--}----module.exports = { createSourcesClient, SourcesError };@@@ -36,7 +36,9 @@ function load(env = process.env) {},// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).-- db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },++ // Non-production without DATABASE_URL: the embedded PGlite database in data/pglite++ // (NEWS_PGLITE_DIR, tests: an isolated directory of their own).++ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.NEWS_PGLITE_DIR || '' },valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:news:' },// OpenVibe.Network: SSO (OAuth2 authorization server), JWKS, client-credentials tokens.@@@ -81,6 +83,11 @@ function load(env = process.env) {minIndependentSources: Math.max(int(env.NEWS_MIN_INDEPENDENT_SOURCES, 2), 1),},++ // IndexNow (openvibe-shared/indexnow): when INDEXNOW_KEY is set the key file is served at++ // /<key>.txt and every publish, update, unpublish or delete of a public, indexable page pings++ // the engines. Unset: off — nothing is mounted and nothing is sent.++ indexnow: { key: env.INDEXNOW_KEY || '' },++// Clustering: the time window a new item may join an existing cluster in.clustering: {windowMs: int(env.NEWS_CLUSTER_WINDOW_HOURS, 72) * 3600 * 1000,@@@ -22,8 +22,9 @@* news_story_citations, news_story_reviews, news_story_drafts, news_story_revision_purges,* news_story_discussion_refs (a story's Community thread id, never its comments),* news_index_revisions (publishing packages); news_cluster_audit (every merge, split and reversal);-- * news_ingest_runs (what each webhook delivery, pull and upstream failure did); news_state (the-- * Sources cursor); news_source_status (a display cache of Sources' registry and health);++ * news_ingest_runs (what each webhook delivery, pull and upstream failure did); news_ingest_cursor++ * (the chassis' Sources change cursor; news_state holds the pre-chassis key it was carried from);++ * news_source_status (a display cache of Sources' registry and health);* event_outbox and idempotency_receipts (openvibe-sdk); subject_projections (Network names).*/const fs = require('fs');@@@ -45,9 +46,10 @@ const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');async function openDb(config, { log = console, registry } = {}) {if (!config.db.url) {if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh news)');-- log.warn(`[News] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);-- fs.mkdirSync(DEV_PGLITE, { recursive: true });-- const db = createDb({ pglite: DEV_PGLITE, service: 'news', registry, log });++ const dir = config.db.pgliteDir || DEV_PGLITE;++ log.warn(`[News] DATABASE_URL unset: embedded PGlite database in ${dir} (development only, one process)`);++ fs.mkdirSync(dir, { recursive: true });++ const db = createDb({ pglite: dir, service: 'news', registry, log });await db.migrate({ dir: MIGRATIONS, log });return db;}@@@ -63,9 +65,6 @@ async function openDb(config, { log = console, registry } = {}) {*/function createStore(db, { now = () => Date.now() } = {}) {const revisions = createRevisionStore(db, { prefix: 'news_story', now });-- const getStateStmt = db.prepare('SELECT value FROM news_state WHERE key = ?');-- const setStateStmt = db.prepare(`INSERT INTO news_state (key, value, updated_at) VALUES (?, ?, ?)-- ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`);return {db,now,@@@ -75,13 +74,6 @@ function createStore(db, { now = () => Date.now() } = {}) {sequencer: createIndexSequencer(db, { prefix: 'news', now }),discussion: createDiscussionRefs(db, { prefix: 'news_story', now }),tx: async (fn) => await db.tx(() => fn()),-- async getState(key, def = null) {-- const r = await getStateStmt.get(key);-- return r ? JSON.parse(r.value) : def;-- },-- async setState(key, value) {-- await setStateStmt.run(key, JSON.stringify(value), now());-- },close: () => db.close(),};}@@@ -18,7 +18,7 @@* Every change emits news.cluster.updated (internal) in the same transaction.*/const { ids } = require('openvibe-contracts');--const text = require('./text');++const { normalize } = require('openvibe-publishing/ingest');const { ApiError } = require('../http/errors');const newClusterId = (now) => `clu_${ids.ulid(now)}`;@@@ -28,7 +28,7 @@ const parse = (s, d) => { try { return s ? JSON.parse(s) : d; } catch { return d/** Terms and entities of one item (headline + the licensed summary we store). */function signature(item) {const t = `${item.headline || ''}. ${item.summary || ''}`;-- return { terms: text.terms(t), entities: text.entities(item.headline || '').concat(text.entities(item.summary || '')).filter((v, i, a) => a.indexOf(v) === i).sort() };++ return { terms: normalize.terms(t), entities: normalize.entities(item.headline || '').concat(normalize.entities(item.summary || '')).filter((v, i, a) => a.indexOf(v) === i).sort() };}/** The time an item is placed at: when the source says it was published, else when News first saw it. */ddeleted file mode 100644@@@ -1,290 +0,0 @@--'use strict';----/**-- * Ingestion: OpenVibe.Sources items (category news) → news_source_items → dedupe → clusters.-- *-- * Two ways in, one effect:-- * - the Events webhook (sources.item.created|updated|removed, sources.fetch.failed), each event-- * applied once through the inbox (http/webhook.js);-- * - the cursor pull (worker), the backstop for missed deliveries: GET /api/v1/items in change-- * order from the stored cursor.-- * apply() is idempotent on (Sources item id, Sources revision): a replay, or the same item arriving-- * by both paths, changes nothing and emits nothing.-- *-- * What is stored (licensing): headline, canonical URL, outlet, authors, published_at, the licence-- * and terms notes, and a short summary ONLY when those notes explicitly allow short summaries-- * (text.licensedSummary). Every other field of the Sources item — whatever an adapter mapped into-- * `fields` — is ignored, so an article body can never be stored, shown, fed or indexed.-- *-- * Failure is a recorded state (news_ingest_runs + news.source.failed), never a replacement: a-- * failed fetch or an unreadable item creates no source item, no cluster and no story text.-- *-- * Upstream changes to an item a story cites (a revision, a removal) go to stories.onUpstreamChange,-- * which flags the story and prepares a pending revision; published text is never changed silently.-- * A removal can also hide the story's Community thread (domain/discussion.js, after the commit).-- */--const { ids } = require('openvibe-contracts');--const text = require('./text');----const NEAR_TITLE = { anyOutlet: 0.9, sameOutlet: 0.75, windowMs: 48 * 3600 * 1000 };--const newItemId = (now) => `nsi_${ids.ulid(now)}`;----function createIngest({ store, config, clusters, outbox, sources, discussion = null, log = console }) {-- const { db } = store;-- let stories = null; // set by app.js (stories depend on ingest's items too)-- const q = {-- bySourcesId: db.prepare('SELECT * FROM news_source_items WHERE sources_item_id = ?'),-- byId: db.prepare('SELECT * FROM news_source_items WHERE id = ?'),-- byUrl: db.prepare("SELECT * FROM news_source_items WHERE url_key = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),-- byHash: db.prepare("SELECT * FROM news_source_items WHERE content_hash = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),-- recent: db.prepare("SELECT * FROM news_source_items WHERE status <> 'removed' AND first_seen_at >= ? ORDER BY first_seen_at, seq"),-- status: db.prepare('SELECT * FROM news_source_status WHERE source_key = ?'),-- putStatus: db.prepare(`INSERT INTO news_source_status (source_key, name, homepage_url, status, stale, last_success_at, refreshed_at)-- VALUES (@source_key, @name, @homepage_url, @status, @stale, @last_success_at, @now)-- ON CONFLICT (source_key) DO UPDATE SET name = COALESCE(excluded.name, news_source_status.name), homepage_url = COALESCE(excluded.homepage_url, news_source_status.homepage_url),-- status = COALESCE(excluded.status, news_source_status.status), stale = COALESCE(excluded.stale, news_source_status.stale),-- last_success_at = COALESCE(excluded.last_success_at, news_source_status.last_success_at), refreshed_at = excluded.refreshed_at`),-- run: db.prepare(`INSERT INTO news_ingest_runs (origin, state, source_key, sources_item_id, error_code, detail, counts, cursor_before, cursor_after, at)-- VALUES (@origin, @state, @source_key, @sources_item_id, @error_code, @detail, @counts, @cursor_before, @cursor_after, @at)`),-- lastPull: db.prepare("SELECT * FROM news_ingest_runs WHERE origin = 'pull' ORDER BY id DESC LIMIT 1"),-- };---- async function recordRun(r) {-- await q.run.run({ source_key: null, sources_item_id: null, error_code: null, detail: null, counts: null, cursor_before: null, cursor_after: null, ...r, counts: r.counts ? JSON.stringify(r.counts) : null, at: store.now() });-- }---- async function failedEvent({ origin, sourceKey = null, sourcesItemId = null, state, errorCode = null, detail = null, httpStatus = null, extra = {} }) {-- await outbox.emit({-- event_type: 'news.source.failed', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'important',-- subject: sourceKey ? { type: 'source', id: sourceKey } : { type: 'ingest', id: origin },-- payload: { origin, source_key: sourceKey, sources_item_id: sourcesItemId, state, error_code: errorCode, http_status: httpStatus, detail: detail ? String(detail).slice(0, 500) : null, ...extra },-- });-- }---- async function outletFor(item) {-- const st = await q.status.get(item.source_key);-- return (st && st.name) || text.hostOf(item.canonical_url) || item.source_key;-- }---- /** The fields News keeps from a Sources item. Nothing else of the item is read. */-- async function normalise(item) {-- const prov = item.provenance || {};-- const headline = String(item.title || '').replace(/\s+/g, ' ').trim().slice(0, 500);-- const lic = text.licensedSummary(item.summary, { termsNote: prov.terms_note, licenseNote: prov.license_note, maxChars: config.licensing.summaryMaxChars });-- const authors = Array.isArray(item.authors) ? item.authors.map((a) => String(a || '').replace(/\s+/g, ' ').trim().slice(0, 200)).filter(Boolean).slice(0, 10) : [];-- const published = item.published_at && Number.isFinite(Date.parse(item.published_at)) ? new Date(Date.parse(item.published_at)).toISOString() : null;-- return {-- headline,-- canonical_url: item.canonical_url || null,-- url_key: text.urlKey(item.canonical_url),-- outlet: await outletFor(item),-- authors: JSON.stringify(authors),-- published_at: published,-- summary: lic.summary,-- summary_basis: lic.basis,-- license_note: prov.license_note ? String(prov.license_note).slice(0, 2000) : null,-- terms_note: prov.terms_note ? String(prov.terms_note).slice(0, 2000) : null,-- content_hash: prov.content_hash || null,-- title_key: text.titleKey(headline),-- retrieved_at: prov.retrieved_at || null,-- };-- }---- /** Is this new item a duplicate of one we have? → null | { of, rule, detail } */-- async function dedupe(n, sourceKey, at) {-- if (n.url_key) {-- const hit = await q.byUrl.get(n.url_key);-- if (hit) return { of: hit.duplicate_of || hit.id, rule: 'canonical_url', detail: n.url_key };-- }-- if (n.content_hash) {-- const hit = await q.byHash.get(n.content_hash);-- if (hit) return { of: hit.duplicate_of || hit.id, rule: 'content_hash', detail: n.content_hash };-- }-- const mine = text.shingles(n.headline);-- const host = text.hostOf(n.canonical_url);-- let best = null;-- for (const c of await q.recent.all(Math.min(at, store.now()) - NEAR_TITLE.windowMs)) {-- const t = clusters.itemTime(c);-- if (Math.abs(t - at) > NEAR_TITLE.windowMs) continue;-- const score = text.jaccard(mine, text.shingles(c.headline));-- const sameOutlet = c.source_key === sourceKey || (host && text.hostOf(c.canonical_url) === host);-- const threshold = sameOutlet ? NEAR_TITLE.sameOutlet : NEAR_TITLE.anyOutlet;-- if (score >= threshold && (!best || score > best.score)) best = { c, score, sameOutlet };-- }-- if (best) return { of: best.c.duplicate_of || best.c.id, rule: 'near_title', detail: `jaccard ${best.score.toFixed(2)} of headline 2-shingles${best.sameOutlet ? ' (same outlet)' : ''}` };-- return null;-- }---- async function ingestedEvent(row, action, extra = {}) {-- await outbox.emit({-- event_type: 'news.source.ingested', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'low',-- subject: { type: 'source_item', id: row.id },-- payload: {-- item_id: row.id, action, status: row.status,-- sources_item: { service: 'sources', type: 'item', id: row.sources_item_id, revision: row.sources_revision },-- source_key: row.source_key, canonical_url: row.canonical_url, duplicate_of: row.duplicate_of,-- dedupe_rule: row.dedupe ? JSON.parse(row.dedupe).rule : null, cluster_id: row.cluster_id, ...extra,-- },-- });-- }---- /**-- * Apply one Sources item (inside a transaction the caller may already hold). Returns-- * { outcome: created|updated|removed|unchanged|ignored|rejected, item?, reason? }.-- */-- async function apply(item, { origin = 'pull' } = {}) {-- return await db.tx(async () => {-- if (!item || typeof item.id !== 'string') return { outcome: 'rejected', reason: 'item without id' };-- if (item.category && item.category !== config.sources.category) return { outcome: 'ignored', reason: `category ${item.category}` };-- const revision = Number.isInteger(item.revision) ? item.revision : 1;-- const cur = await q.bySourcesId.get(item.id);-- if (item.removed) return await applyRemoval({ sourcesItemId: item.id, reason: item.removed.reason || 'removed upstream', revision, origin });-- if (cur && cur.sources_revision >= revision) return { outcome: 'unchanged', item: cur };-- const n = await normalise(item);-- if (!n.headline) {-- await recordRun({ origin, state: 'rejected', source_key: item.source_key || null, sources_item_id: item.id, error_code: 'item.no_headline', detail: 'a Sources item without a title cannot be shown' });-- await failedEvent({ origin, sourceKey: item.source_key || null, sourcesItemId: item.id, state: 'rejected', errorCode: 'item.no_headline' });-- return { outcome: 'rejected', reason: 'no headline' };-- }-- const now = store.now();-- if (cur) {-- if (cur.status === 'removed') return { outcome: 'unchanged', item: cur }; // removal is sticky here too-- await db.prepare(`UPDATE news_source_items SET sources_revision = @rev, canonical_url = @canonical_url, url_key = @url_key, headline = @headline,-- outlet = @outlet, authors = @authors, published_at = @published_at, summary = @summary, summary_basis = @summary_basis,-- license_note = @license_note, terms_note = @terms_note, content_hash = @content_hash, title_key = @title_key,-- retrieved_at = @retrieved_at, upstream_updated_at = @now, updated_at = @now WHERE id = @id`)-- .run({ ...n, rev: revision, now, id: cur.id });-- const row = await q.byId.get(cur.id);-- if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);-- await ingestedEvent(row, 'updated', { previous_revision: cur.sources_revision });-- const affected = stories ? await stories.onUpstreamChange(row, 'source_updated', { previous: cur }) : [];-- return { outcome: 'updated', item: row, stories: affected };-- }-- const at = (() => { const p = n.published_at ? Date.parse(n.published_at) : NaN; return Number.isFinite(p) ? p : now; })();-- const dup = await dedupe(n, item.source_key, at);-- const id = newItemId(now);-- await db.prepare(`INSERT INTO news_source_items (id, sources_item_id, sources_revision, source_key, canonical_url, url_key, headline, outlet, authors,-- published_at, summary, summary_basis, license_note, terms_note, content_hash, title_key, status, duplicate_of, dedupe,-- retrieved_at, first_seen_at, updated_at)-- VALUES (@id, @sources_item_id, @rev, @source_key, @canonical_url, @url_key, @headline, @outlet, @authors,-- @published_at, @summary, @summary_basis, @license_note, @terms_note, @content_hash, @title_key, @status, @duplicate_of, @dedupe,-- @retrieved_at, @now, @now)`)-- .run({ ...n, id, sources_item_id: item.id, rev: revision, source_key: String(item.source_key || 'unknown'),-- status: dup ? 'duplicate' : 'active', duplicate_of: dup ? dup.of : null, dedupe: dup ? JSON.stringify({ rule: dup.rule, detail: dup.detail }) : null, now });-- await clusters.assign(await q.byId.get(id));-- const row = await q.byId.get(id);-- await ingestedEvent(row, 'created');-- return { outcome: 'created', item: row };-- });-- }---- /** A Sources removal (takedown, licence, error): sticky; the stored summary is dropped. */-- async function applyRemoval({ sourcesItemId, reason, revision = null, origin = 'webhook' }) {-- return await db.tx(async () => {-- const cur = await q.bySourcesId.get(sourcesItemId);-- if (!cur) return { outcome: 'ignored', reason: 'never ingested' };-- if (cur.status === 'removed') return { outcome: 'unchanged', item: cur };-- // A story resting on it may stop being open for comments (domain/discussion.js).-- if (discussion) await discussion.watchSource(cur.id);-- const now = store.now();-- await db.prepare(`UPDATE news_source_items SET status = 'removed', summary = NULL, summary_basis = 'removed_upstream', removed_at = ?, removed_reason = ?,-- sources_revision = GREATEST(sources_revision, COALESCE(?, sources_revision)), updated_at = ? WHERE id = ?`)-- .run(now, String(reason || 'removed upstream').slice(0, 500), revision, now, cur.id);-- const row = await q.byId.get(cur.id);-- if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);-- await ingestedEvent(row, 'removed', { reason: row.removed_reason, origin });-- const affected = stories ? await stories.onUpstreamChange(row, 'source_removed', { previous: cur }) : [];-- return { outcome: 'removed', item: row, stories: affected };-- });-- }---- /** sources.fetch.failed for a news source: recorded and relayed; nothing else changes. */-- async function upstreamFailure(payload) {-- return await db.tx(async () => {-- if (payload.category && payload.category !== config.sources.category) return { outcome: 'ignored' };-- await recordRun({ origin: 'sources', state: String(payload.state || 'failed').slice(0, 40), source_key: payload.source_key || null, error_code: payload.error_code || null,-- detail: `run ${payload.run_id || '?'}${payload.http_status ? ` HTTP ${payload.http_status}` : ''}${payload.consecutive_failures ? `, ${payload.consecutive_failures} consecutive failures` : ''}` });-- await q.putStatus.run({ source_key: payload.source_key || 'unknown', name: null, homepage_url: null, status: 'failing', stale: null, last_success_at: null, now: store.now() });-- await failedEvent({ origin: 'sources', sourceKey: payload.source_key || null, state: payload.state || 'failed', errorCode: payload.error_code || null, httpStatus: payload.http_status || null,-- extra: { run_id: payload.run_id || null, consecutive_failures: payload.consecutive_failures || null } });-- return { outcome: 'recorded' };-- });-- }---- /** Remember what a Sources page says about its sources' health (display only). */-- async function rememberSources(map) {-- for (const [key, h] of Object.entries(map || {})) {-- await q.putStatus.run({ source_key: key, name: null, homepage_url: null, status: h.status || null, stale: h.stale == null ? null : (h.stale ? 1 : 0), last_success_at: h.last_success_at || null, now: store.now() });-- }-- }---- /** Fetch registry names we do not know yet (needs sources.source.read; failure = keep the host name). */-- async function learnOutlets(keys) {-- for (const key of [...new Set(keys)]) {-- const st = await q.status.get(key);-- if (st && st.name) continue;-- try {-- const s = await sources.getSource(key);-- await q.putStatus.run({ source_key: key, name: s && s.name ? String(s.name).slice(0, 200) : null, homepage_url: s && s.homepage_url ? String(s.homepage_url) : null,-- status: s && s.health ? s.health.status || null : null, stale: s && s.health && s.health.stale != null ? (s.health.stale ? 1 : 0) : null,-- last_success_at: s && s.health ? s.health.last_success_at || null : null, now: store.now() });-- } catch { /* optional grant; the outlet stays the URL's host */ }-- }-- }---- /**-- * The cursor pull. Reads pages from the stored cursor until Sources says there is no more (or-- * the page budget is spent), applying each item. The cursor moves only past applied pages.-- */-- let pulling = null;-- async function pull() {-- if (pulling) return pulling;-- pulling = (async () => {-- const before = await store.getState('sources_cursor', 0);-- let after = before;-- const counts = { created: 0, updated: 0, removed: 0, unchanged: 0, ignored: 0, rejected: 0, duplicates: 0 };-- try {-- for (let page = 0; page < config.sources.maxPagesPerRun; page++) {-- const body = await sources.listItems({ after, limit: config.sources.pageSize });-- await rememberSources(body.sources);-- await learnOutlets(body.items.map((i) => i.source_key).filter(Boolean));-- await store.tx(async () => {-- for (const item of body.items) {-- const r = await apply(item, { origin: 'pull' });-- counts[r.outcome] = (counts[r.outcome] || 0) + 1;-- if (r.outcome === 'created' && r.item.status === 'duplicate') counts.duplicates++;-- }-- await store.setState('sources_cursor', body.next_after);-- });-- after = body.next_after;-- if (!body.more) break;-- }-- await store.tx(async () => await recordRun({ origin: 'pull', state: 'ok', counts, cursor_before: before, cursor_after: after }));-- return { ok: true, counts, cursor: after };-- } catch (err) {-- const prev = await q.lastPull.get();-- await store.tx(async () => {-- await recordRun({ origin: 'pull', state: 'failed', error_code: err.code || 'pull.error', detail: err.message, counts, cursor_before: before, cursor_after: after });-- // One event per outage, not one per retry.-- if (!prev || prev.state !== 'failed') await failedEvent({ origin: 'pull', state: 'failed', errorCode: err.code || 'pull.error', detail: err.message, httpStatus: err.status || null });-- });-- log.warn(`[News] Sources pull failed (${err.code || 'error'}): ${err.message}`);-- return { ok: false, error: err.code || 'pull.error', detail: err.message, counts, cursor: after };-- } finally { pulling = null; }-- })();-- return pulling;-- }---- return {-- NEAR_TITLE,-- setStories(s) { stories = s; },-- normalise, dedupe, apply, applyRemoval, upstreamFailure, pull, recordRun, failedEvent, learnOutlets, rememberSources,-- get: async (id) => await q.byId.get(String(id || '')) || null,-- bySourcesId: async (id) => await q.bySourcesId.get(String(id || '')) || null,-- runs: async ({ limit = 30 } = {}) => await db.prepare('SELECT * FROM news_ingest_runs ORDER BY id DESC LIMIT ?').all(limit),-- sourceStatus: async () => await db.prepare('SELECT * FROM news_source_status ORDER BY source_key').all(),-- lastPull: async () => await q.lastPull.get() || null,-- };--}----module.exports = { createIngest, NEAR_TITLE };@@@ -9,7 +9,7 @@** News' gate policy: at least NEWS_MIN_INDEPENDENT_SOURCES (2) independent sources, at least 40* words. The sources counted are the Sources items the published revision's paragraphs cite and-- * that are still live upstream, grouped by text.independentSources: items from the same Sources++ * that are still live upstream, grouped by normalize.independentSources: items from the same Sources* source, the same publisher's domain, the same outlet, or copies of the same original report are* one source. A story on fewer is still published and readable, but noindex ('unsourced'), so it* stays out of sitemaps. A paragraph whose every source was removed upstream counts as an@@@ -22,7 +22,8 @@const seo = require('openvibe-publishing/seo');const hooks = require('openvibe-publishing/index-hooks');const authorship = require('openvibe-publishing/authorship');--const { independentSources } = require('./text');++const { createPublication: createPublicationGlue } = require('openvibe-publishing/publication');++const { normalize } = require('openvibe-publishing/ingest');// minSources counts independent sources; config.indexing.minIndependentSources overrides it.const POLICY = Object.freeze({ minWords: 40, requireSources: true, minSources: 2 });@@@ -47,9 +48,13 @@ function disclosure(rec, review) {return { ...d, long: `${d.long.replace(/\.\s*$/, '')}. ${reviewed ? 'Reviewed by a person.' : 'Not yet reviewed by a person.'}` };}--function createPublication({ store, config, outbox }) {++function createPublication({ store, config, outbox, indexnow }) {const { db } = store;const policy = Object.freeze({ ...POLICY, minSources: (config.indexing && config.indexing.minIndependentSources) || POLICY.minSources });++ // The chassis owns the mechanics: the sequencer revision, the search.index-document@1 event, the++ // outbox write and the IndexNow ping — all on the caller's transaction handle (this product's++ // gate, document builder and canonical path stay here).++ const glue = createPublicationGlue({ owner: OWNER, sequencer: store.sequencer, outbox, baseUrl: config.baseUrl, indexnow, db, now: store.now });const storyById = db.prepare('SELECT * FROM news_stories WHERE id = ?');const itemById = db.prepare('SELECT * FROM news_source_items WHERE id = ?');const topicById = db.prepare('SELECT * FROM news_topics WHERE id = ?');@@@ -83,7 +88,7 @@ function createPublication({ store, config, outbox }) {for (const n of ns) if (alive(n)) cited.add(n);if (!ns.some(alive)) unsupported++;}-- const groups = independentSources([...cited].map((n) => live.get(n)));++ const groups = normalize.independentSources([...cited].map((n) => live.get(n)));return { citedLive: cited.size, independent: groups.length, groups, unsupported, live };}@@@ -153,11 +158,7 @@ function createPublication({ store, config, outbox }) {/** Stamp and enqueue the Search document when it changed. Inside the caller's transaction. */async function syncIndex(story, { traceparent } = {}) {const { doc } = await documentFor(story, { forSearch: true });-- const prev = await store.sequencer.current(OWNER, 'story', story.id);-- if (doc.deleted && prev == null) return null;-- const stamped = await store.sequencer.stamp(store.db, doc);-- if (prev != null && stamped.revision === prev) return null;-- return await outbox.emit(hooks.indexEvent({ document: stamped, now: store.now() }), { traceparent });++ return await glue.index(store.db, { document: doc, page: storyPath(story), traceparent });}function snapshot(story) {@@@ -172,14 +173,15 @@ function createPublication({ store, config, outbox }) {const story = await storyById.get(storyId);const after = snapshot(story);let event = null;++ let decision = null;if (after.state === 'retracted' && (!before || before.state !== 'retracted')) {-- const d = await decide(story, await store.revisions.get(story.id, story.published_revision));++ decision = await decide(story, await store.revisions.get(story.id, story.published_revision));event = await outbox.emit({event_type: 'news.story.retracted', version: 1, source: OWNER, actor: actorRef(actor),timestamp: new Date(store.now()).toISOString(),-- visibility: d.listable ? 'public' : 'internal',++ visibility: decision.listable ? 'public' : 'internal',subject: { type: 'story', id: story.id, revision: story.published_revision || 0 },-- payload: { canonical_url: storyUrl(story), publication_state: 'retracted', indexability: hooks.searchIndexability(d), ...extra },++ payload: { canonical_url: storyUrl(story), publication_state: 'retracted', indexability: hooks.searchIndexability(decision), ...extra },}, { traceparent });} else {// A retracted story is still published (with its notice) for the product events.@@@ -187,15 +189,19 @@ function createPublication({ store, config, outbox }) {let action = hooks.actionFor(norm(before), norm(after));if (!action && before && before.state === 'published' && after.state === 'published' && before.url !== after.url) action = 'updated';if (action) {-- const { doc, decision } = await documentFor(story, { forSearch: false });++ const { doc, decision: dec } = await documentFor(story, { forSearch: false });++ decision = dec;const topic = await topicOf(story);-- event = await outbox.emit(hooks.publicationEvent({-- product: OWNER, type: 'story', action, id: story.id, revision: story.published_revision || 0,-- actor: actorRef(actor), document: doc, decision, now: store.now(),++ event = await glue.publication(store.db, {++ type: 'story', action, id: story.id, revision: story.published_revision || 0,++ actor: actorRef(actor), document: doc, decision, traceparent,extra: { ...(topic ? { topic: topic.slug } : {}), ...extra },-- }), { traceparent });++ });}}++ // The chassis pings IndexNow inside syncIndex when the stamped document is an indexable page++ // that appeared or changed, or a page Search already had that went away; never a draft,++ // private or noindex page.await syncIndex(story, { traceparent });return { story, event };}nnew file mode 100644@@@ -0,0 +1,302 @@++'use strict';++++/**++ * OpenVibe.Sources items (category news) → news_source_items → dedupe → clusters.++ *++ * The chassis (openvibe-publishing/ingest) owns the Sources client, the change cursor and the pull++ * loop; this module keeps News' domain half: what a Sources item becomes in News' tables (licensing++ * limits included), dedupe, cluster assignment, upstream changes and the recorded run history.++ *++ * Two ways in, one effect:++ * - the Events webhook (sources.item.created|updated|removed, sources.fetch.failed), each event++ * applied once through the inbox (http/events.js);++ * - the cursor pull (worker), the backstop for missed deliveries: the chassis reads++ * /api/v1/items in change order from the cursor in news_ingest_cursor and calls apply() per item.++ * apply() is idempotent on (Sources item id, Sources revision): a replay, or the same item arriving++ * by both paths, changes nothing and emits nothing.++ *++ * What is stored (licensing): headline, canonical URL, outlet, authors, published_at, the licence++ * and terms notes, and a short summary ONLY when those notes explicitly allow short summaries++ * (normalize.licensedSummary). Every other field of the Sources item — whatever an adapter mapped++ * into `fields` — is ignored, so an article body can never be stored, shown, fed or indexed.++ *++ * Failure is a recorded state (news_ingest_runs + news.source.failed), never a replacement: a++ * failed fetch or an unreadable item creates no source item, no cluster and no story text.++ *++ * Upstream changes to an item a story cites (a revision, a removal) go to stories.onUpstreamChange,++ * which flags the story and prepares a pending revision; published text is never changed silently.++ * A removal can also hide the story's Community thread (domain/discussion.js, after the commit).++ */++const { ids } = require('openvibe-contracts');++const { createChangeCursor, pullChanges, normalize } = require('openvibe-publishing/ingest');++++const NEAR_TITLE = { anyOutlet: 0.9, sameOutlet: 0.75, windowMs: 48 * 3600 * 1000 };++const newItemId = (now) => `nsi_${ids.ulid(now)}`;++const CURSOR = 'sources';++++// The chassis' three pull outcomes: News' own apply() outcome decides which one it is.++const PULL_OUTCOME = { created: 'applied', updated: 'applied', unchanged: 'applied', ignored: 'hold', rejected: 'applied', removed: 'removed' };++++function createIngest({ store, config, clusters, outbox, sources, discussion = null, log = console }) {++ const { db } = store;++ const cursor = createChangeCursor(db, { prefix: 'news', now: store.now });++ let stories = null; // set by app.js (stories depend on ingest's items too)++ const q = {++ bySourcesId: db.prepare('SELECT * FROM news_source_items WHERE sources_item_id = ?'),++ byId: db.prepare('SELECT * FROM news_source_items WHERE id = ?'),++ byUrl: db.prepare("SELECT * FROM news_source_items WHERE url_key = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),++ byHash: db.prepare("SELECT * FROM news_source_items WHERE content_hash = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),++ recent: db.prepare("SELECT * FROM news_source_items WHERE status <> 'removed' AND first_seen_at >= ? ORDER BY first_seen_at, seq"),++ status: db.prepare('SELECT * FROM news_source_status WHERE source_key = ?'),++ putStatus: db.prepare(`INSERT INTO news_source_status (source_key, name, homepage_url, status, stale, last_success_at, refreshed_at)++ VALUES (@source_key, @name, @homepage_url, @status, @stale, @last_success_at, @now)++ ON CONFLICT (source_key) DO UPDATE SET name = COALESCE(excluded.name, news_source_status.name), homepage_url = COALESCE(excluded.homepage_url, news_source_status.homepage_url),++ status = COALESCE(excluded.status, news_source_status.status), stale = COALESCE(excluded.stale, news_source_status.stale),++ last_success_at = COALESCE(excluded.last_success_at, news_source_status.last_success_at), refreshed_at = excluded.refreshed_at`),++ run: db.prepare(`INSERT INTO news_ingest_runs (origin, state, source_key, sources_item_id, error_code, detail, counts, cursor_before, cursor_after, at)++ VALUES (@origin, @state, @source_key, @sources_item_id, @error_code, @detail, @counts, @cursor_before, @cursor_after, @at)`),++ lastPull: db.prepare("SELECT * FROM news_ingest_runs WHERE origin = 'pull' ORDER BY id DESC LIMIT 1"),++ };++++ async function recordRun(r) {++ await q.run.run({ source_key: null, sources_item_id: null, error_code: null, detail: null, counts: null, cursor_before: null, cursor_after: null, ...r, counts: r.counts ? JSON.stringify(r.counts) : null, at: store.now() });++ }++++ async function failedEvent({ origin, sourceKey = null, sourcesItemId = null, state, errorCode = null, detail = null, httpStatus = null, extra = {} }) {++ await outbox.emit({++ event_type: 'news.source.failed', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'important',++ subject: sourceKey ? { type: 'source', id: sourceKey } : { type: 'ingest', id: origin },++ payload: { origin, source_key: sourceKey, sources_item_id: sourcesItemId, state, error_code: errorCode, http_status: httpStatus, detail: detail ? String(detail).slice(0, 500) : null, ...extra },++ });++ }++++ async function outletFor(item) {++ const st = await q.status.get(item.source_key);++ return (st && st.name) || normalize.hostOf(item.canonical_url) || item.source_key;++ }++++ /** The fields News keeps from a Sources item. Nothing else of the item is read. */++ async function normalise(item) {++ const prov = item.provenance || {};++ const headline = String(item.title || '').replace(/\s+/g, ' ').trim().slice(0, 500);++ const lic = normalize.licensedSummary(item.summary, { termsNote: prov.terms_note, licenseNote: prov.license_note, maxChars: config.licensing.summaryMaxChars });++ const authors = Array.isArray(item.authors) ? item.authors.map((a) => String(a || '').replace(/\s+/g, ' ').trim().slice(0, 200)).filter(Boolean).slice(0, 10) : [];++ const published = item.published_at && Number.isFinite(Date.parse(item.published_at)) ? new Date(Date.parse(item.published_at)).toISOString() : null;++ return {++ headline,++ canonical_url: item.canonical_url || null,++ url_key: normalize.urlKey(item.canonical_url),++ outlet: await outletFor(item),++ authors: JSON.stringify(authors),++ published_at: published,++ summary: lic.summary,++ summary_basis: lic.basis,++ license_note: prov.license_note ? String(prov.license_note).slice(0, 2000) : null,++ terms_note: prov.terms_note ? String(prov.terms_note).slice(0, 2000) : null,++ content_hash: prov.content_hash || null,++ title_key: normalize.titleKey(headline),++ retrieved_at: prov.retrieved_at || null,++ };++ }++++ /** Is this new item a duplicate of one we have? → null | { of, rule, detail } */++ async function dedupe(n, sourceKey, at) {++ if (n.url_key) {++ const hit = await q.byUrl.get(n.url_key);++ if (hit) return { of: hit.duplicate_of || hit.id, rule: 'canonical_url', detail: n.url_key };++ }++ if (n.content_hash) {++ const hit = await q.byHash.get(n.content_hash);++ if (hit) return { of: hit.duplicate_of || hit.id, rule: 'content_hash', detail: n.content_hash };++ }++ const mine = normalize.shingles(n.headline);++ const host = normalize.hostOf(n.canonical_url);++ let best = null;++ for (const c of await q.recent.all(Math.min(at, store.now()) - NEAR_TITLE.windowMs)) {++ const t = clusters.itemTime(c);++ if (Math.abs(t - at) > NEAR_TITLE.windowMs) continue;++ const score = normalize.jaccard(mine, normalize.shingles(c.headline));++ const sameOutlet = c.source_key === sourceKey || (host && normalize.hostOf(c.canonical_url) === host);++ const threshold = sameOutlet ? NEAR_TITLE.sameOutlet : NEAR_TITLE.anyOutlet;++ if (score >= threshold && (!best || score > best.score)) best = { c, score, sameOutlet };++ }++ if (best) return { of: best.c.duplicate_of || best.c.id, rule: 'near_title', detail: `jaccard ${best.score.toFixed(2)} of headline 2-shingles${best.sameOutlet ? ' (same outlet)' : ''}` };++ return null;++ }++++ async function ingestedEvent(row, action, extra = {}) {++ await outbox.emit({++ event_type: 'news.source.ingested', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'low',++ subject: { type: 'source_item', id: row.id },++ payload: {++ item_id: row.id, action, status: row.status,++ sources_item: { service: 'sources', type: 'item', id: row.sources_item_id, revision: row.sources_revision },++ source_key: row.source_key, canonical_url: row.canonical_url, duplicate_of: row.duplicate_of,++ dedupe_rule: row.dedupe ? JSON.parse(row.dedupe).rule : null, cluster_id: row.cluster_id, ...extra,++ },++ });++ }++++ /**++ * Apply one Sources item (inside the caller's transaction). Returns++ * { outcome: created|updated|removed|unchanged|ignored|rejected, item?, reason? }.++ */++ async function apply(item, { origin = 'pull' } = {}) {++ return await db.tx(async () => {++ if (!item || typeof item.id !== 'string') return { outcome: 'rejected', reason: 'item without id' };++ if (item.category && item.category !== config.sources.category) return { outcome: 'ignored', reason: `category ${item.category}` };++ const revision = Number.isInteger(item.revision) ? item.revision : 1;++ const cur = await q.bySourcesId.get(item.id);++ if (item.removed) return await applyRemoval({ sourcesItemId: item.id, reason: item.removed.reason || 'removed upstream', revision, origin });++ if (cur && cur.sources_revision >= revision) return { outcome: 'unchanged', item: cur };++ const n = await normalise(item);++ if (!n.headline) {++ await recordRun({ origin, state: 'rejected', source_key: item.source_key || null, sources_item_id: item.id, error_code: 'item.no_headline', detail: 'a Sources item without a title cannot be shown' });++ await failedEvent({ origin, sourceKey: item.source_key || null, sourcesItemId: item.id, state: 'rejected', errorCode: 'item.no_headline' });++ return { outcome: 'rejected', reason: 'no headline' };++ }++ const now = store.now();++ if (cur) {++ if (cur.status === 'removed') return { outcome: 'unchanged', item: cur }; // removal is sticky here too++ await db.prepare(`UPDATE news_source_items SET sources_revision = @rev, canonical_url = @canonical_url, url_key = @url_key, headline = @headline,++ outlet = @outlet, authors = @authors, published_at = @published_at, summary = @summary, summary_basis = @summary_basis,++ license_note = @license_note, terms_note = @terms_note, content_hash = @content_hash, title_key = @title_key,++ retrieved_at = @retrieved_at, upstream_updated_at = @now, updated_at = @now WHERE id = @id`)++ .run({ ...n, rev: revision, now, id: cur.id });++ const row = await q.byId.get(cur.id);++ if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);++ await ingestedEvent(row, 'updated', { previous_revision: cur.sources_revision });++ const affected = stories ? await stories.onUpstreamChange(row, 'source_updated', { previous: cur }) : [];++ return { outcome: 'updated', item: row, stories: affected };++ }++ const at = (() => { const p = n.published_at ? Date.parse(n.published_at) : NaN; return Number.isFinite(p) ? p : now; })();++ const dup = await dedupe(n, item.source_key, at);++ const id = newItemId(now);++ await db.prepare(`INSERT INTO news_source_items (id, sources_item_id, sources_revision, source_key, canonical_url, url_key, headline, outlet, authors,++ published_at, summary, summary_basis, license_note, terms_note, content_hash, title_key, status, duplicate_of, dedupe,++ retrieved_at, first_seen_at, updated_at)++ VALUES (@id, @sources_item_id, @rev, @source_key, @canonical_url, @url_key, @headline, @outlet, @authors,++ @published_at, @summary, @summary_basis, @license_note, @terms_note, @content_hash, @title_key, @status, @duplicate_of, @dedupe,++ @retrieved_at, @now, @now)`)++ .run({ ...n, id, sources_item_id: item.id, rev: revision, source_key: String(item.source_key || 'unknown'),++ status: dup ? 'duplicate' : 'active', duplicate_of: dup ? dup.of : null, dedupe: dup ? JSON.stringify({ rule: dup.rule, detail: dup.detail }) : null, now });++ await clusters.assign(await q.byId.get(id));++ const row = await q.byId.get(id);++ await ingestedEvent(row, 'created');++ return { outcome: 'created', item: row };++ });++ }++++ /** A Sources removal (takedown, licence, error): sticky; the stored summary is dropped. */++ async function applyRemoval({ sourcesItemId, reason, revision = null, origin = 'webhook' }) {++ return await db.tx(async () => {++ const cur = await q.bySourcesId.get(sourcesItemId);++ if (!cur) return { outcome: 'ignored', reason: 'never ingested' };++ if (cur.status === 'removed') return { outcome: 'unchanged', item: cur };++ // A story resting on it may stop being open for comments (domain/discussion.js).++ if (discussion) await discussion.watchSource(cur.id);++ const now = store.now();++ await db.prepare(`UPDATE news_source_items SET status = 'removed', summary = NULL, summary_basis = 'removed_upstream', removed_at = ?, removed_reason = ?,++ sources_revision = GREATEST(sources_revision, COALESCE(?, sources_revision)), updated_at = ? WHERE id = ?`)++ .run(now, String(reason || 'removed upstream').slice(0, 500), revision, now, cur.id);++ const row = await q.byId.get(cur.id);++ if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);++ await ingestedEvent(row, 'removed', { reason: row.removed_reason, origin });++ const affected = stories ? await stories.onUpstreamChange(row, 'source_removed', { previous: cur }) : [];++ return { outcome: 'removed', item: row, stories: affected };++ });++ }++++ /** sources.fetch.failed for a news source: recorded and relayed; nothing else changes. */++ async function upstreamFailure(payload) {++ return await db.tx(async () => {++ if (payload.category && payload.category !== config.sources.category) return { outcome: 'ignored' };++ await recordRun({ origin: 'sources', state: String(payload.state || 'failed').slice(0, 40), source_key: payload.source_key || null, error_code: payload.error_code || null,++ detail: `run ${payload.run_id || '?'}${payload.http_status ? ` HTTP ${payload.http_status}` : ''}${payload.consecutive_failures ? `, ${payload.consecutive_failures} consecutive failures` : ''}` });++ await q.putStatus.run({ source_key: payload.source_key || 'unknown', name: null, homepage_url: null, status: 'failing', stale: null, last_success_at: null, now: store.now() });++ await failedEvent({ origin: 'sources', sourceKey: payload.source_key || null, state: payload.state || 'failed', errorCode: payload.error_code || null, httpStatus: payload.http_status || null,++ extra: { run_id: payload.run_id || null, consecutive_failures: payload.consecutive_failures || null } });++ return { outcome: 'recorded' };++ });++ }++++ /** Remember what a Sources page says about its sources' health (display only). */++ async function rememberSources(map) {++ for (const [key, h] of Object.entries(map || {})) {++ await q.putStatus.run({ source_key: key, name: null, homepage_url: null, status: h.status || null, stale: h.stale == null ? null : (h.stale ? 1 : 0), last_success_at: h.last_success_at || null, now: store.now() });++ }++ }++++ /** Fetch registry names we do not know yet (needs sources.source.read; failure = keep the host name). */++ async function learnOutlets(keys) {++ for (const key of [...new Set(keys)]) {++ const st = await q.status.get(key);++ if (st && st.name) continue;++ try {++ const s = await sources.getSource(key);++ await q.putStatus.run({ source_key: key, name: s && s.name ? String(s.name).slice(0, 200) : null, homepage_url: s && s.homepage_url ? String(s.homepage_url) : null,++ status: s && s.health ? s.health.status || null : null, stale: s && s.health && s.health.stale != null ? (s.health.stale ? 1 : 0) : null,++ last_success_at: s && s.health ? s.health.last_success_at || null : null, now: store.now() });++ } catch { /* optional grant; the outlet stays the URL's host */ }++ }++ }++++ /**++ * The cursor pull. The chassis (openvibe-publishing/ingest.pullChanges) reads pages from the++ * stored cursor until Sources says there is no more (or the page budget is spent), applying each++ * item in its own savepoint and moving the cursor only past applied pages. This module's page++ * source also refreshes the display-only source status; apply() records News' own counts.++ */++ let pulling = null;++ async function pull() {++ if (pulling) return pulling;++ pulling = (async () => {++ const before = await cursor.get(CURSOR);++ const counts = { created: 0, updated: 0, removed: 0, unchanged: 0, ignored: 0, rejected: 0, duplicates: 0, failed: 0 };++ const tally = (r) => { if (!r || !r.outcome) return; counts[r.outcome] = (counts[r.outcome] || 0) + 1; if (r.outcome === 'created' && r.item && r.item.status === 'duplicate') counts.duplicates++; };++ const pageSource = {++ listItems: async (opts) => {++ const body = await sources.listItems(opts);++ await rememberSources(body.sources);++ await learnOutlets((body.items || []).map((i) => i.source_key).filter(Boolean));++ return body;++ },++ };++ try {++ const summary = await pullChanges({++ db, cursor, source: pageSource, name: CURSOR,++ maxPages: config.sources.maxPagesPerRun, pageSize: config.sources.pageSize,++ apply: async (item) => { const r = await apply(item, { origin: 'pull' }); tally(r); return PULL_OUTCOME[r.outcome] || 'applied'; },++ onItem: (_item, err, outcome) => { if (outcome === 'failed') { counts.failed++; log.warn(`[News] item skipped (${err && err.message ? err.message : 'error'})`); } },++ });++ await store.tx(async () => await recordRun({ origin: 'pull', state: 'ok', counts, cursor_before: before, cursor_after: summary.after }));++ return { ok: true, counts, cursor: summary.after };++ } catch (err) {++ const after = await cursor.get(CURSOR);++ const prev = await q.lastPull.get();++ await store.tx(async () => {++ await recordRun({ origin: 'pull', state: 'failed', error_code: err.code || 'pull.error', detail: err.message, counts, cursor_before: before, cursor_after: after });++ // One event per outage, not one per retry.++ if (!prev || prev.state !== 'failed') await failedEvent({ origin: 'pull', state: 'failed', errorCode: err.code || 'pull.error', detail: err.message, httpStatus: err.status || null });++ });++ log.warn(`[News] Sources pull failed (${err.code || 'error'}): ${err.message}`);++ return { ok: false, error: err.code || 'pull.error', detail: err.message, counts, cursor: after };++ } finally { pulling = null; }++ })();++ return pulling;++ }++++ return {++ NEAR_TITLE,++ setStories(s) { stories = s; },++ normalise, dedupe, apply, applyRemoval, upstreamFailure, pull, recordRun, failedEvent, learnOutlets, rememberSources,++ cursor: async () => await cursor.get(CURSOR),++ get: async (id) => await q.byId.get(String(id || '')) || null,++ bySourcesId: async (id) => await q.bySourcesId.get(String(id || '')) || null,++ runs: async ({ limit = 30 } = {}) => await db.prepare('SELECT * FROM news_ingest_runs ORDER BY id DESC LIMIT ?').all(limit),++ sourceStatus: async () => await db.prepare('SELECT * FROM news_source_status ORDER BY source_key').all(),++ lastPull: async () => await q.lastPull.get() || null,++ };++}++++module.exports = { createIngest, NEAR_TITLE, CURSOR };ddeleted file mode 100644@@@ -1,197 +0,0 @@--'use strict';----/**-- * Deterministic text helpers for ingestion: URL keys and title keys for dedupe, word shingles for-- * near-duplicate headlines, the terms and named entities clustering compares, the licensing rule-- * for source summaries, and which cited sources are independent (the index gate). Pure functions: the same input always gives the same output, so every-- * dedupe and cluster decision can be recomputed and explained.-- */----const STOPWORDS = new Set(('a about above after again against all am an and any are as at be because been before being below between both but by '-- + 'can could did do does doing down during each few for from further had has have having he her here hers him his how i if in into is it its '-- + 'itself just me more most my no nor not now of off on once only or other our ours out over own same she should so some such than that the '-- + 'their theirs them then there these they this those through to too under until up very was we were what when where which while who whom '-- + 'why will with would you your yours says said say new news report reports update updates live latest via amid over after more first year '-- + 'years today yesterday week one two three four five six seven eight nine ten'--).split(/\s+/));----// Capitalised words that start sentences or headlines far more often than they name something.--const NOT_ENTITIES = new Set(['the', 'a', 'an', 'in', 'on', 'at', 'as', 'how', 'why', 'what', 'when', 'where', 'who', 'new', 'breaking', 'update', 'live', 'watch', 'exclusive', 'opinion', 'analysis', 'report']);----/** Lowercase, strip accents and punctuation, collapse spaces. */--function fold(s) {-- return String(s == null ? '' : s).normalize('NFKD').replace(/[̀-ͯ]/g, '')-- .toLowerCase().replace(/[’']/g, '').replace(/[^a-z0-9]+/g, ' ').trim();--}----/** A headline normalised for comparison. */--function titleKey(title) { return fold(title); }----const TRACKING = /^(utm_[a-z]+|fbclid|gclid|mc_cid|mc_eid|ref|ref_src|cmpid|ocid|igshid|smid)$/i;----/**-- * The URL key dedupe compares: https, lowercase host without "www.", no fragment, no tracking-- * parameters, remaining parameters sorted, no trailing slash. null for anything that is not http(s).-- */--function urlKey(url) {-- if (!url) return null;-- let u;-- try { u = new URL(String(url)); } catch { return null; }-- if (u.protocol !== 'http:' && u.protocol !== 'https:') return null;-- const host = u.hostname.toLowerCase().replace(/^www\./, '');-- const params = [...u.searchParams.entries()].filter(([k]) => !TRACKING.test(k)).sort(([a, x], [b, y]) => (a < b ? -1 : a > b ? 1 : x < y ? -1 : x > y ? 1 : 0));-- const q = params.length ? `?${new URLSearchParams(params).toString()}` : '';-- let p = u.pathname.replace(/\/{2,}/g, '/');-- if (p.length > 1) p = p.replace(/\/+$/, '');-- return `https://${host}${u.port && u.port !== '443' && u.port !== '80' ? `:${u.port}` : ''}${p}${q}`;--}----/** The outlet shown when the registry has no name: the URL's host without "www.". */--function hostOf(url) {-- try { return new URL(String(url)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return null; }--}----// Second-level labels under a two-letter country code that are not a publisher's own name--// (bbc.co.uk, abc.net.au): the publisher is the label before them.--const COUNTRY_SECOND_LEVEL = new Set(['co', 'com', 'org', 'net', 'gov', 'edu', 'ac', 'ne', 'or', 'go', 'gob', 'nic', 'mil']);----/**-- * The publisher's domain of a URL: its host without subdomains (news.example.com and-- * www.example.com → example.com; news.bbc.co.uk → bbc.co.uk). Deliberately coarse: two sites on-- * one shared host (two blogs on one platform) count as one publisher, never the other way round.-- */--function publisherDomain(url) {-- const host = hostOf(url);-- if (!host) return null;-- if (/^[\d.]+$/.test(host) || host.startsWith('[')) return host;-- const labels = host.split('.').filter(Boolean);-- if (labels.length <= 2) return labels.join('.');-- const n = labels.length;-- const take = labels[n - 1].length === 2 && COUNTRY_SECOND_LEVEL.has(labels[n - 2]) ? 3 : 2;-- return labels.slice(-take).join('.');--}----/**-- * Independent sources among source items (news_source_items rows): items are one source when they-- * share any of-- * - the OpenVibe.Sources source (source_key: the same feed or registry entry),-- * - the publisher's domain of their canonical URL (publisherDomain),-- * - the outlet name (folded),-- * - the original they duplicate (duplicate_of: the same report syndicated or copied),-- * and the relation is transitive. → groups of item ids, each group one independent source.-- */--function independentSources(items) {-- const parent = new Map();-- const find = (k) => { while (parent.get(k) !== k) { parent.set(k, parent.get(parent.get(k))); k = parent.get(k); } return k; };-- const union = (a, b) => { const x = find(a); const y = find(b); if (x !== y) parent.set(y, x); };-- for (const it of items) {-- const node = `item:${it.id}`;-- if (!parent.has(node)) parent.set(node, node);-- const domain = publisherDomain(it.canonical_url);-- const outlet = fold(it.outlet);-- const keys = [`source:${it.source_key}`, `original:${it.duplicate_of || it.id}`, domain ? `domain:${domain}` : null, outlet ? `outlet:${outlet}` : null].filter(Boolean);-- for (const k of keys) {-- if (!parent.has(k)) parent.set(k, k);-- union(node, k);-- }-- }-- const groups = new Map();-- for (const it of items) {-- const root = find(`item:${it.id}`);-- if (!groups.has(root)) groups.set(root, []);-- groups.get(root).push(it.id);-- }-- return [...groups.values()];--}----/** Very light stemming: plural and possessive endings, so "rockets" and "rocket" meet. */--function stem(w) {-- if (w.length > 4 && w.endsWith('ies')) return `${w.slice(0, -3)}y`;-- if (w.length > 3 && w.endsWith('s') && !w.endsWith('ss') && !w.endsWith('us') && !w.endsWith('is')) return w.slice(0, -1);-- return w;--}----/** Content terms of a text: folded words of 3+ letters (or any number), minus stopwords, stemmed, unique, sorted. */--function terms(text) {-- const out = new Set();-- for (const w of fold(text).split(' ')) {-- if (!w || STOPWORDS.has(w)) continue;-- if (w.length < 3 && !/^\d+$/.test(w)) continue;-- out.add(stem(w));-- }-- return [...out].sort();--}----/**-- * Named entities, deterministically: runs of Capitalised words (and ALL-CAPS acronyms of 2+-- * letters) in the original text, folded. A single capitalised word at the very start of the text-- * counts only if it is an acronym or appears capitalised again later. Sorted, unique.-- */--function entities(text) {-- const src = String(text || '');-- const tokens = src.split(/\s+/).filter(Boolean);-- const out = new Set();-- let run = [];-- const flush = () => {-- if (run.length) {-- const words = run.map((r) => r.word);-- const startsText = run[0].index === 0;-- const acronym = words.some((w) => /^[A-Z0-9]{2,}$/.test(w));-- if (!(startsText && words.length === 1 && !acronym && !laterCapitalised(words[0], run[0].index))) {-- const f = fold(words.join(' '));-- if (f && !NOT_ENTITIES.has(f) && !STOPWORDS.has(f)) out.add(f);-- }-- }-- run = [];-- };-- const laterCapitalised = (w, idx) => tokens.slice(idx + 1).some((t) => t.replace(/[^\p{L}\p{N}]/gu, '') === w);-- tokens.forEach((raw, index) => {-- const word = raw.replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, '');-- const cap = /^\p{Lu}/u.test(word) && !NOT_ENTITIES.has(word.toLowerCase());-- if (word && cap) run.push({ word, index }); else flush();-- if (/[.!?:;,]$/.test(raw)) flush();-- });-- flush();-- return [...out].sort();--}----/** Word 2-shingles of a folded headline (single-word headlines give the word itself). */--function shingles(title) {-- const w = titleKey(title).split(' ').filter(Boolean);-- if (w.length < 2) return new Set(w);-- const s = new Set();-- for (let i = 0; i < w.length - 1; i++) s.add(`${w[i]} ${w[i + 1]}`);-- return s;--}----function jaccard(a, b) {-- if (!a.size && !b.size) return 0;-- let inter = 0;-- for (const x of a) if (b.has(x)) inter++;-- return inter / (a.size + b.size - inter);--}----// ── Licensing ─────────────────────────────────────────────────----const ALLOWS_SUMMARY = /\b(short|brief)\s+summar(y|ies)\b|\bsummar(y|ies)\s+(are|is)\s+(allowed|permitted)\b/i;--const FORBIDS_SUMMARY = /\b(no|not|never|without)\s+(short\s+|brief\s+)?summar(y|ies)\b|\bsummar(y|ies)\s+(are|is)\s+not\b|\bheadlines?\s+(and\s+links\s+)?only\b|\btitles?\s+and\s+links\s+only\b/i;----/**-- * What of a source's summary News may keep: a short summary, only when the item's terms or licence-- * note explicitly allows short summaries (and nothing forbids them), cut to maxChars at a word-- * boundary. Article bodies are never kept, whatever the terms say. → { summary, basis }-- */--function licensedSummary(summary, { termsNote, licenseNote, maxChars = 280 } = {}) {-- const notes = `${termsNote || ''}\n${licenseNote || ''}`;-- const text = String(summary || '').replace(/\s+/g, ' ').trim();-- if (!text) return { summary: null, basis: 'none_provided' };-- if (!maxChars) return { summary: null, basis: 'summaries_disabled' };-- if (FORBIDS_SUMMARY.test(notes)) return { summary: null, basis: 'terms_forbid_summaries' };-- if (!ALLOWS_SUMMARY.test(notes)) return { summary: null, basis: 'terms_do_not_allow_summaries' };-- if (text.length <= maxChars) return { summary: text, basis: 'terms_allow_short_summaries' };-- const cut = text.slice(0, maxChars - 1);-- const at = cut.lastIndexOf(' ');-- return { summary: `${(at > maxChars * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.-]+$/, '')}…`, basis: 'terms_allow_short_summaries' };--}----module.exports = { fold, titleKey, urlKey, hostOf, publisherDomain, independentSources, terms, entities, shingles, jaccard, licensedSummary, stem, STOPWORDS };@@@ -259,7 +259,7 @@ function createApi(ctx) {router.get('/ingest', guard('news.cluster.read'), run(async (req) => {editorRead(req);return {-- cursor: await ctx.store.getState('sources_cursor', 0),++ cursor: await ingest.cursor(),runs: (await ingest.runs({ limit: 50 })).map((r) => ({ ...r, counts: parse(r.counts, null), at: iso(r.at) })),sources: (await ingest.sourceStatus()).map((s) => ({ ...s, stale: s.stale == null ? null : Boolean(s.stale), refreshed_at: iso(s.refreshed_at) })),};@@@ -82,7 +82,7 @@ function createEditorRoutes(ctx) {page(req, res, 'Editor desk', editor.dashboard({clusters: await Promise.all((await clusters.recent({ limit: 40 })).map(async (c) => ({ ...c, count: (await clusters.members(c.id)).length }))),stories: await stories.listAll({ limit: 100 }), flags, runs: await ingest.runs({ limit: 20 }), sources: await ingest.sourceStatus(),-- cursor: await store.getState('sources_cursor', 0), csrf: csrf(req), message: flashOf(req), aiEnabled: Boolean(ai && ai.enabled),++ cursor: await ingest.cursor(), csrf: csrf(req), message: flashOf(req), aiEnabled: Boolean(ai && ai.enabled),pullOn: Boolean(config.worker.enabled && config.sources.pullIntervalMs),}));}));nnew file mode 100644@@@ -0,0 +1,100 @@++'use strict';++++/**++ * Events consumer: POST /internal/events, the endpoint of News' OpenVibe.Events subscriptions++ * (topic patterns `sources.item.*` and `sources.fetch.failed`, see scripts/subscribe.js).++ *++ * sources.item.created|updated the item is read from Sources (sources.item.read) and applied++ * sources.item.removed applied from the payload (the removal is sticky)++ * sources.fetch.failed recorded; relayed as news.source.failed; no text is made++ *++ * The signature (X-OpenVibe-Signature v2, ±300 s) and the exactly-once inbox come from the chassis++ * (openvibe-publishing/ingest.createEventConsumer): the receipt (consumer, event_id) and the change++ * commit in one PostgreSQL transaction, so a redelivery changes nothing. When Sources cannot be read,++ * the delivery is answered 503 so Events retries it, and the failure is recorded (and relayed as++ * news.source.failed on the first attempt). The cursor pull catches anything that never arrives.++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { createEventConsumer } = require('openvibe-publishing/ingest');++++const CONSUMER = 'news-sources';++const EVT_RE = /^evt_[0-9A-HJKMNP-TV-Z]{26}$/;++++function createEvents({ config, store, ingest, sources, log = console }) {++ const router = express.Router();++ const consumer = createEventConsumer({ db: store.db, secrets: config.events.webhookSecrets, consumer: CONSUMER, now: store.now });++ const { inbox, verify } = consumer;++++ router.post('/internal/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {++ const ctx = req.ov;++ res.set('Cache-Control', 'no-store');++ if (!config.events.webhookSecrets.length) return http.sendProblem(res, 503, 'news.webhook_disabled', { detail: 'NEWS_EVENTS_SECRET is not set', ctx });++ const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0);++ // v2 only: signature over "<t>.<raw body>" and t within ±300 s (a replayed or v1-only delivery fails).++ const delivery = verify(raw, req.headers);++ if (!delivery) return http.sendProblem(res, 401, 'news.bad_signature', { detail: 'X-OpenVibe-Signature-V2 does not verify or is outside the replay window', ctx });++ const event = delivery.event;++ if (!event || typeof event.event_id !== 'string' || !EVT_RE.test(event.event_id)) {++ return http.sendProblem(res, 400, 'news.bad_delivery', { detail: 'body must be { event: <envelope>, seq }', ctx });++ }++ const attempt = Number(req.get('x-openvibe-delivery-attempt')) || 1;++ const type = String(event.event_type || '');++ const p = event.payload && typeof event.payload === 'object' ? event.payload : {};++ const fromSources = event.source === 'sources';++++ try {++ if (!fromSources || !/^sources\.(item\.(created|updated|removed)|fetch\.failed)$/.test(type)) {++ await inbox.once(CONSUMER, event.event_id, () => null);++ return res.status(204).end();++ }++ if (type === 'sources.item.removed') {++ await inbox.once(CONSUMER, event.event_id, async () => {++ const r = await ingest.applyRemoval({ sourcesItemId: String(p.item_id || ''), reason: p.reason, revision: Number.isInteger(p.revision) ? p.revision : null, origin: 'webhook' });++ await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: p.source_key || null, sources_item_id: p.item_id || null });++ return r;++ });++ return res.status(204).end();++ }++ if (type === 'sources.fetch.failed') {++ await inbox.once(CONSUMER, event.event_id, () => ingest.upstreamFailure(p));++ return res.status(204).end();++ }++ // created | updated++ if (p.category && p.category !== config.sources.category) {++ await inbox.once(CONSUMER, event.event_id, () => null);++ return res.status(204).end();++ }++ let fetched;++ try {++ if (p.source_key) await ingest.learnOutlets([String(p.source_key)]);++ fetched = await sources.getItem(String(p.item_id || ''));++ } catch (err) {++ await store.tx(async () => {++ await ingest.recordRun({ origin: 'webhook', state: 'failed', source_key: p.source_key || null, sources_item_id: p.item_id || null, error_code: err.code || 'sources.error', detail: `${type} ${event.event_id} attempt ${attempt}: ${err.message}` });++ if (attempt === 1) await ingest.failedEvent({ origin: 'webhook', sourceKey: p.source_key || null, sourcesItemId: p.item_id || null, state: 'failed', errorCode: err.code || 'sources.error', detail: err.message, httpStatus: err.status || null });++ });++ if (err.status === 404) {++ // Sources no longer has it: nothing to apply, and retrying cannot help.++ await inbox.once(CONSUMER, event.event_id, () => null);++ return res.status(204).end();++ }++ return http.sendProblem(res, 503, 'news.sources_unavailable', { detail: 'OpenVibe.Sources could not be read; retry later', ctx });++ }++ if (fetched.source) await ingest.rememberSources({ [fetched.item.source_key]: fetched.source });++ await inbox.once(CONSUMER, event.event_id, async () => {++ const r = await ingest.apply(fetched.item, { origin: 'webhook' });++ await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: fetched.item.source_key || null, sources_item_id: fetched.item.id, detail: r.reason || null });++ return r;++ });++ return res.status(204).end();++ } catch (err) {++ log.error('[News] webhook failed:', err && err.stack ? err.stack : err);++ return http.sendProblem(res, 500, 'internal.error', { detail: 'Internal error', ctx });++ }++ });++++ return router;++}++++module.exports = { createEvents, CONSUMER };ddeleted file mode 100644@@@ -1,101 +0,0 @@--'use strict';----/**-- * Events consumer: POST /internal/events, the endpoint of News' OpenVibe.Events subscriptions-- * (topic patterns `sources.item.*` and `sources.fetch.failed`, see scripts/subscribe.js).-- *-- * sources.item.created|updated the item is read from Sources (sources.item.read) and applied-- * sources.item.removed applied from the payload (the removal is sticky)-- * sources.fetch.failed recorded; relayed as news.source.failed; no text is made-- *-- * Deliveries are verified with X-OpenVibe-Signature against NEWS_EVENTS_SECRET (comma-separated-- * during rotation). Exactly once: the inbox receipt (consumer, event_id) and the change commit in-- * one PostgreSQL transaction; a redelivery is answered 204 and changes nothing. When Sources cannot be-- * read, the delivery is answered 503 so Events retries it, and the failure is recorded (and relayed-- * as news.source.failed on the first attempt). The cursor pull catches anything that never arrives.-- */--const express = require('express');--const { http } = require('openvibe-contracts');--const { verifyDeliveryV2, createPgInbox } = require('openvibe-sdk/events');----const CONSUMER = 'news-sources';--const EVT_RE = /^evt_[0-9A-HJKMNP-TV-Z]{26}$/;----function createWebhook({ config, store, ingest, sources, log = console }) {-- const router = express.Router();-- const inbox = createPgInbox(store.db, { now: store.now }); // idempotency_receipts: migrations/0001_initial.sql---- router.post('/internal/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {-- const ctx = req.ov;-- res.set('Cache-Control', 'no-store');-- const secrets = config.events.webhookSecrets;-- if (!secrets.length) return http.sendProblem(res, 503, 'news.webhook_disabled', { detail: 'NEWS_EVENTS_SECRET is not set', ctx });-- const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0);-- // v2 only: signature over "<t>.<raw body>" and t within ±300 s (a replayed or v1-only delivery fails).-- if (!secrets.some((s) => verifyDeliveryV2(raw, req.headers, s))) return http.sendProblem(res, 401, 'news.bad_signature', { detail: 'X-OpenVibe-Signature-V2 does not verify or is outside the replay window', ctx });-- let body;-- try { body = JSON.parse(raw.toString('utf8')); } catch { body = null; }-- const event = body && body.event;-- if (!event || typeof event.event_id !== 'string' || !EVT_RE.test(event.event_id)) {-- return http.sendProblem(res, 400, 'news.bad_delivery', { detail: 'body must be { event: <envelope>, seq }', ctx });-- }-- const attempt = Number(req.get('x-openvibe-delivery-attempt')) || 1;-- const type = String(event.event_type || '');-- const p = event.payload && typeof event.payload === 'object' ? event.payload : {};-- const fromSources = event.source === 'sources';---- try {-- if (!fromSources || !/^sources\.(item\.(created|updated|removed)|fetch\.failed)$/.test(type)) {-- await inbox.once(CONSUMER, event.event_id, () => null);-- return res.status(204).end();-- }-- if (type === 'sources.item.removed') {-- await inbox.once(CONSUMER, event.event_id, async () => {-- const r = await ingest.applyRemoval({ sourcesItemId: String(p.item_id || ''), reason: p.reason, revision: Number.isInteger(p.revision) ? p.revision : null, origin: 'webhook' });-- await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: p.source_key || null, sources_item_id: p.item_id || null });-- return r;-- });-- return res.status(204).end();-- }-- if (type === 'sources.fetch.failed') {-- await inbox.once(CONSUMER, event.event_id, () => ingest.upstreamFailure(p));-- return res.status(204).end();-- }-- // created | updated-- if (p.category && p.category !== config.sources.category) {-- await inbox.once(CONSUMER, event.event_id, () => null);-- return res.status(204).end();-- }-- let fetched;-- try {-- if (p.source_key) await ingest.learnOutlets([String(p.source_key)]);-- fetched = await sources.getItem(String(p.item_id || ''));-- } catch (err) {-- await store.tx(async () => {-- await ingest.recordRun({ origin: 'webhook', state: 'failed', source_key: p.source_key || null, sources_item_id: p.item_id || null, error_code: err.code || 'sources.error', detail: `${type} ${event.event_id} attempt ${attempt}: ${err.message}` });-- if (attempt === 1) await ingest.failedEvent({ origin: 'webhook', sourceKey: p.source_key || null, sourcesItemId: p.item_id || null, state: 'failed', errorCode: err.code || 'sources.error', detail: err.message, httpStatus: err.status || null });-- });-- if (err.status === 404) {-- // Sources no longer has it: nothing to apply, and retrying cannot help.-- await inbox.once(CONSUMER, event.event_id, () => null);-- return res.status(204).end();-- }-- return http.sendProblem(res, 503, 'news.sources_unavailable', { detail: 'OpenVibe.Sources could not be read; retry later', ctx });-- }-- if (fetched.source) await ingest.rememberSources({ [fetched.item.source_key]: fetched.source });-- await inbox.once(CONSUMER, event.event_id, async () => {-- const r = await ingest.apply(fetched.item, { origin: 'webhook' });-- await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: fetched.item.source_key || null, sources_item_id: fetched.item.id, detail: r.reason || null });-- return r;-- });-- return res.status(204).end();-- } catch (err) {-- log.error('[News] webhook failed:', err && err.stack ? err.stack : err);-- return http.sendProblem(res, 500, 'internal.error', { detail: 'Internal error', ctx });-- }-- });---- return router;--}----module.exports = { createWebhook, CONSUMER };nnew file mode 100644@@@ -0,0 +1,111 @@++'use strict';++/**++ * T9 J2: News runs on the openvibe-publishing 1.1.0 chassis. This proves the conversion:++ * - the local copies the brief's §7 lists are gone (nothing under server/ requires them);++ * - a dry-run ingest from the captured Sources fixture produces the same domain rows as the++ * pre-conversion code (test/fixtures/ingest-golden.json, captured before the change);++ * - the Search document validates against search.index-document@1, with its provenance and++ * visibility;++ * - the outbox row and the state change share the caller's transaction (a rollback drops both).++ */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const contracts = require('openvibe-contracts');++const { boot, check, done } = require('./helpers/boot');++const { feedFixture, reviseFixture, project } = require('./helpers/golden');++++const ROOT = path.join(__dirname, '..');++const GOLDEN = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'ingest-golden.json'), 'utf8'));++++// The brief's §7 deletion list for News: the chassis (openvibe-publishing/ingest and /publication)++// owns their code now. The harness denies rm, so the files may remain on disk; they must not be++// required by any live server file.++const DELETED = ['server/domain/ingest.js', 'server/domain/text.js', 'server/http/webhook.js', 'server/clients/sources.js']++ .map((f) => path.join(ROOT, f));++++const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';++++function jsFiles(dir, out = []) {++ for (const e of fs.readdirSync(dir, { withFileTypes: true })) {++ const p = path.join(dir, e.name);++ if (e.isDirectory()) jsFiles(p, out);++ else if (e.name.endsWith('.js')) out.push(p);++ }++ return out;++}++function relativeRequires(file) {++ const src = fs.readFileSync(file, 'utf8');++ return [...src.matchAll(/require\(\s*['"]([^'"]+)['"]\s*\)/g)].map((m) => m[1]).filter((s) => s.startsWith('.'));++}++++/** An editor's story resting on exactly these Sources items (cited in the one paragraph), published. */++async function storyOn(t, headline, items) {++ let r = await t.api('/stories', { json: { headline } });++ assert.strictEqual(r.status, 201, r.text);++ const story = r.json().story;++ for (const it of items) {++ r = await t.api(`/stories/${story.id}/sources`, { json: { item: it.id } });++ assert.ok(r.status === 200 || r.status === 201, r.text);++ }++ r = await t.api(`/stories/${story.id}/revisions`, { json: { body: `${PARA} [${items.map((_, i) => i + 1).join(', ')}]`, expected_revision: 0 } });++ assert.strictEqual(r.status, 201, r.text);++ r = await t.api(`/stories/${story.id}/publish`, { json: {} });++ assert.strictEqual(r.status, 200, r.text);++ return story;++}++++(async () => {++ await check('the local chassis copies are gone: no live server file requires them', () => {++ const gone = new Set(DELETED);++ for (const file of jsFiles(path.join(ROOT, 'server'))) {++ if (gone.has(file)) continue; // the (undeletable) files themselves++ for (const spec of relativeRequires(file)) {++ let resolved;++ try { resolved = require.resolve(path.resolve(path.dirname(file), spec)); } catch { continue; }++ assert.ok(!gone.has(resolved), `${path.relative(ROOT, file)} still requires ${spec}`);++ }++ }++ // ...and the chassis entry points are the ones the app uses.++ assert.match(fs.readFileSync(path.join(ROOT, 'server', 'app.js'), 'utf8'), /openvibe-publishing\/ingest/);++ assert.match(fs.readFileSync(path.join(ROOT, 'server', 'app.js'), 'utf8'), /\.\/domain\/source-items/);++ assert.match(fs.readFileSync(path.join(ROOT, 'server', 'domain', 'publication.js'), 'utf8'), /openvibe-publishing\/publication/);++ });++++ const t = await boot();++ const reports = feedFixture(t);++ await t.pull();++ reviseFixture(t, reports);++ await t.pull();++++ await check('a dry-run ingest from the captured Sources fixture produces the same domain rows as before', async () => {++ assert.deepStrictEqual(await project(t), GOLDEN);++ });++++ await check('the Search document validates against search.index-document@1, with its provenance and visibility', async () => {++ const story = await storyOn(t, 'Two new water treatment plants confirmed', [reports.a, reports.c]);++ const doc = (await t.events('news.index_document.upserted')).filter((e) => e.payload.id === story.id).pop().payload;++ const v = contracts.validate('search.index-document@1', doc);++ assert.ok(v.valid, JSON.stringify(v.errors));++ assert.strictEqual(doc.visibility, 'public');++ assert.strictEqual(doc.indexability.decision, 'index');++ assert.deepStrictEqual(doc.provenance.filter((p) => p.service === 'sources').map((p) => p.id).sort(), [reports.a.id, reports.c.id].sort());++ });++++ await check('the outbox row and the state change share the caller\'s transaction (a rollback drops both)', async () => {++ const story = (await t.db().prepare('SELECT * FROM news_stories WHERE state = \'published\' LIMIT 1').get());++ const row = async () => await t.db().prepare('SELECT state, noindex FROM news_stories WHERE id = ?').get(story.id);++ const outboxCount = async () => (await t.db().prepare('SELECT COUNT(*) AS n FROM event_outbox').get()).n;++ const before = await outboxCount();++ await assert.rejects(t.ctx.store.db.tx(async () => {++ await t.ctx.store.db.prepare('UPDATE news_stories SET noindex = 1 WHERE id = ?').run(story.id);++ await t.ctx.publication.syncIndex(await t.db().prepare('SELECT * FROM news_stories WHERE id = ?').get(story.id));++ throw new Error('rollback: the outbox row and the state change must not survive together');++ }));++ assert.strictEqual(await outboxCount(), before, 'no outbox row survives the rollback');++ assert.strictEqual((await row()).noindex, 0, 'the state change rolled back with it');++ });++++ await t.close();++ done();++})().catch((err) => { console.error(err); process.exit(1); });@@@ -53,7 +53,7 @@ const DIR = path.join(ROOT, 'docs', 'capabilities-proposal');});await check('every event type the code emits is declared', async () => {-- const src = ['server/domain/stories.js', 'server/domain/publication.js', 'server/domain/ingest.js', 'server/domain/clusters.js']++ const src = ['server/domain/stories.js', 'server/domain/publication.js', 'server/domain/source-items.js', 'server/domain/clusters.js'].map((f) => fs.readFileSync(path.join(ROOT, f), 'utf8')).join('\n');const found = [...src.matchAll(/event_type: '([a-z_.]+)'/g)].map((m) => m[1]);assert.ok(found.length >= 6);nnew file mode 100644@@@ -0,0 +1,224 @@++{++ "items": [++ {++ "sources_item_id": "itm_00000000000000000000000001",++ "sources_revision": 1,++ "source_key": "wire-a",++ "canonical_url": "https://wire-a.example/space/europa-clipper-launch",++ "url_key": "https://wire-a.example/space/europa-clipper-launch",++ "headline": "NASA launches Europa Clipper probe to Jupiter moon Europa",++ "outlet": "Wire A",++ "authors": "[\"Ana Reporter\"]",++ "published_at": "2026-09-22T09:00:00.000Z",++ "summary": "The Europa Clipper spacecraft lifted off on a Falcon Heavy rocket from Kennedy Space Center.",++ "summary_basis": "terms_allow_short_summaries",++ "license_note": null,++ "terms_note": "Titles, links and short summaries only; link back.",++ "content_hash": "962be516c6a75b667766bd8a9ae13d19c3e7c29db10521f0a934bff6e33b1fae",++ "title_key": "nasa launches europa clipper probe to jupiter moon europa",++ "status": "active",++ "dedupe_rule": null,++ "duplicate_of": null,++ "cluster": 1,++ "cluster_reason_rule": "new_cluster",++ "retrieved_at": "2026-09-22T11:00:00.000Z",++ "first_seen_at": 1790078400000,++ "updated_at": 1790078400000,++ "removed_at": null,++ "removed_reason": null++ },++ {++ "sources_item_id": "itm_00000000000000000000000002",++ "sources_revision": 2,++ "source_key": "paper-b",++ "canonical_url": "https://paper-b.example/2026/09/22/europa-clipper",++ "url_key": "https://paper-b.example/2026/09/22/europa-clipper",++ "headline": "Europa Clipper lifts off as NASA begins long trip to Jupiter",++ "outlet": "Paper B",++ "authors": "[\"Ben Writer\"]",++ "published_at": "2026-09-22T09:30:00.000Z",++ "summary": null,++ "summary_basis": "removed_upstream",++ "license_note": null,++ "terms_note": "Titles, links and short summaries only; link back.",++ "content_hash": "29305d93c699b16209ecd8adce4cba785c905abbadc37d58898209626e384ccc",++ "title_key": "europa clipper lifts off as nasa begins long trip to jupiter",++ "status": "removed",++ "dedupe_rule": null,++ "duplicate_of": null,++ "cluster": 1,++ "cluster_reason_rule": "shared_terms",++ "retrieved_at": "2026-09-22T11:00:00.000Z",++ "first_seen_at": 1790078400000,++ "updated_at": 1790078400000,++ "removed_at": 1790078400000,++ "removed_reason": "licence review"++ },++ {++ "sources_item_id": "itm_00000000000000000000000003",++ "sources_revision": 2,++ "source_key": "site-c",++ "canonical_url": "https://site-c.example/europa-clipper-why",++ "url_key": "https://site-c.example/europa-clipper-why",++ "headline": "Why NASA is sending Europa Clipper to Jupiter, explained",++ "outlet": "Site C",++ "authors": "[]",++ "published_at": "2026-09-22T10:00:00.000Z",++ "summary": null,++ "summary_basis": "none_provided",++ "license_note": null,++ "terms_note": "Titles, links and short summaries only; link back.",++ "content_hash": "2fc0ac9c2865a02385d2e2157455086c28bfecbb6903e5caaffb595cca48fda5",++ "title_key": "why nasa is sending europa clipper to jupiter explained",++ "status": "active",++ "dedupe_rule": null,++ "duplicate_of": null,++ "cluster": 1,++ "cluster_reason_rule": "shared_terms",++ "retrieved_at": "2026-09-22T11:00:00.000Z",++ "first_seen_at": 1790078400000,++ "updated_at": 1790078400000,++ "removed_at": null,++ "removed_reason": null++ },++ {++ "sources_item_id": "itm_00000000000000000000000005",++ "sources_revision": 1,++ "source_key": "paper-b",++ "canonical_url": "https://www.wire-a.example/space/europa-clipper-launch/?utm_source=x",++ "url_key": "https://wire-a.example/space/europa-clipper-launch",++ "headline": "Europa Clipper launch: our live coverage",++ "outlet": "Paper B",++ "authors": "[]",++ "published_at": "2026-09-22T11:00:00.000Z",++ "summary": null,++ "summary_basis": "none_provided",++ "license_note": null,++ "terms_note": "Titles, links and short summaries only; link back.",++ "content_hash": "13d92fe5d237158a8c88a3f4168b9867d8066b57480b95a1f1932c8346082c49",++ "title_key": "europa clipper launch our live coverage",++ "status": "duplicate",++ "dedupe_rule": "canonical_url",++ "duplicate_of": "itm_00000000000000000000000001",++ "cluster": 1,++ "cluster_reason_rule": "duplicate_of",++ "retrieved_at": "2026-09-22T11:00:00.000Z",++ "first_seen_at": 1790078400000,++ "updated_at": 1790078400000,++ "removed_at": null,++ "removed_reason": null++ },++ {++ "sources_item_id": "itm_00000000000000000000000006",++ "sources_revision": 1,++ "source_key": "site-c",++ "canonical_url": "https://mirror.example/copy",++ "url_key": "https://mirror.example/copy",++ "headline": "Syndicated copy",++ "outlet": "Site C",++ "authors": "[]",++ "published_at": null,++ "summary": null,++ "summary_basis": "none_provided",++ "license_note": null,++ "terms_note": "Titles, links and short summaries only; link back.",++ "content_hash": "29305d93c699b16209ecd8adce4cba785c905abbadc37d58898209626e384ccc",++ "title_key": "syndicated copy",++ "status": "duplicate",++ "dedupe_rule": "content_hash",++ "duplicate_of": "itm_00000000000000000000000002",++ "cluster": 1,++ "cluster_reason_rule": "duplicate_of",++ "retrieved_at": "2026-09-22T11:00:00.000Z",++ "first_seen_at": 1790078400000,++ "updated_at": 1790078400000,++ "removed_at": null,++ "removed_reason": null++ },++ {++ "sources_item_id": "itm_00000000000000000000000007",++ "sources_revision": 1,++ "source_key": "wire-a",++ "canonical_url": "https://wire-a.example/space/europa-clipper-launch-updated",++ "url_key": "https://wire-a.example/space/europa-clipper-launch-updated",++ "headline": "NASA launches Europa Clipper probe to Jupiter moon Europa, officials say",++ "outlet": "Wire A",++ "authors": "[]",++ "published_at": "2026-09-22T12:00:00.000Z",++ "summary": null,++ "summary_basis": "none_provided",++ "license_note": null,++ "terms_note": "Titles, links and short summaries only; link back.",++ "content_hash": "d477db3534724f4a397e7caafa010d6da238508f1152fae218cda385336ecd70",++ "title_key": "nasa launches europa clipper probe to jupiter moon europa officials say",++ "status": "duplicate",++ "dedupe_rule": "near_title",++ "duplicate_of": "itm_00000000000000000000000001",++ "cluster": 1,++ "cluster_reason_rule": "duplicate_of",++ "retrieved_at": "2026-09-22T11:00:00.000Z",++ "first_seen_at": 1790078400000,++ "updated_at": 1790078400000,++ "removed_at": null,++ "removed_reason": null++ },++ {++ "sources_item_id": "itm_00000000000000000000000008",++ "sources_revision": 1,++ "source_key": "paper-b",++ "canonical_url": "https://paper-b.example/bike-lanes",++ "url_key": "https://paper-b.example/bike-lanes",++ "headline": "City council approves new bike lanes downtown",++ "outlet": "Paper B",++ "authors": "[]",++ "published_at": "2026-09-22T12:00:00.000Z",++ "summary": null,++ "summary_basis": "none_provided",++ "license_note": null,++ "terms_note": "Titles, links and short summaries only; link back.",++ "content_hash": "176afd88e4219b4e923b30298ab1c82bbc08bf49375d138188ad7960a1906fb7",++ "title_key": "city council approves new bike lanes downtown",++ "status": "active",++ "dedupe_rule": null,++ "duplicate_of": null,++ "cluster": 2,++ "cluster_reason_rule": "new_cluster",++ "retrieved_at": "2026-09-22T11:00:00.000Z",++ "first_seen_at": 1790078400000,++ "updated_at": 1790078400000,++ "removed_at": null,++ "removed_reason": null++ }++ ],++ "clusters": [++ {++ "index": 1,++ "label": "europa clipper · jupiter · nasa",++ "status": "open",++ "terms": "{\"center\":1,\"clipper\":4,\"europa\":4,\"falcon\":1,\"heavy\":1,\"jupiter\":3,\"kennedy\":1,\"launche\":2,\"lifted\":1,\"moon\":2,\"nasa\":3,\"probe\":2,\"rocket\":1,\"space\":1,\"spacecraft\":1,\"explained\":1,\"sending\":1,\"coverage\":1,\"launch\":1,\"copy\":1,\"syndicated\":1,\"official\":1}",++ "entities": "{\"europa\":2,\"europa clipper\":4,\"falcon heavy\":1,\"jupiter\":3,\"kennedy space center\":1,\"nasa\":3}",++ "window_start": 1790067600000,++ "window_end": 1790078400000,++ "members": [++ "itm_00000000000000000000000001",++ "itm_00000000000000000000000002",++ "itm_00000000000000000000000003",++ "itm_00000000000000000000000005",++ "itm_00000000000000000000000006",++ "itm_00000000000000000000000007"++ ]++ },++ {++ "index": 2,++ "label": "approve · bike · city · council",++ "status": "open",++ "terms": "{\"approve\":1,\"bike\":1,\"city\":1,\"council\":1,\"downtown\":1,\"lane\":1}",++ "entities": "{}",++ "window_start": 1790078400000,++ "window_end": 1790078400000,++ "members": [++ "itm_00000000000000000000000008"++ ]++ }++ ]++}@@@ -56,7 +56,7 @@ async function boot(opts = {}) {let built = null;async function start() {const config = configLib.load(env);-- built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow });++ built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow, indexnow: opts.indexnow });await require('openvibe-sdk/auth').jwksClient(`${config.networkInternalUrl}/api/.well-known/jwks`).keys();server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });t.base = `http://127.0.0.1:${server.address().port}`;nnew file mode 100644@@@ -0,0 +1,70 @@++'use strict';++/**++ * The ingest golden fixture: a deterministic Sources feed and the canonical projection of the++ * domain rows it produces (news_source_items + news_story_clusters + a removed item's live state).++ * The same fixture and the same projection run before and after the chassis conversion, so++ * test/chassis.test.js can compare the projection to test/fixtures/ingest-golden.json (captured++ * from the pre-conversion code). Volatile values (ULID ids, cluster ids) are mapped to their stable++ * Sources item ids, so only domain meaning is compared.++ */++const { launchReports } = require('./boot');++++const parse = (s, d) => { try { return s ? JSON.parse(s) : d; } catch { return d; } };++++/** Add the fixture's items to the Sources mock (same order and fields as the capture run). → reports */++function feedFixture(t) {++ const reports = launchReports(t.sources);++ t.sources.addItem({ source_key: 'blog-x', category: 'blog', title: 'A blog item that News must ignore' });++ t.sources.addItem({ source_key: 'paper-b', title: 'Europa Clipper launch: our live coverage', url: 'https://www.wire-a.example/space/europa-clipper-launch/?utm_source=x', published_at: '2026-09-22T11:00:00Z' });++ t.sources.addItem({ source_key: 'site-c', title: 'Syndicated copy', url: 'https://mirror.example/copy', content_hash: reports.b.provenance.content_hash });++ t.sources.addItem({ source_key: 'wire-a', title: 'NASA launches Europa Clipper probe to Jupiter moon Europa, officials say', url: 'https://wire-a.example/space/europa-clipper-launch-updated', published_at: '2026-09-22T12:00:00Z' });++ t.sources.addItem({ source_key: 'paper-b', title: 'City council approves new bike lanes downtown', url: 'https://paper-b.example/bike-lanes', published_at: '2026-09-22T12:00:00Z' });++ t.sources.addItem({ source_key: 'wire-a', title: '' });++ return reports;++}++++/** A revision upstream (title change) and a removal, applied by a second pull. */++function reviseFixture(t, reports) {++ t.sources.updateItem(reports.c.id, { title: 'Why NASA is sending Europa Clipper to Jupiter, explained' });++ t.sources.removeItem(reports.b.id, 'licence review');++}++++/** The canonical projection of the domain rows. Deterministic in everything but ids. → { items, clusters } */++async function project(t) {++ const db = t.db();++ const items = await db.prepare('SELECT * FROM news_source_items ORDER BY sources_item_id').all();++ const srcOf = new Map(items.map((r) => [r.id, r.sources_item_id]));++ const membersByCluster = new Map();++ for (const r of items) {++ if (!r.cluster_id) continue;++ if (!membersByCluster.has(r.cluster_id)) membersByCluster.set(r.cluster_id, []);++ membersByCluster.get(r.cluster_id).push(r.sources_item_id);++ }++ const clusterIndex = new Map([...membersByCluster.entries()]++ .map(([id, ms]) => [id, ms.slice().sort()[0]])++ .sort((a, b) => (a[1] < b[1] ? -1 : a[1] > b[1] ? 1 : 0))++ .map(([id], i) => [id, i + 1]));++ const clusters = await db.prepare('SELECT * FROM news_story_clusters ORDER BY id').all();++ return {++ items: items.map((r) => ({++ sources_item_id: r.sources_item_id, sources_revision: r.sources_revision, source_key: r.source_key,++ canonical_url: r.canonical_url, url_key: r.url_key, headline: r.headline, outlet: r.outlet,++ authors: r.authors, published_at: r.published_at, summary: r.summary, summary_basis: r.summary_basis,++ license_note: r.license_note, terms_note: r.terms_note, content_hash: r.content_hash, title_key: r.title_key,++ status: r.status, dedupe_rule: r.dedupe ? parse(r.dedupe, {}).rule : null,++ duplicate_of: r.duplicate_of ? (srcOf.get(r.duplicate_of) || null) : null,++ cluster: r.cluster_id ? clusterIndex.get(r.cluster_id) : null,++ cluster_reason_rule: r.cluster_reason ? parse(r.cluster_reason, {}).rule : null,++ retrieved_at: r.retrieved_at, first_seen_at: r.first_seen_at, updated_at: r.updated_at,++ removed_at: r.removed_at || null, removed_reason: r.removed_reason || null,++ })),++ clusters: [...membersByCluster.entries()]++ .sort((a, b) => clusterIndex.get(a[0]) - clusterIndex.get(b[0]))++ .map(([id, ms]) => {++ const c = clusters.find((x) => x.id === id) || {};++ return { index: clusterIndex.get(id), label: c.label || null, status: c.status || null, terms: c.terms || null, entities: c.entities || null, window_start: c.window_start || null, window_end: c.window_end || null, members: ms.slice().sort() };++ }),++ };++}++++module.exports = { feedFixture, reviseFixture, project };@@@ -8,7 +8,7 @@*/const assert = require('assert');const { boot, check, done } = require('./helpers/boot');--const text = require('../server/domain/text');++const text = require('openvibe-publishing/ingest').normalize;const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';nnew file mode 100644@@@ -0,0 +1,84 @@++'use strict';++/**++ * IndexNow (openvibe-shared/indexnow): INDEXNOW_KEY unset → the feature is off (no key route, nothing++ * sent). With a key, the key file is served at /<key>.txt as text/plain and publishing an indexable++ * story pings the engines with the story's path and the sitemap. Drafts and noindex pages never ping.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++++const KEY = 'k'.repeat(32);++const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';++++/** An editor's story resting on exactly these Sources items (cited), published. */++async function storyOn(t, headline, items) {++ let r = await t.api('/stories', { json: { headline } });++ assert.strictEqual(r.status, 201, r.text);++ const story = r.json().story;++ for (const it of items) {++ r = await t.api(`/stories/${story.id}/sources`, { json: { item: it.id } });++ assert.ok(r.status === 200 || r.status === 201, r.text);++ }++ r = await t.api(`/stories/${story.id}/revisions`, { json: { body: `${PARA} [${items.map((_, i) => i + 1).join(', ')}]`, expected_revision: 0 } });++ assert.strictEqual(r.status, 201, r.text);++ r = await t.api(`/stories/${story.id}/publish`, { json: {} });++ assert.strictEqual(r.status, 200, r.text);++ return story;++}++++(async () => {++ const off = await boot();++ await check('without a key IndexNow is off: no key route and nothing sent', async () => {++ assert.strictEqual(off.ctx.indexnow.enabled, false);++ const res = await off.get(`/${KEY}.txt`);++ assert.strictEqual(res.status, 404, res.text);++ });++ await off.close();++++ const on = await boot({ env: { INDEXNOW_KEY: KEY } });++ await check('with a key the key file answers text/plain with the key', async () => {++ assert.strictEqual(on.ctx.indexnow.enabled, true);++ const res = await on.get(`/${KEY}.txt`);++ assert.strictEqual(res.status, 200, res.text);++ assert.match(res.headers.get('content-type'), /text\/plain/);++ assert.strictEqual(res.text, KEY);++ });++ await on.close();++++ // A spy in place of the module's HTTP send: records every pingSoon batch.++ const pings = [];++ const spy = {++ enabled: true,++ keyFile: (_req, _res, next) => next(),++ pingSoon: (urls) => { const a = Array.isArray(urls) ? urls : [urls]; pings.push(...a); return a.length; },++ ping: async () => ({ sent: 0, status: 0 }),++ flush: async () => ({ sent: 0, status: 0 }),++ };++ const t = await boot({ indexnow: spy });++ t.sources.addSource('alpha-news', { name: 'Alpha News' });++ t.sources.addSource('solo-wire', { name: 'Solo Wire' });++ const alpha = t.sources.addItem({ source_key: 'alpha-news', title: 'Water agency says two new treatment plants will open next spring', url: 'https://alpha.example/a/water-agency-plants' });++ const solo = t.sources.addItem({ source_key: 'solo-wire', title: 'Water authority to open two treatment plants next spring', url: 'https://solo-wire.example/water/plants' });++ await t.pull();++++ await check('a draft never pings', async () => {++ const r = await t.api('/stories', { json: { headline: 'A draft story about water' } });++ assert.strictEqual(r.status, 201, r.text);++ assert.deepStrictEqual(pings, []);++ });++++ await check('a publish pings the page path and the sitemap', async () => {++ const story = await storyOn(t, 'Two new water treatment plants confirmed', [alpha, solo]);++ assert.ok(pings.includes(`https://openvibe.news/stories/${story.slug}`), JSON.stringify(pings));++ assert.ok(pings.includes('https://openvibe.news/sitemap.xml'), JSON.stringify(pings));++ });++++ await check('a published but noindex story never pings', async () => {++ pings.length = 0;++ await storyOn(t, 'Two treatment plants to open next spring', [alpha]);++ assert.deepStrictEqual(pings, [], 'a noindex page is never pinged');++ });++ await t.close();++++ done();++})().catch((err) => { console.error(err); process.exit(1); });@@@ -37,7 +37,7 @@ const { boot, check, done, launchReports } = require('./helpers/boot');assert.strictEqual(why.rule, 'shared_terms');assert.ok(why.shared_entities.includes('europa clipper'), JSON.stringify(why));assert.ok(why.score >= 3);-- assert.strictEqual(await t.ctx.store.getState('sources_cursor'), reports.c.change_seq);++ assert.strictEqual(await t.ctx.ingest.cursor(), reports.c.change_seq);assert.strictEqual((await t.events('news.source.ingested')).length, 3);assert.ok((await t.events('news.cluster.updated')).length >= 3);});@@@ -46,7 +46,7 @@ const { boot, check, done, launchReports } = require('./helpers/boot');const before = (await t.events()).length;const r = await t.pull();assert.strictEqual(r.counts.created || 0, 0);-- await t.ctx.store.setState('sources_cursor', 0);++ await t.db().prepare("UPDATE news_ingest_cursor SET cursor = 0 WHERE name = 'sources'").run();const again = await t.pull();assert.strictEqual(again.counts.unchanged, 3);assert.strictEqual((await t.events()).length, before);@@@ -8,7 +8,7 @@*/const assert = require('assert');const { boot, check, done } = require('./helpers/boot');--const { licensedSummary } = require('../server/domain/text');++const { licensedSummary } = require('openvibe-publishing/ingest').normalize;const BODY_SENTINEL = 'SECRET-FULL-ARTICLE-BODY-7f3a';const TAIL_SENTINEL = 'SECRET-SUMMARY-TAIL-91c2';nnew file mode 100644@@@ -0,0 +1,63 @@++'use strict';++// Size budgets for openvibe.news's home page (roadmap WS-T task 1, openvibe-shared/perf-budget): the++// server as it runs (a fresh database), measured without a browser. Budgets sit a little above the++// measurement below; raising one is a decision to state in the commit.++// node test/perf-budget.test.js++const assert = require('assert');++const fs = require('fs');++const net = require('net');++const os = require('os');++const path = require('path');++const { spawn } = require('child_process');++const { measure, check, format } = require('openvibe-shared/perf-budget');++++const BUDGETS = {++ htmlRawKB: 22, // measured 17.4 (fresh database)++ htmlBrotliKB: 5.5, // 4.5++ jsFiles: 5, // 4++ jsRawKB: 240, // 212.2++ jsBrotliKB: 55, // 49.9++ cssFiles: 2, // 1++ cssRawKB: 6.5, // 5.2++ cssBrotliKB: 2, // 1.4++ externalFiles: 1, // 0++};++++const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });++++(async () => {++ // A database of its own (not the shared dev PGlite in data/pglite, and not whatever the caller's++ // DATABASE_URL names), so the measurement is the server on a fresh database and the run leaves nothing.++ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-budget-'));++ const pgliteDir = path.join(dir, 'pglite');++ const port = await freePort();++ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {++ cwd: path.join(__dirname, '..'),++ env: {++ ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test',++ DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', NEWS_PGLITE_DIR: pgliteDir,++ NEWS_WORKER: 'off', OV_OAUTH_CLIENT_SECRET: 'test-secret', NEWS_FORM_SECRET: 'test-form-secret',++ },++ stdio: ['ignore', 'ignore', 'pipe'],++ });++ let stderr = '';++ child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });++ const base = `http://127.0.0.1:${port}`;++ try {++ let up = false;++ for (let i = 0; i < 150 && !up; i++) {++ up = await fetch(`${base}/api/health`).then((r) => r.ok).catch(() => false);++ if (!up) await new Promise((r) => setTimeout(r, 100));++ }++ assert.ok(up, `the server did not start:\n${stderr}`);++ assert.ok(fs.existsSync(pgliteDir), 'the server did not use the isolated database (NEWS_PGLITE_DIR)');++ const m = await measure({ base });++ const over = check(m, BUDGETS);++ assert.deepStrictEqual(over, [], format(m, over));++ console.log(format(m));++ console.log('perf budget: all checks passed');++ } finally {++ child.kill();++ fs.rmSync(dir, { recursive: true, force: true });++ }++})().catch((err) => { console.error(err); process.exitCode = 1; });@@@ -131,7 +131,7 @@ const BODY = [await check('an unchanged upstream refetch (same revision) flags nothing', async () => {const before = (await t.db().prepare('SELECT COUNT(*) AS n FROM news_editorial_flags').get()).n;-- await t.ctx.store.setState('sources_cursor', 0);++ await t.db().prepare("UPDATE news_ingest_cursor SET cursor = 0 WHERE name = 'sources'").run();await t.pull();assert.strictEqual((await t.db().prepare('SELECT COUNT(*) AS n FROM news_editorial_flags').get()).n, before);});
@@ -50,8 +50,9 @@ Other tables in the same database:
@@ -50,8 +50,9 @@ Other tables in the same database:
an author).
an author).
- `news_cluster_audit`: every merge, split and reversal with exactly which items moved.
- `news_cluster_audit`: every merge, split and reversal with exactly which items moved.
- `news_ingest_runs`: what every webhook delivery, pull and upstream fetch failure did.
- `news_ingest_runs`: what every webhook delivery, pull and upstream fetch failure did.
-- `news_state` (the Sources cursor), `news_source_status` (display cache of Sources' registry
- names and health), `event_outbox` and `idempotency_receipts` (openvibe-sdk),
+- `news_ingest_cursor` (the OpenVibe.Sources change cursor, `openvibe-publishing/ingest`; `news_state`
+ keeps the pre-chassis key it was carried from), `news_source_status` (display cache of Sources'
+ registry names and health), `event_outbox` and `idempotency_receipts` (openvibe-sdk),
`subject_projections` (Network names). None of them is authority for anything outside News.
`subject_projections` (Network names). None of them is authority for anything outside News.
## Does not own
## Does not own
@@ -176,6 +177,10 @@ Other tables in the same database:
@@ -176,6 +177,10 @@ Other tables in the same database:
- JSON-LD `NewsArticle` from real fields only: headline, dates, the editors named in the
- JSON-LD `NewsArticle` from real fields only: headline, dates, the editors named in the
revision's authorship (only when the Network name is known), the section (topic), the cited
revision's authorship (only when the Network name is known), the section (topic), the cited
source URLs, and OpenVibe.News as publisher. Missing fields are omitted.
source URLs, and OpenVibe.News as publisher. Missing fields are omitted.
+- IndexNow (`openvibe-shared/indexnow`): with `INDEXNOW_KEY` set, the key file is served at
+ `/<key>.txt` and publishing, updating or unpublishing an indexable story pings the engines with
+ the story's URL and `/sitemap.xml`. Unset, the feature is off — nothing is mounted and nothing is
+ sent. Drafts, private and noindex pages are never pinged (`test/indexnow.test.js`).
- Drafts, unpublished, retracted and noindex stories never appear in sitemaps or Search; feeds
- Drafts, unpublished, retracted and noindex stories never appear in sitemaps or Search; feeds
list published and (labelled) retracted stories, never a draft.
list published and (labelled) retracted stories, never a draft.
@@ -224,8 +229,10 @@ under [Grants the Network must hold](#grants-the-network-must-hold).
@@ -224,8 +229,10 @@ under [Grants the Network must hold](#grants-the-network-must-hold).
- **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through
- **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through
`openvibe-sdk/db`; Valkey holds the per-actor limit counters (optional: without `VALKEY_URL` they count per process).
`openvibe-sdk/db`; Valkey holds the per-actor limit counters (optional: without `VALKEY_URL` they count per process).
-- **Packages** (pinned by release tarball): `openvibe-publishing` v1.0.0 (async PostgreSQL stores: revisions,
- citations, authorship, seo, index-hooks, ssr, taxonomy slugify), `openvibe-contracts` v0.79.0,
+- **Packages** (pinned by release tarball): `openvibe-publishing` v1.1.0 (async PostgreSQL stores: revisions,
+ citations, authorship, seo, index-hooks, ssr, taxonomy slugify, and the shared ingest/publication
+ chassis: the Sources client, the change-cursor pull, the signed events consumer and the
+ gate → document → outbox glue), `openvibe-contracts` v0.79.0,
`openvibe-shared` v2.2.0 (Frame, app icon, footer, legal, release, metrics, ready, seo),
`openvibe-shared` v2.2.0 (Frame, app icon, footer, legal, release, metrics, ready, seo),
`openvibe-sdk` v0.25.0 (db with after-commit hooks, PostgreSQL events outbox and inbox, webhook signatures v2,
`openvibe-sdk` v0.25.0 (db with after-commit hooks, PostgreSQL events outbox and inbox, webhook signatures v2,
service tokens, Network sign-in (`openvibe-sdk/sso`) and the service outbox (`openvibe-sdk/events`), per-actor limits, testing).
service tokens, Network sign-in (`openvibe-sdk/sso`) and the service outbox (`openvibe-sdk/events`), per-actor limits, testing).
@@ -389,7 +396,7 @@ First install (done once; kept for a rebuild):
@@ -389,7 +396,7 @@ First install (done once; kept for a rebuild):
`NEWS_EDITORS`, `NEWS_FORM_SECRET`, `NEWS_EVENTS_SECRET` (`openssl rand -hex 32`),
`NEWS_EDITORS`, `NEWS_FORM_SECRET`, `NEWS_EVENTS_SECRET` (`openssl rand -hex 32`),
`EVENTS_URL=http://127.0.0.1:4300`, `OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720`,
`EVENTS_URL=http://127.0.0.1:4300`, `OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720`,
`BASE_URL=https://openvibe.news`, `OV_COMMUNITY_INTERNAL_URL=http://127.0.0.1:4200`, and
`BASE_URL=https://openvibe.news`, `OV_COMMUNITY_INTERNAL_URL=http://127.0.0.1:4200`, and
- optionally `OV_AI_INTERNAL_URL`.
+ optionally `OV_AI_INTERNAL_URL` and `INDEXNOW_KEY`.
2. **Network:** create (or give a secret to) the OAuth client `news` with redirect
2. **Network:** create (or give a secret to) the OAuth client `news` with redirect
`https://openvibe.news/auth/callback`, and add the grants listed above.
`https://openvibe.news/auth/callback`, and add the grants listed above.
3. **Search:** make sure `news` is in `SEARCH_EVENT_OWNERS`.
3. **Search:** make sure `news` is in `SEARCH_EVENT_OWNERS`.
@@ -9,7 +9,7 @@
@@ -9,7 +9,7 @@
"domain": "openvibe.news",
"domain": "openvibe.news",
"contracts": "openvibe-contracts v0.79.0",
"contracts": "openvibe-contracts v0.79.0",
"packages": [
"packages": [
- "openvibe-publishing v1.0.0",
+ "openvibe-publishing v1.1.0",
"openvibe-contracts v0.79.0",
"openvibe-contracts v0.79.0",
"openvibe-shared v2.2.0",
"openvibe-shared v2.2.0",
"openvibe-sdk v0.25.0"
"openvibe-sdk v0.25.0"
new file mode 100644
new file mode 100644
@@ -0,0 +1,20 @@
@@ -0,0 +1,20 @@
+-- phase: expand
+-- The ingest cursor moves from news_state ('sources_cursor') to the chassis' own table
+-- (openvibe-publishing/ingest, prefix news → news_ingest_cursor). Additive: the old key is read
+-- once here so a deployment continues from the cursor it had, never from zero. Generated from
+-- require('openvibe-publishing').schema({ ingest: 'news' }).
+
+CREATE TABLE IF NOT EXISTS news_ingest_cursor (
+ name text COLLATE "C" NOT NULL,
+ cursor bigint NOT NULL,
+ updated_at bigint NOT NULL,
+ PRIMARY KEY (name)
+);
+
+-- Continuity: carry the cursor already stored under the old key, once. news_state stores JSON, so a
+-- numeric value is plain digits; anything else is ignored (the chassis cursor then starts empty).
+INSERT INTO news_ingest_cursor (name, cursor, updated_at)
+SELECT 'sources', value::bigint, (EXTRACT(EPOCH FROM now()) * 1000)::bigint
+FROM news_state
+WHERE key = 'sources_cursor' AND value ~ '^[0-9]+$'
+ON CONFLICT (name) DO NOTHING;
@@ -16,7 +16,7 @@
@@ -16,7 +16,7 @@
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.79.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.79.0",
- "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.0.0",
+ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.1.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.25.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.25.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.2.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.2.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -1163,9 +1163,9 @@
@@ -1163,9 +1163,9 @@
}
}
},
},
"node_modules/openvibe-publishing": {
"node_modules/openvibe-publishing": {
- "version": "1.0.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.0.0",
- "integrity": "sha512-g4mU0PPtI4y38NcsdppEbrXTrCYp0rPmIYWyxLxh4GpEYhn48jMWq2+cMQA2uwZCeeZlB3UiXVrQNQyQLiliwg==",
+ "version": "1.1.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.1.0",
+ "integrity": "sha512-bJsVl82hDivazoiM/np9ckKJE4V5F4OxyIiDwB6xuhK8M6P42sGU+V88rFCkyiI4BVQiKv9xs4GVx8Fj3pyjsg==",
"license": "MIT",
"license": "MIT",
"engines": {
"engines": {
"node": ">=20"
"node": ">=20"
@@ -30,7 +30,7 @@
@@ -30,7 +30,7 @@
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.79.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.79.0",
- "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.0.0",
+ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.1.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.25.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.25.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.2.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.2.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -17,18 +17,19 @@ const contracts = require('openvibe-contracts');
@@ -17,18 +17,19 @@ const contracts = require('openvibe-contracts');
const { createSsoClient } = require('openvibe-sdk/sso');
const { createSsoClient } = require('openvibe-sdk/sso');
const { createServiceOutbox } = require('openvibe-sdk/events');
const { createServiceOutbox } = require('openvibe-sdk/events');
+const { createSourcesClient } = require('openvibe-publishing/ingest');
+const { createIndexNow } = require('openvibe-shared/indexnow');
const configLib = require('./config');
const configLib = require('./config');
const { openStore } = require('./db');
const { openStore } = require('./db');
const { createViewerResolver } = require('./auth/viewer');
const { createViewerResolver } = require('./auth/viewer');
const { createPeople } = require('./clients/network');
const { createPeople } = require('./clients/network');
-const { createSourcesClient } = require('./clients/sources');
const { createAi } = require('./clients/ai');
const { createAi } = require('./clients/ai');
const { createCommunity } = require('./clients/community');
const { createCommunity } = require('./clients/community');
const { createPublication } = require('./domain/publication');
const { createPublication } = require('./domain/publication');
const { createDiscussion } = require('./domain/discussion');
const { createDiscussion } = require('./domain/discussion');
const { createClusters } = require('./domain/clusters');
const { createClusters } = require('./domain/clusters');
-const { createIngest } = require('./domain/ingest');
+const { createIngest } = require('./domain/source-items');
const { createStories } = require('./domain/stories');
const { createStories } = require('./domain/stories');
const { createTopics } = require('./domain/topics');
const { createTopics } = require('./domain/topics');
const { createReading } = require('./domain/reading');
const { createReading } = require('./domain/reading');
@@ -36,7 +37,7 @@ const { createPublicRoutes } = require('./http/public');
@@ -36,7 +37,7 @@ const { createPublicRoutes } = require('./http/public');
const { createEditorRoutes } = require('./http/editor');
const { createEditorRoutes } = require('./http/editor');
const { createApi } = require('./http/api');
const { createApi } = require('./http/api');
const { createDiscoveryRoutes } = require('./http/discovery');
const { createDiscoveryRoutes } = require('./http/discovery');
-const { createWebhook } = require('./http/webhook');
+const { createEvents } = require('./http/events');
const { createActorLimits } = require('./http/actor-limits');
const { createActorLimits } = require('./http/actor-limits');
const { createNewsReadiness } = require('./observability');
const { createNewsReadiness } = require('./observability');
const { createWorker } = require('./worker');
const { createWorker } = require('./worker');
@@ -62,7 +63,11 @@ async function createApp(opts = {}) {
@@ -62,7 +63,11 @@ async function createApp(opts = {}) {
intervalMs: config.events.intervalMs, log, now: store.now,
intervalMs: config.events.intervalMs, log, now: store.now,
...(fetchImpl ? { fetch: fetchImpl } : {}),
...(fetchImpl ? { fetch: fetchImpl } : {}),
});
});
- const publication = createPublication({ store, config, outbox });
+ // IndexNow (openvibe-shared/indexnow): created once at boot from INDEXNOW_KEY; unset → off
+ // (nothing mounted, nothing sent). The key file is served at /<key>.txt and publication.js
+ // pings the engines when an indexable page appears, changes or goes away.
+ const indexnow = opts.indexnow || createIndexNow({ host: config.baseUrl, key: config.indexnow.key, ...(fetchImpl ? { fetch: fetchImpl } : {}), log });
+ const publication = createPublication({ store, config, outbox, indexnow });
const clusters = createClusters({ store, config, outbox });
const clusters = createClusters({ store, config, outbox });
const sources = createSourcesClient({ config, fetchImpl });
const sources = createSourcesClient({ config, fetchImpl });
const ai = createAi({ config, fetchImpl });
const ai = createAi({ config, fetchImpl });
@@ -91,7 +96,7 @@ async function createApp(opts = {}) {
@@ -91,7 +96,7 @@ async function createApp(opts = {}) {
const viewers = createViewerResolver({ auth, config, people });
const viewers = createViewerResolver({ auth, config, people });
const worker = createWorker({ config, ingest, outbox, log });
const worker = createWorker({ config, ingest, outbox, log });
- const ctx = { config, store, outbox, publication, clusters, sources, ai, community, discussion, ingest, stories, topics, people, reading, auth, viewers, worker };
+ const ctx = { config, store, outbox, publication, clusters, sources, ai, community, discussion, ingest, stories, topics, people, reading, auth, viewers, worker, indexnow };
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
@@ -138,9 +143,11 @@ async function createApp(opts = {}) {
@@ -138,9 +143,11 @@ async function createApp(opts = {}) {
release.mount(app, { registry: metrics.registry });
release.mount(app, { registry: metrics.registry });
const readiness = createNewsReadiness({ store, outbox, ingest, config, release: release.release, valkey: ctx.valkey });
const readiness = createNewsReadiness({ store, outbox, ingest, config, release: release.release, valkey: ctx.valkey });
app.get('/api/ready', readiness.handler);
app.get('/api/ready', readiness.handler);
+ // GET /<key>.txt — the IndexNow key file (only when a key is configured; it serves itself).
+ if (indexnow.enabled) app.use(indexnow.keyFile);
// ── Events webhook (raw body; before any other body parser) ─
// ── Events webhook (raw body; before any other body parser) ─
- app.use(createWebhook({ config, store, ingest, sources, log }));
+ app.use(createEvents({ config, store, ingest, sources, log }));
app.use(cookieParser());
app.use(cookieParser());
deleted file mode 100644
deleted file mode 100644
@@ -1,77 +0,0 @@
@@ -1,77 +0,0 @@
-'use strict';
-
-/**
- * OpenVibe.Sources client (host-local API, service tokens for audience openvibe.sources).
- *
- * sources.item.read GET /api/v1/items?category=news&after=&limit=&include_removed=1 (change order)
- * GET /api/v1/items/:id
- * sources.source.read GET /api/v1/sources/:key (optional: outlet names and source health)
- *
- * Failures are thrown as errors with a stable `code` (sources.unavailable, sources.http_<status>,
- * sources.bad_response); the caller records them. Nothing here invents an item.
- */
-const { serviceAuth } = require('openvibe-contracts');
-
-class SourcesError extends Error {
- constructor(code, message, status = null) { super(message); this.code = code; this.status = status; }
-}
-
-function createSourcesClient({ config, fetchImpl = globalThis.fetch }) {
- const enabled = Boolean(config.oauth.clientSecret && config.sources.internalUrl);
- const token = (scope) => (enabled ? serviceAuth.createTokenClient({
- tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,
- audience: 'openvibe.sources', scope, fetchImpl,
- }) : null);
- const itemTokens = token('sources.item.read');
- const sourceTokens = token('sources.source.read');
-
- async function call(tokens, path) {
- if (!tokens) throw new SourcesError('sources.not_configured', 'OV_OAUTH_CLIENT_SECRET or OV_SOURCES_INTERNAL_URL is not set');
- let auth;
- try { auth = await tokens.authHeaders(); } catch (err) {
- throw new SourcesError('sources.token_unavailable', `No service token for OpenVibe.Sources: ${err.message}`);
- }
- let res;
- try {
- res = await fetchImpl(`${config.sources.internalUrl}${path}`, {
- headers: { Accept: 'application/json', ...auth },
- signal: AbortSignal.timeout(config.sources.timeoutMs),
- });
- } catch (err) {
- throw new SourcesError('sources.unavailable', `OpenVibe.Sources is unreachable: ${err.message}`);
- }
- if (res.status === 401 && tokens.invalidate) tokens.invalidate();
- const body = await res.json().catch(() => null);
- if (!res.ok) throw new SourcesError(`sources.http_${res.status}`, `OpenVibe.Sources answered ${res.status}${body && (body.code || body.detail) ? `: ${body.code || ''} ${body.detail || ''}`.trimEnd() : ''}`, res.status);
- if (!body || typeof body !== 'object') throw new SourcesError('sources.bad_response', 'OpenVibe.Sources sent no JSON');
- return body;
- }
-
- return {
- enabled,
- SourcesError,
-
- /** One page of news items in change order. → { items, next_after, more, sources } */
- async listItems({ after = 0, limit = 100 } = {}) {
- const qs = new URLSearchParams({ category: config.sources.category, after: String(after), limit: String(limit), include_removed: '1' });
- const body = await call(itemTokens, `/api/v1/items?${qs}`);
- if (!Array.isArray(body.items) || !Number.isInteger(body.next_after)) throw new SourcesError('sources.bad_response', 'items page without items/next_after');
- return body;
- },
-
- /** One item (with its source's health). → { item, source } */
- async getItem(id) {
- const body = await call(itemTokens, `/api/v1/items/${encodeURIComponent(id)}`);
- if (!body.item || typeof body.item.id !== 'string') throw new SourcesError('sources.bad_response', 'item response without item');
- return body;
- },
-
- /** A registry record (name, homepage, health). Needs sources.source.read; callers treat failure as "unknown". */
- async getSource(key) {
- const body = await call(sourceTokens, `/api/v1/sources/${encodeURIComponent(key)}`);
- return body.source || body;
- },
- };
-}
-
-module.exports = { createSourcesClient, SourcesError };
@@ -36,7 +36,9 @@ function load(env = process.env) {
@@ -36,7 +36,9 @@ function load(env = process.env) {
},
},
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
// PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
- db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' },
+ // Non-production without DATABASE_URL: the embedded PGlite database in data/pglite
+ // (NEWS_PGLITE_DIR, tests: an isolated directory of their own).
+ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.NEWS_PGLITE_DIR || '' },
valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:news:' },
valkey: { url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:news:' },
// OpenVibe.Network: SSO (OAuth2 authorization server), JWKS, client-credentials tokens.
// OpenVibe.Network: SSO (OAuth2 authorization server), JWKS, client-credentials tokens.
@@ -81,6 +83,11 @@ function load(env = process.env) {
@@ -81,6 +83,11 @@ function load(env = process.env) {
minIndependentSources: Math.max(int(env.NEWS_MIN_INDEPENDENT_SOURCES, 2), 1),
minIndependentSources: Math.max(int(env.NEWS_MIN_INDEPENDENT_SOURCES, 2), 1),
},
},
+ // IndexNow (openvibe-shared/indexnow): when INDEXNOW_KEY is set the key file is served at
+ // /<key>.txt and every publish, update, unpublish or delete of a public, indexable page pings
+ // the engines. Unset: off — nothing is mounted and nothing is sent.
+ indexnow: { key: env.INDEXNOW_KEY || '' },
+
// Clustering: the time window a new item may join an existing cluster in.
// Clustering: the time window a new item may join an existing cluster in.
clustering: {
clustering: {
windowMs: int(env.NEWS_CLUSTER_WINDOW_HOURS, 72) * 3600 * 1000,
windowMs: int(env.NEWS_CLUSTER_WINDOW_HOURS, 72) * 3600 * 1000,
@@ -22,8 +22,9 @@
@@ -22,8 +22,9 @@
* news_story_citations, news_story_reviews, news_story_drafts, news_story_revision_purges,
* news_story_citations, news_story_reviews, news_story_drafts, news_story_revision_purges,
* news_story_discussion_refs (a story's Community thread id, never its comments),
* news_story_discussion_refs (a story's Community thread id, never its comments),
* news_index_revisions (publishing packages); news_cluster_audit (every merge, split and reversal);
* news_index_revisions (publishing packages); news_cluster_audit (every merge, split and reversal);
- * news_ingest_runs (what each webhook delivery, pull and upstream failure did); news_state (the
- * Sources cursor); news_source_status (a display cache of Sources' registry and health);
+ * news_ingest_runs (what each webhook delivery, pull and upstream failure did); news_ingest_cursor
+ * (the chassis' Sources change cursor; news_state holds the pre-chassis key it was carried from);
+ * news_source_status (a display cache of Sources' registry and health);
* event_outbox and idempotency_receipts (openvibe-sdk); subject_projections (Network names).
* event_outbox and idempotency_receipts (openvibe-sdk); subject_projections (Network names).
*/
*/
const fs = require('fs');
const fs = require('fs');
@@ -45,9 +46,10 @@ const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
@@ -45,9 +46,10 @@ const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
async function openDb(config, { log = console, registry } = {}) {
async function openDb(config, { log = console, registry } = {}) {
if (!config.db.url) {
if (!config.db.url) {
if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh news)');
if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh news)');
- log.warn(`[News] DATABASE_URL unset: embedded PGlite database in ${DEV_PGLITE} (development only, one process)`);
- fs.mkdirSync(DEV_PGLITE, { recursive: true });
- const db = createDb({ pglite: DEV_PGLITE, service: 'news', registry, log });
+ const dir = config.db.pgliteDir || DEV_PGLITE;
+ log.warn(`[News] DATABASE_URL unset: embedded PGlite database in ${dir} (development only, one process)`);
+ fs.mkdirSync(dir, { recursive: true });
+ const db = createDb({ pglite: dir, service: 'news', registry, log });
await db.migrate({ dir: MIGRATIONS, log });
await db.migrate({ dir: MIGRATIONS, log });
return db;
return db;
}
}
@@ -63,9 +65,6 @@ async function openDb(config, { log = console, registry } = {}) {
@@ -63,9 +65,6 @@ async function openDb(config, { log = console, registry } = {}) {
*/
*/
function createStore(db, { now = () => Date.now() } = {}) {
function createStore(db, { now = () => Date.now() } = {}) {
const revisions = createRevisionStore(db, { prefix: 'news_story', now });
const revisions = createRevisionStore(db, { prefix: 'news_story', now });
- const getStateStmt = db.prepare('SELECT value FROM news_state WHERE key = ?');
- const setStateStmt = db.prepare(`INSERT INTO news_state (key, value, updated_at) VALUES (?, ?, ?)
- ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`);
return {
return {
db,
db,
now,
now,
@@ -75,13 +74,6 @@ function createStore(db, { now = () => Date.now() } = {}) {
@@ -75,13 +74,6 @@ function createStore(db, { now = () => Date.now() } = {}) {
sequencer: createIndexSequencer(db, { prefix: 'news', now }),
sequencer: createIndexSequencer(db, { prefix: 'news', now }),
discussion: createDiscussionRefs(db, { prefix: 'news_story', now }),
discussion: createDiscussionRefs(db, { prefix: 'news_story', now }),
tx: async (fn) => await db.tx(() => fn()),
tx: async (fn) => await db.tx(() => fn()),
- async getState(key, def = null) {
- const r = await getStateStmt.get(key);
- return r ? JSON.parse(r.value) : def;
- },
- async setState(key, value) {
- await setStateStmt.run(key, JSON.stringify(value), now());
- },
close: () => db.close(),
close: () => db.close(),
};
};
}
}
@@ -18,7 +18,7 @@
@@ -18,7 +18,7 @@
* Every change emits news.cluster.updated (internal) in the same transaction.
* Every change emits news.cluster.updated (internal) in the same transaction.
*/
*/
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
-const text = require('./text');
+const { normalize } = require('openvibe-publishing/ingest');
const { ApiError } = require('../http/errors');
const { ApiError } = require('../http/errors');
const newClusterId = (now) => `clu_${ids.ulid(now)}`;
const newClusterId = (now) => `clu_${ids.ulid(now)}`;
@@ -28,7 +28,7 @@ const parse = (s, d) => { try { return s ? JSON.parse(s) : d; } catch { return d
@@ -28,7 +28,7 @@ const parse = (s, d) => { try { return s ? JSON.parse(s) : d; } catch { return d
/** Terms and entities of one item (headline + the licensed summary we store). */
/** Terms and entities of one item (headline + the licensed summary we store). */
function signature(item) {
function signature(item) {
const t = `${item.headline || ''}. ${item.summary || ''}`;
const t = `${item.headline || ''}. ${item.summary || ''}`;
- return { terms: text.terms(t), entities: text.entities(item.headline || '').concat(text.entities(item.summary || '')).filter((v, i, a) => a.indexOf(v) === i).sort() };
+ return { terms: normalize.terms(t), entities: normalize.entities(item.headline || '').concat(normalize.entities(item.summary || '')).filter((v, i, a) => a.indexOf(v) === i).sort() };
}
}
/** The time an item is placed at: when the source says it was published, else when News first saw it. */
/** The time an item is placed at: when the source says it was published, else when News first saw it. */
deleted file mode 100644
deleted file mode 100644
@@ -1,290 +0,0 @@
@@ -1,290 +0,0 @@
-'use strict';
-
-/**
- * Ingestion: OpenVibe.Sources items (category news) → news_source_items → dedupe → clusters.
- *
- * Two ways in, one effect:
- * - the Events webhook (sources.item.created|updated|removed, sources.fetch.failed), each event
- * applied once through the inbox (http/webhook.js);
- * - the cursor pull (worker), the backstop for missed deliveries: GET /api/v1/items in change
- * order from the stored cursor.
- * apply() is idempotent on (Sources item id, Sources revision): a replay, or the same item arriving
- * by both paths, changes nothing and emits nothing.
- *
- * What is stored (licensing): headline, canonical URL, outlet, authors, published_at, the licence
- * and terms notes, and a short summary ONLY when those notes explicitly allow short summaries
- * (text.licensedSummary). Every other field of the Sources item — whatever an adapter mapped into
- * `fields` — is ignored, so an article body can never be stored, shown, fed or indexed.
- *
- * Failure is a recorded state (news_ingest_runs + news.source.failed), never a replacement: a
- * failed fetch or an unreadable item creates no source item, no cluster and no story text.
- *
- * Upstream changes to an item a story cites (a revision, a removal) go to stories.onUpstreamChange,
- * which flags the story and prepares a pending revision; published text is never changed silently.
- * A removal can also hide the story's Community thread (domain/discussion.js, after the commit).
- */
-const { ids } = require('openvibe-contracts');
-const text = require('./text');
-
-const NEAR_TITLE = { anyOutlet: 0.9, sameOutlet: 0.75, windowMs: 48 * 3600 * 1000 };
-const newItemId = (now) => `nsi_${ids.ulid(now)}`;
-
-function createIngest({ store, config, clusters, outbox, sources, discussion = null, log = console }) {
- const { db } = store;
- let stories = null; // set by app.js (stories depend on ingest's items too)
- const q = {
- bySourcesId: db.prepare('SELECT * FROM news_source_items WHERE sources_item_id = ?'),
- byId: db.prepare('SELECT * FROM news_source_items WHERE id = ?'),
- byUrl: db.prepare("SELECT * FROM news_source_items WHERE url_key = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),
- byHash: db.prepare("SELECT * FROM news_source_items WHERE content_hash = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),
- recent: db.prepare("SELECT * FROM news_source_items WHERE status <> 'removed' AND first_seen_at >= ? ORDER BY first_seen_at, seq"),
- status: db.prepare('SELECT * FROM news_source_status WHERE source_key = ?'),
- putStatus: db.prepare(`INSERT INTO news_source_status (source_key, name, homepage_url, status, stale, last_success_at, refreshed_at)
- VALUES (@source_key, @name, @homepage_url, @status, @stale, @last_success_at, @now)
- ON CONFLICT (source_key) DO UPDATE SET name = COALESCE(excluded.name, news_source_status.name), homepage_url = COALESCE(excluded.homepage_url, news_source_status.homepage_url),
- status = COALESCE(excluded.status, news_source_status.status), stale = COALESCE(excluded.stale, news_source_status.stale),
- last_success_at = COALESCE(excluded.last_success_at, news_source_status.last_success_at), refreshed_at = excluded.refreshed_at`),
- run: db.prepare(`INSERT INTO news_ingest_runs (origin, state, source_key, sources_item_id, error_code, detail, counts, cursor_before, cursor_after, at)
- VALUES (@origin, @state, @source_key, @sources_item_id, @error_code, @detail, @counts, @cursor_before, @cursor_after, @at)`),
- lastPull: db.prepare("SELECT * FROM news_ingest_runs WHERE origin = 'pull' ORDER BY id DESC LIMIT 1"),
- };
-
- async function recordRun(r) {
- await q.run.run({ source_key: null, sources_item_id: null, error_code: null, detail: null, counts: null, cursor_before: null, cursor_after: null, ...r, counts: r.counts ? JSON.stringify(r.counts) : null, at: store.now() });
- }
-
- async function failedEvent({ origin, sourceKey = null, sourcesItemId = null, state, errorCode = null, detail = null, httpStatus = null, extra = {} }) {
- await outbox.emit({
- event_type: 'news.source.failed', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'important',
- subject: sourceKey ? { type: 'source', id: sourceKey } : { type: 'ingest', id: origin },
- payload: { origin, source_key: sourceKey, sources_item_id: sourcesItemId, state, error_code: errorCode, http_status: httpStatus, detail: detail ? String(detail).slice(0, 500) : null, ...extra },
- });
- }
-
- async function outletFor(item) {
- const st = await q.status.get(item.source_key);
- return (st && st.name) || text.hostOf(item.canonical_url) || item.source_key;
- }
-
- /** The fields News keeps from a Sources item. Nothing else of the item is read. */
- async function normalise(item) {
- const prov = item.provenance || {};
- const headline = String(item.title || '').replace(/\s+/g, ' ').trim().slice(0, 500);
- const lic = text.licensedSummary(item.summary, { termsNote: prov.terms_note, licenseNote: prov.license_note, maxChars: config.licensing.summaryMaxChars });
- const authors = Array.isArray(item.authors) ? item.authors.map((a) => String(a || '').replace(/\s+/g, ' ').trim().slice(0, 200)).filter(Boolean).slice(0, 10) : [];
- const published = item.published_at && Number.isFinite(Date.parse(item.published_at)) ? new Date(Date.parse(item.published_at)).toISOString() : null;
- return {
- headline,
- canonical_url: item.canonical_url || null,
- url_key: text.urlKey(item.canonical_url),
- outlet: await outletFor(item),
- authors: JSON.stringify(authors),
- published_at: published,
- summary: lic.summary,
- summary_basis: lic.basis,
- license_note: prov.license_note ? String(prov.license_note).slice(0, 2000) : null,
- terms_note: prov.terms_note ? String(prov.terms_note).slice(0, 2000) : null,
- content_hash: prov.content_hash || null,
- title_key: text.titleKey(headline),
- retrieved_at: prov.retrieved_at || null,
- };
- }
-
- /** Is this new item a duplicate of one we have? → null | { of, rule, detail } */
- async function dedupe(n, sourceKey, at) {
- if (n.url_key) {
- const hit = await q.byUrl.get(n.url_key);
- if (hit) return { of: hit.duplicate_of || hit.id, rule: 'canonical_url', detail: n.url_key };
- }
- if (n.content_hash) {
- const hit = await q.byHash.get(n.content_hash);
- if (hit) return { of: hit.duplicate_of || hit.id, rule: 'content_hash', detail: n.content_hash };
- }
- const mine = text.shingles(n.headline);
- const host = text.hostOf(n.canonical_url);
- let best = null;
- for (const c of await q.recent.all(Math.min(at, store.now()) - NEAR_TITLE.windowMs)) {
- const t = clusters.itemTime(c);
- if (Math.abs(t - at) > NEAR_TITLE.windowMs) continue;
- const score = text.jaccard(mine, text.shingles(c.headline));
- const sameOutlet = c.source_key === sourceKey || (host && text.hostOf(c.canonical_url) === host);
- const threshold = sameOutlet ? NEAR_TITLE.sameOutlet : NEAR_TITLE.anyOutlet;
- if (score >= threshold && (!best || score > best.score)) best = { c, score, sameOutlet };
- }
- if (best) return { of: best.c.duplicate_of || best.c.id, rule: 'near_title', detail: `jaccard ${best.score.toFixed(2)} of headline 2-shingles${best.sameOutlet ? ' (same outlet)' : ''}` };
- return null;
- }
-
- async function ingestedEvent(row, action, extra = {}) {
- await outbox.emit({
- event_type: 'news.source.ingested', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'low',
- subject: { type: 'source_item', id: row.id },
- payload: {
- item_id: row.id, action, status: row.status,
- sources_item: { service: 'sources', type: 'item', id: row.sources_item_id, revision: row.sources_revision },
- source_key: row.source_key, canonical_url: row.canonical_url, duplicate_of: row.duplicate_of,
- dedupe_rule: row.dedupe ? JSON.parse(row.dedupe).rule : null, cluster_id: row.cluster_id, ...extra,
- },
- });
- }
-
- /**
- * Apply one Sources item (inside a transaction the caller may already hold). Returns
- * { outcome: created|updated|removed|unchanged|ignored|rejected, item?, reason? }.
- */
- async function apply(item, { origin = 'pull' } = {}) {
- return await db.tx(async () => {
- if (!item || typeof item.id !== 'string') return { outcome: 'rejected', reason: 'item without id' };
- if (item.category && item.category !== config.sources.category) return { outcome: 'ignored', reason: `category ${item.category}` };
- const revision = Number.isInteger(item.revision) ? item.revision : 1;
- const cur = await q.bySourcesId.get(item.id);
- if (item.removed) return await applyRemoval({ sourcesItemId: item.id, reason: item.removed.reason || 'removed upstream', revision, origin });
- if (cur && cur.sources_revision >= revision) return { outcome: 'unchanged', item: cur };
- const n = await normalise(item);
- if (!n.headline) {
- await recordRun({ origin, state: 'rejected', source_key: item.source_key || null, sources_item_id: item.id, error_code: 'item.no_headline', detail: 'a Sources item without a title cannot be shown' });
- await failedEvent({ origin, sourceKey: item.source_key || null, sourcesItemId: item.id, state: 'rejected', errorCode: 'item.no_headline' });
- return { outcome: 'rejected', reason: 'no headline' };
- }
- const now = store.now();
- if (cur) {
- if (cur.status === 'removed') return { outcome: 'unchanged', item: cur }; // removal is sticky here too
- await db.prepare(`UPDATE news_source_items SET sources_revision = @rev, canonical_url = @canonical_url, url_key = @url_key, headline = @headline,
- outlet = @outlet, authors = @authors, published_at = @published_at, summary = @summary, summary_basis = @summary_basis,
- license_note = @license_note, terms_note = @terms_note, content_hash = @content_hash, title_key = @title_key,
- retrieved_at = @retrieved_at, upstream_updated_at = @now, updated_at = @now WHERE id = @id`)
- .run({ ...n, rev: revision, now, id: cur.id });
- const row = await q.byId.get(cur.id);
- if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);
- await ingestedEvent(row, 'updated', { previous_revision: cur.sources_revision });
- const affected = stories ? await stories.onUpstreamChange(row, 'source_updated', { previous: cur }) : [];
- return { outcome: 'updated', item: row, stories: affected };
- }
- const at = (() => { const p = n.published_at ? Date.parse(n.published_at) : NaN; return Number.isFinite(p) ? p : now; })();
- const dup = await dedupe(n, item.source_key, at);
- const id = newItemId(now);
- await db.prepare(`INSERT INTO news_source_items (id, sources_item_id, sources_revision, source_key, canonical_url, url_key, headline, outlet, authors,
- published_at, summary, summary_basis, license_note, terms_note, content_hash, title_key, status, duplicate_of, dedupe,
- retrieved_at, first_seen_at, updated_at)
- VALUES (@id, @sources_item_id, @rev, @source_key, @canonical_url, @url_key, @headline, @outlet, @authors,
- @published_at, @summary, @summary_basis, @license_note, @terms_note, @content_hash, @title_key, @status, @duplicate_of, @dedupe,
- @retrieved_at, @now, @now)`)
- .run({ ...n, id, sources_item_id: item.id, rev: revision, source_key: String(item.source_key || 'unknown'),
- status: dup ? 'duplicate' : 'active', duplicate_of: dup ? dup.of : null, dedupe: dup ? JSON.stringify({ rule: dup.rule, detail: dup.detail }) : null, now });
- await clusters.assign(await q.byId.get(id));
- const row = await q.byId.get(id);
- await ingestedEvent(row, 'created');
- return { outcome: 'created', item: row };
- });
- }
-
- /** A Sources removal (takedown, licence, error): sticky; the stored summary is dropped. */
- async function applyRemoval({ sourcesItemId, reason, revision = null, origin = 'webhook' }) {
- return await db.tx(async () => {
- const cur = await q.bySourcesId.get(sourcesItemId);
- if (!cur) return { outcome: 'ignored', reason: 'never ingested' };
- if (cur.status === 'removed') return { outcome: 'unchanged', item: cur };
- // A story resting on it may stop being open for comments (domain/discussion.js).
- if (discussion) await discussion.watchSource(cur.id);
- const now = store.now();
- await db.prepare(`UPDATE news_source_items SET status = 'removed', summary = NULL, summary_basis = 'removed_upstream', removed_at = ?, removed_reason = ?,
- sources_revision = GREATEST(sources_revision, COALESCE(?, sources_revision)), updated_at = ? WHERE id = ?`)
- .run(now, String(reason || 'removed upstream').slice(0, 500), revision, now, cur.id);
- const row = await q.byId.get(cur.id);
- if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);
- await ingestedEvent(row, 'removed', { reason: row.removed_reason, origin });
- const affected = stories ? await stories.onUpstreamChange(row, 'source_removed', { previous: cur }) : [];
- return { outcome: 'removed', item: row, stories: affected };
- });
- }
-
- /** sources.fetch.failed for a news source: recorded and relayed; nothing else changes. */
- async function upstreamFailure(payload) {
- return await db.tx(async () => {
- if (payload.category && payload.category !== config.sources.category) return { outcome: 'ignored' };
- await recordRun({ origin: 'sources', state: String(payload.state || 'failed').slice(0, 40), source_key: payload.source_key || null, error_code: payload.error_code || null,
- detail: `run ${payload.run_id || '?'}${payload.http_status ? ` HTTP ${payload.http_status}` : ''}${payload.consecutive_failures ? `, ${payload.consecutive_failures} consecutive failures` : ''}` });
- await q.putStatus.run({ source_key: payload.source_key || 'unknown', name: null, homepage_url: null, status: 'failing', stale: null, last_success_at: null, now: store.now() });
- await failedEvent({ origin: 'sources', sourceKey: payload.source_key || null, state: payload.state || 'failed', errorCode: payload.error_code || null, httpStatus: payload.http_status || null,
- extra: { run_id: payload.run_id || null, consecutive_failures: payload.consecutive_failures || null } });
- return { outcome: 'recorded' };
- });
- }
-
- /** Remember what a Sources page says about its sources' health (display only). */
- async function rememberSources(map) {
- for (const [key, h] of Object.entries(map || {})) {
- await q.putStatus.run({ source_key: key, name: null, homepage_url: null, status: h.status || null, stale: h.stale == null ? null : (h.stale ? 1 : 0), last_success_at: h.last_success_at || null, now: store.now() });
- }
- }
-
- /** Fetch registry names we do not know yet (needs sources.source.read; failure = keep the host name). */
- async function learnOutlets(keys) {
- for (const key of [...new Set(keys)]) {
- const st = await q.status.get(key);
- if (st && st.name) continue;
- try {
- const s = await sources.getSource(key);
- await q.putStatus.run({ source_key: key, name: s && s.name ? String(s.name).slice(0, 200) : null, homepage_url: s && s.homepage_url ? String(s.homepage_url) : null,
- status: s && s.health ? s.health.status || null : null, stale: s && s.health && s.health.stale != null ? (s.health.stale ? 1 : 0) : null,
- last_success_at: s && s.health ? s.health.last_success_at || null : null, now: store.now() });
- } catch { /* optional grant; the outlet stays the URL's host */ }
- }
- }
-
- /**
- * The cursor pull. Reads pages from the stored cursor until Sources says there is no more (or
- * the page budget is spent), applying each item. The cursor moves only past applied pages.
- */
- let pulling = null;
- async function pull() {
- if (pulling) return pulling;
- pulling = (async () => {
- const before = await store.getState('sources_cursor', 0);
- let after = before;
- const counts = { created: 0, updated: 0, removed: 0, unchanged: 0, ignored: 0, rejected: 0, duplicates: 0 };
- try {
- for (let page = 0; page < config.sources.maxPagesPerRun; page++) {
- const body = await sources.listItems({ after, limit: config.sources.pageSize });
- await rememberSources(body.sources);
- await learnOutlets(body.items.map((i) => i.source_key).filter(Boolean));
- await store.tx(async () => {
- for (const item of body.items) {
- const r = await apply(item, { origin: 'pull' });
- counts[r.outcome] = (counts[r.outcome] || 0) + 1;
- if (r.outcome === 'created' && r.item.status === 'duplicate') counts.duplicates++;
- }
- await store.setState('sources_cursor', body.next_after);
- });
- after = body.next_after;
- if (!body.more) break;
- }
- await store.tx(async () => await recordRun({ origin: 'pull', state: 'ok', counts, cursor_before: before, cursor_after: after }));
- return { ok: true, counts, cursor: after };
- } catch (err) {
- const prev = await q.lastPull.get();
- await store.tx(async () => {
- await recordRun({ origin: 'pull', state: 'failed', error_code: err.code || 'pull.error', detail: err.message, counts, cursor_before: before, cursor_after: after });
- // One event per outage, not one per retry.
- if (!prev || prev.state !== 'failed') await failedEvent({ origin: 'pull', state: 'failed', errorCode: err.code || 'pull.error', detail: err.message, httpStatus: err.status || null });
- });
- log.warn(`[News] Sources pull failed (${err.code || 'error'}): ${err.message}`);
- return { ok: false, error: err.code || 'pull.error', detail: err.message, counts, cursor: after };
- } finally { pulling = null; }
- })();
- return pulling;
- }
-
- return {
- NEAR_TITLE,
- setStories(s) { stories = s; },
- normalise, dedupe, apply, applyRemoval, upstreamFailure, pull, recordRun, failedEvent, learnOutlets, rememberSources,
- get: async (id) => await q.byId.get(String(id || '')) || null,
- bySourcesId: async (id) => await q.bySourcesId.get(String(id || '')) || null,
- runs: async ({ limit = 30 } = {}) => await db.prepare('SELECT * FROM news_ingest_runs ORDER BY id DESC LIMIT ?').all(limit),
- sourceStatus: async () => await db.prepare('SELECT * FROM news_source_status ORDER BY source_key').all(),
- lastPull: async () => await q.lastPull.get() || null,
- };
-}
-
-module.exports = { createIngest, NEAR_TITLE };
@@ -9,7 +9,7 @@
@@ -9,7 +9,7 @@
*
*
* News' gate policy: at least NEWS_MIN_INDEPENDENT_SOURCES (2) independent sources, at least 40
* News' gate policy: at least NEWS_MIN_INDEPENDENT_SOURCES (2) independent sources, at least 40
* words. The sources counted are the Sources items the published revision's paragraphs cite and
* words. The sources counted are the Sources items the published revision's paragraphs cite and
- * that are still live upstream, grouped by text.independentSources: items from the same Sources
+ * that are still live upstream, grouped by normalize.independentSources: items from the same Sources
* source, the same publisher's domain, the same outlet, or copies of the same original report are
* source, the same publisher's domain, the same outlet, or copies of the same original report are
* one source. A story on fewer is still published and readable, but noindex ('unsourced'), so it
* one source. A story on fewer is still published and readable, but noindex ('unsourced'), so it
* stays out of sitemaps. A paragraph whose every source was removed upstream counts as an
* stays out of sitemaps. A paragraph whose every source was removed upstream counts as an
@@ -22,7 +22,8 @@
@@ -22,7 +22,8 @@
const seo = require('openvibe-publishing/seo');
const seo = require('openvibe-publishing/seo');
const hooks = require('openvibe-publishing/index-hooks');
const hooks = require('openvibe-publishing/index-hooks');
const authorship = require('openvibe-publishing/authorship');
const authorship = require('openvibe-publishing/authorship');
-const { independentSources } = require('./text');
+const { createPublication: createPublicationGlue } = require('openvibe-publishing/publication');
+const { normalize } = require('openvibe-publishing/ingest');
// minSources counts independent sources; config.indexing.minIndependentSources overrides it.
// minSources counts independent sources; config.indexing.minIndependentSources overrides it.
const POLICY = Object.freeze({ minWords: 40, requireSources: true, minSources: 2 });
const POLICY = Object.freeze({ minWords: 40, requireSources: true, minSources: 2 });
@@ -47,9 +48,13 @@ function disclosure(rec, review) {
@@ -47,9 +48,13 @@ function disclosure(rec, review) {
return { ...d, long: `${d.long.replace(/\.\s*$/, '')}. ${reviewed ? 'Reviewed by a person.' : 'Not yet reviewed by a person.'}` };
return { ...d, long: `${d.long.replace(/\.\s*$/, '')}. ${reviewed ? 'Reviewed by a person.' : 'Not yet reviewed by a person.'}` };
}
}
-function createPublication({ store, config, outbox }) {
+function createPublication({ store, config, outbox, indexnow }) {
const { db } = store;
const { db } = store;
const policy = Object.freeze({ ...POLICY, minSources: (config.indexing && config.indexing.minIndependentSources) || POLICY.minSources });
const policy = Object.freeze({ ...POLICY, minSources: (config.indexing && config.indexing.minIndependentSources) || POLICY.minSources });
+ // The chassis owns the mechanics: the sequencer revision, the search.index-document@1 event, the
+ // outbox write and the IndexNow ping — all on the caller's transaction handle (this product's
+ // gate, document builder and canonical path stay here).
+ const glue = createPublicationGlue({ owner: OWNER, sequencer: store.sequencer, outbox, baseUrl: config.baseUrl, indexnow, db, now: store.now });
const storyById = db.prepare('SELECT * FROM news_stories WHERE id = ?');
const storyById = db.prepare('SELECT * FROM news_stories WHERE id = ?');
const itemById = db.prepare('SELECT * FROM news_source_items WHERE id = ?');
const itemById = db.prepare('SELECT * FROM news_source_items WHERE id = ?');
const topicById = db.prepare('SELECT * FROM news_topics WHERE id = ?');
const topicById = db.prepare('SELECT * FROM news_topics WHERE id = ?');
@@ -83,7 +88,7 @@ function createPublication({ store, config, outbox }) {
@@ -83,7 +88,7 @@ function createPublication({ store, config, outbox }) {
for (const n of ns) if (alive(n)) cited.add(n);
for (const n of ns) if (alive(n)) cited.add(n);
if (!ns.some(alive)) unsupported++;
if (!ns.some(alive)) unsupported++;
}
}
- const groups = independentSources([...cited].map((n) => live.get(n)));
+ const groups = normalize.independentSources([...cited].map((n) => live.get(n)));
return { citedLive: cited.size, independent: groups.length, groups, unsupported, live };
return { citedLive: cited.size, independent: groups.length, groups, unsupported, live };
}
}
@@ -153,11 +158,7 @@ function createPublication({ store, config, outbox }) {
@@ -153,11 +158,7 @@ function createPublication({ store, config, outbox }) {
/** Stamp and enqueue the Search document when it changed. Inside the caller's transaction. */
/** Stamp and enqueue the Search document when it changed. Inside the caller's transaction. */
async function syncIndex(story, { traceparent } = {}) {
async function syncIndex(story, { traceparent } = {}) {
const { doc } = await documentFor(story, { forSearch: true });
const { doc } = await documentFor(story, { forSearch: true });
- const prev = await store.sequencer.current(OWNER, 'story', story.id);
- if (doc.deleted && prev == null) return null;
- const stamped = await store.sequencer.stamp(store.db, doc);
- if (prev != null && stamped.revision === prev) return null;
- return await outbox.emit(hooks.indexEvent({ document: stamped, now: store.now() }), { traceparent });
+ return await glue.index(store.db, { document: doc, page: storyPath(story), traceparent });
}
}
function snapshot(story) {
function snapshot(story) {
@@ -172,14 +173,15 @@ function createPublication({ store, config, outbox }) {
@@ -172,14 +173,15 @@ function createPublication({ store, config, outbox }) {
const story = await storyById.get(storyId);
const story = await storyById.get(storyId);
const after = snapshot(story);
const after = snapshot(story);
let event = null;
let event = null;
+ let decision = null;
if (after.state === 'retracted' && (!before || before.state !== 'retracted')) {
if (after.state === 'retracted' && (!before || before.state !== 'retracted')) {
- const d = await decide(story, await store.revisions.get(story.id, story.published_revision));
+ decision = await decide(story, await store.revisions.get(story.id, story.published_revision));
event = await outbox.emit({
event = await outbox.emit({
event_type: 'news.story.retracted', version: 1, source: OWNER, actor: actorRef(actor),
event_type: 'news.story.retracted', version: 1, source: OWNER, actor: actorRef(actor),
timestamp: new Date(store.now()).toISOString(),
timestamp: new Date(store.now()).toISOString(),
- visibility: d.listable ? 'public' : 'internal',
+ visibility: decision.listable ? 'public' : 'internal',
subject: { type: 'story', id: story.id, revision: story.published_revision || 0 },
subject: { type: 'story', id: story.id, revision: story.published_revision || 0 },
- payload: { canonical_url: storyUrl(story), publication_state: 'retracted', indexability: hooks.searchIndexability(d), ...extra },
+ payload: { canonical_url: storyUrl(story), publication_state: 'retracted', indexability: hooks.searchIndexability(decision), ...extra },
}, { traceparent });
}, { traceparent });
} else {
} else {
// A retracted story is still published (with its notice) for the product events.
// A retracted story is still published (with its notice) for the product events.
@@ -187,15 +189,19 @@ function createPublication({ store, config, outbox }) {
@@ -187,15 +189,19 @@ function createPublication({ store, config, outbox }) {
let action = hooks.actionFor(norm(before), norm(after));
let action = hooks.actionFor(norm(before), norm(after));
if (!action && before && before.state === 'published' && after.state === 'published' && before.url !== after.url) action = 'updated';
if (!action && before && before.state === 'published' && after.state === 'published' && before.url !== after.url) action = 'updated';
if (action) {
if (action) {
- const { doc, decision } = await documentFor(story, { forSearch: false });
+ const { doc, decision: dec } = await documentFor(story, { forSearch: false });
+ decision = dec;
const topic = await topicOf(story);
const topic = await topicOf(story);
- event = await outbox.emit(hooks.publicationEvent({
- product: OWNER, type: 'story', action, id: story.id, revision: story.published_revision || 0,
- actor: actorRef(actor), document: doc, decision, now: store.now(),
+ event = await glue.publication(store.db, {
+ type: 'story', action, id: story.id, revision: story.published_revision || 0,
+ actor: actorRef(actor), document: doc, decision, traceparent,
extra: { ...(topic ? { topic: topic.slug } : {}), ...extra },
extra: { ...(topic ? { topic: topic.slug } : {}), ...extra },
- }), { traceparent });
+ });
}
}
}
}
+ // The chassis pings IndexNow inside syncIndex when the stamped document is an indexable page
+ // that appeared or changed, or a page Search already had that went away; never a draft,
+ // private or noindex page.
await syncIndex(story, { traceparent });
await syncIndex(story, { traceparent });
return { story, event };
return { story, event };
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,302 @@
@@ -0,0 +1,302 @@
+'use strict';
+
+/**
+ * OpenVibe.Sources items (category news) → news_source_items → dedupe → clusters.
+ *
+ * The chassis (openvibe-publishing/ingest) owns the Sources client, the change cursor and the pull
+ * loop; this module keeps News' domain half: what a Sources item becomes in News' tables (licensing
+ * limits included), dedupe, cluster assignment, upstream changes and the recorded run history.
+ *
+ * Two ways in, one effect:
+ * - the Events webhook (sources.item.created|updated|removed, sources.fetch.failed), each event
+ * applied once through the inbox (http/events.js);
+ * - the cursor pull (worker), the backstop for missed deliveries: the chassis reads
+ * /api/v1/items in change order from the cursor in news_ingest_cursor and calls apply() per item.
+ * apply() is idempotent on (Sources item id, Sources revision): a replay, or the same item arriving
+ * by both paths, changes nothing and emits nothing.
+ *
+ * What is stored (licensing): headline, canonical URL, outlet, authors, published_at, the licence
+ * and terms notes, and a short summary ONLY when those notes explicitly allow short summaries
+ * (normalize.licensedSummary). Every other field of the Sources item — whatever an adapter mapped
+ * into `fields` — is ignored, so an article body can never be stored, shown, fed or indexed.
+ *
+ * Failure is a recorded state (news_ingest_runs + news.source.failed), never a replacement: a
+ * failed fetch or an unreadable item creates no source item, no cluster and no story text.
+ *
+ * Upstream changes to an item a story cites (a revision, a removal) go to stories.onUpstreamChange,
+ * which flags the story and prepares a pending revision; published text is never changed silently.
+ * A removal can also hide the story's Community thread (domain/discussion.js, after the commit).
+ */
+const { ids } = require('openvibe-contracts');
+const { createChangeCursor, pullChanges, normalize } = require('openvibe-publishing/ingest');
+
+const NEAR_TITLE = { anyOutlet: 0.9, sameOutlet: 0.75, windowMs: 48 * 3600 * 1000 };
+const newItemId = (now) => `nsi_${ids.ulid(now)}`;
+const CURSOR = 'sources';
+
+// The chassis' three pull outcomes: News' own apply() outcome decides which one it is.
+const PULL_OUTCOME = { created: 'applied', updated: 'applied', unchanged: 'applied', ignored: 'hold', rejected: 'applied', removed: 'removed' };
+
+function createIngest({ store, config, clusters, outbox, sources, discussion = null, log = console }) {
+ const { db } = store;
+ const cursor = createChangeCursor(db, { prefix: 'news', now: store.now });
+ let stories = null; // set by app.js (stories depend on ingest's items too)
+ const q = {
+ bySourcesId: db.prepare('SELECT * FROM news_source_items WHERE sources_item_id = ?'),
+ byId: db.prepare('SELECT * FROM news_source_items WHERE id = ?'),
+ byUrl: db.prepare("SELECT * FROM news_source_items WHERE url_key = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),
+ byHash: db.prepare("SELECT * FROM news_source_items WHERE content_hash = ? AND status <> 'removed' ORDER BY first_seen_at, seq LIMIT 1"),
+ recent: db.prepare("SELECT * FROM news_source_items WHERE status <> 'removed' AND first_seen_at >= ? ORDER BY first_seen_at, seq"),
+ status: db.prepare('SELECT * FROM news_source_status WHERE source_key = ?'),
+ putStatus: db.prepare(`INSERT INTO news_source_status (source_key, name, homepage_url, status, stale, last_success_at, refreshed_at)
+ VALUES (@source_key, @name, @homepage_url, @status, @stale, @last_success_at, @now)
+ ON CONFLICT (source_key) DO UPDATE SET name = COALESCE(excluded.name, news_source_status.name), homepage_url = COALESCE(excluded.homepage_url, news_source_status.homepage_url),
+ status = COALESCE(excluded.status, news_source_status.status), stale = COALESCE(excluded.stale, news_source_status.stale),
+ last_success_at = COALESCE(excluded.last_success_at, news_source_status.last_success_at), refreshed_at = excluded.refreshed_at`),
+ run: db.prepare(`INSERT INTO news_ingest_runs (origin, state, source_key, sources_item_id, error_code, detail, counts, cursor_before, cursor_after, at)
+ VALUES (@origin, @state, @source_key, @sources_item_id, @error_code, @detail, @counts, @cursor_before, @cursor_after, @at)`),
+ lastPull: db.prepare("SELECT * FROM news_ingest_runs WHERE origin = 'pull' ORDER BY id DESC LIMIT 1"),
+ };
+
+ async function recordRun(r) {
+ await q.run.run({ source_key: null, sources_item_id: null, error_code: null, detail: null, counts: null, cursor_before: null, cursor_after: null, ...r, counts: r.counts ? JSON.stringify(r.counts) : null, at: store.now() });
+ }
+
+ async function failedEvent({ origin, sourceKey = null, sourcesItemId = null, state, errorCode = null, detail = null, httpStatus = null, extra = {} }) {
+ await outbox.emit({
+ event_type: 'news.source.failed', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'important',
+ subject: sourceKey ? { type: 'source', id: sourceKey } : { type: 'ingest', id: origin },
+ payload: { origin, source_key: sourceKey, sources_item_id: sourcesItemId, state, error_code: errorCode, http_status: httpStatus, detail: detail ? String(detail).slice(0, 500) : null, ...extra },
+ });
+ }
+
+ async function outletFor(item) {
+ const st = await q.status.get(item.source_key);
+ return (st && st.name) || normalize.hostOf(item.canonical_url) || item.source_key;
+ }
+
+ /** The fields News keeps from a Sources item. Nothing else of the item is read. */
+ async function normalise(item) {
+ const prov = item.provenance || {};
+ const headline = String(item.title || '').replace(/\s+/g, ' ').trim().slice(0, 500);
+ const lic = normalize.licensedSummary(item.summary, { termsNote: prov.terms_note, licenseNote: prov.license_note, maxChars: config.licensing.summaryMaxChars });
+ const authors = Array.isArray(item.authors) ? item.authors.map((a) => String(a || '').replace(/\s+/g, ' ').trim().slice(0, 200)).filter(Boolean).slice(0, 10) : [];
+ const published = item.published_at && Number.isFinite(Date.parse(item.published_at)) ? new Date(Date.parse(item.published_at)).toISOString() : null;
+ return {
+ headline,
+ canonical_url: item.canonical_url || null,
+ url_key: normalize.urlKey(item.canonical_url),
+ outlet: await outletFor(item),
+ authors: JSON.stringify(authors),
+ published_at: published,
+ summary: lic.summary,
+ summary_basis: lic.basis,
+ license_note: prov.license_note ? String(prov.license_note).slice(0, 2000) : null,
+ terms_note: prov.terms_note ? String(prov.terms_note).slice(0, 2000) : null,
+ content_hash: prov.content_hash || null,
+ title_key: normalize.titleKey(headline),
+ retrieved_at: prov.retrieved_at || null,
+ };
+ }
+
+ /** Is this new item a duplicate of one we have? → null | { of, rule, detail } */
+ async function dedupe(n, sourceKey, at) {
+ if (n.url_key) {
+ const hit = await q.byUrl.get(n.url_key);
+ if (hit) return { of: hit.duplicate_of || hit.id, rule: 'canonical_url', detail: n.url_key };
+ }
+ if (n.content_hash) {
+ const hit = await q.byHash.get(n.content_hash);
+ if (hit) return { of: hit.duplicate_of || hit.id, rule: 'content_hash', detail: n.content_hash };
+ }
+ const mine = normalize.shingles(n.headline);
+ const host = normalize.hostOf(n.canonical_url);
+ let best = null;
+ for (const c of await q.recent.all(Math.min(at, store.now()) - NEAR_TITLE.windowMs)) {
+ const t = clusters.itemTime(c);
+ if (Math.abs(t - at) > NEAR_TITLE.windowMs) continue;
+ const score = normalize.jaccard(mine, normalize.shingles(c.headline));
+ const sameOutlet = c.source_key === sourceKey || (host && normalize.hostOf(c.canonical_url) === host);
+ const threshold = sameOutlet ? NEAR_TITLE.sameOutlet : NEAR_TITLE.anyOutlet;
+ if (score >= threshold && (!best || score > best.score)) best = { c, score, sameOutlet };
+ }
+ if (best) return { of: best.c.duplicate_of || best.c.id, rule: 'near_title', detail: `jaccard ${best.score.toFixed(2)} of headline 2-shingles${best.sameOutlet ? ' (same outlet)' : ''}` };
+ return null;
+ }
+
+ async function ingestedEvent(row, action, extra = {}) {
+ await outbox.emit({
+ event_type: 'news.source.ingested', actor: { type: 'service', id: 'news' }, visibility: 'internal', priority: 'low',
+ subject: { type: 'source_item', id: row.id },
+ payload: {
+ item_id: row.id, action, status: row.status,
+ sources_item: { service: 'sources', type: 'item', id: row.sources_item_id, revision: row.sources_revision },
+ source_key: row.source_key, canonical_url: row.canonical_url, duplicate_of: row.duplicate_of,
+ dedupe_rule: row.dedupe ? JSON.parse(row.dedupe).rule : null, cluster_id: row.cluster_id, ...extra,
+ },
+ });
+ }
+
+ /**
+ * Apply one Sources item (inside the caller's transaction). Returns
+ * { outcome: created|updated|removed|unchanged|ignored|rejected, item?, reason? }.
+ */
+ async function apply(item, { origin = 'pull' } = {}) {
+ return await db.tx(async () => {
+ if (!item || typeof item.id !== 'string') return { outcome: 'rejected', reason: 'item without id' };
+ if (item.category && item.category !== config.sources.category) return { outcome: 'ignored', reason: `category ${item.category}` };
+ const revision = Number.isInteger(item.revision) ? item.revision : 1;
+ const cur = await q.bySourcesId.get(item.id);
+ if (item.removed) return await applyRemoval({ sourcesItemId: item.id, reason: item.removed.reason || 'removed upstream', revision, origin });
+ if (cur && cur.sources_revision >= revision) return { outcome: 'unchanged', item: cur };
+ const n = await normalise(item);
+ if (!n.headline) {
+ await recordRun({ origin, state: 'rejected', source_key: item.source_key || null, sources_item_id: item.id, error_code: 'item.no_headline', detail: 'a Sources item without a title cannot be shown' });
+ await failedEvent({ origin, sourceKey: item.source_key || null, sourcesItemId: item.id, state: 'rejected', errorCode: 'item.no_headline' });
+ return { outcome: 'rejected', reason: 'no headline' };
+ }
+ const now = store.now();
+ if (cur) {
+ if (cur.status === 'removed') return { outcome: 'unchanged', item: cur }; // removal is sticky here too
+ await db.prepare(`UPDATE news_source_items SET sources_revision = @rev, canonical_url = @canonical_url, url_key = @url_key, headline = @headline,
+ outlet = @outlet, authors = @authors, published_at = @published_at, summary = @summary, summary_basis = @summary_basis,
+ license_note = @license_note, terms_note = @terms_note, content_hash = @content_hash, title_key = @title_key,
+ retrieved_at = @retrieved_at, upstream_updated_at = @now, updated_at = @now WHERE id = @id`)
+ .run({ ...n, rev: revision, now, id: cur.id });
+ const row = await q.byId.get(cur.id);
+ if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);
+ await ingestedEvent(row, 'updated', { previous_revision: cur.sources_revision });
+ const affected = stories ? await stories.onUpstreamChange(row, 'source_updated', { previous: cur }) : [];
+ return { outcome: 'updated', item: row, stories: affected };
+ }
+ const at = (() => { const p = n.published_at ? Date.parse(n.published_at) : NaN; return Number.isFinite(p) ? p : now; })();
+ const dup = await dedupe(n, item.source_key, at);
+ const id = newItemId(now);
+ await db.prepare(`INSERT INTO news_source_items (id, sources_item_id, sources_revision, source_key, canonical_url, url_key, headline, outlet, authors,
+ published_at, summary, summary_basis, license_note, terms_note, content_hash, title_key, status, duplicate_of, dedupe,
+ retrieved_at, first_seen_at, updated_at)
+ VALUES (@id, @sources_item_id, @rev, @source_key, @canonical_url, @url_key, @headline, @outlet, @authors,
+ @published_at, @summary, @summary_basis, @license_note, @terms_note, @content_hash, @title_key, @status, @duplicate_of, @dedupe,
+ @retrieved_at, @now, @now)`)
+ .run({ ...n, id, sources_item_id: item.id, rev: revision, source_key: String(item.source_key || 'unknown'),
+ status: dup ? 'duplicate' : 'active', duplicate_of: dup ? dup.of : null, dedupe: dup ? JSON.stringify({ rule: dup.rule, detail: dup.detail }) : null, now });
+ await clusters.assign(await q.byId.get(id));
+ const row = await q.byId.get(id);
+ await ingestedEvent(row, 'created');
+ return { outcome: 'created', item: row };
+ });
+ }
+
+ /** A Sources removal (takedown, licence, error): sticky; the stored summary is dropped. */
+ async function applyRemoval({ sourcesItemId, reason, revision = null, origin = 'webhook' }) {
+ return await db.tx(async () => {
+ const cur = await q.bySourcesId.get(sourcesItemId);
+ if (!cur) return { outcome: 'ignored', reason: 'never ingested' };
+ if (cur.status === 'removed') return { outcome: 'unchanged', item: cur };
+ // A story resting on it may stop being open for comments (domain/discussion.js).
+ if (discussion) await discussion.watchSource(cur.id);
+ const now = store.now();
+ await db.prepare(`UPDATE news_source_items SET status = 'removed', summary = NULL, summary_basis = 'removed_upstream', removed_at = ?, removed_reason = ?,
+ sources_revision = GREATEST(sources_revision, COALESCE(?, sources_revision)), updated_at = ? WHERE id = ?`)
+ .run(now, String(reason || 'removed upstream').slice(0, 500), revision, now, cur.id);
+ const row = await q.byId.get(cur.id);
+ if (row.cluster_id) await clusters.recompute((await clusters.resolve(row.cluster_id)).id);
+ await ingestedEvent(row, 'removed', { reason: row.removed_reason, origin });
+ const affected = stories ? await stories.onUpstreamChange(row, 'source_removed', { previous: cur }) : [];
+ return { outcome: 'removed', item: row, stories: affected };
+ });
+ }
+
+ /** sources.fetch.failed for a news source: recorded and relayed; nothing else changes. */
+ async function upstreamFailure(payload) {
+ return await db.tx(async () => {
+ if (payload.category && payload.category !== config.sources.category) return { outcome: 'ignored' };
+ await recordRun({ origin: 'sources', state: String(payload.state || 'failed').slice(0, 40), source_key: payload.source_key || null, error_code: payload.error_code || null,
+ detail: `run ${payload.run_id || '?'}${payload.http_status ? ` HTTP ${payload.http_status}` : ''}${payload.consecutive_failures ? `, ${payload.consecutive_failures} consecutive failures` : ''}` });
+ await q.putStatus.run({ source_key: payload.source_key || 'unknown', name: null, homepage_url: null, status: 'failing', stale: null, last_success_at: null, now: store.now() });
+ await failedEvent({ origin: 'sources', sourceKey: payload.source_key || null, state: payload.state || 'failed', errorCode: payload.error_code || null, httpStatus: payload.http_status || null,
+ extra: { run_id: payload.run_id || null, consecutive_failures: payload.consecutive_failures || null } });
+ return { outcome: 'recorded' };
+ });
+ }
+
+ /** Remember what a Sources page says about its sources' health (display only). */
+ async function rememberSources(map) {
+ for (const [key, h] of Object.entries(map || {})) {
+ await q.putStatus.run({ source_key: key, name: null, homepage_url: null, status: h.status || null, stale: h.stale == null ? null : (h.stale ? 1 : 0), last_success_at: h.last_success_at || null, now: store.now() });
+ }
+ }
+
+ /** Fetch registry names we do not know yet (needs sources.source.read; failure = keep the host name). */
+ async function learnOutlets(keys) {
+ for (const key of [...new Set(keys)]) {
+ const st = await q.status.get(key);
+ if (st && st.name) continue;
+ try {
+ const s = await sources.getSource(key);
+ await q.putStatus.run({ source_key: key, name: s && s.name ? String(s.name).slice(0, 200) : null, homepage_url: s && s.homepage_url ? String(s.homepage_url) : null,
+ status: s && s.health ? s.health.status || null : null, stale: s && s.health && s.health.stale != null ? (s.health.stale ? 1 : 0) : null,
+ last_success_at: s && s.health ? s.health.last_success_at || null : null, now: store.now() });
+ } catch { /* optional grant; the outlet stays the URL's host */ }
+ }
+ }
+
+ /**
+ * The cursor pull. The chassis (openvibe-publishing/ingest.pullChanges) reads pages from the
+ * stored cursor until Sources says there is no more (or the page budget is spent), applying each
+ * item in its own savepoint and moving the cursor only past applied pages. This module's page
+ * source also refreshes the display-only source status; apply() records News' own counts.
+ */
+ let pulling = null;
+ async function pull() {
+ if (pulling) return pulling;
+ pulling = (async () => {
+ const before = await cursor.get(CURSOR);
+ const counts = { created: 0, updated: 0, removed: 0, unchanged: 0, ignored: 0, rejected: 0, duplicates: 0, failed: 0 };
+ const tally = (r) => { if (!r || !r.outcome) return; counts[r.outcome] = (counts[r.outcome] || 0) + 1; if (r.outcome === 'created' && r.item && r.item.status === 'duplicate') counts.duplicates++; };
+ const pageSource = {
+ listItems: async (opts) => {
+ const body = await sources.listItems(opts);
+ await rememberSources(body.sources);
+ await learnOutlets((body.items || []).map((i) => i.source_key).filter(Boolean));
+ return body;
+ },
+ };
+ try {
+ const summary = await pullChanges({
+ db, cursor, source: pageSource, name: CURSOR,
+ maxPages: config.sources.maxPagesPerRun, pageSize: config.sources.pageSize,
+ apply: async (item) => { const r = await apply(item, { origin: 'pull' }); tally(r); return PULL_OUTCOME[r.outcome] || 'applied'; },
+ onItem: (_item, err, outcome) => { if (outcome === 'failed') { counts.failed++; log.warn(`[News] item skipped (${err && err.message ? err.message : 'error'})`); } },
+ });
+ await store.tx(async () => await recordRun({ origin: 'pull', state: 'ok', counts, cursor_before: before, cursor_after: summary.after }));
+ return { ok: true, counts, cursor: summary.after };
+ } catch (err) {
+ const after = await cursor.get(CURSOR);
+ const prev = await q.lastPull.get();
+ await store.tx(async () => {
+ await recordRun({ origin: 'pull', state: 'failed', error_code: err.code || 'pull.error', detail: err.message, counts, cursor_before: before, cursor_after: after });
+ // One event per outage, not one per retry.
+ if (!prev || prev.state !== 'failed') await failedEvent({ origin: 'pull', state: 'failed', errorCode: err.code || 'pull.error', detail: err.message, httpStatus: err.status || null });
+ });
+ log.warn(`[News] Sources pull failed (${err.code || 'error'}): ${err.message}`);
+ return { ok: false, error: err.code || 'pull.error', detail: err.message, counts, cursor: after };
+ } finally { pulling = null; }
+ })();
+ return pulling;
+ }
+
+ return {
+ NEAR_TITLE,
+ setStories(s) { stories = s; },
+ normalise, dedupe, apply, applyRemoval, upstreamFailure, pull, recordRun, failedEvent, learnOutlets, rememberSources,
+ cursor: async () => await cursor.get(CURSOR),
+ get: async (id) => await q.byId.get(String(id || '')) || null,
+ bySourcesId: async (id) => await q.bySourcesId.get(String(id || '')) || null,
+ runs: async ({ limit = 30 } = {}) => await db.prepare('SELECT * FROM news_ingest_runs ORDER BY id DESC LIMIT ?').all(limit),
+ sourceStatus: async () => await db.prepare('SELECT * FROM news_source_status ORDER BY source_key').all(),
+ lastPull: async () => await q.lastPull.get() || null,
+ };
+}
+
+module.exports = { createIngest, NEAR_TITLE, CURSOR };
deleted file mode 100644
deleted file mode 100644
@@ -1,197 +0,0 @@
@@ -1,197 +0,0 @@
-'use strict';
-
-/**
- * Deterministic text helpers for ingestion: URL keys and title keys for dedupe, word shingles for
- * near-duplicate headlines, the terms and named entities clustering compares, the licensing rule
- * for source summaries, and which cited sources are independent (the index gate). Pure functions: the same input always gives the same output, so every
- * dedupe and cluster decision can be recomputed and explained.
- */
-
-const STOPWORDS = new Set(('a about above after again against all am an and any are as at be because been before being below between both but by '
- + 'can could did do does doing down during each few for from further had has have having he her here hers him his how i if in into is it its '
- + 'itself just me more most my no nor not now of off on once only or other our ours out over own same she should so some such than that the '
- + 'their theirs them then there these they this those through to too under until up very was we were what when where which while who whom '
- + 'why will with would you your yours says said say new news report reports update updates live latest via amid over after more first year '
- + 'years today yesterday week one two three four five six seven eight nine ten'
-).split(/\s+/));
-
-// Capitalised words that start sentences or headlines far more often than they name something.
-const NOT_ENTITIES = new Set(['the', 'a', 'an', 'in', 'on', 'at', 'as', 'how', 'why', 'what', 'when', 'where', 'who', 'new', 'breaking', 'update', 'live', 'watch', 'exclusive', 'opinion', 'analysis', 'report']);
-
-/** Lowercase, strip accents and punctuation, collapse spaces. */
-function fold(s) {
- return String(s == null ? '' : s).normalize('NFKD').replace(/[̀-ͯ]/g, '')
- .toLowerCase().replace(/[’']/g, '').replace(/[^a-z0-9]+/g, ' ').trim();
-}
-
-/** A headline normalised for comparison. */
-function titleKey(title) { return fold(title); }
-
-const TRACKING = /^(utm_[a-z]+|fbclid|gclid|mc_cid|mc_eid|ref|ref_src|cmpid|ocid|igshid|smid)$/i;
-
-/**
- * The URL key dedupe compares: https, lowercase host without "www.", no fragment, no tracking
- * parameters, remaining parameters sorted, no trailing slash. null for anything that is not http(s).
- */
-function urlKey(url) {
- if (!url) return null;
- let u;
- try { u = new URL(String(url)); } catch { return null; }
- if (u.protocol !== 'http:' && u.protocol !== 'https:') return null;
- const host = u.hostname.toLowerCase().replace(/^www\./, '');
- const params = [...u.searchParams.entries()].filter(([k]) => !TRACKING.test(k)).sort(([a, x], [b, y]) => (a < b ? -1 : a > b ? 1 : x < y ? -1 : x > y ? 1 : 0));
- const q = params.length ? `?${new URLSearchParams(params).toString()}` : '';
- let p = u.pathname.replace(/\/{2,}/g, '/');
- if (p.length > 1) p = p.replace(/\/+$/, '');
- return `https://${host}${u.port && u.port !== '443' && u.port !== '80' ? `:${u.port}` : ''}${p}${q}`;
-}
-
-/** The outlet shown when the registry has no name: the URL's host without "www.". */
-function hostOf(url) {
- try { return new URL(String(url)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return null; }
-}
-
-// Second-level labels under a two-letter country code that are not a publisher's own name
-// (bbc.co.uk, abc.net.au): the publisher is the label before them.
-const COUNTRY_SECOND_LEVEL = new Set(['co', 'com', 'org', 'net', 'gov', 'edu', 'ac', 'ne', 'or', 'go', 'gob', 'nic', 'mil']);
-
-/**
- * The publisher's domain of a URL: its host without subdomains (news.example.com and
- * www.example.com → example.com; news.bbc.co.uk → bbc.co.uk). Deliberately coarse: two sites on
- * one shared host (two blogs on one platform) count as one publisher, never the other way round.
- */
-function publisherDomain(url) {
- const host = hostOf(url);
- if (!host) return null;
- if (/^[\d.]+$/.test(host) || host.startsWith('[')) return host;
- const labels = host.split('.').filter(Boolean);
- if (labels.length <= 2) return labels.join('.');
- const n = labels.length;
- const take = labels[n - 1].length === 2 && COUNTRY_SECOND_LEVEL.has(labels[n - 2]) ? 3 : 2;
- return labels.slice(-take).join('.');
-}
-
-/**
- * Independent sources among source items (news_source_items rows): items are one source when they
- * share any of
- * - the OpenVibe.Sources source (source_key: the same feed or registry entry),
- * - the publisher's domain of their canonical URL (publisherDomain),
- * - the outlet name (folded),
- * - the original they duplicate (duplicate_of: the same report syndicated or copied),
- * and the relation is transitive. → groups of item ids, each group one independent source.
- */
-function independentSources(items) {
- const parent = new Map();
- const find = (k) => { while (parent.get(k) !== k) { parent.set(k, parent.get(parent.get(k))); k = parent.get(k); } return k; };
- const union = (a, b) => { const x = find(a); const y = find(b); if (x !== y) parent.set(y, x); };
- for (const it of items) {
- const node = `item:${it.id}`;
- if (!parent.has(node)) parent.set(node, node);
- const domain = publisherDomain(it.canonical_url);
- const outlet = fold(it.outlet);
- const keys = [`source:${it.source_key}`, `original:${it.duplicate_of || it.id}`, domain ? `domain:${domain}` : null, outlet ? `outlet:${outlet}` : null].filter(Boolean);
- for (const k of keys) {
- if (!parent.has(k)) parent.set(k, k);
- union(node, k);
- }
- }
- const groups = new Map();
- for (const it of items) {
- const root = find(`item:${it.id}`);
- if (!groups.has(root)) groups.set(root, []);
- groups.get(root).push(it.id);
- }
- return [...groups.values()];
-}
-
-/** Very light stemming: plural and possessive endings, so "rockets" and "rocket" meet. */
-function stem(w) {
- if (w.length > 4 && w.endsWith('ies')) return `${w.slice(0, -3)}y`;
- if (w.length > 3 && w.endsWith('s') && !w.endsWith('ss') && !w.endsWith('us') && !w.endsWith('is')) return w.slice(0, -1);
- return w;
-}
-
-/** Content terms of a text: folded words of 3+ letters (or any number), minus stopwords, stemmed, unique, sorted. */
-function terms(text) {
- const out = new Set();
- for (const w of fold(text).split(' ')) {
- if (!w || STOPWORDS.has(w)) continue;
- if (w.length < 3 && !/^\d+$/.test(w)) continue;
- out.add(stem(w));
- }
- return [...out].sort();
-}
-
-/**
- * Named entities, deterministically: runs of Capitalised words (and ALL-CAPS acronyms of 2+
- * letters) in the original text, folded. A single capitalised word at the very start of the text
- * counts only if it is an acronym or appears capitalised again later. Sorted, unique.
- */
-function entities(text) {
- const src = String(text || '');
- const tokens = src.split(/\s+/).filter(Boolean);
- const out = new Set();
- let run = [];
- const flush = () => {
- if (run.length) {
- const words = run.map((r) => r.word);
- const startsText = run[0].index === 0;
- const acronym = words.some((w) => /^[A-Z0-9]{2,}$/.test(w));
- if (!(startsText && words.length === 1 && !acronym && !laterCapitalised(words[0], run[0].index))) {
- const f = fold(words.join(' '));
- if (f && !NOT_ENTITIES.has(f) && !STOPWORDS.has(f)) out.add(f);
- }
- }
- run = [];
- };
- const laterCapitalised = (w, idx) => tokens.slice(idx + 1).some((t) => t.replace(/[^\p{L}\p{N}]/gu, '') === w);
- tokens.forEach((raw, index) => {
- const word = raw.replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, '');
- const cap = /^\p{Lu}/u.test(word) && !NOT_ENTITIES.has(word.toLowerCase());
- if (word && cap) run.push({ word, index }); else flush();
- if (/[.!?:;,]$/.test(raw)) flush();
- });
- flush();
- return [...out].sort();
-}
-
-/** Word 2-shingles of a folded headline (single-word headlines give the word itself). */
-function shingles(title) {
- const w = titleKey(title).split(' ').filter(Boolean);
- if (w.length < 2) return new Set(w);
- const s = new Set();
- for (let i = 0; i < w.length - 1; i++) s.add(`${w[i]} ${w[i + 1]}`);
- return s;
-}
-
-function jaccard(a, b) {
- if (!a.size && !b.size) return 0;
- let inter = 0;
- for (const x of a) if (b.has(x)) inter++;
- return inter / (a.size + b.size - inter);
-}
-
-// ── Licensing ─────────────────────────────────────────────────
-
-const ALLOWS_SUMMARY = /\b(short|brief)\s+summar(y|ies)\b|\bsummar(y|ies)\s+(are|is)\s+(allowed|permitted)\b/i;
-const FORBIDS_SUMMARY = /\b(no|not|never|without)\s+(short\s+|brief\s+)?summar(y|ies)\b|\bsummar(y|ies)\s+(are|is)\s+not\b|\bheadlines?\s+(and\s+links\s+)?only\b|\btitles?\s+and\s+links\s+only\b/i;
-
-/**
- * What of a source's summary News may keep: a short summary, only when the item's terms or licence
- * note explicitly allows short summaries (and nothing forbids them), cut to maxChars at a word
- * boundary. Article bodies are never kept, whatever the terms say. → { summary, basis }
- */
-function licensedSummary(summary, { termsNote, licenseNote, maxChars = 280 } = {}) {
- const notes = `${termsNote || ''}\n${licenseNote || ''}`;
- const text = String(summary || '').replace(/\s+/g, ' ').trim();
- if (!text) return { summary: null, basis: 'none_provided' };
- if (!maxChars) return { summary: null, basis: 'summaries_disabled' };
- if (FORBIDS_SUMMARY.test(notes)) return { summary: null, basis: 'terms_forbid_summaries' };
- if (!ALLOWS_SUMMARY.test(notes)) return { summary: null, basis: 'terms_do_not_allow_summaries' };
- if (text.length <= maxChars) return { summary: text, basis: 'terms_allow_short_summaries' };
- const cut = text.slice(0, maxChars - 1);
- const at = cut.lastIndexOf(' ');
- return { summary: `${(at > maxChars * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.-]+$/, '')}…`, basis: 'terms_allow_short_summaries' };
-}
-
-module.exports = { fold, titleKey, urlKey, hostOf, publisherDomain, independentSources, terms, entities, shingles, jaccard, licensedSummary, stem, STOPWORDS };
@@ -259,7 +259,7 @@ function createApi(ctx) {
@@ -259,7 +259,7 @@ function createApi(ctx) {
router.get('/ingest', guard('news.cluster.read'), run(async (req) => {
router.get('/ingest', guard('news.cluster.read'), run(async (req) => {
editorRead(req);
editorRead(req);
return {
return {
- cursor: await ctx.store.getState('sources_cursor', 0),
+ cursor: await ingest.cursor(),
runs: (await ingest.runs({ limit: 50 })).map((r) => ({ ...r, counts: parse(r.counts, null), at: iso(r.at) })),
runs: (await ingest.runs({ limit: 50 })).map((r) => ({ ...r, counts: parse(r.counts, null), at: iso(r.at) })),
sources: (await ingest.sourceStatus()).map((s) => ({ ...s, stale: s.stale == null ? null : Boolean(s.stale), refreshed_at: iso(s.refreshed_at) })),
sources: (await ingest.sourceStatus()).map((s) => ({ ...s, stale: s.stale == null ? null : Boolean(s.stale), refreshed_at: iso(s.refreshed_at) })),
};
};
@@ -82,7 +82,7 @@ function createEditorRoutes(ctx) {
@@ -82,7 +82,7 @@ function createEditorRoutes(ctx) {
page(req, res, 'Editor desk', editor.dashboard({
page(req, res, 'Editor desk', editor.dashboard({
clusters: await Promise.all((await clusters.recent({ limit: 40 })).map(async (c) => ({ ...c, count: (await clusters.members(c.id)).length }))),
clusters: await Promise.all((await clusters.recent({ limit: 40 })).map(async (c) => ({ ...c, count: (await clusters.members(c.id)).length }))),
stories: await stories.listAll({ limit: 100 }), flags, runs: await ingest.runs({ limit: 20 }), sources: await ingest.sourceStatus(),
stories: await stories.listAll({ limit: 100 }), flags, runs: await ingest.runs({ limit: 20 }), sources: await ingest.sourceStatus(),
- cursor: await store.getState('sources_cursor', 0), csrf: csrf(req), message: flashOf(req), aiEnabled: Boolean(ai && ai.enabled),
+ cursor: await ingest.cursor(), csrf: csrf(req), message: flashOf(req), aiEnabled: Boolean(ai && ai.enabled),
pullOn: Boolean(config.worker.enabled && config.sources.pullIntervalMs),
pullOn: Boolean(config.worker.enabled && config.sources.pullIntervalMs),
}));
}));
}));
}));
new file mode 100644
new file mode 100644
@@ -0,0 +1,100 @@
@@ -0,0 +1,100 @@
+'use strict';
+
+/**
+ * Events consumer: POST /internal/events, the endpoint of News' OpenVibe.Events subscriptions
+ * (topic patterns `sources.item.*` and `sources.fetch.failed`, see scripts/subscribe.js).
+ *
+ * sources.item.created|updated the item is read from Sources (sources.item.read) and applied
+ * sources.item.removed applied from the payload (the removal is sticky)
+ * sources.fetch.failed recorded; relayed as news.source.failed; no text is made
+ *
+ * The signature (X-OpenVibe-Signature v2, ±300 s) and the exactly-once inbox come from the chassis
+ * (openvibe-publishing/ingest.createEventConsumer): the receipt (consumer, event_id) and the change
+ * commit in one PostgreSQL transaction, so a redelivery changes nothing. When Sources cannot be read,
+ * the delivery is answered 503 so Events retries it, and the failure is recorded (and relayed as
+ * news.source.failed on the first attempt). The cursor pull catches anything that never arrives.
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { createEventConsumer } = require('openvibe-publishing/ingest');
+
+const CONSUMER = 'news-sources';
+const EVT_RE = /^evt_[0-9A-HJKMNP-TV-Z]{26}$/;
+
+function createEvents({ config, store, ingest, sources, log = console }) {
+ const router = express.Router();
+ const consumer = createEventConsumer({ db: store.db, secrets: config.events.webhookSecrets, consumer: CONSUMER, now: store.now });
+ const { inbox, verify } = consumer;
+
+ router.post('/internal/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {
+ const ctx = req.ov;
+ res.set('Cache-Control', 'no-store');
+ if (!config.events.webhookSecrets.length) return http.sendProblem(res, 503, 'news.webhook_disabled', { detail: 'NEWS_EVENTS_SECRET is not set', ctx });
+ const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0);
+ // v2 only: signature over "<t>.<raw body>" and t within ±300 s (a replayed or v1-only delivery fails).
+ const delivery = verify(raw, req.headers);
+ if (!delivery) return http.sendProblem(res, 401, 'news.bad_signature', { detail: 'X-OpenVibe-Signature-V2 does not verify or is outside the replay window', ctx });
+ const event = delivery.event;
+ if (!event || typeof event.event_id !== 'string' || !EVT_RE.test(event.event_id)) {
+ return http.sendProblem(res, 400, 'news.bad_delivery', { detail: 'body must be { event: <envelope>, seq }', ctx });
+ }
+ const attempt = Number(req.get('x-openvibe-delivery-attempt')) || 1;
+ const type = String(event.event_type || '');
+ const p = event.payload && typeof event.payload === 'object' ? event.payload : {};
+ const fromSources = event.source === 'sources';
+
+ try {
+ if (!fromSources || !/^sources\.(item\.(created|updated|removed)|fetch\.failed)$/.test(type)) {
+ await inbox.once(CONSUMER, event.event_id, () => null);
+ return res.status(204).end();
+ }
+ if (type === 'sources.item.removed') {
+ await inbox.once(CONSUMER, event.event_id, async () => {
+ const r = await ingest.applyRemoval({ sourcesItemId: String(p.item_id || ''), reason: p.reason, revision: Number.isInteger(p.revision) ? p.revision : null, origin: 'webhook' });
+ await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: p.source_key || null, sources_item_id: p.item_id || null });
+ return r;
+ });
+ return res.status(204).end();
+ }
+ if (type === 'sources.fetch.failed') {
+ await inbox.once(CONSUMER, event.event_id, () => ingest.upstreamFailure(p));
+ return res.status(204).end();
+ }
+ // created | updated
+ if (p.category && p.category !== config.sources.category) {
+ await inbox.once(CONSUMER, event.event_id, () => null);
+ return res.status(204).end();
+ }
+ let fetched;
+ try {
+ if (p.source_key) await ingest.learnOutlets([String(p.source_key)]);
+ fetched = await sources.getItem(String(p.item_id || ''));
+ } catch (err) {
+ await store.tx(async () => {
+ await ingest.recordRun({ origin: 'webhook', state: 'failed', source_key: p.source_key || null, sources_item_id: p.item_id || null, error_code: err.code || 'sources.error', detail: `${type} ${event.event_id} attempt ${attempt}: ${err.message}` });
+ if (attempt === 1) await ingest.failedEvent({ origin: 'webhook', sourceKey: p.source_key || null, sourcesItemId: p.item_id || null, state: 'failed', errorCode: err.code || 'sources.error', detail: err.message, httpStatus: err.status || null });
+ });
+ if (err.status === 404) {
+ // Sources no longer has it: nothing to apply, and retrying cannot help.
+ await inbox.once(CONSUMER, event.event_id, () => null);
+ return res.status(204).end();
+ }
+ return http.sendProblem(res, 503, 'news.sources_unavailable', { detail: 'OpenVibe.Sources could not be read; retry later', ctx });
+ }
+ if (fetched.source) await ingest.rememberSources({ [fetched.item.source_key]: fetched.source });
+ await inbox.once(CONSUMER, event.event_id, async () => {
+ const r = await ingest.apply(fetched.item, { origin: 'webhook' });
+ await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: fetched.item.source_key || null, sources_item_id: fetched.item.id, detail: r.reason || null });
+ return r;
+ });
+ return res.status(204).end();
+ } catch (err) {
+ log.error('[News] webhook failed:', err && err.stack ? err.stack : err);
+ return http.sendProblem(res, 500, 'internal.error', { detail: 'Internal error', ctx });
+ }
+ });
+
+ return router;
+}
+
+module.exports = { createEvents, CONSUMER };
deleted file mode 100644
deleted file mode 100644
@@ -1,101 +0,0 @@
@@ -1,101 +0,0 @@
-'use strict';
-
-/**
- * Events consumer: POST /internal/events, the endpoint of News' OpenVibe.Events subscriptions
- * (topic patterns `sources.item.*` and `sources.fetch.failed`, see scripts/subscribe.js).
- *
- * sources.item.created|updated the item is read from Sources (sources.item.read) and applied
- * sources.item.removed applied from the payload (the removal is sticky)
- * sources.fetch.failed recorded; relayed as news.source.failed; no text is made
- *
- * Deliveries are verified with X-OpenVibe-Signature against NEWS_EVENTS_SECRET (comma-separated
- * during rotation). Exactly once: the inbox receipt (consumer, event_id) and the change commit in
- * one PostgreSQL transaction; a redelivery is answered 204 and changes nothing. When Sources cannot be
- * read, the delivery is answered 503 so Events retries it, and the failure is recorded (and relayed
- * as news.source.failed on the first attempt). The cursor pull catches anything that never arrives.
- */
-const express = require('express');
-const { http } = require('openvibe-contracts');
-const { verifyDeliveryV2, createPgInbox } = require('openvibe-sdk/events');
-
-const CONSUMER = 'news-sources';
-const EVT_RE = /^evt_[0-9A-HJKMNP-TV-Z]{26}$/;
-
-function createWebhook({ config, store, ingest, sources, log = console }) {
- const router = express.Router();
- const inbox = createPgInbox(store.db, { now: store.now }); // idempotency_receipts: migrations/0001_initial.sql
-
- router.post('/internal/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {
- const ctx = req.ov;
- res.set('Cache-Control', 'no-store');
- const secrets = config.events.webhookSecrets;
- if (!secrets.length) return http.sendProblem(res, 503, 'news.webhook_disabled', { detail: 'NEWS_EVENTS_SECRET is not set', ctx });
- const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0);
- // v2 only: signature over "<t>.<raw body>" and t within ±300 s (a replayed or v1-only delivery fails).
- if (!secrets.some((s) => verifyDeliveryV2(raw, req.headers, s))) return http.sendProblem(res, 401, 'news.bad_signature', { detail: 'X-OpenVibe-Signature-V2 does not verify or is outside the replay window', ctx });
- let body;
- try { body = JSON.parse(raw.toString('utf8')); } catch { body = null; }
- const event = body && body.event;
- if (!event || typeof event.event_id !== 'string' || !EVT_RE.test(event.event_id)) {
- return http.sendProblem(res, 400, 'news.bad_delivery', { detail: 'body must be { event: <envelope>, seq }', ctx });
- }
- const attempt = Number(req.get('x-openvibe-delivery-attempt')) || 1;
- const type = String(event.event_type || '');
- const p = event.payload && typeof event.payload === 'object' ? event.payload : {};
- const fromSources = event.source === 'sources';
-
- try {
- if (!fromSources || !/^sources\.(item\.(created|updated|removed)|fetch\.failed)$/.test(type)) {
- await inbox.once(CONSUMER, event.event_id, () => null);
- return res.status(204).end();
- }
- if (type === 'sources.item.removed') {
- await inbox.once(CONSUMER, event.event_id, async () => {
- const r = await ingest.applyRemoval({ sourcesItemId: String(p.item_id || ''), reason: p.reason, revision: Number.isInteger(p.revision) ? p.revision : null, origin: 'webhook' });
- await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: p.source_key || null, sources_item_id: p.item_id || null });
- return r;
- });
- return res.status(204).end();
- }
- if (type === 'sources.fetch.failed') {
- await inbox.once(CONSUMER, event.event_id, () => ingest.upstreamFailure(p));
- return res.status(204).end();
- }
- // created | updated
- if (p.category && p.category !== config.sources.category) {
- await inbox.once(CONSUMER, event.event_id, () => null);
- return res.status(204).end();
- }
- let fetched;
- try {
- if (p.source_key) await ingest.learnOutlets([String(p.source_key)]);
- fetched = await sources.getItem(String(p.item_id || ''));
- } catch (err) {
- await store.tx(async () => {
- await ingest.recordRun({ origin: 'webhook', state: 'failed', source_key: p.source_key || null, sources_item_id: p.item_id || null, error_code: err.code || 'sources.error', detail: `${type} ${event.event_id} attempt ${attempt}: ${err.message}` });
- if (attempt === 1) await ingest.failedEvent({ origin: 'webhook', sourceKey: p.source_key || null, sourcesItemId: p.item_id || null, state: 'failed', errorCode: err.code || 'sources.error', detail: err.message, httpStatus: err.status || null });
- });
- if (err.status === 404) {
- // Sources no longer has it: nothing to apply, and retrying cannot help.
- await inbox.once(CONSUMER, event.event_id, () => null);
- return res.status(204).end();
- }
- return http.sendProblem(res, 503, 'news.sources_unavailable', { detail: 'OpenVibe.Sources could not be read; retry later', ctx });
- }
- if (fetched.source) await ingest.rememberSources({ [fetched.item.source_key]: fetched.source });
- await inbox.once(CONSUMER, event.event_id, async () => {
- const r = await ingest.apply(fetched.item, { origin: 'webhook' });
- await ingest.recordRun({ origin: 'webhook', state: r.outcome, source_key: fetched.item.source_key || null, sources_item_id: fetched.item.id, detail: r.reason || null });
- return r;
- });
- return res.status(204).end();
- } catch (err) {
- log.error('[News] webhook failed:', err && err.stack ? err.stack : err);
- return http.sendProblem(res, 500, 'internal.error', { detail: 'Internal error', ctx });
- }
- });
-
- return router;
-}
-
-module.exports = { createWebhook, CONSUMER };
new file mode 100644
new file mode 100644
@@ -0,0 +1,111 @@
@@ -0,0 +1,111 @@
+'use strict';
+/**
+ * T9 J2: News runs on the openvibe-publishing 1.1.0 chassis. This proves the conversion:
+ * - the local copies the brief's §7 lists are gone (nothing under server/ requires them);
+ * - a dry-run ingest from the captured Sources fixture produces the same domain rows as the
+ * pre-conversion code (test/fixtures/ingest-golden.json, captured before the change);
+ * - the Search document validates against search.index-document@1, with its provenance and
+ * visibility;
+ * - the outbox row and the state change share the caller's transaction (a rollback drops both).
+ */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const contracts = require('openvibe-contracts');
+const { boot, check, done } = require('./helpers/boot');
+const { feedFixture, reviseFixture, project } = require('./helpers/golden');
+
+const ROOT = path.join(__dirname, '..');
+const GOLDEN = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'ingest-golden.json'), 'utf8'));
+
+// The brief's §7 deletion list for News: the chassis (openvibe-publishing/ingest and /publication)
+// owns their code now. The harness denies rm, so the files may remain on disk; they must not be
+// required by any live server file.
+const DELETED = ['server/domain/ingest.js', 'server/domain/text.js', 'server/http/webhook.js', 'server/clients/sources.js']
+ .map((f) => path.join(ROOT, f));
+
+const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';
+
+function jsFiles(dir, out = []) {
+ for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
+ const p = path.join(dir, e.name);
+ if (e.isDirectory()) jsFiles(p, out);
+ else if (e.name.endsWith('.js')) out.push(p);
+ }
+ return out;
+}
+function relativeRequires(file) {
+ const src = fs.readFileSync(file, 'utf8');
+ return [...src.matchAll(/require\(\s*['"]([^'"]+)['"]\s*\)/g)].map((m) => m[1]).filter((s) => s.startsWith('.'));
+}
+
+/** An editor's story resting on exactly these Sources items (cited in the one paragraph), published. */
+async function storyOn(t, headline, items) {
+ let r = await t.api('/stories', { json: { headline } });
+ assert.strictEqual(r.status, 201, r.text);
+ const story = r.json().story;
+ for (const it of items) {
+ r = await t.api(`/stories/${story.id}/sources`, { json: { item: it.id } });
+ assert.ok(r.status === 200 || r.status === 201, r.text);
+ }
+ r = await t.api(`/stories/${story.id}/revisions`, { json: { body: `${PARA} [${items.map((_, i) => i + 1).join(', ')}]`, expected_revision: 0 } });
+ assert.strictEqual(r.status, 201, r.text);
+ r = await t.api(`/stories/${story.id}/publish`, { json: {} });
+ assert.strictEqual(r.status, 200, r.text);
+ return story;
+}
+
+(async () => {
+ await check('the local chassis copies are gone: no live server file requires them', () => {
+ const gone = new Set(DELETED);
+ for (const file of jsFiles(path.join(ROOT, 'server'))) {
+ if (gone.has(file)) continue; // the (undeletable) files themselves
+ for (const spec of relativeRequires(file)) {
+ let resolved;
+ try { resolved = require.resolve(path.resolve(path.dirname(file), spec)); } catch { continue; }
+ assert.ok(!gone.has(resolved), `${path.relative(ROOT, file)} still requires ${spec}`);
+ }
+ }
+ // ...and the chassis entry points are the ones the app uses.
+ assert.match(fs.readFileSync(path.join(ROOT, 'server', 'app.js'), 'utf8'), /openvibe-publishing\/ingest/);
+ assert.match(fs.readFileSync(path.join(ROOT, 'server', 'app.js'), 'utf8'), /\.\/domain\/source-items/);
+ assert.match(fs.readFileSync(path.join(ROOT, 'server', 'domain', 'publication.js'), 'utf8'), /openvibe-publishing\/publication/);
+ });
+
+ const t = await boot();
+ const reports = feedFixture(t);
+ await t.pull();
+ reviseFixture(t, reports);
+ await t.pull();
+
+ await check('a dry-run ingest from the captured Sources fixture produces the same domain rows as before', async () => {
+ assert.deepStrictEqual(await project(t), GOLDEN);
+ });
+
+ await check('the Search document validates against search.index-document@1, with its provenance and visibility', async () => {
+ const story = await storyOn(t, 'Two new water treatment plants confirmed', [reports.a, reports.c]);
+ const doc = (await t.events('news.index_document.upserted')).filter((e) => e.payload.id === story.id).pop().payload;
+ const v = contracts.validate('search.index-document@1', doc);
+ assert.ok(v.valid, JSON.stringify(v.errors));
+ assert.strictEqual(doc.visibility, 'public');
+ assert.strictEqual(doc.indexability.decision, 'index');
+ assert.deepStrictEqual(doc.provenance.filter((p) => p.service === 'sources').map((p) => p.id).sort(), [reports.a.id, reports.c.id].sort());
+ });
+
+ await check('the outbox row and the state change share the caller\'s transaction (a rollback drops both)', async () => {
+ const story = (await t.db().prepare('SELECT * FROM news_stories WHERE state = \'published\' LIMIT 1').get());
+ const row = async () => await t.db().prepare('SELECT state, noindex FROM news_stories WHERE id = ?').get(story.id);
+ const outboxCount = async () => (await t.db().prepare('SELECT COUNT(*) AS n FROM event_outbox').get()).n;
+ const before = await outboxCount();
+ await assert.rejects(t.ctx.store.db.tx(async () => {
+ await t.ctx.store.db.prepare('UPDATE news_stories SET noindex = 1 WHERE id = ?').run(story.id);
+ await t.ctx.publication.syncIndex(await t.db().prepare('SELECT * FROM news_stories WHERE id = ?').get(story.id));
+ throw new Error('rollback: the outbox row and the state change must not survive together');
+ }));
+ assert.strictEqual(await outboxCount(), before, 'no outbox row survives the rollback');
+ assert.strictEqual((await row()).noindex, 0, 'the state change rolled back with it');
+ });
+
+ await t.close();
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -53,7 +53,7 @@ const DIR = path.join(ROOT, 'docs', 'capabilities-proposal');
@@ -53,7 +53,7 @@ const DIR = path.join(ROOT, 'docs', 'capabilities-proposal');
});
});
await check('every event type the code emits is declared', async () => {
await check('every event type the code emits is declared', async () => {
- const src = ['server/domain/stories.js', 'server/domain/publication.js', 'server/domain/ingest.js', 'server/domain/clusters.js']
+ const src = ['server/domain/stories.js', 'server/domain/publication.js', 'server/domain/source-items.js', 'server/domain/clusters.js']
.map((f) => fs.readFileSync(path.join(ROOT, f), 'utf8')).join('\n');
.map((f) => fs.readFileSync(path.join(ROOT, f), 'utf8')).join('\n');
const found = [...src.matchAll(/event_type: '([a-z_.]+)'/g)].map((m) => m[1]);
const found = [...src.matchAll(/event_type: '([a-z_.]+)'/g)].map((m) => m[1]);
assert.ok(found.length >= 6);
assert.ok(found.length >= 6);
new file mode 100644
new file mode 100644
@@ -0,0 +1,224 @@
@@ -0,0 +1,224 @@
+{
+ "items": [
+ {
+ "sources_item_id": "itm_00000000000000000000000001",
+ "sources_revision": 1,
+ "source_key": "wire-a",
+ "canonical_url": "https://wire-a.example/space/europa-clipper-launch",
+ "url_key": "https://wire-a.example/space/europa-clipper-launch",
+ "headline": "NASA launches Europa Clipper probe to Jupiter moon Europa",
+ "outlet": "Wire A",
+ "authors": "[\"Ana Reporter\"]",
+ "published_at": "2026-09-22T09:00:00.000Z",
+ "summary": "The Europa Clipper spacecraft lifted off on a Falcon Heavy rocket from Kennedy Space Center.",
+ "summary_basis": "terms_allow_short_summaries",
+ "license_note": null,
+ "terms_note": "Titles, links and short summaries only; link back.",
+ "content_hash": "962be516c6a75b667766bd8a9ae13d19c3e7c29db10521f0a934bff6e33b1fae",
+ "title_key": "nasa launches europa clipper probe to jupiter moon europa",
+ "status": "active",
+ "dedupe_rule": null,
+ "duplicate_of": null,
+ "cluster": 1,
+ "cluster_reason_rule": "new_cluster",
+ "retrieved_at": "2026-09-22T11:00:00.000Z",
+ "first_seen_at": 1790078400000,
+ "updated_at": 1790078400000,
+ "removed_at": null,
+ "removed_reason": null
+ },
+ {
+ "sources_item_id": "itm_00000000000000000000000002",
+ "sources_revision": 2,
+ "source_key": "paper-b",
+ "canonical_url": "https://paper-b.example/2026/09/22/europa-clipper",
+ "url_key": "https://paper-b.example/2026/09/22/europa-clipper",
+ "headline": "Europa Clipper lifts off as NASA begins long trip to Jupiter",
+ "outlet": "Paper B",
+ "authors": "[\"Ben Writer\"]",
+ "published_at": "2026-09-22T09:30:00.000Z",
+ "summary": null,
+ "summary_basis": "removed_upstream",
+ "license_note": null,
+ "terms_note": "Titles, links and short summaries only; link back.",
+ "content_hash": "29305d93c699b16209ecd8adce4cba785c905abbadc37d58898209626e384ccc",
+ "title_key": "europa clipper lifts off as nasa begins long trip to jupiter",
+ "status": "removed",
+ "dedupe_rule": null,
+ "duplicate_of": null,
+ "cluster": 1,
+ "cluster_reason_rule": "shared_terms",
+ "retrieved_at": "2026-09-22T11:00:00.000Z",
+ "first_seen_at": 1790078400000,
+ "updated_at": 1790078400000,
+ "removed_at": 1790078400000,
+ "removed_reason": "licence review"
+ },
+ {
+ "sources_item_id": "itm_00000000000000000000000003",
+ "sources_revision": 2,
+ "source_key": "site-c",
+ "canonical_url": "https://site-c.example/europa-clipper-why",
+ "url_key": "https://site-c.example/europa-clipper-why",
+ "headline": "Why NASA is sending Europa Clipper to Jupiter, explained",
+ "outlet": "Site C",
+ "authors": "[]",
+ "published_at": "2026-09-22T10:00:00.000Z",
+ "summary": null,
+ "summary_basis": "none_provided",
+ "license_note": null,
+ "terms_note": "Titles, links and short summaries only; link back.",
+ "content_hash": "2fc0ac9c2865a02385d2e2157455086c28bfecbb6903e5caaffb595cca48fda5",
+ "title_key": "why nasa is sending europa clipper to jupiter explained",
+ "status": "active",
+ "dedupe_rule": null,
+ "duplicate_of": null,
+ "cluster": 1,
+ "cluster_reason_rule": "shared_terms",
+ "retrieved_at": "2026-09-22T11:00:00.000Z",
+ "first_seen_at": 1790078400000,
+ "updated_at": 1790078400000,
+ "removed_at": null,
+ "removed_reason": null
+ },
+ {
+ "sources_item_id": "itm_00000000000000000000000005",
+ "sources_revision": 1,
+ "source_key": "paper-b",
+ "canonical_url": "https://www.wire-a.example/space/europa-clipper-launch/?utm_source=x",
+ "url_key": "https://wire-a.example/space/europa-clipper-launch",
+ "headline": "Europa Clipper launch: our live coverage",
+ "outlet": "Paper B",
+ "authors": "[]",
+ "published_at": "2026-09-22T11:00:00.000Z",
+ "summary": null,
+ "summary_basis": "none_provided",
+ "license_note": null,
+ "terms_note": "Titles, links and short summaries only; link back.",
+ "content_hash": "13d92fe5d237158a8c88a3f4168b9867d8066b57480b95a1f1932c8346082c49",
+ "title_key": "europa clipper launch our live coverage",
+ "status": "duplicate",
+ "dedupe_rule": "canonical_url",
+ "duplicate_of": "itm_00000000000000000000000001",
+ "cluster": 1,
+ "cluster_reason_rule": "duplicate_of",
+ "retrieved_at": "2026-09-22T11:00:00.000Z",
+ "first_seen_at": 1790078400000,
+ "updated_at": 1790078400000,
+ "removed_at": null,
+ "removed_reason": null
+ },
+ {
+ "sources_item_id": "itm_00000000000000000000000006",
+ "sources_revision": 1,
+ "source_key": "site-c",
+ "canonical_url": "https://mirror.example/copy",
+ "url_key": "https://mirror.example/copy",
+ "headline": "Syndicated copy",
+ "outlet": "Site C",
+ "authors": "[]",
+ "published_at": null,
+ "summary": null,
+ "summary_basis": "none_provided",
+ "license_note": null,
+ "terms_note": "Titles, links and short summaries only; link back.",
+ "content_hash": "29305d93c699b16209ecd8adce4cba785c905abbadc37d58898209626e384ccc",
+ "title_key": "syndicated copy",
+ "status": "duplicate",
+ "dedupe_rule": "content_hash",
+ "duplicate_of": "itm_00000000000000000000000002",
+ "cluster": 1,
+ "cluster_reason_rule": "duplicate_of",
+ "retrieved_at": "2026-09-22T11:00:00.000Z",
+ "first_seen_at": 1790078400000,
+ "updated_at": 1790078400000,
+ "removed_at": null,
+ "removed_reason": null
+ },
+ {
+ "sources_item_id": "itm_00000000000000000000000007",
+ "sources_revision": 1,
+ "source_key": "wire-a",
+ "canonical_url": "https://wire-a.example/space/europa-clipper-launch-updated",
+ "url_key": "https://wire-a.example/space/europa-clipper-launch-updated",
+ "headline": "NASA launches Europa Clipper probe to Jupiter moon Europa, officials say",
+ "outlet": "Wire A",
+ "authors": "[]",
+ "published_at": "2026-09-22T12:00:00.000Z",
+ "summary": null,
+ "summary_basis": "none_provided",
+ "license_note": null,
+ "terms_note": "Titles, links and short summaries only; link back.",
+ "content_hash": "d477db3534724f4a397e7caafa010d6da238508f1152fae218cda385336ecd70",
+ "title_key": "nasa launches europa clipper probe to jupiter moon europa officials say",
+ "status": "duplicate",
+ "dedupe_rule": "near_title",
+ "duplicate_of": "itm_00000000000000000000000001",
+ "cluster": 1,
+ "cluster_reason_rule": "duplicate_of",
+ "retrieved_at": "2026-09-22T11:00:00.000Z",
+ "first_seen_at": 1790078400000,
+ "updated_at": 1790078400000,
+ "removed_at": null,
+ "removed_reason": null
+ },
+ {
+ "sources_item_id": "itm_00000000000000000000000008",
+ "sources_revision": 1,
+ "source_key": "paper-b",
+ "canonical_url": "https://paper-b.example/bike-lanes",
+ "url_key": "https://paper-b.example/bike-lanes",
+ "headline": "City council approves new bike lanes downtown",
+ "outlet": "Paper B",
+ "authors": "[]",
+ "published_at": "2026-09-22T12:00:00.000Z",
+ "summary": null,
+ "summary_basis": "none_provided",
+ "license_note": null,
+ "terms_note": "Titles, links and short summaries only; link back.",
+ "content_hash": "176afd88e4219b4e923b30298ab1c82bbc08bf49375d138188ad7960a1906fb7",
+ "title_key": "city council approves new bike lanes downtown",
+ "status": "active",
+ "dedupe_rule": null,
+ "duplicate_of": null,
+ "cluster": 2,
+ "cluster_reason_rule": "new_cluster",
+ "retrieved_at": "2026-09-22T11:00:00.000Z",
+ "first_seen_at": 1790078400000,
+ "updated_at": 1790078400000,
+ "removed_at": null,
+ "removed_reason": null
+ }
+ ],
+ "clusters": [
+ {
+ "index": 1,
+ "label": "europa clipper · jupiter · nasa",
+ "status": "open",
+ "terms": "{\"center\":1,\"clipper\":4,\"europa\":4,\"falcon\":1,\"heavy\":1,\"jupiter\":3,\"kennedy\":1,\"launche\":2,\"lifted\":1,\"moon\":2,\"nasa\":3,\"probe\":2,\"rocket\":1,\"space\":1,\"spacecraft\":1,\"explained\":1,\"sending\":1,\"coverage\":1,\"launch\":1,\"copy\":1,\"syndicated\":1,\"official\":1}",
+ "entities": "{\"europa\":2,\"europa clipper\":4,\"falcon heavy\":1,\"jupiter\":3,\"kennedy space center\":1,\"nasa\":3}",
+ "window_start": 1790067600000,
+ "window_end": 1790078400000,
+ "members": [
+ "itm_00000000000000000000000001",
+ "itm_00000000000000000000000002",
+ "itm_00000000000000000000000003",
+ "itm_00000000000000000000000005",
+ "itm_00000000000000000000000006",
+ "itm_00000000000000000000000007"
+ ]
+ },
+ {
+ "index": 2,
+ "label": "approve · bike · city · council",
+ "status": "open",
+ "terms": "{\"approve\":1,\"bike\":1,\"city\":1,\"council\":1,\"downtown\":1,\"lane\":1}",
+ "entities": "{}",
+ "window_start": 1790078400000,
+ "window_end": 1790078400000,
+ "members": [
+ "itm_00000000000000000000000008"
+ ]
+ }
+ ]
+}
@@ -56,7 +56,7 @@ async function boot(opts = {}) {
@@ -56,7 +56,7 @@ async function boot(opts = {}) {
let built = null;
let built = null;
async function start() {
async function start() {
const config = configLib.load(env);
const config = configLib.load(env);
- built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow });
+ built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow, indexnow: opts.indexnow });
await require('openvibe-sdk/auth').jwksClient(`${config.networkInternalUrl}/api/.well-known/jwks`).keys();
await require('openvibe-sdk/auth').jwksClient(`${config.networkInternalUrl}/api/.well-known/jwks`).keys();
server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
t.base = `http://127.0.0.1:${server.address().port}`;
t.base = `http://127.0.0.1:${server.address().port}`;
new file mode 100644
new file mode 100644
@@ -0,0 +1,70 @@
@@ -0,0 +1,70 @@
+'use strict';
+/**
+ * The ingest golden fixture: a deterministic Sources feed and the canonical projection of the
+ * domain rows it produces (news_source_items + news_story_clusters + a removed item's live state).
+ * The same fixture and the same projection run before and after the chassis conversion, so
+ * test/chassis.test.js can compare the projection to test/fixtures/ingest-golden.json (captured
+ * from the pre-conversion code). Volatile values (ULID ids, cluster ids) are mapped to their stable
+ * Sources item ids, so only domain meaning is compared.
+ */
+const { launchReports } = require('./boot');
+
+const parse = (s, d) => { try { return s ? JSON.parse(s) : d; } catch { return d; } };
+
+/** Add the fixture's items to the Sources mock (same order and fields as the capture run). → reports */
+function feedFixture(t) {
+ const reports = launchReports(t.sources);
+ t.sources.addItem({ source_key: 'blog-x', category: 'blog', title: 'A blog item that News must ignore' });
+ t.sources.addItem({ source_key: 'paper-b', title: 'Europa Clipper launch: our live coverage', url: 'https://www.wire-a.example/space/europa-clipper-launch/?utm_source=x', published_at: '2026-09-22T11:00:00Z' });
+ t.sources.addItem({ source_key: 'site-c', title: 'Syndicated copy', url: 'https://mirror.example/copy', content_hash: reports.b.provenance.content_hash });
+ t.sources.addItem({ source_key: 'wire-a', title: 'NASA launches Europa Clipper probe to Jupiter moon Europa, officials say', url: 'https://wire-a.example/space/europa-clipper-launch-updated', published_at: '2026-09-22T12:00:00Z' });
+ t.sources.addItem({ source_key: 'paper-b', title: 'City council approves new bike lanes downtown', url: 'https://paper-b.example/bike-lanes', published_at: '2026-09-22T12:00:00Z' });
+ t.sources.addItem({ source_key: 'wire-a', title: '' });
+ return reports;
+}
+
+/** A revision upstream (title change) and a removal, applied by a second pull. */
+function reviseFixture(t, reports) {
+ t.sources.updateItem(reports.c.id, { title: 'Why NASA is sending Europa Clipper to Jupiter, explained' });
+ t.sources.removeItem(reports.b.id, 'licence review');
+}
+
+/** The canonical projection of the domain rows. Deterministic in everything but ids. → { items, clusters } */
+async function project(t) {
+ const db = t.db();
+ const items = await db.prepare('SELECT * FROM news_source_items ORDER BY sources_item_id').all();
+ const srcOf = new Map(items.map((r) => [r.id, r.sources_item_id]));
+ const membersByCluster = new Map();
+ for (const r of items) {
+ if (!r.cluster_id) continue;
+ if (!membersByCluster.has(r.cluster_id)) membersByCluster.set(r.cluster_id, []);
+ membersByCluster.get(r.cluster_id).push(r.sources_item_id);
+ }
+ const clusterIndex = new Map([...membersByCluster.entries()]
+ .map(([id, ms]) => [id, ms.slice().sort()[0]])
+ .sort((a, b) => (a[1] < b[1] ? -1 : a[1] > b[1] ? 1 : 0))
+ .map(([id], i) => [id, i + 1]));
+ const clusters = await db.prepare('SELECT * FROM news_story_clusters ORDER BY id').all();
+ return {
+ items: items.map((r) => ({
+ sources_item_id: r.sources_item_id, sources_revision: r.sources_revision, source_key: r.source_key,
+ canonical_url: r.canonical_url, url_key: r.url_key, headline: r.headline, outlet: r.outlet,
+ authors: r.authors, published_at: r.published_at, summary: r.summary, summary_basis: r.summary_basis,
+ license_note: r.license_note, terms_note: r.terms_note, content_hash: r.content_hash, title_key: r.title_key,
+ status: r.status, dedupe_rule: r.dedupe ? parse(r.dedupe, {}).rule : null,
+ duplicate_of: r.duplicate_of ? (srcOf.get(r.duplicate_of) || null) : null,
+ cluster: r.cluster_id ? clusterIndex.get(r.cluster_id) : null,
+ cluster_reason_rule: r.cluster_reason ? parse(r.cluster_reason, {}).rule : null,
+ retrieved_at: r.retrieved_at, first_seen_at: r.first_seen_at, updated_at: r.updated_at,
+ removed_at: r.removed_at || null, removed_reason: r.removed_reason || null,
+ })),
+ clusters: [...membersByCluster.entries()]
+ .sort((a, b) => clusterIndex.get(a[0]) - clusterIndex.get(b[0]))
+ .map(([id, ms]) => {
+ const c = clusters.find((x) => x.id === id) || {};
+ return { index: clusterIndex.get(id), label: c.label || null, status: c.status || null, terms: c.terms || null, entities: c.entities || null, window_start: c.window_start || null, window_end: c.window_end || null, members: ms.slice().sort() };
+ }),
+ };
+}
+
+module.exports = { feedFixture, reviseFixture, project };
@@ -8,7 +8,7 @@
@@ -8,7 +8,7 @@
*/
*/
const assert = require('assert');
const assert = require('assert');
const { boot, check, done } = require('./helpers/boot');
const { boot, check, done } = require('./helpers/boot');
-const text = require('../server/domain/text');
+const text = require('openvibe-publishing/ingest').normalize;
const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';
const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';
new file mode 100644
new file mode 100644
@@ -0,0 +1,84 @@
@@ -0,0 +1,84 @@
+'use strict';
+/**
+ * IndexNow (openvibe-shared/indexnow): INDEXNOW_KEY unset → the feature is off (no key route, nothing
+ * sent). With a key, the key file is served at /<key>.txt as text/plain and publishing an indexable
+ * story pings the engines with the story's path and the sitemap. Drafts and noindex pages never ping.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+
+const KEY = 'k'.repeat(32);
+const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';
+
+/** An editor's story resting on exactly these Sources items (cited), published. */
+async function storyOn(t, headline, items) {
+ let r = await t.api('/stories', { json: { headline } });
+ assert.strictEqual(r.status, 201, r.text);
+ const story = r.json().story;
+ for (const it of items) {
+ r = await t.api(`/stories/${story.id}/sources`, { json: { item: it.id } });
+ assert.ok(r.status === 200 || r.status === 201, r.text);
+ }
+ r = await t.api(`/stories/${story.id}/revisions`, { json: { body: `${PARA} [${items.map((_, i) => i + 1).join(', ')}]`, expected_revision: 0 } });
+ assert.strictEqual(r.status, 201, r.text);
+ r = await t.api(`/stories/${story.id}/publish`, { json: {} });
+ assert.strictEqual(r.status, 200, r.text);
+ return story;
+}
+
+(async () => {
+ const off = await boot();
+ await check('without a key IndexNow is off: no key route and nothing sent', async () => {
+ assert.strictEqual(off.ctx.indexnow.enabled, false);
+ const res = await off.get(`/${KEY}.txt`);
+ assert.strictEqual(res.status, 404, res.text);
+ });
+ await off.close();
+
+ const on = await boot({ env: { INDEXNOW_KEY: KEY } });
+ await check('with a key the key file answers text/plain with the key', async () => {
+ assert.strictEqual(on.ctx.indexnow.enabled, true);
+ const res = await on.get(`/${KEY}.txt`);
+ assert.strictEqual(res.status, 200, res.text);
+ assert.match(res.headers.get('content-type'), /text\/plain/);
+ assert.strictEqual(res.text, KEY);
+ });
+ await on.close();
+
+ // A spy in place of the module's HTTP send: records every pingSoon batch.
+ const pings = [];
+ const spy = {
+ enabled: true,
+ keyFile: (_req, _res, next) => next(),
+ pingSoon: (urls) => { const a = Array.isArray(urls) ? urls : [urls]; pings.push(...a); return a.length; },
+ ping: async () => ({ sent: 0, status: 0 }),
+ flush: async () => ({ sent: 0, status: 0 }),
+ };
+ const t = await boot({ indexnow: spy });
+ t.sources.addSource('alpha-news', { name: 'Alpha News' });
+ t.sources.addSource('solo-wire', { name: 'Solo Wire' });
+ const alpha = t.sources.addItem({ source_key: 'alpha-news', title: 'Water agency says two new treatment plants will open next spring', url: 'https://alpha.example/a/water-agency-plants' });
+ const solo = t.sources.addItem({ source_key: 'solo-wire', title: 'Water authority to open two treatment plants next spring', url: 'https://solo-wire.example/water/plants' });
+ await t.pull();
+
+ await check('a draft never pings', async () => {
+ const r = await t.api('/stories', { json: { headline: 'A draft story about water' } });
+ assert.strictEqual(r.status, 201, r.text);
+ assert.deepStrictEqual(pings, []);
+ });
+
+ await check('a publish pings the page path and the sitemap', async () => {
+ const story = await storyOn(t, 'Two new water treatment plants confirmed', [alpha, solo]);
+ assert.ok(pings.includes(`https://openvibe.news/stories/${story.slug}`), JSON.stringify(pings));
+ assert.ok(pings.includes('https://openvibe.news/sitemap.xml'), JSON.stringify(pings));
+ });
+
+ await check('a published but noindex story never pings', async () => {
+ pings.length = 0;
+ await storyOn(t, 'Two treatment plants to open next spring', [alpha]);
+ assert.deepStrictEqual(pings, [], 'a noindex page is never pinged');
+ });
+ await t.close();
+
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -37,7 +37,7 @@ const { boot, check, done, launchReports } = require('./helpers/boot');
@@ -37,7 +37,7 @@ const { boot, check, done, launchReports } = require('./helpers/boot');
assert.strictEqual(why.rule, 'shared_terms');
assert.strictEqual(why.rule, 'shared_terms');
assert.ok(why.shared_entities.includes('europa clipper'), JSON.stringify(why));
assert.ok(why.shared_entities.includes('europa clipper'), JSON.stringify(why));
assert.ok(why.score >= 3);
assert.ok(why.score >= 3);
- assert.strictEqual(await t.ctx.store.getState('sources_cursor'), reports.c.change_seq);
+ assert.strictEqual(await t.ctx.ingest.cursor(), reports.c.change_seq);
assert.strictEqual((await t.events('news.source.ingested')).length, 3);
assert.strictEqual((await t.events('news.source.ingested')).length, 3);
assert.ok((await t.events('news.cluster.updated')).length >= 3);
assert.ok((await t.events('news.cluster.updated')).length >= 3);
});
});
@@ -46,7 +46,7 @@ const { boot, check, done, launchReports } = require('./helpers/boot');
@@ -46,7 +46,7 @@ const { boot, check, done, launchReports } = require('./helpers/boot');
const before = (await t.events()).length;
const before = (await t.events()).length;
const r = await t.pull();
const r = await t.pull();
assert.strictEqual(r.counts.created || 0, 0);
assert.strictEqual(r.counts.created || 0, 0);
- await t.ctx.store.setState('sources_cursor', 0);
+ await t.db().prepare("UPDATE news_ingest_cursor SET cursor = 0 WHERE name = 'sources'").run();
const again = await t.pull();
const again = await t.pull();
assert.strictEqual(again.counts.unchanged, 3);
assert.strictEqual(again.counts.unchanged, 3);
assert.strictEqual((await t.events()).length, before);
assert.strictEqual((await t.events()).length, before);
@@ -8,7 +8,7 @@
@@ -8,7 +8,7 @@
*/
*/
const assert = require('assert');
const assert = require('assert');
const { boot, check, done } = require('./helpers/boot');
const { boot, check, done } = require('./helpers/boot');
-const { licensedSummary } = require('../server/domain/text');
+const { licensedSummary } = require('openvibe-publishing/ingest').normalize;
const BODY_SENTINEL = 'SECRET-FULL-ARTICLE-BODY-7f3a';
const BODY_SENTINEL = 'SECRET-FULL-ARTICLE-BODY-7f3a';
const TAIL_SENTINEL = 'SECRET-SUMMARY-TAIL-91c2';
const TAIL_SENTINEL = 'SECRET-SUMMARY-TAIL-91c2';
new file mode 100644
new file mode 100644
@@ -0,0 +1,63 @@
@@ -0,0 +1,63 @@
+'use strict';
+// Size budgets for openvibe.news's home page (roadmap WS-T task 1, openvibe-shared/perf-budget): the
+// server as it runs (a fresh database), measured without a browser. Budgets sit a little above the
+// measurement below; raising one is a decision to state in the commit.
+// node test/perf-budget.test.js
+const assert = require('assert');
+const fs = require('fs');
+const net = require('net');
+const os = require('os');
+const path = require('path');
+const { spawn } = require('child_process');
+const { measure, check, format } = require('openvibe-shared/perf-budget');
+
+const BUDGETS = {
+ htmlRawKB: 22, // measured 17.4 (fresh database)
+ htmlBrotliKB: 5.5, // 4.5
+ jsFiles: 5, // 4
+ jsRawKB: 240, // 212.2
+ jsBrotliKB: 55, // 49.9
+ cssFiles: 2, // 1
+ cssRawKB: 6.5, // 5.2
+ cssBrotliKB: 2, // 1.4
+ externalFiles: 1, // 0
+};
+
+const freePort = () => new Promise((resolve) => { const s = net.createServer(); s.listen(0, '127.0.0.1', () => { const { port } = s.address(); s.close(() => resolve(port)); }); });
+
+(async () => {
+ // A database of its own (not the shared dev PGlite in data/pglite, and not whatever the caller's
+ // DATABASE_URL names), so the measurement is the server on a fresh database and the run leaves nothing.
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-budget-'));
+ const pgliteDir = path.join(dir, 'pglite');
+ const port = await freePort();
+ const child = spawn(process.execPath, [path.join(__dirname, '..', 'server', 'index.js')], {
+ cwd: path.join(__dirname, '..'),
+ env: {
+ ...process.env, PORT: String(port), HOST: '127.0.0.1', NODE_ENV: 'test',
+ DATABASE_URL: '', DATABASE_DIRECT_URL: '', VALKEY_URL: '', NEWS_PGLITE_DIR: pgliteDir,
+ NEWS_WORKER: 'off', OV_OAUTH_CLIENT_SECRET: 'test-secret', NEWS_FORM_SECRET: 'test-form-secret',
+ },
+ stdio: ['ignore', 'ignore', 'pipe'],
+ });
+ let stderr = '';
+ child.stderr.on('data', (d) => { stderr = (stderr + d).slice(-2000); });
+ const base = `http://127.0.0.1:${port}`;
+ try {
+ let up = false;
+ for (let i = 0; i < 150 && !up; i++) {
+ up = await fetch(`${base}/api/health`).then((r) => r.ok).catch(() => false);
+ if (!up) await new Promise((r) => setTimeout(r, 100));
+ }
+ assert.ok(up, `the server did not start:\n${stderr}`);
+ assert.ok(fs.existsSync(pgliteDir), 'the server did not use the isolated database (NEWS_PGLITE_DIR)');
+ const m = await measure({ base });
+ const over = check(m, BUDGETS);
+ assert.deepStrictEqual(over, [], format(m, over));
+ console.log(format(m));
+ console.log('perf budget: all checks passed');
+ } finally {
+ child.kill();
+ fs.rmSync(dir, { recursive: true, force: true });
+ }
+})().catch((err) => { console.error(err); process.exitCode = 1; });
@@ -131,7 +131,7 @@ const BODY = [
@@ -131,7 +131,7 @@ const BODY = [
await check('an unchanged upstream refetch (same revision) flags nothing', async () => {
await check('an unchanged upstream refetch (same revision) flags nothing', async () => {
const before = (await t.db().prepare('SELECT COUNT(*) AS n FROM news_editorial_flags').get()).n;
const before = (await t.db().prepare('SELECT COUNT(*) AS n FROM news_editorial_flags').get()).n;
- await t.ctx.store.setState('sources_cursor', 0);
+ await t.db().prepare("UPDATE news_ingest_cursor SET cursor = 0 WHERE name = 'sources'").run();
await t.pull();
await t.pull();
assert.strictEqual((await t.db().prepare('SELECT COUNT(*) AS n FROM news_editorial_flags').get()).n, before);
assert.strictEqual((await t.db().prepare('SELECT COUNT(*) AS n FROM news_editorial_flags').get()).n, before);
});
});