openvibers/openvibe.news Commits
Index gate needs two independent sources (NEWS_MIN_INDEPENDENT_SOURCES, default 2)
Changes
@@@ -42,6 +42,10 @@ OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720# Licensing: the longest source summary kept (and only when the source's terms allow short summaries)#NEWS_SUMMARY_MAX_CHARS=280++# Indexing: independent sources a story must cite to be indexable (fewer: published but noindex,++# out of sitemaps). Items from one Sources source, one publisher's domain or one outlet, or copies++# of one report, count once. Minimum 1.++#NEWS_MIN_INDEPENDENT_SOURCES=2# Clustering window#NEWS_CLUSTER_WINDOW_HOURS=72@@@ -160,8 +160,19 @@ Other tables in the same database:### Discoverability (roadmap §32)- Robots meta, canonical and `X-Robots-Tag` come from the `openvibe-publishing/seo` gate with-- explicit reasons. News' policy: at least one live cited source, at least 40 words; unsupported-- claims, retraction, an editor's noindex and unreviewed AI output are never indexable.++ explicit reasons. News' policy: at least two independent live cited sources++ (`NEWS_MIN_INDEPENDENT_SOURCES`, default 2), at least 40 words; unsupported claims, retraction,++ an editor's noindex and unreviewed AI output are never indexable.++- **Independent sources** (`text.independentSources`): the live Sources items the published++ revision's paragraphs cite, where items count as one source when they share the OpenVibe.Sources++ source (`source_key`: the same feed or registry entry), the publisher's domain of their canonical++ URL (`news.example.com` and `www.example.com` are `example.com`; `news.bbc.co.uk` is++ `bbc.co.uk`), the outlet name, or the original report they duplicate (a syndicated or copied++ report dedupe marked as a duplicate). The relation is transitive and deliberately coarse: two++ sites on one shared host count once. A story on fewer independent sources is still published,++ listed and readable, but `noindex` with the reason `unsourced` (for example `1 of 2 sources`),++ so it is out of the sitemaps and its Search document says noindex. Setting the minimum to 1++ restores the one-source gate.- JSON-LD `NewsArticle` from real fields only: headline, dates, the editors named in therevision's authorship (only when the Network name is known), the section (topic), the citedsource URLs, and OpenVibe.News as publisher. Missing fields are omitted.@@@ -250,6 +261,7 @@ Each grant is `[client, capability, audience]`:| Licensed material never leaks: bodies and summaries beyond the allowance never reach the database, pages, JSON, feeds, sitemaps, the API, events or Search. | `test/licensing.test.js` || Dedupe by canonical URL, content hash and near-duplicate headline; idempotent replays. | `test/ingest.test.js` || Clusters are deterministic and explained; merges and splits are audited and reversible. | `test/clusters.test.js` |++| The index gate counts independent sources: a single-source story, two items of one Sources source, two sources of one publisher's domain and a report with its syndicated copy are published but noindex and out of the sitemap; two independent sources are indexable; `NEWS_MIN_INDEPENDENT_SOURCES=1` restores the old gate. | `test/indexability.test.js` || Retraction: visible at its URL with the notice, noindex with its reason, out of sitemaps and Search, event emitted, final. | `test/retraction.test.js` || Useful without JavaScript: the whole editorial journey with forms, and public pages complete in HTML. | `test/nojs.test.js` || AI output is a draft that needs a person's review (AI-assisted too); uncited claims dropped; failure makes no text. | `test/ai.test.js` |@@@ -66,6 +66,13 @@ function load(env = process.env) {summaryMaxChars: Math.min(Math.max(int(env.NEWS_SUMMARY_MAX_CHARS, 280), 0), 500),},++ // Indexing: how many independent sources (distinct Sources sources, publishers' domains and++ // outlets, not copies of one report) a story's paragraphs must cite before search engines,++ // sitemaps and Search may treat it as indexable. Fewer: published, but noindex.++ indexing: {++ minIndependentSources: Math.max(int(env.NEWS_MIN_INDEPENDENT_SOURCES, 2), 1),++ },++// Clustering: the time window a new item may join an existing cluster in.clustering: {windowMs: int(env.NEWS_CLUSTER_WINDOW_HOURS, 72) * 3600 * 1000,@@@ -7,10 +7,14 @@* - the Search document and the publication events (openvibe-publishing/index-hooks), enqueued* in the same transaction as the change through the SDK outbox*-- * News' gate policy: at least one source (citations are the Sources items the published revision-- * cites and that are still live upstream), at least 40 words; a paragraph whose every source was-- * removed upstream counts as an unsupported claim (noindex until an editor revises); a retraction-- * is noindex; AI-generated text is hidden until a person reviews it.++ * News' gate policy: at least NEWS_MIN_INDEPENDENT_SOURCES (2) independent sources, at least 40++ * words. The sources counted are the Sources items the published revision's paragraphs cite and++ * that are still live upstream, grouped by text.independentSources: items from the same Sources++ * source, the same publisher's domain, the same outlet, or copies of the same original report are++ * one source. A story on fewer is still published and readable, but noindex ('unsourced'), so it++ * stays out of sitemaps. A paragraph whose every source was removed upstream counts as an++ * unsupported claim (noindex until an editor revises); a retraction is noindex; AI-generated text++ * is hidden until a person reviews it.** Search receives ONLY published (not retracted), listable stories. Everything else is a tombstone,* so a retraction or an unpublish removes the old copy; a story that was never indexed gets no tombstone.@@@ -18,8 +22,10 @@const seo = require('openvibe-publishing/seo');const hooks = require('openvibe-publishing/index-hooks');const authorship = require('openvibe-publishing/authorship');++const { independentSources } = require('./text');--const POLICY = Object.freeze({ minWords: 40, requireSources: true, minSources: 1 });++// minSources counts independent sources; config.indexing.minIndependentSources overrides it.++const POLICY = Object.freeze({ minWords: 40, requireSources: true, minSources: 2 });const OWNER = 'news';/**@@@ -43,6 +49,7 @@ function disclosure(rec, review) {function createPublication({ store, config, outbox }) {const { db } = store;++ const policy = Object.freeze({ ...POLICY, minSources: (config.indexing && config.indexing.minIndependentSources) || POLICY.minSources });const storyById = db.prepare('SELECT * FROM news_stories WHERE id = ?');const itemById = db.prepare('SELECT * FROM news_source_items WHERE id = ?');const topicById = db.prepare('SELECT * FROM news_topics WHERE id = ?');@@@ -76,7 +83,8 @@ function createPublication({ store, config, outbox }) {for (const n of ns) if (alive(n)) cited.add(n);if (!ns.some(alive)) unsupported++;}-- return { citedLive: cited.size, unsupported, live };++ const groups = independentSources([...cited].map((n) => live.get(n)));++ return { citedLive: cited.size, independent: groups.length, groups, unsupported, live };}const plain = (rev) => (rev ? `${rev.fields.headline || ''}\n${paragraphsOf(rev).map((p) => p.text).join('\n')}` : '');@@@ -97,13 +105,13 @@ function createPublication({ store, config, outbox }) {visibility: 'public',canonicalUrl: storyUrl(story),text: plain(rev),-- citationCount: sup.citedLive,++ citationCount: sup.independent,unsupportedClaims: sup.unsupported,retracted: (state || story.state) === 'retracted',noindex: Boolean(story.noindex),};if (rec) Object.assign(facts, gateAuthorship(rec, reviewOf(story, rev)));-- return seo.evaluate(facts, { policy: POLICY, now: store.now() });++ return seo.evaluate(facts, { policy, now: store.now() });}function topicOf(story) { return story.topic_id ? topicById.get(story.topic_id) : null; }@@@ -193,7 +201,7 @@ function createPublication({ store, config, outbox }) {}return {-- POLICY, OWNER, abs, storyPath, storyUrl, topicPath, feedId,++ POLICY: policy, OWNER, abs, storyPath, storyUrl, topicPath, feedId,authorshipOf, reviewOf, paragraphsOf, snapshotOf, liveSources, support, decide, topicOf,documentFor, syncIndex, snapshot, afterChange, plain,};@@@ -2,8 +2,8 @@/*** Deterministic text helpers for ingestion: URL keys and title keys for dedupe, word shingles for-- * near-duplicate headlines, the terms and named entities clustering compares, and the licensing-- * rule for source summaries. Pure functions: the same input always gives the same output, so every++ * near-duplicate headlines, the terms and named entities clustering compares, the licensing rule++ * for source summaries, and which cited sources are independent (the index gate). Pure functions: the same input always gives the same output, so every* dedupe and cluster decision can be recomputed and explained.*/@@@ -51,6 +51,59 @@ function hostOf(url) {try { return new URL(String(url)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return null; }}++// Second-level labels under a two-letter country code that are not a publisher's own name++// (bbc.co.uk, abc.net.au): the publisher is the label before them.++const COUNTRY_SECOND_LEVEL = new Set(['co', 'com', 'org', 'net', 'gov', 'edu', 'ac', 'ne', 'or', 'go', 'gob', 'nic', 'mil']);++++/**++ * The publisher's domain of a URL: its host without subdomains (news.example.com and++ * www.example.com → example.com; news.bbc.co.uk → bbc.co.uk). Deliberately coarse: two sites on++ * one shared host (two blogs on one platform) count as one publisher, never the other way round.++ */++function publisherDomain(url) {++ const host = hostOf(url);++ if (!host) return null;++ if (/^[\d.]+$/.test(host) || host.startsWith('[')) return host;++ const labels = host.split('.').filter(Boolean);++ if (labels.length <= 2) return labels.join('.');++ const n = labels.length;++ const take = labels[n - 1].length === 2 && COUNTRY_SECOND_LEVEL.has(labels[n - 2]) ? 3 : 2;++ return labels.slice(-take).join('.');++}++++/**++ * Independent sources among source items (news_source_items rows): items are one source when they++ * share any of++ * - the OpenVibe.Sources source (source_key: the same feed or registry entry),++ * - the publisher's domain of their canonical URL (publisherDomain),++ * - the outlet name (folded),++ * - the original they duplicate (duplicate_of: the same report syndicated or copied),++ * and the relation is transitive. → groups of item ids, each group one independent source.++ */++function independentSources(items) {++ const parent = new Map();++ const find = (k) => { while (parent.get(k) !== k) { parent.set(k, parent.get(parent.get(k))); k = parent.get(k); } return k; };++ const union = (a, b) => { const x = find(a); const y = find(b); if (x !== y) parent.set(y, x); };++ for (const it of items) {++ const node = `item:${it.id}`;++ if (!parent.has(node)) parent.set(node, node);++ const domain = publisherDomain(it.canonical_url);++ const outlet = fold(it.outlet);++ const keys = [`source:${it.source_key}`, `original:${it.duplicate_of || it.id}`, domain ? `domain:${domain}` : null, outlet ? `outlet:${outlet}` : null].filter(Boolean);++ for (const k of keys) {++ if (!parent.has(k)) parent.set(k, k);++ union(node, k);++ }++ }++ const groups = new Map();++ for (const it of items) {++ const root = find(`item:${it.id}`);++ if (!groups.has(root)) groups.set(root, []);++ groups.get(root).push(it.id);++ }++ return [...groups.values()];++}++/** Very light stemming: plural and possessive endings, so "rockets" and "rocket" meet. */function stem(w) {if (w.length > 4 && w.endsWith('ies')) return `${w.slice(0, -3)}y`;@@@ -141,4 +194,4 @@ function licensedSummary(summary, { termsNote, licenseNote, maxChars = 280 } = {return { summary: `${(at > maxChars * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.-]+$/, '')}…`, basis: 'terms_allow_short_summaries' };}--module.exports = { fold, titleKey, urlKey, hostOf, terms, entities, shingles, jaccard, licensedSummary, stem, STOPWORDS };++module.exports = { fold, titleKey, urlKey, hostOf, publisherDomain, independentSources, terms, entities, shingles, jaccard, licensedSummary, stem, STOPWORDS };nnew file mode 100644@@@ -0,0 +1,126 @@++'use strict';++/**++ * The index gate needs independent sources: a story's paragraphs must cite at least++ * NEWS_MIN_INDEPENDENT_SOURCES (2) live sources that are not the same Sources source, the same++ * publisher's domain, the same outlet, or copies of one report. A story on fewer is published and++ * readable, but noindex ('unsourced'), out of the sitemaps, and Search is told it is noindex.++ * Setting the minimum to 1 restores the old behaviour.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const text = require('../server/domain/text');++++const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';++++/** An editor's story resting on exactly these Sources items (cited in the one paragraph). */++async function storyOn(t, headline, items) {++ let r = await t.api('/stories', { json: { headline } });++ assert.strictEqual(r.status, 201, r.text);++ const story = r.json().story;++ for (const it of items) {++ r = await t.api(`/stories/${story.id}/sources`, { json: { item: it.id } });++ assert.ok(r.status === 200 || r.status === 201, r.text);++ }++ r = await t.api(`/stories/${story.id}/revisions`, { json: { body: `${PARA} [${items.map((_, i) => i + 1).join(', ')}]`, expected_revision: 0 } });++ assert.strictEqual(r.status, 201, r.text);++ r = await t.api(`/stories/${story.id}/publish`, { json: {} });++ assert.strictEqual(r.status, 200, r.text);++ return story;++}++++(async () => {++ const t = await boot();++ const S = t.sources;++ S.addSource('solo-wire', { name: 'Solo Wire' });++ S.addSource('one-feed', { name: 'One Feed' });++ S.addSource('paper-world', { name: 'The Paper: World' });++ S.addSource('paper-tech', { name: 'The Paper: Tech' });++ S.addSource('alpha-news', { name: 'Alpha News' });++ S.addSource('beta-daily', { name: 'Beta Daily' });++ const solo = S.addItem({ source_key: 'solo-wire', title: 'Water authority to open two treatment plants next spring', url: 'https://solo-wire.example/water/plants' });++ const feed1 = S.addItem({ source_key: 'one-feed', title: 'Valley water restrictions to end as plants come online', url: 'https://one-feed.example/2026/valley-water' });++ const feed2 = S.addItem({ source_key: 'one-feed', title: 'Residents welcome news of added treatment capacity', url: 'https://one-feed.example/2026/residents-react' });++ const world = S.addItem({ source_key: 'paper-world', title: 'Regional authority approves treatment plant expansion', url: 'https://www.thepaper.example/world/plant-expansion' });++ const tech = S.addItem({ source_key: 'paper-tech', title: 'Inside the membrane technology behind the new plants', url: 'https://tech.thepaper.example/membranes' });++ const wire = S.addItem({ source_key: 'alpha-news', title: 'Water agency says two new treatment plants will open next spring', url: 'https://alpha.example/a/water-agency-plants' });++ const copy = S.addItem({ source_key: 'beta-daily', title: 'Water agency says two new treatment plants will open next spring', url: 'https://beta.example/b/syndicated-water' });++ const beta = S.addItem({ source_key: 'beta-daily', title: 'Budget vote clears way for valley water projects', url: 'https://beta.example/b/budget-vote' });++ await t.pull();++ const row = (it) => t.db().prepare('SELECT * FROM news_source_items WHERE sources_item_id = ?').get(it.id);++ const sitemap = async () => (await t.get('/sitemaps/stories.xml')).text;++ const indexEvent = (story) => t.events('news.index_document.upserted').filter((e) => e.payload.id === story.id).pop();++++ await check('independence is concrete: same Sources source, same publisher domain, same outlet or a copy of one report count once', async () => {++ assert.strictEqual(text.publisherDomain('https://tech.thepaper.example/x'), 'thepaper.example');++ assert.strictEqual(text.publisherDomain('https://news.bbc.co.uk/x'), 'bbc.co.uk');++ assert.strictEqual(text.publisherDomain('https://www.bbc.co.uk/x'), 'bbc.co.uk');++ assert.strictEqual(row(copy).status, 'duplicate', 'the syndicated copy is a near-duplicate of the Alpha report');++ assert.strictEqual(row(copy).duplicate_of, row(wire).id);++ const n = (items) => text.independentSources(items.map(row)).length;++ assert.strictEqual(n([feed1, feed2]), 1);++ assert.strictEqual(n([world, tech]), 1);++ assert.strictEqual(n([wire, copy]), 1);++ assert.strictEqual(n([wire, copy, beta]), 1, 'the copy ties Beta Daily to the Alpha report: still one');++ assert.strictEqual(n([solo, wire]), 2);++ assert.strictEqual(n([solo, feed1, world, wire]), 4);++ });++++ await check('a single-source story is published and readable but noindex, and absent from the sitemap', async () => {++ const story = await storyOn(t, 'Two treatment plants to open next spring', [solo]);++ const page = await t.get(`/stories/${story.slug}`);++ assert.strictEqual(page.status, 200);++ assert.match(page.text, /Officials confirmed on Tuesday/);++ assert.match(page.text, /<meta name="robots" content="noindex, follow">/);++ assert.strictEqual(page.headers.get('x-robots-tag'), 'noindex, follow');++ const json = (await t.get(`/stories/${story.slug}.json`)).json();++ assert.strictEqual(json.state, 'published');++ assert.deepStrictEqual(json.indexability.reasons.map((r) => [r.code, r.detail]), [['unsourced', '1 of 2 sources']]);++ assert.doesNotMatch(await sitemap(), new RegExp(story.slug));++ assert.strictEqual(indexEvent(story).payload.indexability.decision, 'noindex', 'Search is told it is noindex');++ assert.match((await t.get('/')).text, new RegExp(`href="/stories/${story.slug}"`), 'still listed for readers');++ });++++ await check('two items from the same Sources source are one source: still noindex', async () => {++ const story = await storyOn(t, 'Valley water restrictions to end', [feed1, feed2]);++ const json = (await t.get(`/stories/${story.slug}.json`)).json();++ assert.strictEqual(json.sources.length, 2);++ assert.deepStrictEqual(json.indexability.reasons.map((r) => [r.code, r.detail]), [['unsourced', '1 of 2 sources']]);++ assert.doesNotMatch(await sitemap(), new RegExp(story.slug));++ });++++ await check('two Sources sources of one publisher (one domain), or a report and its syndicated copy, are still noindex', async () => {++ const paper = await storyOn(t, 'Treatment plant expansion approved', [world, tech]);++ assert.deepStrictEqual((await t.get(`/stories/${paper.slug}.json`)).json().indexability.reasons.map((r) => r.code), ['unsourced']);++ const syndicated = await storyOn(t, 'Water agency confirms spring opening', [wire, copy]);++ assert.deepStrictEqual((await t.get(`/stories/${syndicated.slug}.json`)).json().indexability.reasons.map((r) => r.code), ['unsourced']);++ const sm = await sitemap();++ assert.doesNotMatch(sm, new RegExp(paper.slug));++ assert.doesNotMatch(sm, new RegExp(syndicated.slug));++ });++++ await check('two independent sources make the story indexable and put it in the sitemap', async () => {++ const story = await storyOn(t, 'Two new water treatment plants confirmed', [solo, wire]);++ const page = await t.get(`/stories/${story.slug}`);++ assert.match(page.text, /<meta name="robots" content="index, follow">/);++ assert.strictEqual(page.headers.get('x-robots-tag'), null);++ assert.deepStrictEqual((await t.get(`/stories/${story.slug}.json`)).json().indexability, { indexable: true, robots: 'index, follow', reasons: [] });++ assert.match(await sitemap(), new RegExp(`<loc>https://openvibe.news/stories/${story.slug}</loc>`));++ assert.strictEqual(indexEvent(story).payload.indexability.decision, 'index');++ });++ await t.close();++++ const one = await boot({ env: { NEWS_MIN_INDEPENDENT_SOURCES: '1' } });++ await check('NEWS_MIN_INDEPENDENT_SOURCES=1 restores the one-source gate', async () => {++ assert.strictEqual(one.ctx.publication.POLICY.minSources, 1);++ one.sources.addSource('solo-wire', { name: 'Solo Wire' });++ const it = one.sources.addItem({ source_key: 'solo-wire', title: 'Water authority to open two treatment plants next spring', url: 'https://solo-wire.example/water/plants' });++ await one.pull();++ const story = await storyOn(one, 'Two treatment plants to open next spring', [it]);++ const json = (await one.get(`/stories/${story.slug}.json`)).json();++ assert.strictEqual(json.indexability.indexable, true, JSON.stringify(json.indexability));++ assert.match((await one.get('/sitemaps/stories.xml')).text, new RegExp(story.slug));++ });++ await one.close();++++ done();++})();@@@ -106,7 +106,8 @@ const BODY = [});await check('the editor revises without the removed source; publishing needs a correction note, then resolves both flags', async () => {-- const fixed = BODY.split('\n\n').slice(0, 2).map((p) => p.replace('[1, 2]', '[2]')).join('\n\n');++ // [1] leaves; the first paragraph now rests on [2] and [3], two independent outlets (the index gate needs two).++ const fixed = BODY.split('\n\n').slice(0, 2).map((p) => p.replace('[1, 2]', '[2, 3]')).join('\n\n');r = await t.api(`/stories/${story.id}/sources/${t.db().prepare('SELECT id FROM news_source_items WHERE sources_item_id = ?').get(reports.a.id).id}`, { method: 'DELETE' });assert.strictEqual(r.status, 200, r.text);r = await t.api(`/stories/${story.id}/revisions`, { json: { body: fixed.replace('2030', '2031'), expected_revision: 3 } });
@@ -42,6 +42,10 @@ OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720
@@ -42,6 +42,10 @@ OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720
# Licensing: the longest source summary kept (and only when the source's terms allow short summaries)
# Licensing: the longest source summary kept (and only when the source's terms allow short summaries)
#NEWS_SUMMARY_MAX_CHARS=280
#NEWS_SUMMARY_MAX_CHARS=280
+# Indexing: independent sources a story must cite to be indexable (fewer: published but noindex,
+# out of sitemaps). Items from one Sources source, one publisher's domain or one outlet, or copies
+# of one report, count once. Minimum 1.
+#NEWS_MIN_INDEPENDENT_SOURCES=2
# Clustering window
# Clustering window
#NEWS_CLUSTER_WINDOW_HOURS=72
#NEWS_CLUSTER_WINDOW_HOURS=72
@@ -160,8 +160,19 @@ Other tables in the same database:
@@ -160,8 +160,19 @@ Other tables in the same database:
### Discoverability (roadmap §32)
### Discoverability (roadmap §32)
- Robots meta, canonical and `X-Robots-Tag` come from the `openvibe-publishing/seo` gate with
- Robots meta, canonical and `X-Robots-Tag` come from the `openvibe-publishing/seo` gate with
- explicit reasons. News' policy: at least one live cited source, at least 40 words; unsupported
- claims, retraction, an editor's noindex and unreviewed AI output are never indexable.
+ explicit reasons. News' policy: at least two independent live cited sources
+ (`NEWS_MIN_INDEPENDENT_SOURCES`, default 2), at least 40 words; unsupported claims, retraction,
+ an editor's noindex and unreviewed AI output are never indexable.
+- **Independent sources** (`text.independentSources`): the live Sources items the published
+ revision's paragraphs cite, where items count as one source when they share the OpenVibe.Sources
+ source (`source_key`: the same feed or registry entry), the publisher's domain of their canonical
+ URL (`news.example.com` and `www.example.com` are `example.com`; `news.bbc.co.uk` is
+ `bbc.co.uk`), the outlet name, or the original report they duplicate (a syndicated or copied
+ report dedupe marked as a duplicate). The relation is transitive and deliberately coarse: two
+ sites on one shared host count once. A story on fewer independent sources is still published,
+ listed and readable, but `noindex` with the reason `unsourced` (for example `1 of 2 sources`),
+ so it is out of the sitemaps and its Search document says noindex. Setting the minimum to 1
+ restores the one-source gate.
- JSON-LD `NewsArticle` from real fields only: headline, dates, the editors named in the
- JSON-LD `NewsArticle` from real fields only: headline, dates, the editors named in the
revision's authorship (only when the Network name is known), the section (topic), the cited
revision's authorship (only when the Network name is known), the section (topic), the cited
source URLs, and OpenVibe.News as publisher. Missing fields are omitted.
source URLs, and OpenVibe.News as publisher. Missing fields are omitted.
@@ -250,6 +261,7 @@ Each grant is `[client, capability, audience]`:
@@ -250,6 +261,7 @@ Each grant is `[client, capability, audience]`:
| Licensed material never leaks: bodies and summaries beyond the allowance never reach the database, pages, JSON, feeds, sitemaps, the API, events or Search. | `test/licensing.test.js` |
| Licensed material never leaks: bodies and summaries beyond the allowance never reach the database, pages, JSON, feeds, sitemaps, the API, events or Search. | `test/licensing.test.js` |
| Dedupe by canonical URL, content hash and near-duplicate headline; idempotent replays. | `test/ingest.test.js` |
| Dedupe by canonical URL, content hash and near-duplicate headline; idempotent replays. | `test/ingest.test.js` |
| Clusters are deterministic and explained; merges and splits are audited and reversible. | `test/clusters.test.js` |
| Clusters are deterministic and explained; merges and splits are audited and reversible. | `test/clusters.test.js` |
+| The index gate counts independent sources: a single-source story, two items of one Sources source, two sources of one publisher's domain and a report with its syndicated copy are published but noindex and out of the sitemap; two independent sources are indexable; `NEWS_MIN_INDEPENDENT_SOURCES=1` restores the old gate. | `test/indexability.test.js` |
| Retraction: visible at its URL with the notice, noindex with its reason, out of sitemaps and Search, event emitted, final. | `test/retraction.test.js` |
| Retraction: visible at its URL with the notice, noindex with its reason, out of sitemaps and Search, event emitted, final. | `test/retraction.test.js` |
| Useful without JavaScript: the whole editorial journey with forms, and public pages complete in HTML. | `test/nojs.test.js` |
| Useful without JavaScript: the whole editorial journey with forms, and public pages complete in HTML. | `test/nojs.test.js` |
| AI output is a draft that needs a person's review (AI-assisted too); uncited claims dropped; failure makes no text. | `test/ai.test.js` |
| AI output is a draft that needs a person's review (AI-assisted too); uncited claims dropped; failure makes no text. | `test/ai.test.js` |
@@ -66,6 +66,13 @@ function load(env = process.env) {
@@ -66,6 +66,13 @@ function load(env = process.env) {
summaryMaxChars: Math.min(Math.max(int(env.NEWS_SUMMARY_MAX_CHARS, 280), 0), 500),
summaryMaxChars: Math.min(Math.max(int(env.NEWS_SUMMARY_MAX_CHARS, 280), 0), 500),
},
},
+ // Indexing: how many independent sources (distinct Sources sources, publishers' domains and
+ // outlets, not copies of one report) a story's paragraphs must cite before search engines,
+ // sitemaps and Search may treat it as indexable. Fewer: published, but noindex.
+ indexing: {
+ minIndependentSources: Math.max(int(env.NEWS_MIN_INDEPENDENT_SOURCES, 2), 1),
+ },
+
// Clustering: the time window a new item may join an existing cluster in.
// Clustering: the time window a new item may join an existing cluster in.
clustering: {
clustering: {
windowMs: int(env.NEWS_CLUSTER_WINDOW_HOURS, 72) * 3600 * 1000,
windowMs: int(env.NEWS_CLUSTER_WINDOW_HOURS, 72) * 3600 * 1000,
@@ -7,10 +7,14 @@
@@ -7,10 +7,14 @@
* - the Search document and the publication events (openvibe-publishing/index-hooks), enqueued
* - the Search document and the publication events (openvibe-publishing/index-hooks), enqueued
* in the same transaction as the change through the SDK outbox
* in the same transaction as the change through the SDK outbox
*
*
- * News' gate policy: at least one source (citations are the Sources items the published revision
- * cites and that are still live upstream), at least 40 words; a paragraph whose every source was
- * removed upstream counts as an unsupported claim (noindex until an editor revises); a retraction
- * is noindex; AI-generated text is hidden until a person reviews it.
+ * News' gate policy: at least NEWS_MIN_INDEPENDENT_SOURCES (2) independent sources, at least 40
+ * words. The sources counted are the Sources items the published revision's paragraphs cite and
+ * that are still live upstream, grouped by text.independentSources: items from the same Sources
+ * source, the same publisher's domain, the same outlet, or copies of the same original report are
+ * one source. A story on fewer is still published and readable, but noindex ('unsourced'), so it
+ * stays out of sitemaps. A paragraph whose every source was removed upstream counts as an
+ * unsupported claim (noindex until an editor revises); a retraction is noindex; AI-generated text
+ * is hidden until a person reviews it.
*
*
* Search receives ONLY published (not retracted), listable stories. Everything else is a tombstone,
* Search receives ONLY published (not retracted), listable stories. Everything else is a tombstone,
* so a retraction or an unpublish removes the old copy; a story that was never indexed gets no tombstone.
* so a retraction or an unpublish removes the old copy; a story that was never indexed gets no tombstone.
@@ -18,8 +22,10 @@
@@ -18,8 +22,10 @@
const seo = require('openvibe-publishing/seo');
const seo = require('openvibe-publishing/seo');
const hooks = require('openvibe-publishing/index-hooks');
const hooks = require('openvibe-publishing/index-hooks');
const authorship = require('openvibe-publishing/authorship');
const authorship = require('openvibe-publishing/authorship');
+const { independentSources } = require('./text');
-const POLICY = Object.freeze({ minWords: 40, requireSources: true, minSources: 1 });
+// minSources counts independent sources; config.indexing.minIndependentSources overrides it.
+const POLICY = Object.freeze({ minWords: 40, requireSources: true, minSources: 2 });
const OWNER = 'news';
const OWNER = 'news';
/**
/**
@@ -43,6 +49,7 @@ function disclosure(rec, review) {
@@ -43,6 +49,7 @@ function disclosure(rec, review) {
function createPublication({ store, config, outbox }) {
function createPublication({ store, config, outbox }) {
const { db } = store;
const { db } = store;
+ const policy = Object.freeze({ ...POLICY, minSources: (config.indexing && config.indexing.minIndependentSources) || POLICY.minSources });
const storyById = db.prepare('SELECT * FROM news_stories WHERE id = ?');
const storyById = db.prepare('SELECT * FROM news_stories WHERE id = ?');
const itemById = db.prepare('SELECT * FROM news_source_items WHERE id = ?');
const itemById = db.prepare('SELECT * FROM news_source_items WHERE id = ?');
const topicById = db.prepare('SELECT * FROM news_topics WHERE id = ?');
const topicById = db.prepare('SELECT * FROM news_topics WHERE id = ?');
@@ -76,7 +83,8 @@ function createPublication({ store, config, outbox }) {
@@ -76,7 +83,8 @@ function createPublication({ store, config, outbox }) {
for (const n of ns) if (alive(n)) cited.add(n);
for (const n of ns) if (alive(n)) cited.add(n);
if (!ns.some(alive)) unsupported++;
if (!ns.some(alive)) unsupported++;
}
}
- return { citedLive: cited.size, unsupported, live };
+ const groups = independentSources([...cited].map((n) => live.get(n)));
+ return { citedLive: cited.size, independent: groups.length, groups, unsupported, live };
}
}
const plain = (rev) => (rev ? `${rev.fields.headline || ''}\n${paragraphsOf(rev).map((p) => p.text).join('\n')}` : '');
const plain = (rev) => (rev ? `${rev.fields.headline || ''}\n${paragraphsOf(rev).map((p) => p.text).join('\n')}` : '');
@@ -97,13 +105,13 @@ function createPublication({ store, config, outbox }) {
@@ -97,13 +105,13 @@ function createPublication({ store, config, outbox }) {
visibility: 'public',
visibility: 'public',
canonicalUrl: storyUrl(story),
canonicalUrl: storyUrl(story),
text: plain(rev),
text: plain(rev),
- citationCount: sup.citedLive,
+ citationCount: sup.independent,
unsupportedClaims: sup.unsupported,
unsupportedClaims: sup.unsupported,
retracted: (state || story.state) === 'retracted',
retracted: (state || story.state) === 'retracted',
noindex: Boolean(story.noindex),
noindex: Boolean(story.noindex),
};
};
if (rec) Object.assign(facts, gateAuthorship(rec, reviewOf(story, rev)));
if (rec) Object.assign(facts, gateAuthorship(rec, reviewOf(story, rev)));
- return seo.evaluate(facts, { policy: POLICY, now: store.now() });
+ return seo.evaluate(facts, { policy, now: store.now() });
}
}
function topicOf(story) { return story.topic_id ? topicById.get(story.topic_id) : null; }
function topicOf(story) { return story.topic_id ? topicById.get(story.topic_id) : null; }
@@ -193,7 +201,7 @@ function createPublication({ store, config, outbox }) {
@@ -193,7 +201,7 @@ function createPublication({ store, config, outbox }) {
}
}
return {
return {
- POLICY, OWNER, abs, storyPath, storyUrl, topicPath, feedId,
+ POLICY: policy, OWNER, abs, storyPath, storyUrl, topicPath, feedId,
authorshipOf, reviewOf, paragraphsOf, snapshotOf, liveSources, support, decide, topicOf,
authorshipOf, reviewOf, paragraphsOf, snapshotOf, liveSources, support, decide, topicOf,
documentFor, syncIndex, snapshot, afterChange, plain,
documentFor, syncIndex, snapshot, afterChange, plain,
};
};
@@ -2,8 +2,8 @@
@@ -2,8 +2,8 @@
/**
/**
* Deterministic text helpers for ingestion: URL keys and title keys for dedupe, word shingles for
* Deterministic text helpers for ingestion: URL keys and title keys for dedupe, word shingles for
- * near-duplicate headlines, the terms and named entities clustering compares, and the licensing
- * rule for source summaries. Pure functions: the same input always gives the same output, so every
+ * near-duplicate headlines, the terms and named entities clustering compares, the licensing rule
+ * for source summaries, and which cited sources are independent (the index gate). Pure functions: the same input always gives the same output, so every
* dedupe and cluster decision can be recomputed and explained.
* dedupe and cluster decision can be recomputed and explained.
*/
*/
@@ -51,6 +51,59 @@ function hostOf(url) {
@@ -51,6 +51,59 @@ function hostOf(url) {
try { return new URL(String(url)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return null; }
try { return new URL(String(url)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return null; }
}
}
+// Second-level labels under a two-letter country code that are not a publisher's own name
+// (bbc.co.uk, abc.net.au): the publisher is the label before them.
+const COUNTRY_SECOND_LEVEL = new Set(['co', 'com', 'org', 'net', 'gov', 'edu', 'ac', 'ne', 'or', 'go', 'gob', 'nic', 'mil']);
+
+/**
+ * The publisher's domain of a URL: its host without subdomains (news.example.com and
+ * www.example.com → example.com; news.bbc.co.uk → bbc.co.uk). Deliberately coarse: two sites on
+ * one shared host (two blogs on one platform) count as one publisher, never the other way round.
+ */
+function publisherDomain(url) {
+ const host = hostOf(url);
+ if (!host) return null;
+ if (/^[\d.]+$/.test(host) || host.startsWith('[')) return host;
+ const labels = host.split('.').filter(Boolean);
+ if (labels.length <= 2) return labels.join('.');
+ const n = labels.length;
+ const take = labels[n - 1].length === 2 && COUNTRY_SECOND_LEVEL.has(labels[n - 2]) ? 3 : 2;
+ return labels.slice(-take).join('.');
+}
+
+/**
+ * Independent sources among source items (news_source_items rows): items are one source when they
+ * share any of
+ * - the OpenVibe.Sources source (source_key: the same feed or registry entry),
+ * - the publisher's domain of their canonical URL (publisherDomain),
+ * - the outlet name (folded),
+ * - the original they duplicate (duplicate_of: the same report syndicated or copied),
+ * and the relation is transitive. → groups of item ids, each group one independent source.
+ */
+function independentSources(items) {
+ const parent = new Map();
+ const find = (k) => { while (parent.get(k) !== k) { parent.set(k, parent.get(parent.get(k))); k = parent.get(k); } return k; };
+ const union = (a, b) => { const x = find(a); const y = find(b); if (x !== y) parent.set(y, x); };
+ for (const it of items) {
+ const node = `item:${it.id}`;
+ if (!parent.has(node)) parent.set(node, node);
+ const domain = publisherDomain(it.canonical_url);
+ const outlet = fold(it.outlet);
+ const keys = [`source:${it.source_key}`, `original:${it.duplicate_of || it.id}`, domain ? `domain:${domain}` : null, outlet ? `outlet:${outlet}` : null].filter(Boolean);
+ for (const k of keys) {
+ if (!parent.has(k)) parent.set(k, k);
+ union(node, k);
+ }
+ }
+ const groups = new Map();
+ for (const it of items) {
+ const root = find(`item:${it.id}`);
+ if (!groups.has(root)) groups.set(root, []);
+ groups.get(root).push(it.id);
+ }
+ return [...groups.values()];
+}
+
/** Very light stemming: plural and possessive endings, so "rockets" and "rocket" meet. */
/** Very light stemming: plural and possessive endings, so "rockets" and "rocket" meet. */
function stem(w) {
function stem(w) {
if (w.length > 4 && w.endsWith('ies')) return `${w.slice(0, -3)}y`;
if (w.length > 4 && w.endsWith('ies')) return `${w.slice(0, -3)}y`;
@@ -141,4 +194,4 @@ function licensedSummary(summary, { termsNote, licenseNote, maxChars = 280 } = {
@@ -141,4 +194,4 @@ function licensedSummary(summary, { termsNote, licenseNote, maxChars = 280 } = {
return { summary: `${(at > maxChars * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.-]+$/, '')}…`, basis: 'terms_allow_short_summaries' };
return { summary: `${(at > maxChars * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.-]+$/, '')}…`, basis: 'terms_allow_short_summaries' };
}
}
-module.exports = { fold, titleKey, urlKey, hostOf, terms, entities, shingles, jaccard, licensedSummary, stem, STOPWORDS };
+module.exports = { fold, titleKey, urlKey, hostOf, publisherDomain, independentSources, terms, entities, shingles, jaccard, licensedSummary, stem, STOPWORDS };
new file mode 100644
new file mode 100644
@@ -0,0 +1,126 @@
@@ -0,0 +1,126 @@
+'use strict';
+/**
+ * The index gate needs independent sources: a story's paragraphs must cite at least
+ * NEWS_MIN_INDEPENDENT_SOURCES (2) live sources that are not the same Sources source, the same
+ * publisher's domain, the same outlet, or copies of one report. A story on fewer is published and
+ * readable, but noindex ('unsourced'), out of the sitemaps, and Search is told it is noindex.
+ * Setting the minimum to 1 restores the old behaviour.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const text = require('../server/domain/text');
+
+const PARA = 'Officials confirmed on Tuesday that the regional water authority will open two new treatment plants next spring, adding capacity for roughly 400,000 residents across the valley and ending the seasonal restrictions of recent summers. The authority said construction is on schedule and within the approved budget.';
+
+/** An editor's story resting on exactly these Sources items (cited in the one paragraph). */
+async function storyOn(t, headline, items) {
+ let r = await t.api('/stories', { json: { headline } });
+ assert.strictEqual(r.status, 201, r.text);
+ const story = r.json().story;
+ for (const it of items) {
+ r = await t.api(`/stories/${story.id}/sources`, { json: { item: it.id } });
+ assert.ok(r.status === 200 || r.status === 201, r.text);
+ }
+ r = await t.api(`/stories/${story.id}/revisions`, { json: { body: `${PARA} [${items.map((_, i) => i + 1).join(', ')}]`, expected_revision: 0 } });
+ assert.strictEqual(r.status, 201, r.text);
+ r = await t.api(`/stories/${story.id}/publish`, { json: {} });
+ assert.strictEqual(r.status, 200, r.text);
+ return story;
+}
+
+(async () => {
+ const t = await boot();
+ const S = t.sources;
+ S.addSource('solo-wire', { name: 'Solo Wire' });
+ S.addSource('one-feed', { name: 'One Feed' });
+ S.addSource('paper-world', { name: 'The Paper: World' });
+ S.addSource('paper-tech', { name: 'The Paper: Tech' });
+ S.addSource('alpha-news', { name: 'Alpha News' });
+ S.addSource('beta-daily', { name: 'Beta Daily' });
+ const solo = S.addItem({ source_key: 'solo-wire', title: 'Water authority to open two treatment plants next spring', url: 'https://solo-wire.example/water/plants' });
+ const feed1 = S.addItem({ source_key: 'one-feed', title: 'Valley water restrictions to end as plants come online', url: 'https://one-feed.example/2026/valley-water' });
+ const feed2 = S.addItem({ source_key: 'one-feed', title: 'Residents welcome news of added treatment capacity', url: 'https://one-feed.example/2026/residents-react' });
+ const world = S.addItem({ source_key: 'paper-world', title: 'Regional authority approves treatment plant expansion', url: 'https://www.thepaper.example/world/plant-expansion' });
+ const tech = S.addItem({ source_key: 'paper-tech', title: 'Inside the membrane technology behind the new plants', url: 'https://tech.thepaper.example/membranes' });
+ const wire = S.addItem({ source_key: 'alpha-news', title: 'Water agency says two new treatment plants will open next spring', url: 'https://alpha.example/a/water-agency-plants' });
+ const copy = S.addItem({ source_key: 'beta-daily', title: 'Water agency says two new treatment plants will open next spring', url: 'https://beta.example/b/syndicated-water' });
+ const beta = S.addItem({ source_key: 'beta-daily', title: 'Budget vote clears way for valley water projects', url: 'https://beta.example/b/budget-vote' });
+ await t.pull();
+ const row = (it) => t.db().prepare('SELECT * FROM news_source_items WHERE sources_item_id = ?').get(it.id);
+ const sitemap = async () => (await t.get('/sitemaps/stories.xml')).text;
+ const indexEvent = (story) => t.events('news.index_document.upserted').filter((e) => e.payload.id === story.id).pop();
+
+ await check('independence is concrete: same Sources source, same publisher domain, same outlet or a copy of one report count once', async () => {
+ assert.strictEqual(text.publisherDomain('https://tech.thepaper.example/x'), 'thepaper.example');
+ assert.strictEqual(text.publisherDomain('https://news.bbc.co.uk/x'), 'bbc.co.uk');
+ assert.strictEqual(text.publisherDomain('https://www.bbc.co.uk/x'), 'bbc.co.uk');
+ assert.strictEqual(row(copy).status, 'duplicate', 'the syndicated copy is a near-duplicate of the Alpha report');
+ assert.strictEqual(row(copy).duplicate_of, row(wire).id);
+ const n = (items) => text.independentSources(items.map(row)).length;
+ assert.strictEqual(n([feed1, feed2]), 1);
+ assert.strictEqual(n([world, tech]), 1);
+ assert.strictEqual(n([wire, copy]), 1);
+ assert.strictEqual(n([wire, copy, beta]), 1, 'the copy ties Beta Daily to the Alpha report: still one');
+ assert.strictEqual(n([solo, wire]), 2);
+ assert.strictEqual(n([solo, feed1, world, wire]), 4);
+ });
+
+ await check('a single-source story is published and readable but noindex, and absent from the sitemap', async () => {
+ const story = await storyOn(t, 'Two treatment plants to open next spring', [solo]);
+ const page = await t.get(`/stories/${story.slug}`);
+ assert.strictEqual(page.status, 200);
+ assert.match(page.text, /Officials confirmed on Tuesday/);
+ assert.match(page.text, /<meta name="robots" content="noindex, follow">/);
+ assert.strictEqual(page.headers.get('x-robots-tag'), 'noindex, follow');
+ const json = (await t.get(`/stories/${story.slug}.json`)).json();
+ assert.strictEqual(json.state, 'published');
+ assert.deepStrictEqual(json.indexability.reasons.map((r) => [r.code, r.detail]), [['unsourced', '1 of 2 sources']]);
+ assert.doesNotMatch(await sitemap(), new RegExp(story.slug));
+ assert.strictEqual(indexEvent(story).payload.indexability.decision, 'noindex', 'Search is told it is noindex');
+ assert.match((await t.get('/')).text, new RegExp(`href="/stories/${story.slug}"`), 'still listed for readers');
+ });
+
+ await check('two items from the same Sources source are one source: still noindex', async () => {
+ const story = await storyOn(t, 'Valley water restrictions to end', [feed1, feed2]);
+ const json = (await t.get(`/stories/${story.slug}.json`)).json();
+ assert.strictEqual(json.sources.length, 2);
+ assert.deepStrictEqual(json.indexability.reasons.map((r) => [r.code, r.detail]), [['unsourced', '1 of 2 sources']]);
+ assert.doesNotMatch(await sitemap(), new RegExp(story.slug));
+ });
+
+ await check('two Sources sources of one publisher (one domain), or a report and its syndicated copy, are still noindex', async () => {
+ const paper = await storyOn(t, 'Treatment plant expansion approved', [world, tech]);
+ assert.deepStrictEqual((await t.get(`/stories/${paper.slug}.json`)).json().indexability.reasons.map((r) => r.code), ['unsourced']);
+ const syndicated = await storyOn(t, 'Water agency confirms spring opening', [wire, copy]);
+ assert.deepStrictEqual((await t.get(`/stories/${syndicated.slug}.json`)).json().indexability.reasons.map((r) => r.code), ['unsourced']);
+ const sm = await sitemap();
+ assert.doesNotMatch(sm, new RegExp(paper.slug));
+ assert.doesNotMatch(sm, new RegExp(syndicated.slug));
+ });
+
+ await check('two independent sources make the story indexable and put it in the sitemap', async () => {
+ const story = await storyOn(t, 'Two new water treatment plants confirmed', [solo, wire]);
+ const page = await t.get(`/stories/${story.slug}`);
+ assert.match(page.text, /<meta name="robots" content="index, follow">/);
+ assert.strictEqual(page.headers.get('x-robots-tag'), null);
+ assert.deepStrictEqual((await t.get(`/stories/${story.slug}.json`)).json().indexability, { indexable: true, robots: 'index, follow', reasons: [] });
+ assert.match(await sitemap(), new RegExp(`<loc>https://openvibe.news/stories/${story.slug}</loc>`));
+ assert.strictEqual(indexEvent(story).payload.indexability.decision, 'index');
+ });
+ await t.close();
+
+ const one = await boot({ env: { NEWS_MIN_INDEPENDENT_SOURCES: '1' } });
+ await check('NEWS_MIN_INDEPENDENT_SOURCES=1 restores the one-source gate', async () => {
+ assert.strictEqual(one.ctx.publication.POLICY.minSources, 1);
+ one.sources.addSource('solo-wire', { name: 'Solo Wire' });
+ const it = one.sources.addItem({ source_key: 'solo-wire', title: 'Water authority to open two treatment plants next spring', url: 'https://solo-wire.example/water/plants' });
+ await one.pull();
+ const story = await storyOn(one, 'Two treatment plants to open next spring', [it]);
+ const json = (await one.get(`/stories/${story.slug}.json`)).json();
+ assert.strictEqual(json.indexability.indexable, true, JSON.stringify(json.indexability));
+ assert.match((await one.get('/sitemaps/stories.xml')).text, new RegExp(story.slug));
+ });
+ await one.close();
+
+ done();
+})();
@@ -106,7 +106,8 @@ const BODY = [
@@ -106,7 +106,8 @@ const BODY = [
});
});
await check('the editor revises without the removed source; publishing needs a correction note, then resolves both flags', async () => {
await check('the editor revises without the removed source; publishing needs a correction note, then resolves both flags', async () => {
- const fixed = BODY.split('\n\n').slice(0, 2).map((p) => p.replace('[1, 2]', '[2]')).join('\n\n');
+ // [1] leaves; the first paragraph now rests on [2] and [3], two independent outlets (the index gate needs two).
+ const fixed = BODY.split('\n\n').slice(0, 2).map((p) => p.replace('[1, 2]', '[2, 3]')).join('\n\n');
r = await t.api(`/stories/${story.id}/sources/${t.db().prepare('SELECT id FROM news_source_items WHERE sources_item_id = ?').get(reports.a.id).id}`, { method: 'DELETE' });
r = await t.api(`/stories/${story.id}/sources/${t.db().prepare('SELECT id FROM news_source_items WHERE sources_item_id = ?').get(reports.a.id).id}`, { method: 'DELETE' });
assert.strictEqual(r.status, 200, r.text);
assert.strictEqual(r.status, 200, r.text);
r = await t.api(`/stories/${story.id}/revisions`, { json: { body: fixed.replace('2030', '2031'), expected_revision: 3 } });
r = await t.api(`/stories/${story.id}/revisions`, { json: { body: fixed.replace('2030', '2031'), expected_revision: 3 } });