openvibers/openvibe.news README.md
README.md
| 1 | # OpenVibe.News |
| 2 | |
| 3 | > Source-backed stories: clustered coverage, cited paragraphs, perspectives, timelines and a |
| 4 | > public correction history. |
| 5 | |
| 6 | **Live site:** https://openvibe.news · **API docs:** https://openvibe.services · **For AI:** https://openvibe.news/llms.txt |
| 7 | **Run it:** `npm ci && npm run dev` (Node 22 or newer) · **Tests:** `npm test` |
| 8 | |
| 9 | **Status:** alpha (roadmap Wave 17, News half). The service runs and its tests pass. It is |
| 10 | **public at `openvibe.news` since 2026-10-09**, after an independent pre-launch security review whose |
| 11 | fixes shipped first; the domain no longer shows the OpenVibe.Sites placeholder. Its capabilities and |
| 12 | service manifest are registered in openvibe-contracts v0.97.0. |
| 13 | Since 2026-10-11 production ingests from the running OpenVibe.Sources service (NASA news releases, GOV.UK |
| 14 | news, the EU press corner): 70 source items in 18 clusters, 12 topics, and no stories yet, because a story |
| 15 | exists only once an editor opens one from a cluster and publishes it (below). |
| 16 | **Domain:** `openvibe.news` · **Port:** 4820 · **Service id:** `news` |
| 17 | **Plan:** OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §12.7; roadmap §15.13, §29, §32. |
| 18 | **License:** AGPL-3.0 (same as every OpenVibe service). |
| 19 | |
| 20 | ## Purpose |
| 21 | |
| 22 | A publication product, not an autonomous headline generator. OpenVibe.Sources collects news items |
| 23 | (headline, link, outlet, dates, terms). News normalises them, removes duplicates, groups them into |
| 24 | explainable clusters, and gives editors the tools to write stories from them. Every paragraph an |
| 25 | editor publishes cites the source items it rests on. A story with no sources cannot be published. |
| 26 | When a source changes or is withdrawn upstream, the story is flagged and a pending revision is |
| 27 | prepared; published text never changes silently. Full article text from sources is never stored, |
| 28 | shown, fed or indexed. |
| 29 | |
| 30 | News publishes nothing until an editor does: the only seed is the topic list. Until then, and beside the stories after, **the wire** (`/wire`, and "From the wire" on the front page) shows what the sources report, grouped by cluster: each line is the outlet's own headline, outlet, date and link (the fields the licensing rules let News keep), never News' writing; duplicates and reports removed upstream leave it; `/wire` is `noindex, follow`. |
| 31 | |
| 32 | ## AI briefs and Live |
| 33 | |
| 34 | The owner's April ask for News was multi-source summaries with tone and perspective controls and a live breaking-news |
| 35 | view. Stories stay editor-written (below); beside them News now has: |
| 36 | |
| 37 | - **AI briefs** (`/briefs`, `/briefs/:cluster`; [server/domain/briefs.js](server/domain/briefs.js), |
| 38 | `migrations/0004_cluster_briefs.sql`): what a cluster's reports say, written by OpenVibe.AI and labelled as AI on |
| 39 | every page. Every paragraph cites the reports it rests on, numbered in the table below it; claims the model did not |
| 40 | cite are dropped and listed under "What the AI left out"; a run with no cited text makes no brief; the stub |
| 41 | provider's output is never shown. Only News' stored fields are sent (headline, outlet, date, URL and the short summary |
| 42 | the terms allow). The worker writes a neutral summary for every open cluster with two or more reports in the last 72 |
| 43 | hours, six a tick every `NEWS_AI_BRIEFS_INTERVAL_MS` (10 minutes), and writes it again once the cluster changes. |
| 44 | - **Tone** (`?tone=brief|neutral|explained`, `news.summarize_story`'s `tone`) and **how the outlets put it** |
| 45 | (`news.compare_perspectives`, two outlets or more): made when a reader asks, with a POST from this site (five every ten |
| 46 | minutes an address, 40 an hour in all), never by a link a crawler follows. |
| 47 | - **Live** (`/live`): the newest reports as they arrive, grouped, refreshing itself every minute (`Refresh: 60`), with |
| 48 | **Breaking** on a subject three or more reports covered in the last two hours, and links to their AI briefs. |
| 49 | - Briefs and Live are `noindex` by default: they are AI text and other publishers' headlines, not News' writing. A person can review a brief (`POST /api/v1/briefs/:id/reviews`, the `/admin/review` queue); a reviewed summary brief with enough independent outlets becomes indexable and is listed in `/sitemaps/briefs.xml`. A remake after its cluster changes waits for a person again. |
| 50 | `NEWS_AI_BRIEFS=off` turns briefs off. |
| 51 | |
| 52 | ## Owns |
| 53 | |
| 54 | The nine charter tables live in News' own PostgreSQL database (`ov_news` on the host's data role, ADR-035; schema in [migrations/](migrations/)): |
| 55 | |
| 56 | | Charter table | What it is | |
| 57 | |---|---| |
| 58 | | `news_topics` | Owned by News. Canonical topic identities (slug, name, description). The only seed (`seeds/topics.json`). | |
| 59 | | `news_source_items` | Owned by News. One row per Sources item (`sources_item_id` is a typed reference to `itm_…`): headline, canonical URL, outlet, authors, `published_at` as the source states it, the licence and terms notes, and a short summary **only** when those notes allow short summaries. Dedupe outcome (`status`, `duplicate_of`, `dedupe`), cluster and the stored reason for it, upstream revision and removal. No article body. | |
| 60 | | `news_story_clusters` | Owned by News. Clusters with their key terms and entities, time window, status (`open`, `merged`, `dissolved`) and lineage (`merged_into`, `split_from`). | |
| 61 | | `news_stories` | Owned by News. Publication state: slug, state (`draft`, `published`, `unpublished`, `retracted`), published revision, topic, cluster. | |
| 62 | | `news_story_revisions` | `openvibe-publishing/revisions` (prefix `news_story`). Immutable. Each revision holds the headline, the paragraphs with their source numbers, and a snapshot of the source table, timeline and perspectives. | |
| 63 | | `news_story_sources` | Owned by News. The story's source table: which items, their stable numbers `[n]` (never reused), perspective grouping, detachment. | |
| 64 | | `news_perspectives` | Owned by News. Editor-written labels that group a story's sources (for example an outlet group or a position a source states). Never generated. | |
| 65 | | `news_timeline_entries` | Owned by News. Dated entries; each date is one the source states and each entry rests on one attached source. | |
| 66 | | `news_editorial_flags` | Owned by News. `correction`, `update`, `retraction` (public, with notes) and `source_updated`, `source_removed` (upstream changes, with the pending revision). | |
| 67 | |
| 68 | Other tables in the same database: |
| 69 | |
| 70 | - Package companions: `news_story_citations` (one row per paragraph and timeline entry and cited |
| 71 | Sources item, append-only), `news_story_reviews` (a person's approval of AI output), |
| 72 | `news_story_drafts`, `news_story_revision_purges`, `news_index_revisions` (Search sequencer), |
| 73 | `news_story_discussion_refs` (a story's OpenVibe.Community thread id; never a comment, a count or |
| 74 | an author). |
| 75 | - `news_cluster_audit`: every merge, split and reversal with exactly which items moved. |
| 76 | - `news_ingest_runs`: what every webhook delivery, pull and upstream fetch failure did. |
| 77 | - `news_ingest_cursor` (the OpenVibe.Sources change cursor, `openvibe-publishing/ingest`; `news_state` |
| 78 | keeps the pre-chassis key it was carried from), `news_source_status` (display cache of Sources' |
| 79 | registry names and health), `event_outbox` and `idempotency_receipts` (openvibe-sdk), |
| 80 | `subject_projections` (Network names). None of them is authority for anything outside News. |
| 81 | - `telemetry_samples` (migration `0003_telemetry.sql`): News' own request telemetry, openvibe-sdk |
| 82 | 0.46's `platform.telemetry-sample@1` shape. Each HTTP request is aggregated by route template, |
| 83 | method and status class, never by raw URL; `/shared` files, static assets, health, readiness and |
| 84 | `/metrics` are not counted. Once a minute one `http.request` sample per key, plus the |
| 85 | active-requests, p95 latency and event-loop-lag gauges, is written here (`server/app.js` builds |
| 86 | the collector, `server/index.js` flushes it and runs the 14-day retention); rows older than |
| 87 | 14 days are deleted daily. |
| 88 | |
| 89 | ## Does not own |
| 90 | |
| 91 | - **Source adapters and the source registry.** OpenVibe.Sources owns them: which sites are read, |
| 92 | under which terms, how and when. News reads items with `sources.item.read` and never fetches a |
| 93 | site itself. |
| 94 | - **Discussion.** OpenVibe.Community owns the comment threads, their comments and their |
| 95 | moderation. News keeps only each story's thread id and reads the thread on every render. |
| 96 | - **AI generation.** OpenVibe.AI. News only receives drafts. |
| 97 | - **Search.** OpenVibe.Search indexes what News sends through Events. |
| 98 | - **Identity.** OpenVibe.Network: SSO, subjects and service principals. |
| 99 | |
| 100 | ## What works |
| 101 | |
| 102 | - **Ingestion** from OpenVibe.Sources, two ways with one effect: |
| 103 | - an Events webhook (`POST /internal/events`, signed with `NEWS_EVENTS_SECRET`) for |
| 104 | `sources.item.created|updated|removed` and `sources.fetch.failed`, each applied exactly once |
| 105 | through the SDK inbox; |
| 106 | - a cursor pull (`GET /api/v1/items?category=news&after=…&include_removed=1`) every |
| 107 | `NEWS_PULL_INTERVAL_MS`, the backstop for missed deliveries. |
| 108 | Applying an item is idempotent on (Sources item id, revision). Failures are recorded states |
| 109 | (`news_ingest_runs`, `news.source.failed`, `/api/ready`, the editor desk) and never create a |
| 110 | source item, a cluster or any story text. When Sources cannot be read, the delivery is answered |
| 111 | 503 so Events retries it. |
| 112 | - **Licensing.** Only headline, URL, outlet, authors, `published_at`, the notes and — when the |
| 113 | item's terms or licence note explicitly allows short summaries and nothing forbids them — a |
| 114 | summary of at most `NEWS_SUMMARY_MAX_CHARS` (280) are kept. Anything else a Sources item carries |
| 115 | (for example an article body an adapter mapped into `fields`) is never read. A removed item's |
| 116 | summary is dropped. Source summaries appear only in a story's source table; never in feeds, |
| 117 | Search documents or events. |
| 118 | - **Dedupe** (explainable, stored on the item): the same normalised canonical URL (tracking |
| 119 | parameters, `www.`, trailing slashes ignored), the same Sources content hash, or a near-duplicate |
| 120 | headline (Jaccard of word 2-shingles ≥ 0.9, or ≥ 0.75 from the same outlet, within 48 h). |
| 121 | Duplicates are kept for provenance, join their original's cluster and are not attached to new |
| 122 | stories by default. |
| 123 | - **Clusters** (deterministic): an item joins the open cluster, within `NEWS_CLUSTER_WINDOW_HOURS` |
| 124 | (72), with which it shares at least one named entity and two terms, or at least three terms; |
| 125 | entities score 2, terms 1; ties go to the oldest cluster. The shared entities, shared terms and |
| 126 | score are stored on the item and shown on the cluster page. Editors can merge clusters and split |
| 127 | items off; both are audited and reversible (`news_cluster_audit`). |
| 128 | - **Stories** (editorial): |
| 129 | - An editor opens a story from a cluster; its live, non-duplicate items become sources |
| 130 | `[1]`, `[2]`, … . Editors can attach and detach sources; numbers are never reused. |
| 131 | - Paragraphs end with the source numbers they rest on (`… [1, 3]`). A revision with an uncited |
| 132 | paragraph, or citing a number not in the source table, is refused. |
| 133 | - Revisions are immutable with optimistic concurrency (412 on a stale base), diff and preview. |
| 134 | - Publishing is refused while the revision has no sources (`story.unsourced`), cites a source |
| 135 | removed upstream or detached, or is AI-generated or AI-assisted without a person's approval. |
| 136 | - Timeline entries have a date the source states and one source each. Perspectives are labels |
| 137 | an editor writes; readers can re-frame the source table by number, outlet or perspective |
| 138 | (`?group=`), without JavaScript. |
| 139 | - Corrections and updates are notes published with a revision and shown in the story's |
| 140 | correction history. Retraction keeps the story at its URL with a retraction notice, noindex, |
| 141 | out of sitemaps and Search, labelled in listings and feeds; it is final. |
| 142 | - **Upstream changes.** When a cited source is revised or removed in Sources, every story citing it |
| 143 | gets an open flag and a pending revision prepared by the system (same text, refreshed source |
| 144 | entry). The published revision stays exactly as it was, with a notice to readers. A removed |
| 145 | source's headline, summary and link leave the public page at once; a paragraph resting only on |
| 146 | it is marked, and the gate makes the story noindex (`unsupported_claims`) until an editor |
| 147 | publishes a revision that no longer rests on it — with a correction or update note. |
| 148 | - **Comments** (OpenVibe.Community, referenced, never copied): |
| 149 | - A story open for discussion gets the thread Community resolves for EntityRef |
| 150 | `{ service: 'news', type: 'story', id }`, the first time its page is read. Open means published |
| 151 | (indexable or not: a noindex story is still public) with no paragraph resting only on a source |
| 152 | removed upstream. Drafts never get a thread: a Community thread is readable by anyone with its id. |
| 153 | - Only the thread id is stored (`news_story_discussion_refs`). The page reads the thread from |
| 154 | Community on every render, so Community's moderation and deletions are what readers see. When |
| 155 | Community cannot be read the page says comments are unavailable; it never shows an empty or |
| 156 | invented thread. |
| 157 | - Signed-in members comment with a plain form (`POST /stories/:slug/comments`, form token), as |
| 158 | themselves (`X-OV-Subject`, `community.comment.write`). |
| 159 | - Retracting or unpublishing a story, or a removal upstream that leaves a paragraph without a |
| 160 | source, hides its thread; publishing a revision that no longer rests on the removed source (or |
| 161 | republishing) shows it again. The call (`community.comment.moderate`) runs after the change |
| 162 | commits, is best effort, and never fails the change. A retracted story's page says comments are |
| 163 | closed; a story waiting for a revision says they are paused. |
| 164 | - **AI** (optional seams, never publication truth): |
| 165 | - OpenVibe.AI may deliver a draft revision with `X-OV-Origin: ai` (`news.story.revise`), naming |
| 166 | `news.summarize_story` or `news.compare_perspectives` and its run. |
| 167 | - An editor may ask for one (`POST /api/v1/stories/:id/ai-drafts`, or the desk's button when |
| 168 | `OV_AI_INTERNAL_URL` is set). Only News' stored fields are sent. Claims the model did not cite |
| 169 | are dropped and listed as gaps. A failed run makes no text. AI never creates perspective labels. |
| 170 | - AI-generated revisions, and AI-assisted ones (an editor's edit of AI text), cannot be published |
| 171 | and are hidden from indexes until a person (`usr_`) records an approving review of that |
| 172 | revision. Stub-provider output is held too. The disclosure is on the page, in the JSON and in |
| 173 | the Search document. |
| 174 | - With no AI configured, editors write every story from the source items. |
| 175 | |
| 176 | ### Routes (server-rendered, useful without JavaScript) |
| 177 | |
| 178 | | Route | What | |
| 179 | |---|---| |
| 180 | | `/` | published stories, newest first (retracted ones stay, labelled) and the topics | |
| 181 | | `/topics`, `/topics/:slug` | topics and each topic's stories | |
| 182 | | `/stories/:slug`, `/stories/:slug.json` | a story (text with source numbers, source table, timeline, perspectives, corrections, the Community comment thread), and the same story as data | |
| 183 | | `POST /stories/:slug/comments` | comment on the story's Community thread as the signed-in member (form token) | |
| 184 | | `/feed.xml`, `/atom.xml`, `/feed.json`, `/topics/:slug/{feed.xml,atom.xml,feed.json}` | feeds (News' own text only) | |
| 185 | | `/sitemap.xml` → `/sitemaps/stories.xml`, `/sitemaps/topics.xml` | indexable stories and topics only | |
| 186 | | `/robots.txt`, `/llms.txt`, `/llms-full.txt` | the automated-consumer policy and machine orientation (llms-full: the same header plus one section per indexable story) | |
| 187 | | `/edit`, `/edit/stories/:id`, `/clusters/:id` | the editor desk (forms, Network SSO, form tokens; editors only; private, noindex) | |
| 188 | | `/auth/*` | sign-in; the same session layer as OpenVibe.Community and OpenVibe.Blog | |
| 189 | | `/internal/events` | the Events webhook (loopback; nginx does not proxy it) | |
| 190 | | `/api/health`, `/api/ready`, `/release.json`, `/metrics` | health, truthful readiness, release, metrics (`/metrics` loopback only) | |
| 191 | |
| 192 | Sitemap entries carry lastmod from each story's latest publication or topic page's latest change. |
| 193 | |
| 194 | ### Discoverability (roadmap §32) |
| 195 | |
| 196 | - Robots meta, canonical and `X-Robots-Tag` come from the `openvibe-publishing/seo` gate with |
| 197 | explicit reasons. News' policy: at least two independent live cited sources |
| 198 | (`NEWS_MIN_INDEPENDENT_SOURCES`, default 2), at least 40 words; unsupported claims, retraction, |
| 199 | an editor's noindex and unreviewed AI output are never indexable. |
| 200 | - **Independent sources** (`text.independentSources`): the live Sources items the published |
| 201 | revision's paragraphs cite, where items count as one source when they share the OpenVibe.Sources |
| 202 | source (`source_key`: the same feed or registry entry), the publisher's domain of their canonical |
| 203 | URL (`news.example.com` and `www.example.com` are `example.com`; `news.bbc.co.uk` is |
| 204 | `bbc.co.uk`), the outlet name, or the original report they duplicate (a syndicated or copied |
| 205 | report dedupe marked as a duplicate). The relation is transitive and deliberately coarse: two |
| 206 | sites on one shared host count once. A story on fewer independent sources is still published, |
| 207 | listed and readable, but `noindex` with the reason `unsourced` (for example `1 of 2 sources`), |
| 208 | so it is out of the sitemaps and its Search document says noindex. Setting the minimum to 1 |
| 209 | restores the one-source gate. |
| 210 | - JSON-LD `NewsArticle` from real fields only: headline, dates, the editors named in the |
| 211 | revision's authorship (only when the Network name is known), the section (topic), the cited |
| 212 | source URLs, and OpenVibe.News as publisher. Missing fields are omitted. |
| 213 | - IndexNow (`openvibe-shared/indexnow`): with `INDEXNOW_KEY` set, the key file is served at |
| 214 | `/<key>.txt` and publishing, updating or unpublishing an indexable story pings the engines with |
| 215 | the story's URL and `/sitemap.xml`. Unset, the feature is off — nothing is mounted and nothing is |
| 216 | sent. Drafts, private and noindex pages are never pinged (`test/indexnow.test.js`). |
| 217 | - Drafts, unpublished, retracted and noindex stories never appear in sitemaps or Search; feeds |
| 218 | list published and (labelled) retracted stories, never a draft. |
| 219 | |
| 220 | ### Caching |
| 221 | |
| 222 | HTML and JSON vary on `Cookie` and `Authorization`. Published stories and lists rendered for an |
| 223 | anonymous visitor are `public, max-age=60`; everything else (signed-in views, the desk, previews, |
| 224 | refusals, the API) is `private, no-store` with `X-Robots-Tag`. Feeds and sitemaps are public for |
| 225 | five minutes and are never built for a viewer. |
| 226 | |
| 227 | ### Events (SDK outbox, same transaction as the change) |
| 228 | |
| 229 | | Event | Notes | |
| 230 | |---|---| |
| 231 | | `news.source.ingested` | internal; per Sources item created, revised or removed, with the dedupe and cluster outcome | |
| 232 | | `news.source.failed` | internal; a Sources fetch failure of a news source, an unreadable item, or a failed pull (once per outage) | |
| 233 | | `news.cluster.updated` | internal; created, grew, merged, split, reversed | |
| 234 | | `news.story.created`, `news.story.flagged` | internal | |
| 235 | | `news.story.published`, `.updated`, `.unpublished` | `openvibe-publishing/index-hooks` `publicationEvent`; canonical URL, state, indexability, topic; never the text | |
| 236 | | `news.story.retracted` | public for a listable story; the retraction note | |
| 237 | | `news.index_document.upserted` / `.deleted` | `search.index-document@1` documents (provenance: the Sources items and the AI run) and tombstones, with a monotonic index revision | |
| 238 | |
| 239 | ## Capabilities (registered in openvibe-contracts v0.97.0) |
| 240 | |
| 241 | Service tokens use audience `openvibe.news`, one capability per route; the editor the service acts |
| 242 | for goes in `X-OV-Subject` and must be an editor: |
| 243 | |
| 244 | - `news.story.create`, `news.story.read`, `news.story.revise`, `news.story.publish`, `news.story.retract` |
| 245 | - `news.cluster.read`, `news.cluster.manage` |
| 246 | - `news.source.attach`, `news.timeline.update`, `news.perspective.update` |
| 247 | - `news.topic.manage` |
| 248 | |
| 249 | Browser and app user JWTs are judged as editors (`NEWS_EDITORS` subjects and Network admins). |
| 250 | Grants for these ids are decided with the contracts library's matching rule |
| 251 | (`server/auth/capabilities.js`, which delegates straight to `capabilities.check` now that the ids |
| 252 | are released). The ids and the service manifest are released in openvibe-contracts (pinned at |
| 253 | v0.97.0); `docs/capabilities-proposal/` and `docs/service-manifest-proposal.json` stay as the |
| 254 | proposals they were released from. |
| 255 | |
| 256 | Called elsewhere, as the service principal `news`: `sources.item.read` and optionally |
| 257 | `sources.source.read` (Sources), `events.event.publish` and `events.subscription.manage` (Events), |
| 258 | `identity.subject.resolve` (Network), `community.comment.write` and `community.comment.moderate` |
| 259 | (Community) and, only with `OV_AI_INTERNAL_URL`, `ai.run.create` (AI). The list with audiences is |
| 260 | under [Grants the Network must hold](#grants-the-network-must-hold). |
| 261 | |
| 262 | ## Depends on |
| 263 | |
| 264 | - **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through |
| 265 | `openvibe-sdk/db`; Valkey holds the per-actor limit counters (optional: without `VALKEY_URL` they count per process). |
| 266 | - **Packages** (pinned by release tarball): `openvibe-publishing` v1.7.0 (async PostgreSQL stores: revisions, |
| 267 | citations, authorship, seo (ai-summary and llms-full), index-hooks, ssr, taxonomy slugify, and the shared ingest/publication |
| 268 | chassis: the Sources client, the change-cursor pull, the signed events consumer and the |
| 269 | gate → document → outbox glue), `openvibe-contracts` v0.97.0, |
| 270 | `openvibe-shared` v2.6.0 (Frame, app icon, footer, legal, release, metrics, ready, seo), |
| 271 | `openvibe-sdk` v0.67.0 (db with after-commit hooks, PostgreSQL events outbox and inbox, webhook signatures v2, |
| 272 | service tokens, Network sign-in (`openvibe-sdk/sso`) and the service outbox (`openvibe-sdk/events`), per-actor limits, testing). |
| 273 | - **OpenVibe.Sources** (4720): `sources.item.read`; optionally `sources.source.read` (outlet |
| 274 | names and source health; without it the outlet is the URL's host). |
| 275 | - **OpenVibe.Events** (4300): `events.event.publish`; `events.subscription.manage` to create the |
| 276 | `sources.item.*` and `sources.fetch.failed` subscriptions (`npm run subscribe`). |
| 277 | - **OpenVibe.Network** (4000): SSO (OAuth client `news`, redirect `https://openvibe.news/auth/callback`), |
| 278 | JWKS, `identity.subject.resolve` for editor names. |
| 279 | - **OpenVibe.Community** (4200): `community.comment.write` (resolve a story's thread, comment as |
| 280 | the signed-in member) and `community.comment.moderate` (hide and show a story's thread). |
| 281 | Without them a story page says comments are unavailable; nothing else depends on Community. |
| 282 | - **OpenVibe.Search**: consumes `news.index_document.*` through its `*.index_document.*` |
| 283 | subscription; `news` must be in `SEARCH_EVENT_OWNERS`. |
| 284 | - **Optional:** OpenVibe.AI (4700) with `ai.run.create` for namespace `news.*` (a bare `news` only matches a workflow literally named `news`). |
| 285 | |
| 286 | ### Grants the Network must hold |
| 287 | |
| 288 | Each grant is `[client, capability, audience]`: |
| 289 | |
| 290 | - `[news, sources.item.read, openvibe.sources]` |
| 291 | - `[news, sources.source.read, openvibe.sources]` (optional: outlet names and health) |
| 292 | - `[news, events.event.publish, openvibe.events]` |
| 293 | - `[news, events.subscription.manage, openvibe.events]` |
| 294 | - `[news, identity.subject.resolve, openvibe.network]` |
| 295 | - `[news, community.comment.write, openvibe.community]` (comment threads on stories) |
| 296 | - `[news, community.comment.moderate, openvibe.community]` (hide the thread of a retracted, |
| 297 | unpublished or source-removed story) |
| 298 | - `[news, ai.run.create, openvibe.ai]`, namespace `news.*` (optional: only with `OV_AI_INTERNAL_URL`) |
| 299 | - For OpenVibe.AI to deliver drafts: `[ai, news.story.revise, openvibe.news]`, and |
| 300 | `[ai, news.story.read, openvibe.news]` if it reads stories back. |
| 301 | |
| 302 | ## Acceptance (automated: `npm test`) |
| 303 | |
| 304 | | Charter / roadmap requirement | Test | |
| 305 | |---|---| |
| 306 | | Every published claim traces to source records: each paragraph's numbers resolve to Sources items in the page, the JSON, the citations table and the Search document's provenance. | `test/stories.test.js` | |
| 307 | | A story with zero sources cannot be published; an uncited paragraph is refused. | `test/stories.test.js` | |
| 308 | | Failed ingestion never fabricates: failed pulls, unreadable deliveries, `sources.fetch.failed` and title-less items are recorded and create no item, cluster or story text. | `test/ingest.test.js` | |
| 309 | | A source correction or removal triggers a flag and a pending revision; published text is unchanged; a removed source leaves the page; publishing a revision that rests on it is refused; the editor's revision resolves the flags with a note. | `test/upstream.test.js` | |
| 310 | | Community discussion is referenced, not duplicated: one thread per published story, only its id stored, read on render; Community down says comments are unavailable; drafts get no thread; retraction, unpublishing and a removed source hide it after the commit; members comment by form as themselves. | `test/discussion.test.js` | |
| 311 | | Licensed material never leaks: bodies and summaries beyond the allowance never reach the database, pages, JSON, feeds, sitemaps, the API, events or Search. | `test/licensing.test.js` | |
| 312 | | Dedupe by canonical URL, content hash and near-duplicate headline; idempotent replays. | `test/ingest.test.js` | |
| 313 | | Clusters are deterministic and explained; merges and splits are audited and reversible. | `test/clusters.test.js` | |
| 314 | | The index gate counts independent sources: a single-source story, two items of one Sources source, two sources of one publisher's domain and a report with its syndicated copy are published but noindex and out of the sitemap; two independent sources are indexable; `NEWS_MIN_INDEPENDENT_SOURCES=1` restores the old gate. | `test/indexability.test.js` | |
| 315 | | Retraction: visible at its URL with the notice, noindex with its reason, out of sitemaps and Search, event emitted, final. | `test/retraction.test.js` | |
| 316 | | Useful without JavaScript: the whole editorial journey with forms, and public pages complete in HTML. | `test/nojs.test.js` | |
| 317 | | AI output is a draft that needs a person's review (AI-assisted too); uncited claims dropped; failure makes no text. | `test/ai.test.js` | |
| 318 | | The contract proposals are valid and match the code; every emitted event type is declared. | `test/contracts.test.js` | |
| 319 | | Health, readiness, release, robots, llms.txt, llms-full.txt, sitemaps, legal, problems, CORS, topics-only seed. | `test/ops.test.js` | |
| 320 | |
| 321 | Demonstrated in production on 2026-10-11: ingestion from the deployed OpenVibe.Sources (70 items, 18 |
| 322 | clusters). Not yet demonstrated: a published story (it waits on an editor), delivery of a real item |
| 323 | through the running OpenVibe.Events (the `sources.item.*` and `sources.fetch.failed` subscriptions |
| 324 | exist), and an OpenVibe.AI run (OpenVibe.AI runs on the host, but News in production has no |
| 325 | `OV_AI_INTERNAL_URL`; the seam is tested against a mock of its runs API). |
| 326 | |
| 327 | ## Launch rule |
| 328 | |
| 329 | This repository alone doesn't make the product live: the launch is a coordinated change across |
| 330 | OpenVibe.Sites, routing and the ecosystem registry. All of plan §12.12 was met on 2026-10-09, and |
| 331 | the service is now public at `openvibe.news`: |
| 332 | |
| 333 | 1. **Runtime, health, readiness, observability:** done. |
| 334 | 2. **Canonical identity and auth:** done (Network SSO, subjects, service tokens). |
| 335 | 3. **SSR public routes useful without JS:** done. |
| 336 | 4. **Persistence and end-to-end workflows:** done in tests and deployed on the host; since |
| 337 | 2026-10-11 against live Sources items (70 items, 18 clusters). |
| 338 | 5. **Capability and event registration against OpenVibe.Contracts:** done (openvibe-contracts |
| 339 | v0.97.0). |
| 340 | 6. **Migration and seed strategy, threat review, sitemap/robots/feed behaviour:** done. Nothing |
| 341 | to migrate; the seed is topics only; the threat review is below. |
| 342 | 7. **Acceptance tests:** done. |
| 343 | |
| 344 | The launch release removed `openvibe.news` from `OpenVibe.Sites/sites.json`, switched routing |
| 345 | (nginx vhost, DNS, TLS), flipped the Network hub entry and registered maturity in the ecosystem |
| 346 | registry, together. |
| 347 | |
| 348 | ## Security and threat review |
| 349 | |
| 350 | Reporting a vulnerability: [SECURITY.md](SECURITY.md). |
| 351 | |
| 352 | - **Identity:** only verified Network JWTs (offline RS256) and service tokens for audience |
| 353 | `openvibe.news`. A bad service token is refused, never downgraded. Identity never comes from a |
| 354 | body or query; `X-OV-*` headers are ignored for browsers. AI deliveries can only write drafts. |
| 355 | - **Webhook:** signature v2 only: HMAC-SHA256 over `<timestamp>.<raw body>` with |
| 356 | `NEWS_EVENTS_SECRET` (rotation: comma-separated), timestamp within 300 s, constant-time |
| 357 | comparison; a v1-only or stale delivery is refused. `evt_` ids, exactly-once inbox. Only events from source `sources` are |
| 358 | applied. nginx never proxies `/internal/`. |
| 359 | - **CSRF:** SameSite=Lax session cookie plus an HMAC form token on every desk form. |
| 360 | - **XSS:** all HTML goes through `openvibe-publishing/ssr` auto-escaping; source links get |
| 361 | `rel="noopener nofollow"`; helmet CSP. |
| 362 | - **SSRF:** News makes no outbound request to a URL from content. It calls only its configured |
| 363 | Network, Sources, Events, Community and AI hosts. |
| 364 | - **Licensing and privacy:** see "Licensing" above; removed sources are masked everywhere public. |
| 365 | The editor view shows removal reasons; the public view does not. |
| 366 | - **Fabrication:** no seeded or generated stories, ratings or dates; dates come from sources or an |
| 367 | editor citing one; empty feeds use a real time; JSON-LD omits unknown fields. |
| 368 | - **Abuse:** rate limits on `/auth`, `/edit`, `/clusters`, `/api/v1` and the comment form, in |
| 369 | Express and in the nginx reference, and per-actor limits (below). |
| 370 | - **Comments:** Community owns the text and its moderation; News renders it through the same |
| 371 | auto-escaping and stores only the thread id. The comment form needs a signed-in member and the |
| 372 | form token, is rate-limited, and is refused for a story that is not open for discussion. |
| 373 | - **Known gaps:** no per-story Media attachments; the Sources registry name is cached and not |
| 374 | refreshed after the first successful read; the Network does not hold News' two Community grants |
| 375 | yet, so story pages say comments are unavailable until they are added. |
| 376 | |
| 377 | ### Per-actor limits |
| 378 | |
| 379 | `/api/v1`, the editor desk and the comment form also limit who calls them, once `req.viewer` is |
| 380 | resolved and a route's capability guard passed, before any work (for a write, before its body is |
| 381 | read): `server/http/actor-limits.js`, openvibe-sdk/limits, roadmap WS-R task 4. Counted: a person as |
| 382 | `user:usr_…` (their own token or cookie, a first-party service naming them in `X-OV-Subject`, or an |
| 383 | app's `on_behalf_of`); a service or app acting as itself by its principal; anyone else by address. |
| 384 | Signed-out reads keep only the per-address limit (many readers share a carrier or campus address), |
| 385 | and a first-party service reading for itself is not counted on reads. Past a limit: `429` |
| 386 | problem+json `rate_limited` with `Retry-After`, one log line and `news_rate_limited_total{limit,window}`. |
| 387 | A desk form and the API route that do the same thing share one budget. |
| 388 | |
| 389 | | Routes (API and desk form) | Per caller, a minute / an hour | |
| 390 | |---|---| |
| 391 | | Signed-in API reads, desk pages | `NEWS_LIMITS_MINUTE` / `NEWS_LIMITS_HOUR` (120 / 3000) | |
| 392 | | Topic create | 10 / 60 | |
| 393 | | Story create (`POST /stories`, `/clusters/:id/stories`) | 30 / 300 | |
| 394 | | Revisions | 30 / 600 | |
| 395 | | AI drafts (`POST /stories/:id/ai-drafts`, the desk's AI button) | 5 / 60 | |
| 396 | | Publish, unpublish, retract, reviews | 30 / 300 | |
| 397 | | Attach and detach a source | 30 / 300 | |
| 398 | | Perspectives, timeline, flags | 60 / 600 | |
| 399 | | Cluster merge, split, reverse | 30 / 300 | |
| 400 | | Pull now (`POST /edit/pull`) | 2 / 20 | |
| 401 | | Comment (`POST /stories/:slug/comments`, signed-in) | 20 / 300 | |
| 402 | | Diffs (API and desk) | 30 / 600 | |
| 403 | |
| 404 | Never limited per actor: `/api/health`, `/api/ready`, `/release.json`, `/metrics`, sign-in, the public |
| 405 | pages and feeds, and the signed Events deliveries at `/internal/events`. `test/actor-limits.test.js`. |
| 406 | |
| 407 | ## Development |
| 408 | |
| 409 | ```bash |
| 410 | fnm exec --using=22.22.1 npm install |
| 411 | fnm exec --using=22.22.1 npm test # temp databases and in-process mocks, no network |
| 412 | fnm exec --using=22.22.1 npm run dev # http://localhost:4820 (set OV_OAUTH_CLIENT_SECRET to sign in) |
| 413 | ``` |
| 414 | |
| 415 | ## Deploy (for the lead) |
| 416 | |
| 417 | Production deploys with `sudo ovhost deploy news` on the host (strategy `git-checkout`: fetch, |
| 418 | fast-forward `/opt/openvibe.news`, install on a lockfile change, restart, wait for `/api/ready`). |
| 419 | The unit is `openvibe-news.service` on `127.0.0.1:4820`, the env file `/etc/openvibe/news.env`. The database is |
| 420 | `ov_news` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh news` writes its settings); the |
| 421 | release migrates it at boot. |
| 422 | Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer 2xx after the |
| 423 | restart; afterwards `sudo ovhost rollback news --to <sha>`. Migrations only add tables and columns. |
| 424 | |
| 425 | First install (done once; kept for a rebuild): |
| 426 | |
| 427 | 1. **Code and config:** put the code at `/opt/openvibe.news` and run `npm ci --omit=dev` on Node 22. |
| 428 | Create `/etc/openvibe/news.env` (0600) from `.env.example` with `OV_OAUTH_CLIENT_SECRET`, |
| 429 | `NEWS_EDITORS`, `NEWS_FORM_SECRET`, `NEWS_EVENTS_SECRET` (`openssl rand -hex 32`), |
| 430 | `EVENTS_URL=http://127.0.0.1:4300`, `OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720`, |
| 431 | `BASE_URL=https://openvibe.news`, `OV_COMMUNITY_INTERNAL_URL=http://127.0.0.1:4200`, and |
| 432 | optionally `OV_AI_INTERNAL_URL` and `INDEXNOW_KEY`. |
| 433 | 2. **Network:** create (or give a secret to) the OAuth client `news` with redirect |
| 434 | `https://openvibe.news/auth/callback`, and add the grants listed above. |
| 435 | 3. **Search:** make sure `news` is in `SEARCH_EVENT_OWNERS`. |
| 436 | 4. **systemd:** install `deploy/systemd/openvibe-news.service` (port 4820). |
| 437 | 5. **nginx:** install `deploy/nginx/openvibe.news.conf`. `/metrics` and `/internal/` are never proxied. |
| 438 | 6. **Seed and subscribe:** `npm run seed` (topics only; also done on boot), then |
| 439 | `npm run subscribe` (creates the `sources.item.*` and `sources.fetch.failed` subscriptions to |
| 440 | `http://127.0.0.1:4820/internal/events`). The cursor pull starts on its own. |
| 441 | 7. **Sources:** News shows nothing until a news source is enabled in OpenVibe.Sources (terms |
| 442 | verified by a person) and an editor publishes a story. |
| 443 | 8. **Contracts:** done: the capabilities and manifest are released in openvibe-contracts v0.97.0, |
| 444 | and CI's contracts check runs against them. |
| 445 | 9. **Launch:** done on 2026-10-09: `openvibe.news` was removed from OpenVibe.Sites, routing switched |
| 446 | and the Network hub entry flipped (see the launch rule above). |
| 447 | |
| 448 | --- |
| 449 | |
| 450 | Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers). |
| 451 | |
| 452 | <!-- versions:start --> |
| 453 | - openvibe-contracts: v0.419.0 |
| 454 | - openvibe-sdk: v0.161.0 |
| 455 | - openvibe-shared: v4.241.0 |
| 456 | - openvibe-publishing: v1.7.0 |
| 457 | <!-- versions:end --> |
| 458 |