openvibers/openvibe.news AGENTS.md

AGENTS.md

main 11.8 KB · 159 lines · markdown

Raw HistoryBlame Repository
1# AGENTS.md — OpenVibe.News
2
3## Project Overview
4
5OpenVibe.News is the network's source-backed news product: it ingests items from OpenVibe.Sources, dedupes and clusters
6them deterministically, and lets editors open a story from a cluster, write cited paragraphs, attach a timeline and
7perspectives, publish revisions, and correct or retract them in public. Beside editor-written stories it has AI briefs,
8tone/perspective comparisons scheduled by readers, and `/live` breaking coverage. A story is indexable only through
9an explicit gate.
10
11- **Domain:** `openvibe.news` · **Port:** 4820 · **Service id:** `news`
12- **Runtime:** Node 22, Express 4, PostgreSQL 18 through `openvibe-sdk/db` (PGlite in development/tests), Valkey 9
13 optional for shared limits.
14- **Read first:** [README.md](README.md) (long and authoritative), [STATUS.json](STATUS.json), [.env.example](.env.example).
15
16**Network context:** identity and editor names are OpenVibe.Network (OAuth client `news`, PKCE). Source items are
17OpenVibe.Sources (`sources.item.read`; optionally `sources.source.read`); News **never fetches a site itself**. Story
18comments are OpenVibe.Community threads, referenced by id and read on every render — never copied. Drafts come from
19OpenVibe.AI (`ai.run.create`, optional). Search consumes `news.index_document.*` through Events. Events delivery is
20OpenVibe.Events.
21
22## Commands
23
24```bash
25npm ci # install (contracts/sdk/shared/publishing pinned by tag)
26npm run dev # http://localhost:4820 (OV_OAUTH_CLIENT_SECRET to sign in)
27npm start # node server/index.js
28npm test # every test/*.test.js on temp databases with in-process mocks, no network
29npm run test:pg # the same through PostgreSQL + PgBouncer
30npm run seed # topics only (also done on boot)
31npm run subscribe # creates the sources.item.* / sources.fetch.failed subscriptions
32npm test -- stories # only test files whose name contains the word
33node --check <file.js> # syntax check (no linter configured)
34```
35
36No build step: pages are server-rendered; CSS only under [public/css/](public/css/) (`news.css`, `briefs.css`). Every
37page and form works without JavaScript.
38
39## Architecture at a Glance
40
41- **Entry:** [server/index.js](server/index.js) — listener, outbox relay, Sources pull worker, telemetry and graceful
42 stop. `NEWS_DRILL` keeps boot-time work off.
43- **App factory:** [server/app.js](server/app.js) — CSP, SSO/session, `/auth`, `/internal/events`, static/`/shared`,
44 public pages + editor desk + API + discovery, 404/error handler.
45- **Config:** [server/config.js](server/config.js) — `load(env)` is pure; production `/etc/openvibe/news.env`.
46- **Domain:** [server/domain/](server/domain/) — `source-items.js` (ingest, dedupe), `clusters.js`, `stories.js`,
47 `publication.js` (publish/correct/retract), `briefs.js`, `home.js`, `reading.js`, `discussion.js`, `topics.js`,
48 `review-kind.js`, `access.js`, `topic-match.js`; ingest worker [server/worker.js](server/worker.js).
49- **HTTP:** [server/http/public.js](server/http/public.js) (public pages), [server/http/editor.js](server/http/editor.js)
50 (the desk), [server/http/api.js](server/http/api.js), [server/http/discovery.js](server/http/discovery.js),
51 [server/http/events.js](server/http/events.js), [server/http/actor-limits.js](server/http/actor-limits.js).
52- **Clients:** [server/clients/](server/clients/) — `ai.js`, `community.js`, `network.js` (service tokens; no outbound
53 fetch from a page or a user URL).
54- **Auth/forms:** [server/auth/viewer.js](server/auth/viewer.js), [server/auth/capabilities.js](server/auth/capabilities.js)
55 (delegates to `openvibe-contracts` capabilities), [server/auth/forms.js](server/auth/forms.js) (CSRF form tokens).
56- **Rendering:** [server/render/](server/render/) — `layout.js`, `pages.js`, `editor.js`, `home.js`, `briefs.js`,
57 `short.js`; Markdown twins via [server/agent-pages.js](server/agent-pages.js), [server/agent-descriptor.js](server/agent-descriptor.js).
58- **Database:** [server/db.js](server/db.js) and [migrations/](migrations/). Core: `news_topics`, `news_source_items`,
59 `news_story_clusters`, `news_stories`, `news_story_revisions`, `news_story_sources`, `news_perspectives`,
60 `news_timeline_entries`, `news_editorial_flags`, `news_story_citations`, `news_story_reviews`, `news_story_drafts`,
61 `news_story_revision_purges`, `news_index_revisions`, `news_story_discussion_refs`, `news_cluster_audit`,
62 `news_ingest_runs`, `news_ingest_cursor`, `news_source_status`, `news_cluster_briefs`, `news_brief_reviews`,
63 `news_share_defaults`, sharing tables, `event_outbox`, `idempotency_receipts`, `subject_projections`,
64 `account_data_events`, `telemetry_samples`.
65- **Deploy:** [deploy/systemd/openvibe-news.service](deploy/systemd/openvibe-news.service) (127.0.0.1:4820,
66 `/etc/openvibe/news.env`) and [deploy/nginx/openvibe.news.conf](deploy/nginx/openvibe.news.conf). `sudo ovhost deploy
67 news` (git-checkout at `/opt/openvibe.news`, database `ov_news`; migrations only add tables/columns). Run `npm run
68 seed` and `npm run subscribe` on first install; `/internal/` and `/metrics` are never proxied.
69
70## Ownership and Boundaries
71
72News owns its source-item copies, clusters and their audit, stories, immutable revisions, the source table,
73perspectives, timeline, editorial flags, briefs/reviews, the index sequencer, cursor/run/status rows and share rows. It
74does **not** own source adapters or the registry (Sources), comment threads (Community), AI generation (AI), the search
75index (Search), accounts (Network) or event delivery (Events).
76
77Licensing is a hard rule: keep only headline, canonical URL, outlet, authors, `published_at`, the licence/terms notes
78and — only when those notes allow short summaries — a summary of at most `NEWS_SUMMARY_MAX_CHARS` (280). Never read or
79store an article body, and never show a source summary in feeds, Search documents or events. A removed item's summary
80is dropped at once.
81
82## Auth and Capabilities
83
84- Editors are browser/app users whose subject is in `NEWS_EDITORS` or who are Network admins; the editor desk is
85 private and noindex. Service tokens use audience `openvibe.news`, one capability per route, and put the editor they
86 act for in `X-OV-Subject`.
87- Capabilities: `news.story.create|read|revise|publish|retract`, `news.cluster.read|manage`,
88 `news.source.attach`, `news.timeline.update`, `news.perspective.update`, `news.topic.manage`,
89 `news.review.ai_editor`. Match grants with `openvibe-contracts`' capability rule, not a hand-rolled string check.
90- Outbound: `sources.item.read` (and optionally `sources.source.read`), `events.event.publish`,
91 `events.subscription.manage`, `identity.subject.resolve`, `community.comment.write|moderate`,
92 `ai.run.create` (namespace `news.*`, optional).
93- Form posts carry `NEWS_FORM_SECRET` tokens; cookie writes are same-origin. Never bypass the form-token check.
94- Errors are structured with stable codes (`story.unsourced`, `story.stale`, upstream 503 for Events to retry, …).
95
96## Pages, Feeds and API
97
98Pages: `/`, `/topics`, `/topics/:slug`, `/stories/:slug` (+ `.json`), `/briefs`, `/briefs/:cluster`, `/live`,
99`/edit`, `/edit/stories/:id`, `/clusters/:id`, `/admin/review`, `/auth/*`.
100Discovery: `/feed.xml`, `/atom.xml`, `/feed.json` (and per-topic variants), `/sitemap.xml` →
101`/sitemaps/stories.xml`, `/sitemaps/topics.xml`, `/sitemaps/briefs.xml`, `/robots.txt`, `/llms.txt`, `/llms-full.txt`.
102`/internal/events` is loopback; `/api/health`, `/api/ready`, `/release.json`, `/metrics` are machine endpoints.
103
104## Conventions
105
106- CommonJS, 4-space indent, single quotes, semicolons; no linter/formatter. Async access through `openvibe-sdk/db`.
107- Revisions are immutable and use optimistic concurrency (412 on a stale base). Source numbers `[n]` are never reused;
108 a paragraph must cite numbers present in the source table.
109- Publishing is refused when a revision has no sources, cites a detached/removed source, or is AI-generated/assisted
110 without a person's review. Retraction is final: keep the URL, show the notice, noindex, drop from sitemaps/Search.
111- Indexability is the `openvibe-publishing/seo` gate with explicit reasons: at least `NEWS_MIN_INDEPENDENT_SOURCES`
112 (default 2) live independent sources and at least 40 words; unsupported claims, retraction, an editor's noindex and
113 unreviewed AI output are never indexable. Independent sources collapse shared source keys, publisher domains,
114 outlets and duplicates; the relation is transitive.
115- AI briefs label themselves as AI on every page, cite paragraphs and drop uncited claims; a brief is noindex until a
116 person reviews it and it has enough independent outlets, and a remake after the cluster changes waits again.
117- Caching: published anonymous pages `public, max-age=60`; signed-in/desk/preview/refusal/API `private, no-store` with
118 `X-Robots-Tag`; feeds/sitemaps five minutes, never per viewer. HTML/JSON vary on `Cookie` and `Authorization`.
119- JSON-LD uses real fields only; missing ones are omitted. IndexNow is off without `INDEXNOW_KEY` and never pings a
120 draft/private/noindex page.
121- CSP: add origins to `helmet` in [server/app.js](server/app.js); no inline script beyond what the policy allows.
122- Phone first: public pages fit 390 px; forms work without JS. No Font Awesome; use the shared glyphs.
123- "Calm reduces, only Off removes": previews/teasers reduce, only the explicit retraction/unpublish path removes.
124
125## Key Pitfalls
126
127- **Never fetch a source site.** Ingest only through Sources (webhook or cursor pull); an item apply is idempotent on
128 (Sources item id, revision). Failure is a recorded state and a 503 to Events, never a made-up item.
129- **Dedupe and clustering are deterministic and stored** — same normalised URL/content hash/near-duplicate headline;
130 clusters by shared entities/terms in a 72 h window. Do not replace either with an AI guess; merges/splits must be
131 audited and reversible.
132- **Upstream change:** a cited source revised/removed opens a flag and a pending same-text revision; the published
133 revision stays as it was, the page carries a notice, a removed source's headline/summary/link leave at once, and
134 the story becomes noindex `unsupported_claims` until an editor publishes the fix.
135- **Editor desk is private/noindex**; a form token is required and the same limits apply as the API.
136- **AI**: the stub provider's output must never be shown; claims the model did not cite are dropped and listed. Briefs
137 and `/live` stay noindex by default.
138- **Drill mode** (`NEWS_DRILL`) keeps the worker, outbox relay, brief timer and subscriptions off; add new timers.
139- **Account deletion** removes personal rows/receipts through `/internal/events`; redelivery changes nothing.
140- **Migration 0010** adds the `event_outbox` rejected partial index; keep it.
141
142## Testing
143
144`npm test` runs each file in its own process on temp databases with mock Sources/Events/Community/AI. The suite covers
145ingest/idempotency/licensing/dedupe/clusters, story revisions/publish/retract/upstream change, briefs and reviews,
146indexability/sitemaps/llms/JSON-LD/IndexNow, comments, caching/no-JS, capabilities/contracts, SSO, links, CSP, perf
147budgets, sharing and account data. `npm run test:pg` for PostgreSQL. Run the narrowest file first, then the suite;
148always `node --check` changed files.
149
150## Documentation
151
152- [README.md](README.md) — purpose, AI briefs/Live, ownership, what works, routes, discoverability, caching, events,
153 capabilities/grants, acceptance, launch rule, security, limits, deploy.
154- [docs/capabilities-proposal/](docs/capabilities-proposal/) and
155 [docs/service-manifest-proposal.json](docs/service-manifest-proposal.json) — the released capability/manifest
156 proposals.
157- [seeds/topics.json](seeds/topics.json) — the only topic seed.
158- [STATUS.json](STATUS.json), [.env.example](.env.example), [CHANGELOG.md](CHANGELOG.md), [SECURITY.md](SECURITY.md).
159