openvibers/openvibe.report Commits
Report as the analytics authority, expand and shadow (REP-2) (#4)
Changes
@@@ -6,13 +6,43 @@ HOST=127.0.0.1BASE_URL=https://openvibe.reportTRUST_PROXY=2--# OpenVibe.Network's open stats (GET /api/v1/stats?days=7|30|90). Both sources serve these read APIs on loopback;--# the defaults are the internalOrigin values in openvibe-contracts' manifests/services/{network,live}.json.++# Report's own PostgreSQL (ADR-035). The runtime role serves through PgBouncer on DATABASE_URL; migrations run++# on DATABASE_DIRECT_URL as the owner. Without DATABASE_URL development uses an embedded PGlite database.++DATABASE_URL=++DATABASE_DIRECT_URL=++#REPORT_PGLITE_DIR=./data/pglite++++# Which source the pages and GET /api/v1/stats answer from: network (today's behaviour) or report (after cutover).++REPORT_SOURCE=network++++# OpenVibe.Network's open stats (GET /api/v1/stats?days=7|30|90). Both sources serve their read APIs on loopback.OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000++# The public origin, used only for the OAuth token endpoint (Report's service token to Billing).++OV_NETWORK_URL=https://openvibe.network# OpenVibe.Live's Rundown and after-show reports (GET /api/rundown/*, /api/recap/*).OV_LIVE_INTERNAL_URL=http://127.0.0.1:3000++# OpenVibe.Billing's read API. Report quotes Billing and never computes money. The service needs the grants++# billing.stats.read (GET /api/v1/stats) and billing.balance.read (GET /api/v1/transactions?subject=…).++OV_BILLING_INTERNAL_URL=http://127.0.0.1:4600++# Report's Network OAuth client (client_credentials → audience openvibe.billing). The secret stays in the env file.++OV_OAUTH_CLIENT_ID=report++OV_OAUTH_CLIENT_SECRET=++# An explicit Billing bearer token for an operator run (instead of the OAuth client); never commit it.++#REPORT_BILLING_TOKEN=++++# OpenVibe.Events → Report (POST /internal/events): live.stream.ended and the frame beacon. Comma-separated for++# rotation, 32+ characters each; unset makes the route answer 503.++REPORT_EVENTS_SECRET=++++# The ingest and shadow jobs (on by default; REPORT_JOBS=0 turns both off).++#REPORT_JOBS=1++#REPORT_JOBS_ON_BOOT=1++#REPORT_INGEST_INTERVAL_MS=300000++#REPORT_SHADOW_WINDOW_DAYS=7++#REPORT_SHADOW_INTERVAL_MS=900000++# Optional: the per-address API limit (default 120 a minute) and the readiness probe timeout (default 2500 ms).#REPORT_API_RATE_LIMIT_PER_MIN=120#REPORT_READY_TIMEOUT_MS=2500@@@ -8,4 +8,5 @@ node_modules/*.pem*.keycoverage/++/data/.DS_Store@@@ -2,8 +2,9 @@> Every number on OpenVibe, honest.--**Status:** v0.1.0 service skeleton (see [STATUS.json](STATUS.json)). The routes, the API and the plumbing are--complete; the views are plain placeholders the main session will design.++**Status:** v0.1.0 service, now with its own analytics database and a shadow of Network's numbers (REP-2, plan++2026-10-28 §4.8 and §5.4). The pages still read where they read today; `REPORT_SOURCE=report` switches the numbers++when the cutover is approved.**Domain:** `openvibe.report` · **Port:** 5080 · **Service id:** `report` · **Env prefix:** `OV_` / `REPORT_`**License:** AGPL-3.0 (same as every OpenVibe service).@@@ -12,29 +13,54 @@ about, and each stream's after-show report.## Purpose--OpenVibe.Report is a stateless read-only front. It has no database, no sign-in and no events. It asks two loopback--APIs for their public answers, holds each answer in this process for a few minutes, and renders it:++OpenVibe.Report is the analytics authority in shadow:++++- It **ingests** the analytics OpenVibe.Network records — `live.stream.ended` for creator analytics and the shared++ navbar's anonymous frame beacon — and Network's open-stats answer (`GET /api/v1/stats?days=7|30|90`) into its own++ PostgreSQL tables (migrations/).++- It **computes** the same numbers Network answers, site-wide, per site and per channel, by UTC day.++- It **quotes money from OpenVibe.Billing** through Billing's read API; it never derives a price, fee or amount.++- It **shadows**: a job compares the same window from Network and from Report's tables and records every difference++ in `report_shadow_diffs`.++++The pages and `GET /api/v1/stats` answer from `REPORT_SOURCE`, which defaults to `network`: today's behaviour. Setting++`REPORT_SOURCE=report` makes them answer from Report's tables. The Rundown and after-show reports still come from++OpenVibe.Live in both modes.| Source | Read | Used for ||---|---|---|--| **OpenVibe.Network** | `GET /api/v1/stats?days=7\|30\|90` | `/`, `/stats`, `GET /api/v1/stats` |++| **OpenVibe.Network** | `GET /api/v1/stats?days=7\|30\|90` | `/`, `/stats`, `GET /api/v1/stats` while `REPORT_SOURCE=network`; also the ingest source |++| **Report's tables** | `report_metric_days`, `report_metric_windows`, `report_streams` | the same routes while `REPORT_SOURCE=report` || **OpenVibe.Live** — The Rundown | `GET /api/rundown/`, `/api/rundown/topics`, `/api/rundown/t/:slug`, `/api/rundown/e/:streamId`, `/api/rundown/s/:username` | `/rundown`, `/rundown/t/:slug`, `/rundown/e/:streamId`, `/rundown/s/:username` || **OpenVibe.Live** — after-show reports | `GET /api/recap/:streamId`, `GET /api/recap/channel/:username` | `/after-show/:streamId`, `/c/:username` |++| **OpenVibe.Billing** | `GET /api/v1/stats`, `GET /api/v1/transactions?subject=…` | the revenue read for a channel; network money counts |A source that does not answer is null, after one short log line; the page (or API route) becomes a 404. The servicenever invents a zero and never throws a source error at the reader.++## What Report owns now++++- **The analytics tables** in `migrations/0001_initial.sql`: `report_inbox` (event ids already handled),++ `report_events`, `report_streams`, `report_frame_days`, `report_metric_days`, `report_metric_windows`,++ `report_metric_json`, `report_dimensions`, `report_shadow_runs` and `report_shadow_diffs`.++- **The computation** in `server/analytics/`: `ingest.js` (idempotent writes), `query.js` (Network's stats shape and++ a channel's creator analytics), `shadow.js` (the comparison).++- **The jobs** in `server/jobs/analytics.js`: ingest every five minutes (7, 30 and 90-day windows) and shadow every++ fifteen minutes (7 days by default).++## Does not own--- **The numbers** — OpenVibe.Network's open stats. Report quotes them.++- **Money** — OpenVibe.Billing. Report reads it with `billing.stats.read` and `billing.balance.read`; it holds no++ ledger and computes no price.- **The Rundown and the after-show reports** — OpenVibe.Live. Report quotes them, including their counts and text.--- **Identity, sessions and writes** — Report has none. It does not read cookies and has no POST routes.++- **Identity, sessions and writes** — Report has no sign-in, no cookies and no public write route. `POST++ /internal/events` is loopback-only and HMAC-verified.- **The Shared Frame** — `openvibe-shared` (navbar, footer, theme-loader, static files), served from this site's ownpinned copy at `/shared/*`.## Pages--All server-rendered placeholders in `server/views/index.js`; every dynamic value goes through `esc()`.++All server-rendered in `server/views/index.js`; every dynamic value goes through `esc()`.| Page | What it shows ||---|---|@@@ -52,11 +78,10 @@ status 404. Pages are `Cache-Control: public, max-age=120`.## API--All under `/api/v1`, CORS `*`, `Cache-Control: public, max-age=120`, 120 requests a minute per address, and the--source's answer as it came:++All under `/api/v1`, CORS `*`, `Cache-Control: public, max-age=120`, 120 requests a minute per address:```--GET /api/v1/stats?days=7|30|90++GET /api/v1/stats?days=7|30|90 REPORT_SOURCE=network|reportGET /api/v1/rundownGET /api/v1/rundown/topicsGET /api/v1/rundown/t/:slug@@@ -65,14 +90,32 @@ GET /api/v1/c/:username```A bad parameter or a missing source answer is a 404 `application/problem+json` body built with--`openvibe-contracts` `http.sendProblem`. The API is read-only; other methods get 405.++`openvibe-contracts` `http.sendProblem`. The public API is read-only; other methods get 405.--## Crawl artifacts++### Internal (loopback only)--- `/robots.txt` — the public pages are crawlable; `/api/` is not; the sitemap is named.--- `/sitemap.xml` — `/`, `/stats`, `/rundown` and every `/rundown/t/<slug>` from the Rundown's topics (the three-- static pages remain when the topics source is down). `lastmod` comes from `STATUS.json`.--- `/llms.txt` — the site, its pages and its machine-readable answers.++```++POST /internal/events OpenVibe.Events delivery (v2 HMAC, REPORT_EVENTS_SECRET)++GET /internal/shadow?days=7&limit=5 the latest shadow runs and their differences++POST /internal/shadow?days=7 run a comparison now++GET /internal/metrics?scope=&id=&days= the flat report.metric@1 rows++GET /internal/channels/:channel/analytics a channel's creator analytics plus Billing revenue++```++++nginx refuses `/internal/` from outside; the app also checks the socket is loopback. The flat rows are proposed as++`report.metric@1` in [docs/report.metric.v1.json](docs/report.metric.v1.json) for Contracts to own.++++## Ingest and idempotency++++- `live.stream.ended` → one `report_streams` row (same fields as Network's `creator_streams`) and the channel/day++ counts. A redelivered event id is a no-op; a corrected copy with a new event id replaces the stream and does not++ count it twice.++- `network.frame.hit` / `network.frame.day` → `report_frame_days` (day, host, hits).++- `GET /api/v1/stats` → `report_metric_days`, `report_metric_windows`, `report_metric_json` and site dimensions,++ keyed by the answer's content hash. Polling the same numbers again is a no-op.++- Every delivery is claimed in `report_inbox` (openvibe-sdk/events `createPgInbox`) in the same transaction as its++ writes.++- Billing events are deliberately not ingested: money comes from Billing's read API.## Configuration@@@ -85,8 +128,22 @@ Every value comes from the environment (production: `/etc/openvibe/report.env`).| `HOST` | `127.0.0.1` | listen address || `BASE_URL` | `https://openvibe.report` in production, otherwise `http://localhost:5080` | canonical links, sitemap and llms.txt || `TRUST_PROXY` | `2` | Express trust proxy (the app sets no cookies, but the per-address limit uses the client address) |--| `OV_NETWORK_INTERNAL_URL` | `http://127.0.0.1:4000` | OpenVibe.Network's loopback origin (its manifest's `internalOrigin`) |--| `OV_LIVE_INTERNAL_URL` | `http://127.0.0.1:3000` | OpenVibe.Live's loopback origin (its manifest's `internalOrigin`) |++| `DATABASE_URL` | unset in development | Report's PostgreSQL through PgBouncer; without it development uses PGlite |++| `DATABASE_DIRECT_URL` | unset | the owner connection migrations run on |++| `REPORT_PGLITE_DIR` | `./data/pglite` | the embedded development database |++| `REPORT_SOURCE` | `network` | `network` keeps today's behaviour; `report` answers from Report's tables |++| `OV_NETWORK_INTERNAL_URL` | `http://127.0.0.1:4000` | Network's loopback origin (its manifest's `internalOrigin`) |++| `OV_NETWORK_URL` | `https://openvibe.network` | the OAuth token endpoint's origin |++| `OV_LIVE_INTERNAL_URL` | `http://127.0.0.1:3000` | Live's loopback origin |++| `OV_BILLING_INTERNAL_URL` | `http://127.0.0.1:4600` | Billing's loopback origin |++| `OV_OAUTH_CLIENT_ID` / `OV_OAUTH_CLIENT_SECRET` | `report` / unset | Report's client-credentials token for `openvibe.billing` |++| `REPORT_BILLING_TOKEN` | unset | an explicit Billing bearer token for an operator run |++| `REPORT_EVENTS_SECRET` | unset | the HMAC secret for `POST /internal/events` (32+ chars, comma list for rotation) |++| `REPORT_JOBS` | `1` | `0` turns the ingest and shadow jobs off |++| `REPORT_JOBS_ON_BOOT` | `1` | run both jobs once shortly after boot |++| `REPORT_INGEST_INTERVAL_MS` | `300000` | how often the ingest job runs |++| `REPORT_SHADOW_INTERVAL_MS` | `900000` | how often the shadow job runs |++| `REPORT_SHADOW_WINDOW_DAYS` | `7` | the shadow window (7, 30 or 90) || `REPORT_API_RATE_LIMIT_PER_MIN` | `120` | per-address API requests a minute || `REPORT_READY_TIMEOUT_MS` | `2500` | how long a readiness probe may take || `RELEASE_COMMIT` | unset | `/release.json`'s release; otherwise read from git in the checkout |@@@ -95,16 +152,30 @@ Every value comes from the environment (production: `/etc/openvibe/report.env`).```shnpm install--npm start # http://127.0.0.1:5080--npm test # every test/*.test.js, in its own process, with both sources stubbed++npm start # http://127.0.0.1:5080; development uses the embedded PGlite database++npm test # every test/*.test.js, in its own process```--The tests need no database, no network and no running OpenVibe site: `test/helpers/boot.js` supplies a URL-aware--stub for both sources. They cover the clients' cache hit, expiry, failure and parameter rules; every page route and--its 404s; the API shapes, CORS, cache and rate limit; robots/sitemap/llms; and health/readiness/static caching.++Most tests stub both sources and pass `store: null`, so they need no database. The analytics tests use++`test/helpers/db.js` (`createTestDb` from openvibe-sdk/testing) with a migrated PGlite, or the PostgreSQL containers++with `REPORT_TEST_STORE=pg`. They cover ingest idempotency, every metric against a fixture's expected numbers, the++shadow job's difference and clean run, `GET /internal/shadow`, the source flag, the Billing revenue read with a stub,++and the existing pages/API/health/crawl behaviour.`node --check` is run over `server/`, `test/` and `public/` by CI (`.github/workflows/ci.yml`, the shared--OpenVibe.Shared test workflow, no database service).++OpenVibe.Shared test workflow).++++## Cutover++++1. Run the ingest and shadow jobs long enough to cover a full 90-day window, and check++ `GET /internal/shadow` has zero differences (or only explained ones).++2. Create Report's first-party grants on Network: `billing.stats.read` and `billing.balance.read` for the `report`++ service (and the Network read grants when Report starts pulling creator analytics directly).++3. Set `REPORT_SOURCE=report` in `/etc/openvibe/report.env` and restart; keep the Network ingest running.++4. Leave Network writing during the dual-write window. Live's `ANALYTICS_SOURCE` can then move to Report; Network's++ `/stats` becomes a 301 to `openvibe.report/stats`.++5. After the N-1 window, a Contracts migration removes Network's analytics tables. Rollback before that is++ `REPORT_SOURCE=network` and a restart.## Deploy@@@ -114,15 +185,17 @@ The reference files are in `deploy/`:`EnvironmentFile=/etc/openvibe/report.env`, `PORT=5080`, `SIGTERM` with a 10 s drain.- `deploy/nginx/openvibe.report.conf` — HTTP → HTTPS, `www` → apex 301, Let's Encrypt certificates under`/etc/letsencrypt/live/openvibe.report/`, proxy to `127.0.0.1:5080`, the same security headers as Help's vhost,-- and `/api/` GET-only with a coarse rate limit.++ `/api/` GET-only with a coarse rate limit, and `/internal/` refused from outside.--`/api/ready` reports `network` and `live` as optional checks: one or both down is `status: "degraded"` with HTTP 200,--so a stateless front is never marked not-ready just because a source is unavailable.++`/api/ready` reports `network` and `live` as optional checks: one or both down is `status: "degraded"` with HTTP 200.++When `DATABASE_URL` is configured, the database check is included and is required only after `REPORT_SOURCE=report`.## SecurityEvery source text is escaped before it reaches HTML. Path parameters are checked against fixed patterns before any--request. Report holds no accounts, no cookies, no database and no write routes. See [SECURITY.md](SECURITY.md).++request. Report holds no accounts and no cookies. Its database has public channel facts and counts only; event++payloads with money or personal data are not stored. `/internal/` is loopback-only, and `POST /internal/events`++verifies the Events v2 HMAC. See [SECURITY.md](SECURITY.md).<!-- versions:start -->- openvibe-contracts: v0.225.0@@@ -16,9 +16,11 @@ Every OpenVibe service and site (openvibe.live, openvibe.network, openvibe.toolsopenvibe.games and the other openvibe.* domains) and every repository under github.com/OpenVibers. Machine-readablecontact details are at `/.well-known/security.txt` on each site.--For OpenVibe.Report specifically: the service reads two internal APIs and has no database, no accounts and no--writes. The interesting surface is the path parameters it forwards (validated before a request), the HTML it renders--(every source text is escaped by `server/views/index.js`) and the two source origins it is configured with.++For OpenVibe.Report specifically: the service reads Network, Live and Billing over loopback, keeps its own++analytics tables (counts and public channel facts only; no money payloads), and has no accounts. The interesting++surface is the path parameters it forwards (validated before a request), the HTML it renders (every source text is++escaped by `server/views/index.js`), the `/internal/` routes (loopback-only; `POST /internal/events` verifies the++Events v2 HMAC) and the origins and service token it is configured with. A Billing or Events secret is never logged.## Supported versions@@@ -1,11 +1,11 @@{"repository": "OpenVibers/OpenVibe.Report","version": "0.1.0",-- "stage": "skeleton",++ "stage": "analytics-authority-shadow","code": true,"wave": "W21",-- "plan": "OpenVibe.Report: a stateless front for the network's open numbers, The Rundown and every stream's after-show report",-- "runtime": "node 22, express 4, no database, no sessions, no events",++ "plan": "REP-2 (plan 2026-10-28 §4.8 and §5.4): Report's own analytics database, ingest, shadow comparison and the Billing revenue read; no cutover",++ "runtime": "node 22, express 4, PostgreSQL (PGlite in development), no sessions","port": "5080","domain": "openvibe.report","contracts": "openvibe-contracts v0.225.0",@@@ -14,38 +14,51 @@"packages": ["openvibe-contracts v0.225.0","openvibe-sdk v0.70.0",-- "openvibe-shared v4.33.0"++ "openvibe-shared v4.33.0",++ "pg v8.23.1",++ "pglite v0.5.13 (dev)"],-- "summary": "The Report room of the OpenVibe network. A stateless front with no database, no sign-in and no events: it reads OpenVibe.Network's open stats (GET /api/v1/stats?days=7|30|90) and OpenVibe.Live's Rundown (GET /api/rundown/, /topics, /t/:slug, /e/:streamId, /s/:username) and after-show reports (GET /api/recap/:streamId, /api/recap/channel/:username) over loopback, caches each answer for 2 to 10 minutes in this process, and shows it as the source answered it. Network and Live failures answer null and a page or API 404, never a made-up zero. The pages are plain placeholders the main session will design; the routes, the JSON API, the crawl artifacts, readiness (degraded, never 503, when a source is down) and the deploy files are complete.",++ "summary": "OpenVibe.Report keeps its own PostgreSQL analytics store (migrations/), ingests Network's live.stream.ended and frame beacon plus its open-stats answer idempotently (report_inbox), computes the same site-wide, per-site and per-channel numbers by day, quotes Billing's revenue through Billing's read API, and runs a shadow job that records every difference from Network in report_shadow_diffs with GET /internal/shadow on loopback. REPORT_SOURCE=network (default) keeps the pages on today's Network numbers; REPORT_SOURCE=report answers them from Report's tables. The Rundown and after-show pages still read OpenVibe.Live.","works": [-- "The three sources read through two clients with injectable fetch, 8 s timeouts and an in-memory cache (stats 5 min, Rundown home 2 min, topics 5 min, a topic 5 min, an episode 5 min, an after-show report 10 min, a channel's reports 5 min; 1000 entries, oldest out)",-- "Path parameters validated before a request: slug ^[a-z0-9-]{1,80}$, a positive integer stream id, username ^[A-Za-z0-9_]{1,32}$",-- "Pages: /, /stats?days=7|30|90, /rundown, /rundown/t/:slug, /rundown/e/:streamId, /rundown/s/:username, /after-show/:streamId, /c/:username; a bad parameter or a missing source answer is the 404 view",-- "JSON API: /api/v1/stats, /rundown, /rundown/topics, /rundown/t/:slug, /after-show/:streamId, /c/:username — the source's answer, CORS *, public for two minutes, 120 requests a minute per address, problem+json 404s",-- "Plumbing: helmet with Help's CSP, /api/health, /api/ready (Network and Live optional → degraded, never 503), /release.json, /release-metrics, /metrics on loopback, /shared/*, /css and /js (immutable only with the matching ?v=), robots.txt, sitemap.xml, llms.txt, graceful stop",-- "Deploy files: systemd unit openvibe-report on 127.0.0.1:5080 and an nginx vhost for openvibe.report + www (apex redirect, certs, proxy, GET-only API)",-- "Tests: the clients' cache, expiry, failure and validation; every page 200 and its 404s; the API shapes, CORS and cache; robots/sitemap/llms; health, readiness and static cache"++ "Database: server/db.js (openvibe-sdk/db, migrations at boot; PostgreSQL through PgBouncer, PGlite in development) and test/helpers/db.js (createTestDb)",++ "Tables: report_inbox, report_events, report_streams, report_frame_days, report_metric_days, report_metric_windows, report_metric_json, report_dimensions, report_shadow_runs, report_shadow_diffs",++ "Ingest: live.stream.ended → report_streams and channel/day counts; network.frame.hit/day → report_frame_days; GET /api/v1/stats → day and window metrics, JSON lists and site dimensions; every write claimed in report_inbox in the same transaction",++ "Read: reportStats() reproduces GET /api/v1/stats (totals, sites, pageviews_daily, accounts, coins, live, busiest pages); channelStats() reproduces creator analytics from report_streams and adds Billing revenue; metrics() returns report.metric@1 rows",++ "Billing: GET /api/v1/stats with billing.stats.read; GET /api/v1/transactions?subject= with billing.balance.read for settled creator_payable revenue/refunds; service token via Network client-credentials or REPORT_BILLING_TOKEN; never derives a price",++ "Shadow: server/analytics/shadow.js compares the same window from Network and Report, records report_shadow_runs + report_shadow_diffs; GET/POST /internal/shadow (loopback)",++ "Jobs: ingest 7/30/90-day windows every 5 minutes and shadow every 15 minutes (server/jobs/analytics.js), on by default; REPORT_JOBS=0 turns them off",++ "Events: POST /internal/events with Events v2 HMAC, loopback only",++ "Source switch: REPORT_SOURCE=network|report (default network) for /, /stats and GET /api/v1/stats",++ "Tests: ingest idempotency, every metric from a fixture, shadow difference and clean run, source flag, Billing stub, signed Events delivery, and the existing pages/API/health/crawl tests"],"notYet": [-- "The real page designs: v1 ships plain placeholder views, as the task asked; the main session designs the room itself",-- "A report manifest in openvibe-contracts: this skeleton does not register one, and CI skips the contracts-service check until it lands",-- "Any write path: Report has no accounts, no reactions and no saved views; it is a read-only front"++ "Cutover: REPORT_SOURCE stays network; Network and Live keep answering until the shadow is clean for a full window",++ "Report's first-party grants on Network (billing.stats.read, billing.balance.read, and the Network analytics grant when creator analytics is pulled directly) are not created by this change",++ "No report service manifest or capabilities are registered in openvibe-contracts yet; report.metric@1 is proposed in docs/report.metric.v1.json",++ "The ingest job reads Network's open-stats answer; it does not call Network's creator analytics endpoint (channels come from live.stream.ended)",++ "The Rundown and after-show pages remain Live reads until Report takes those products"],++ "capabilitiesUsed": ["billing.stats.read", "billing.balance.read"],"capabilitiesRegistered": false,"serviceManifestRegistered": false,"deployed": false,"placeholderInSites": false,"features": [++ "analytics ingest with an event-id inbox",++ "site and channel metrics by day",++ "Billing revenue read",++ "shadow comparison and /internal/shadow",++ "REPORT_SOURCE switch",++ "report.metric@1","open network stats","The Rundown","after-show reports",-- "source clients with caches",-- "server-rendered placeholder pages",++ "server-rendered pages","JSON API","crawl artifacts","readiness","deploy files"],-- "deployment": "planned: openvibe-ovh, unit openvibe-report.service on 127.0.0.1:5080 (git checkout /opt/openvibe.report, env /etc/openvibe/report.env), public at https://openvibe.report",++ "deployment": "planned: openvibe-ovh, unit openvibe-report.service on 127.0.0.1:5080 (git checkout /opt/openvibe.report, env /etc/openvibe/report.env), public at https://openvibe.report; DATABASE_URL/DATABASE_DIRECT_URL from OpenVibe.Host roles/data add-service.sh report","updated": "2026-10-28"}@@@ -77,7 +77,7 @@ server {# ── Prometheus metrics: host-local scrapers only (http://127.0.0.1:5080/metrics) ──location ~* ^/metrics(/|$) { return 404; }-- # ── Internal delivery would be loopback-only; Report has none, so refuse it from outside ──++ # ── Internal delivery is loopback-only (Events, shadow); refuse it from outside ──location ~* ^/internal(/|$) { return 404; }# ── Pages, crawl artifacts, static assets, release metadata ──@@@ -1,7 +1,7 @@[Unit]Description=OpenVibe.Report — Every number on OpenVibe, honest.--# Network and Live should be up first, but Report degrades honestly without either: the pages and the API answer--# what they can and /api/ready says degraded (never 503), because neither source is required to serve a page.++# Network, Live and Billing should be up first, but Report degrades honestly without either read API. The database++# is required after REPORT_SOURCE=report; before that it is the shadow store.After=network.target openvibe-network.service openvibe-live.serviceWants=openvibe-network.service openvibe-live.servicennew file mode 100644@@@ -0,0 +1,45 @@++{++ "$schema": "https://json-schema.org/draft/2020-12/schema",++ "$id": "https://openvibe.network/contracts/report/metric.v1.json",++ "title": "ReportMetric",++ "description": "report.metric@1: one number OpenVibe.Report computed or ingested, with its scope and either its UTC day (a day series) or its window. Counts only. Money values are Billing's, read through Billing's API; Report never derives them.",++ "type": "object",++ "required": ["metric", "scope", "scope_id", "value", "unit", "source"],++ "additionalProperties": false,++ "properties": {++ "metric": {++ "type": "string",++ "pattern": "^[a-z][a-z0-9_]{1,60}$",++ "description": "The number's name: pageviews, accounts_new, coins_earned, live_now, streams, stream_seconds, messages, watch_minutes, traffic_p95_ms, revenue_bits, refund_bits, ..."++ },++ "scope": { "enum": ["network", "site", "channel", "stream"] },++ "scope_id": {++ "type": "string",++ "minLength": 1,++ "maxLength": 200,++ "description": "network for the network scope; a site id (network, live, ...), a channel username, or a stream id."++ },++ "day": {++ "type": "string",++ "format": "date",++ "description": "The UTC day this number belongs to. Present on a day row."++ },++ "window_days": {++ "type": "integer",++ "enum": [7, 30, 90],++ "description": "The window this number is for. Present on a window row."++ },++ "value": { "type": "number" },++ "unit": { "enum": ["count", "seconds", "minutes", "bits", "cents", "milliseconds", "percent"] },++ "source": {++ "type": "string",++ "minLength": 1,++ "maxLength": 40,++ "description": "network-stats, live, billing, report, or the job that wrote it."++ }++ },++ "oneOf": [++ { "required": ["day"], "not": { "required": ["window_days"] } },++ { "required": ["window_days"], "not": { "required": ["day"] } }++ ]++}nnew file mode 100644@@@ -0,0 +1,138 @@++-- phase: expand++-- OpenVibe.Report's analytics authority (REP-2, plan 2026-10-28 §4.8 and §5.4).++-- Counts and public channel facts only. Money is never stored here: it is read from OpenVibe.Billing.++++-- The inbox of event ids already handled. openvibe-sdk/events createPgInbox() uses++-- (consumer, event_id, processed_at); the extra columns are audit context, never a secret.++CREATE TABLE report_inbox (++ consumer text NOT NULL,++ event_id text NOT NULL,++ processed_at bigint NOT NULL,++ source text,++ topic text,++ payload_hash text,++ received_at timestamptz NOT NULL DEFAULT now(),++ PRIMARY KEY (consumer, event_id)++);++CREATE INDEX report_inbox_received ON report_inbox (received_at);++++-- The raw event audit trail (public facts only: no viewer ids, no IP addresses, no billing payloads).++CREATE TABLE report_events (++ event_id text COLLATE "C" PRIMARY KEY,++ source text COLLATE "C" NOT NULL,++ topic text COLLATE "C" NOT NULL,++ occurred_at timestamptz,++ payload jsonb NOT NULL DEFAULT '{}'::jsonb,++ received_at timestamptz NOT NULL DEFAULT now()++);++CREATE INDEX report_events_topic_day ON report_events (topic, occurred_at);++++-- One row per ended stream; the same fields Network's creator_streams keeps (from live.stream.ended).++CREATE TABLE report_streams (++ stream_id bigint PRIMARY KEY,++ creator_subject text COLLATE "C",++ channel text COLLATE "C" NOT NULL,++ title text,++ category text,++ started_at timestamptz,++ ended_at timestamptz,++ duration_seconds bigint NOT NULL DEFAULT 0,++ peak_viewers bigint,++ avg_viewers numeric,++ unique_chatters bigint,++ messages bigint,++ watch_minutes bigint,++ payload jsonb NOT NULL DEFAULT '{}'::jsonb,++ received_at timestamptz NOT NULL DEFAULT now()++);++CREATE INDEX report_streams_channel_day ON report_streams (lower(channel), started_at DESC);++CREATE INDEX report_streams_creator_day ON report_streams (creator_subject, started_at DESC);++++-- The navbar's anonymous beacon, aggregated by day and host.++CREATE TABLE report_frame_days (++ day date NOT NULL,++ host text COLLATE "C" NOT NULL,++ hits bigint NOT NULL DEFAULT 0,++ source text COLLATE "C" NOT NULL DEFAULT 'frame',++ updated_at timestamptz NOT NULL DEFAULT now(),++ PRIMARY KEY (day, host)++);++++-- A number by day for one scope: scope = network | site | channel | stream.++CREATE TABLE report_metric_days (++ metric text COLLATE "C" NOT NULL,++ scope text COLLATE "C" NOT NULL,++ scope_id text COLLATE "C" NOT NULL,++ day date NOT NULL,++ value numeric NOT NULL,++ unit text COLLATE "C" NOT NULL DEFAULT 'count',++ source text COLLATE "C" NOT NULL,++ updated_at timestamptz NOT NULL DEFAULT now(),++ PRIMARY KEY (metric, scope, scope_id, day)++);++CREATE INDEX report_metric_days_scope_day ON report_metric_days (scope, scope_id, day);++++-- A number for one window (7, 30 or 90 days) when the source published a window total.++CREATE TABLE report_metric_windows (++ metric text COLLATE "C" NOT NULL,++ scope text COLLATE "C" NOT NULL,++ scope_id text COLLATE "C" NOT NULL,++ window_days integer NOT NULL,++ value numeric NOT NULL,++ unit text COLLATE "C" NOT NULL DEFAULT 'count',++ source text COLLATE "C" NOT NULL,++ updated_at timestamptz NOT NULL DEFAULT now(),++ PRIMARY KEY (metric, scope, scope_id, window_days)++);++++-- Lists that are not one number (the busiest route templates, for example).++CREATE TABLE report_metric_json (++ metric text COLLATE "C" NOT NULL,++ scope text COLLATE "C" NOT NULL,++ scope_id text COLLATE "C" NOT NULL,++ window_days integer NOT NULL,++ value jsonb NOT NULL,++ source text COLLATE "C" NOT NULL,++ updated_at timestamptz NOT NULL DEFAULT now(),++ PRIMARY KEY (metric, scope, scope_id, window_days)++);++++-- Public labels for a scope: a site's name, host and URL, a channel's display name.++CREATE TABLE report_dimensions (++ kind text COLLATE "C" NOT NULL,++ id text COLLATE "C" NOT NULL,++ doc jsonb NOT NULL,++ updated_at timestamptz NOT NULL DEFAULT now(),++ PRIMARY KEY (kind, id)++);++++-- One shadow comparison run and the differences it found. A run with zero differences is a clean run.++CREATE TABLE report_shadow_runs (++ run_id text COLLATE "C" PRIMARY KEY,++ run_seq bigint GENERATED BY DEFAULT AS IDENTITY UNIQUE,++ window_days integer NOT NULL,++ status text COLLATE "C" NOT NULL,++ checked integer NOT NULL DEFAULT 0,++ differences integer NOT NULL DEFAULT 0,++ error text,++ started_at timestamptz NOT NULL,++ finished_at timestamptz++);++CREATE INDEX report_shadow_runs_finished ON report_shadow_runs (finished_at DESC);++++CREATE TABLE report_shadow_diffs (++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,++ run_id text COLLATE "C" NOT NULL REFERENCES report_shadow_runs(run_id) ON DELETE CASCADE,++ metric text COLLATE "C" NOT NULL,++ scope text COLLATE "C" NOT NULL,++ scope_id text COLLATE "C" NOT NULL,++ day date,++ window_days integer,++ network_value numeric,++ report_value numeric,++ delta numeric,++ network_json jsonb,++ report_json jsonb,++ created_at timestamptz NOT NULL DEFAULT now()++);++CREATE INDEX report_shadow_diffs_run ON report_shadow_diffs (run_id, metric, scope, scope_id);@@@ -15,12 +15,23 @@"helmet": "^8.1.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0"++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0",++ "pg": "^8.23.1"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.13"},"engines": {"node": ">=22.0.0"}},++ "node_modules/@electric-sql/pglite": {++ "version": "0.5.13",++ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.13.tgz",++ "integrity": "sha512-4rZuxK1k07w6sSpJzjMri0B9+HuXuC7V9pxVB51pR0T9KPqXg6TD0mdaEwBA9rkcL4jdht4tGDf/wYzAiGhT7g==",++ "devOptional": true,++ "license": "Apache-2.0"++ },"node_modules/accepts": {"version": "1.3.8","resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",@@@ -840,6 +851,134 @@"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==","license": "MIT"},++ "node_modules/pg": {++ "version": "8.23.1",++ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.1.tgz",++ "integrity": "sha512-aL96AHANtWjPLDOLqnhx+ngp9+UK7ETEU8VJrDCGvsSSi/mGLcWYsS6Herg7lmaBJe4uwrfqsa7gTEFaSizDoQ==",++ "license": "MIT",++ "dependencies": {++ "pg-connection-string": "^2.14.1",++ "pg-pool": "^3.14.0",++ "pg-protocol": "^1.16.1",++ "pg-types": "2.2.0",++ "pgpass": "1.0.5"++ },++ "engines": {++ "node": ">= 16.0.0"++ },++ "optionalDependencies": {++ "pg-cloudflare": "^1.4.1"++ },++ "peerDependencies": {++ "pg-native": ">=3.0.1"++ },++ "peerDependenciesMeta": {++ "pg-native": {++ "optional": true++ }++ }++ },++ "node_modules/pg-cloudflare": {++ "version": "1.4.1",++ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.1.tgz",++ "integrity": "sha512-6PQbsFWZcp9EmJEwy5cGQ2La+AMWpP46lgbb8X+U/XsHIUweYDNCpeuKck5RxL2MdVFi7krbbEi5nX4Zh7JhrQ==",++ "license": "MIT",++ "optional": true++ },++ "node_modules/pg-connection-string": {++ "version": "2.14.1",++ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.1.tgz",++ "integrity": "sha512-qR3kGNPBLpCNtz0evbKA0Y/MRFXwSSdT+pTJvYp/bXTcReZbvX1kzF0IyTc1QnxqF7AZbOeBhNL8R5mYQZV/MA==",++ "license": "MIT"++ },++ "node_modules/pg-int8": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",++ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",++ "license": "ISC",++ "engines": {++ "node": ">=4.0.0"++ }++ },++ "node_modules/pg-pool": {++ "version": "3.14.0",++ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",++ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",++ "license": "MIT",++ "peerDependencies": {++ "pg": ">=8.0"++ }++ },++ "node_modules/pg-protocol": {++ "version": "1.16.1",++ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.1.tgz",++ "integrity": "sha512-p9VOFMiHB/ZbJATetbg+99PxssTVSQRnyuPSQ67mN1+1KBOjZaZ83ZQzltnxPhJwSsC3nwVjJ10DVJlerbFzLg==",++ "license": "MIT"++ },++ "node_modules/pg-types": {++ "version": "2.2.0",++ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",++ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",++ "license": "MIT",++ "dependencies": {++ "pg-int8": "1.0.1",++ "postgres-array": "~2.0.0",++ "postgres-bytea": "~1.0.0",++ "postgres-date": "~1.0.4",++ "postgres-interval": "^1.1.0"++ },++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/pgpass": {++ "version": "1.0.5",++ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",++ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",++ "license": "MIT",++ "dependencies": {++ "split2": "^4.1.0"++ }++ },++ "node_modules/postgres-array": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",++ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/postgres-bytea": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",++ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-date": {++ "version": "1.0.7",++ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",++ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-interval": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",++ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",++ "license": "MIT",++ "dependencies": {++ "xtend": "^4.0.0"++ },++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/proxy-addr": {"version": "2.0.8","resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",@@@ -1067,6 +1206,15 @@"url": "https://github.com/sponsors/ljharb"}},++ "node_modules/split2": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",++ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",++ "license": "ISC",++ "engines": {++ "node": ">= 10.x"++ }++ },"node_modules/statuses": {"version": "2.0.2","resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",@@@ -1124,6 +1272,15 @@"engines": {"node": ">= 0.8"}++ },++ "node_modules/xtend": {++ "version": "4.0.2",++ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",++ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.4"++ }}}}@@@ -21,9 +21,13 @@"helmet": "^8.1.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",-- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0"++ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0",++ "pg": "^8.23.1"},"engines": {"node": ">=22.0.0"++ },++ "devDependencies": {++ "@electric-sql/pglite": "^0.5.13"}}nnew file mode 100644@@@ -0,0 +1,50 @@++'use strict';++++/**++ * Report's analytics authority: the source switch and the one object pages, the API, the ingest job and the shadow++ * job share.++ *++ * stats(days) REPORT_SOURCE=network → Network's answer (today's behaviour); report → computed from Report's tables.++ * channel always computed from Report's own report_streams, plus Billing's revenue read.++ * metrics the flat report.metric@1 rows.++ *++ * The default is `network`: the pages keep reading where they read today while Report shadows. The cutover is one++ * environment value, REPORT_SOURCE=report.++ */++const query = require('./query');++const ingest = require('./ingest');++++function createAnalytics({ config = {}, store = null, network = null, billing = null, now = () => Date.now(), log = console } = {}) {++ const source = config.source === 'report' ? 'report' : 'network';++++ function warn(what, err) {++ try { (log.warn || log.error || console.warn).call(log, `[OpenVibe.Report] ${what} failed: ${String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 200)}`); } catch { /* logging must never fail a page */ }++ }++++ async function stats(days) {++ try {++ if (source === 'report') return store ? await query.reportStats(store, days, { now }) : null;++ return network && typeof network.stats === 'function' ? await network.stats(days) : null;++ } catch (err) { warn('stats', err); return null; }++ }++++ async function channel(name, days, opts = {}) {++ try { return store ? await query.channelStats(store, name, days, { now, billing, ...opts }) : null; } catch (err) { warn('channel analytics', err); return null; }++ }++++ async function metrics(opts = {}) {++ try { return store ? await query.metrics(store, { now, ...opts }) : null; } catch (err) { warn('metrics', err); return null; }++ }++++ return {++ source, store, billing, network,++ stats, channel, metrics,++ reportStats: (days, opts = {}) => (store ? query.reportStats(store, days, { now, ...opts }) : null),++ recordEvent: (event, opts = {}) => (store ? ingest.recordEvent(store, event, { now, ...opts }) : Promise.resolve({ duplicate: false, outcome: 'no_database' })),++ ingestStats: (payload, opts = {}) => (store ? ingest.ingestStats(store, payload, { now, ...opts }) : Promise.resolve({ ok: false, reason: 'no_database' })),++ ingestNetwork: (opts = {}) => (store && network ? ingest.ingestNetwork(store, network, { now, ...opts }) : Promise.resolve({ ok: false, reason: 'no_database' })),++ log,++ };++}++++module.exports = { createAnalytics };nnew file mode 100644@@@ -0,0 +1,339 @@++'use strict';++++/**++ * Report's ingest: the analytics facts OpenVibe.Network records today, written to Report's own tables.++ *++ * live.stream.ended one report_streams row and its channel/day counts (Network's creator-analytics input)++ * the frame beacon report_frame_days (day, host, hits) from a daily snapshot or a single hit++ * Network's open stats the daily page views, accounts and OpenCoins, and the window totals, kept as numbers++ *++ * Every delivery goes through report_inbox (openvibe-sdk/events createPgInbox) in the same transaction as the++ * writes: a redelivered event id changes nothing. A stats snapshot is keyed by its own content hash, so polling++ * the same numbers again is a no-op while new numbers replace the stored ones.++ *++ * Billing events are consciously not ingested: Report reads money from Billing's API, never from an event.++ */++const crypto = require('crypto');++const { createPgInbox } = require('openvibe-sdk/events');++const { normaliseDays } = require('../data/source');++++const INBOX = 'report-events';++const STATS_INBOX = 'network-stats';++const DAY = 86_400_000;++const ALLOWED_TOPICS = new Set(['live.stream.ended', 'live.stream.started', 'network.frame.hit', 'network.frame.day', 'frame.hit']);++++const hashOf = (value) => crypto.createHash('sha256').update(JSON.stringify(value)).digest('hex');++const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));++const int = (v) => (Number.isFinite(Number(v)) && Number(v) >= 0 ? Math.round(Number(v)) : null);++const dayOf = (value) => {++ if (value == null || value === '') return null;++ if (typeof value === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(value)) return value;++ const t = Date.parse(String(value));++ return Number.isFinite(t) ? new Date(t).toISOString().slice(0, 10) : null;++};++const isChannel = (v) => typeof v === 'string' && /^[A-Za-z0-9_]{1,80}$/.test(v);++++function unitOf(metric) {++ if (metric.endsWith('_ms')) return 'milliseconds';++ if (metric === 'change_pct' || metric.endsWith('_pct')) return 'percent';++ if (metric === 'stream_seconds' || metric === 'duration_seconds' || metric.endsWith('_seconds')) return 'seconds';++ if (metric === 'watch_minutes' || metric.endsWith('_minutes')) return 'minutes';++ if (metric.includes('revenue') || metric.includes('refund') || metric.endsWith('_bits')) return 'bits';++ if (metric.includes('cents')) return 'cents';++ return 'count';++}++++function seriesOf(value) {++ if (!Array.isArray(value)) return [];++ const out = [];++ for (const p of value) {++ if (Array.isArray(p)) { const d = dayOf(p[0]); const n = num(p[1]); if (d && n != null) out.push({ day: d, n }); continue; }++ if (!p || typeof p !== 'object') continue;++ const d = dayOf(p.day || p.date);++ const n = num(p.n != null ? p.n : p.value);++ if (d && n != null) out.push({ day: d, n });++ }++ return out;++}++++/**++ * The same numbers in one flat shape, used both to write the tables and by the shadow job to compare them.++ * A row is a day number or a window number; nothing is invented for a field the source did not send.++ */++function metricRowsFromStats(stats) {++ const rows = [];++ const seen = new Map();++ function add(kind, metric, scope, scopeId, value, unit = unitOf(metric), day = null, windowDays = null) {++ const n = num(value);++ if (n == null || !scopeId) return;++ const row = { kind, metric, scope, scope_id: String(scopeId), value: n, unit, day: kind === 'day' ? day : null, window_days: kind === 'window' ? windowDays : null };++ if (kind === 'day' && !row.day) return;++ if (kind === 'window' && !row.window_days) return;++ seen.set(`${kind}|${scope}|${row.scope_id}|${metric}|${row.day || ''}|${row.window_days || ''}`, row);++ }++ const win = (metric, scope, id, value) => add('window', metric, scope, id, value, unitOf(metric), null, days);++ const day = (metric, scope, id, d, value) => add('day', metric, scope, id, value, unitOf(metric), d, null);++++ const s = stats && typeof stats === 'object' ? stats : {};++ const days = normaliseDays(s.window_days);++ const T = s.totals && typeof s.totals === 'object' ? s.totals : {};++ const A = s.accounts && typeof s.accounts === 'object' ? s.accounts : null;++ const C = s.coins && typeof s.coins === 'object' ? s.coins : null;++ const L = s.live && typeof s.live === 'object' ? s.live : null;++++ win('pageviews', 'network', 'network', T.pageviews);++ win('pageviews_prev', 'network', 'network', T.pageviews_prev);++ win('change_pct', 'network', 'network', T.change_pct);++ win('sites_open', 'network', 'network', T.sites_open);++ win('accounts_total', 'network', 'network', T.accounts);++ win('requests_24h', 'network', 'network', T.requests_24h);++ if (T.services && typeof T.services === 'object') {++ win('services_up', 'network', 'network', T.services.up);++ win('services_checked', 'network', 'network', T.services.checked);++ }++ for (const p of seriesOf(s.pageviews_daily)) day('pageviews', 'network', 'network', p.day, p.n);++++ if (A) {++ win('accounts_total', 'network', 'network', A.total);++ win('accounts_new', 'network', 'network', A.new);++ win('accounts_prev', 'network', 'network', A.prev);++ for (const p of seriesOf(A.daily)) day('accounts_new', 'network', 'network', p.day, p.n);++ }++ if (C) {++ win('coins_earned', 'network', 'network', C.earned);++ win('coins_spent', 'network', 'network', C.spent);++ win('coins_holders', 'network', 'network', C.holders);++ win('coins_transactions', 'network', 'network', C.transactions);++ win('coins_earned_window', 'network', 'network', C.earned_window);++ win('coins_earned_prev', 'network', 'network', C.earned_prev);++ for (const p of seriesOf(C.daily)) day('coins_earned', 'network', 'network', p.day, p.n);++ }++ if (L) {++ for (const k of ['live_now', 'watching', 'streamers', 'streams', 'accounts', 'visitors', 'messages', 'chatted_7d',++ 'vods', 'clips', 'hours_watched', 'hours_archived', 'ai_moments', 'pastes', 'follows']) {++ win(k, 'network', 'network', L[k]);++ }++ }++++ for (const site of Array.isArray(s.sites) ? s.sites : []) {++ if (!site || typeof site !== 'object') continue;++ const id = site.id || site.host || site.short;++ if (!id) continue;++ win('pageviews', 'site', id, site.pageviews);++ win('pageviews_prev', 'site', id, site.pageviews_prev);++ win('change_pct', 'site', id, site.change_pct);++ win('unique_visitors', 'site', id, site.unique_visitors);++ win('signed_in_people', 'site', id, site.signed_in_people);++ for (const p of seriesOf(site.daily)) day('pageviews', 'site', id, p.day, p.n);++ const t = site.traffic_24h && typeof site.traffic_24h === 'object' ? site.traffic_24h : null;++ if (t) {++ win('traffic_requests', 'site', id, t.requests);++ win('traffic_server_errors', 'site', id, t.server_errors);++ win('traffic_p95_ms', 'site', id, t.p95_ms);++ }++ }++ return [...seen.values()];++}++++function jsonRowsFromStats(stats) {++ const rows = [];++ const s = stats && typeof stats === 'object' ? stats : {};++ if (Array.isArray(s.busiest_pages_24h)) {++ rows.push({ metric: 'busiest_pages', scope: 'network', scope_id: 'network', window_days: normaliseDays(s.window_days), value: s.busiest_pages_24h, source: 'network' });++ }++ return rows;++}++++function frameRowsFromStats(stats) {++ const rows = [];++ const s = stats && typeof stats === 'object' ? stats : {};++ for (const site of Array.isArray(s.sites) ? s.sites : []) {++ if (!site || !site.host || !Array.isArray(site.daily)) continue;++ for (const p of seriesOf(site.daily)) rows.push({ day: p.day, host: String(site.host).toLowerCase(), hits: p.n });++ }++ return rows;++}++++function dimensionsFromStats(stats) {++ const docs = [];++ const s = stats && typeof stats === 'object' ? stats : {};++ if (typeof s.note === 'string' && s.note.trim()) docs.push({ kind: 'network', id: 'network', doc: { note: s.note.trim().slice(0, 600) } });++ for (const site of Array.isArray(s.sites) ? s.sites : []) {++ if (!site || typeof site !== 'object') continue;++ const id = site.id || site.host || site.short;++ if (!id) continue;++ docs.push({++ kind: 'site', id,++ doc: {++ id, name: site.name || `OpenVibe.${site.short || id}`, short: site.short || id, host: site.host || null,++ url: site.url || (site.host ? `https://${site.host}/` : null), tagline: site.tagline || '', icon: site.icon || null,++ },++ });++ }++ return docs;++}++++// ── Writes ──────────────────────────────────────────────────++async function setDay(t, row, source) {++ await t.prepare(`INSERT INTO report_metric_days (metric, scope, scope_id, day, value, unit, source)++ VALUES (?, ?, ?, ?, ?, ?, ?)++ ON CONFLICT(metric, scope, scope_id, day) DO UPDATE SET value = excluded.value, unit = excluded.unit, source = excluded.source, updated_at = now()`)++ .run(row.metric, row.scope, row.scope_id, row.day, row.value, row.unit, source);++}++async function addDay(t, metric, scope, id, day, value, unit = unitOf(metric), source = 'live') {++ await t.prepare(`INSERT INTO report_metric_days (metric, scope, scope_id, day, value, unit, source)++ VALUES (?, ?, ?, ?, ?, ?, ?)++ ON CONFLICT(metric, scope, scope_id, day) DO UPDATE SET value = report_metric_days.value + excluded.value, updated_at = now()`)++ .run(metric, scope, id, day, value, unit, source);++}++async function maxDay(t, metric, scope, id, day, value, unit = unitOf(metric), source = 'live') {++ await t.prepare(`INSERT INTO report_metric_days (metric, scope, scope_id, day, value, unit, source)++ VALUES (?, ?, ?, ?, ?, ?, ?)++ ON CONFLICT(metric, scope, scope_id, day) DO UPDATE SET value = GREATEST(report_metric_days.value, excluded.value), updated_at = now()`)++ .run(metric, scope, id, day, value, unit, source);++}++async function setWindow(t, row, source) {++ await t.prepare(`INSERT INTO report_metric_windows (metric, scope, scope_id, window_days, value, unit, source)++ VALUES (?, ?, ?, ?, ?, ?, ?)++ ON CONFLICT(metric, scope, scope_id, window_days) DO UPDATE SET value = excluded.value, unit = excluded.unit, source = excluded.source, updated_at = now()`)++ .run(row.metric, row.scope, row.scope_id, row.window_days, row.value, row.unit, source);++}++async function setJson(t, row) {++ await t.prepare(`INSERT INTO report_metric_json (metric, scope, scope_id, window_days, value, source)++ VALUES (?, ?, ?, ?, ?::jsonb, ?)++ ON CONFLICT(metric, scope, scope_id, window_days) DO UPDATE SET value = excluded.value, source = excluded.source, updated_at = now()`)++ .run(row.metric, row.scope, row.scope_id, row.window_days, JSON.stringify(row.value), row.source);++}++async function setFrame(t, row) {++ await t.prepare(`INSERT INTO report_frame_days (day, host, hits, source) VALUES (?, ?, ?, 'network-stats')++ ON CONFLICT(day, host) DO UPDATE SET hits = excluded.hits, source = excluded.source, updated_at = now()`)++ .run(row.day, row.host, row.hits);++}++async function setDimension(t, row) {++ await t.prepare(`INSERT INTO report_dimensions (kind, id, doc) VALUES (?, ?, ?::jsonb)++ ON CONFLICT(kind, id) DO UPDATE SET doc = excluded.doc, updated_at = now()`)++ .run(row.kind, row.id, JSON.stringify(row.doc));++}++++async function recordFrame(t, topic, payload, occurredAt, source) {++ const p = payload && typeof payload === 'object' ? payload : {};++ const day = dayOf(p.day || occurredAt);++ const host = String(p.host || p.site || '').toLowerCase();++ if (!day || !/^[a-z0-9.-]{1,253}$/.test(host)) return 'ignored:payload';++ const hits = p.hits != null ? int(p.hits) : 1;++ if (hits == null) return 'ignored:payload';++ const increment = (topic === 'network.frame.hit' || topic === 'frame.hit') && p.hits == null;++ const sql = increment++ ? `INSERT INTO report_frame_days (day, host, hits, source) VALUES (?, ?, ?, ?)++ ON CONFLICT(day, host) DO UPDATE SET hits = report_frame_days.hits + excluded.hits, source = excluded.source, updated_at = now()`++ : `INSERT INTO report_frame_days (day, host, hits, source) VALUES (?, ?, ?, ?)++ ON CONFLICT(day, host) DO UPDATE SET hits = excluded.hits, source = excluded.source, updated_at = now()`;++ await t.prepare(sql).run(day, host, hits, source || (increment ? 'frame-hit' : 'frame'));++ return 'frame:recorded';++}++++async function recordStream(t, event, payload, occurredAt, source) {++ const p = payload && typeof payload === 'object' ? payload : {};++ const streamId = Number(p.stream_id);++ const channel = p.channel && typeof p.channel.username === 'string' ? p.channel.username : null;++ if (!Number.isSafeInteger(streamId) || streamId < 1 || !isChannel(channel)) return 'ignored:payload';++ const creator = p.channel && p.channel.subject && typeof p.channel.subject.id === 'string' ? p.channel.subject.id : null;++ const startedMs = Date.parse(p.started_at || occurredAt || '');++ const endedMs = Date.parse(p.ended_at || occurredAt || '');++ if (!Number.isFinite(startedMs) || !Number.isFinite(endedMs) || endedMs < startedMs) return 'ignored:times';++ const stats = p.stats && typeof p.stats === 'object' ? p.stats : {};++ const duration = int(p.duration_seconds) != null ? int(p.duration_seconds) : Math.round((endedMs - startedMs) / 1000);++ const existing = await t.prepare('SELECT stream_id FROM report_streams WHERE stream_id = ?').get(streamId);++ await t.prepare(`INSERT INTO report_streams (stream_id, creator_subject, channel, title, category, started_at, ended_at, duration_seconds,++ peak_viewers, avg_viewers, unique_chatters, messages, watch_minutes, payload)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?::jsonb)++ ON CONFLICT(stream_id) DO UPDATE SET creator_subject = excluded.creator_subject, channel = excluded.channel, title = excluded.title,++ category = excluded.category, started_at = excluded.started_at, ended_at = excluded.ended_at, duration_seconds = excluded.duration_seconds,++ peak_viewers = excluded.peak_viewers, avg_viewers = excluded.avg_viewers, unique_chatters = excluded.unique_chatters,++ messages = excluded.messages, watch_minutes = excluded.watch_minutes, payload = excluded.payload, received_at = now()`)++ .run(streamId, creator, channel, p.title == null ? null : String(p.title).slice(0, 300), p.category == null ? null : String(p.category).slice(0, 100),++ new Date(startedMs).toISOString(), new Date(endedMs).toISOString(), duration,++ int(stats.peak_viewers), num(stats.avg_viewers), int(stats.unique_chatters), int(stats.messages), int(stats.watch_minutes),++ JSON.stringify({ stream_id: streamId, channel: p.channel, title: p.title || null, category: p.category || null, started_at: p.started_at || null, ended_at: p.ended_at || null, duration_seconds: duration, stats }));++ if (existing) return 'stream:updated'; // a corrected copy: replace the facts, never count the stream twice++ const day = new Date(startedMs).toISOString().slice(0, 10);++ await addDay(t, 'streams', 'channel', channel, day, 1, 'count', source || 'live');++ await addDay(t, 'stream_seconds', 'channel', channel, day, duration, 'seconds', source || 'live');++ await addDay(t, 'unique_chatters', 'channel', channel, day, int(stats.unique_chatters) || 0, 'count', source || 'live');++ await addDay(t, 'messages', 'channel', channel, day, int(stats.messages) || 0, 'count', source || 'live');++ await addDay(t, 'watch_minutes', 'channel', channel, day, int(stats.watch_minutes) || 0, 'minutes', source || 'live');++ if (num(stats.avg_viewers) != null) await addDay(t, 'avg_viewers_weighted', 'channel', channel, day, num(stats.avg_viewers) * duration, 'count', source || 'live');++ if (int(stats.peak_viewers) != null) await maxDay(t, 'peak_viewers', 'channel', channel, day, int(stats.peak_viewers), 'count', source || 'live');++ return 'stream:recorded';++}++++/** One Events envelope → the tables. Returns { duplicate, outcome }. */++async function recordEvent(store, event, { now = () => Date.now() } = {}) {++ if (!store || !store.db) throw new TypeError('recordEvent needs an open Report store');++ const e = event && typeof event === 'object' ? event : {};++ const eventId = String(e.event_id || e.id || '');++ const topic = String(e.event_type || e.topic || e.type || '');++ if (!eventId) throw new TypeError('an event_id is required');++ if (!topic) return { duplicate: false, outcome: 'ignored:topic' };++ const source = String(e.source || topic.split('.')[0] || 'events');++ const rawOccurred = e.occurred_at || e.timestamp || (e.payload && (e.payload.ended_at || e.payload.at)) || new Date(now()).toISOString();++ const occurredMs = Date.parse(String(rawOccurred));++ const occurredAt = Number.isFinite(occurredMs) ? new Date(occurredMs).toISOString() : null;++ if (!ALLOWED_TOPICS.has(topic)) {++ // Today's billing and traffic events are not analytics facts Report may derive: a seen id changes nothing.++ const inbox = createPgInbox(store.db, { table: 'report_inbox', now });++ const r = await inbox.once(INBOX, eventId, async () => 'ignored:topic');++ return { duplicate: r.duplicate, outcome: r.result || 'ignored:topic' };++ }++ const payload = e.payload && typeof e.payload === 'object' ? e.payload : {};++ const inbox = createPgInbox(store.db, { table: 'report_inbox', now });++ const r = await inbox.once(INBOX, eventId, async (t) => {++ await t.prepare(`INSERT INTO report_events (event_id, source, topic, occurred_at, payload) VALUES (?, ?, ?, ?, ?::jsonb) ON CONFLICT (event_id) DO NOTHING`)++ .run(eventId, source, topic, occurredAt, JSON.stringify(payload));++ await t.prepare('UPDATE report_inbox SET source = ?, topic = ?, payload_hash = ? WHERE consumer = ? AND event_id = ?')++ .run(source, topic, hashOf(payload), INBOX, eventId);++ if (topic === 'live.stream.ended') return await recordStream(t, e, payload, occurredAt, source);++ if (topic === 'network.frame.hit' || topic === 'network.frame.day' || topic === 'frame.hit') return await recordFrame(t, topic, payload, occurredAt, source);++ return 'event:recorded';++ });++ return { duplicate: r.duplicate, outcome: r.result || 'event:recorded' };++}++++/** Ingest one GET /api/v1/stats answer. Idempotent by its content hash. */++async function ingestStats(store, stats, { now = () => Date.now(), source = 'network-stats' } = {}) {++ if (!store || !store.db) throw new TypeError('ingestStats needs an open Report store');++ if (!stats || typeof stats !== 'object') return { ok: false, reason: 'no stats' };++ const windowDays = normaliseDays(stats.window_days);++ const rows = metricRowsFromStats(stats);++ const docs = jsonRowsFromStats(stats);++ const frames = frameRowsFromStats(stats);++ const dims = dimensionsFromStats(stats);++ const eventId = `stats:${windowDays}:${hashOf({ stats })}`;++ const inbox = createPgInbox(store.db, { table: 'report_inbox', now });++ const r = await inbox.once(STATS_INBOX, eventId, async (t) => {++ for (const row of rows) {++ if (row.kind === 'window') await setWindow(t, row, source);++ else if (row.kind === 'day') await setDay(t, row, source);++ }++ for (const doc of docs) await setJson(t, doc);++ for (const frame of frames) await setFrame(t, frame);++ for (const dim of dims) await setDimension(t, dim);++ await t.prepare('UPDATE report_inbox SET source = ?, topic = ?, payload_hash = ? WHERE consumer = ? AND event_id = ?')++ .run(source, 'network.stats', hashOf(stats), STATS_INBOX, eventId);++ return `stats:${rows.length}`;++ });++ return { ok: true, duplicate: r.duplicate, outcome: r.result, window_days: windowDays, rows: rows.length, event_id: eventId };++}++++/** Pull Network's open stats once and ingest them. Null from Network is "no answer", never zeroes. */++async function ingestNetwork(store, network, { days = 90, now = () => Date.now() } = {}) {++ if (!network || typeof network.stats !== 'function') throw new TypeError('ingestNetwork needs a Network client');++ const stats = await network.stats(days);++ if (!stats) return { ok: false, reason: 'network_did_not_answer' };++ return await ingestStats(store, stats, { now });++}++++module.exports = {++ recordEvent, ingestStats, ingestNetwork, metricRowsFromStats, jsonRowsFromStats, frameRowsFromStats, dimensionsFromStats,++ setDay, addDay, maxDay, setWindow, setJson, setFrame, setDimension, hashOf, dayOf, unitOf, ALLOWED_TOPICS, INBOX, STATS_INBOX, DAY,++};nnew file mode 100644@@@ -0,0 +1,272 @@++'use strict';++++/**++ * Report's read side: the same numbers Network answers, computed from Report's own tables.++ *++ * reportStats(days) GET /api/v1/stats' shape: network totals, sites, accounts, OpenCoins, Live and day series++ * channelStats(channel) creator analytics from the live.stream.ended events, plus Billing's revenue read++ * metrics(scope,id) the flat report.metric@1 rows the shadow job and operators inspect++ *++ * Window totals come from the source's own window answer when it published one (report_metric_windows); a day++ * series is summed from report_metric_days; a channel is aggregated from report_streams. A number that is not++ * present is null / left out, never a made-up zero.++ */++const { normaliseDays } = require('../data/source');++++const DAY = 86_400_000;++const NOTE = 'Counts only, never about a person. Page views come from the shared navbar\'s anonymous counter (day and site) and OpenVibe.Live\'s own analytics; traffic is over 24 hours.';++const SCOPES = new Set(['network', 'site', 'channel', 'stream']);++++const n = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));++const dayOf = (ms) => new Date(ms).toISOString().slice(0, 10);++const round1 = (v) => Math.round(v * 10) / 10;++const change = (now, before) => (before > 0 && now != null ? Math.round(((now - before) / before) * 100) : null);++const sumOrNull = (values) => { let any = false; let total = 0; for (const v of values) if (v != null) { any = true; total += v; } return any ? total : null; };++++function windowOf(nowMs, days) {++ return {++ currentStart: dayOf(nowMs - (days - 1) * DAY),++ currentEnd: dayOf(nowMs),++ prevStart: dayOf(nowMs - (2 * days - 1) * DAY),++ prevEnd: dayOf(nowMs - days * DAY),++ days: Array.from({ length: days }, (_, i) => dayOf(nowMs - (days - 1 - i) * DAY)),++ };++}++++function indexRows(rows, winRows, jsonRows) {++ const day = new Map();++ for (const r of rows) {++ const key = `${r.scope}|${r.scope_id}|${r.metric}`;++ let m = day.get(key);++ if (!m) { m = new Map(); day.set(key, m); }++ m.set(r.day, { value: n(r.value), unit: r.unit, source: r.source });++ }++ const win = new Map();++ for (const r of winRows) win.set(`${r.scope}|${r.scope_id}|${r.metric}`, { value: n(r.value), unit: r.unit, source: r.source });++ const json = new Map();++ for (const r of jsonRows) json.set(`${r.scope}|${r.scope_id}|${r.metric}`, r.value);++ return { day, win, json };++}++++async function loadIndex(store, days, nowMs) {++ const w = windowOf(nowMs, days);++ const rows = await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, value, unit, source FROM report_metric_days WHERE day >= $1`, [w.prevStart]);++ const winRows = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, unit, source FROM report_metric_windows WHERE window_days = $1`, [days]);++ const jsonRows = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, source FROM report_metric_json WHERE window_days = $1`, [days]);++ const dims = await store.db.many(`SELECT kind, id, doc FROM report_dimensions`);++ return { rows, winRows, jsonRows, dims, w, ...indexRows(rows, winRows, jsonRows) };++}++++function dayGet(idx, metric, scope, id, day) {++ const m = idx.day.get(`${scope}|${id}|${metric}`);++ return m && m.get(day) ? m.get(day).value : null;++}++function dayHas(idx, metric, scope, id) {++ const m = idx.day.get(`${scope}|${id}|${metric}`);++ return !!m && m.size > 0;++}++function daySum(idx, metric, scope, id, from, to) {++ const m = idx.day.get(`${scope}|${id}|${metric}`);++ if (!m) return null;++ let total = 0;++ let seen = false;++ for (const [day, v] of m) {++ if (day >= from && day <= to) { total += v.value || 0; seen = true; }++ }++ return seen ? total : null;++}++function daySeries(idx, metric, scope, id, days) {++ return days.map((day) => ({ day, n: dayGet(idx, metric, scope, id, day) || 0 }));++}++function winGet(idx, metric, scope, id) {++ const r = idx.win.get(`${scope}|${id}|${metric}`);++ return r ? r.value : null;++}++function winHas(idx, scope, id) {++ for (const key of idx.win.keys()) if (key.startsWith(`${scope}|${id}|`)) return true;++ return false;++}++++/** GET /api/v1/stats' shape, from Report's tables. Null when no number exists for the window. */++async function reportStats(store, days = 30, { now = () => Date.now() } = {}) {++ if (!store || !store.db) return null;++ const windowDays = normaliseDays(days);++ const nowMs = now();++ const idx = await loadIndex(store, windowDays, nowMs);++ const w = idx.w;++ const anyCurrentDay = idx.rows.some((r) => r.day >= w.currentStart && r.day <= w.currentEnd);++ if (!anyCurrentDay && !idx.winRows.length && !idx.jsonRows.length) return null;++++ const dimOf = (kind, id) => {++ const d = idx.dims.find((x) => x.kind === kind && x.id === id);++ return d && d.doc ? d.doc : null;++ };++ const siteIds = new Set();++ for (const r of idx.rows) if (r.scope === 'site' && r.day >= w.currentStart) siteIds.add(r.scope_id);++ for (const r of idx.winRows) if (r.scope === 'site') siteIds.add(r.scope_id);++ for (const d of idx.dims) if (d.kind === 'site') siteIds.add(d.id);++++ const sites = [];++ for (const id of siteIds) {++ const meta = dimOf('site', id) || { id, name: id, short: id };++ const hasDaily = dayHas(idx, 'pageviews', 'site', id);++ const hasAny = hasDaily || winHas(idx, 'site', id);++ if (!hasAny) continue;++ const pageviews = winGet(idx, 'pageviews', 'site', id) ?? daySum(idx, 'pageviews', 'site', id, w.currentStart, w.currentEnd);++ const prev = winGet(idx, 'pageviews_prev', 'site', id) ?? daySum(idx, 'pageviews', 'site', id, w.prevStart, w.prevEnd);++ const pct = winGet(idx, 'change_pct', 'site', id) ?? change(pageviews, prev);++ const traffic = (winGet(idx, 'traffic_requests', 'site', id) != null || winGet(idx, 'traffic_p95_ms', 'site', id) != null)++ ? {++ requests: winGet(idx, 'traffic_requests', 'site', id),++ server_errors: winGet(idx, 'traffic_server_errors', 'site', id),++ p95_ms: winGet(idx, 'traffic_p95_ms', 'site', id),++ } : null;++ sites.push({++ ...meta,++ pageviews: pageviews == null ? null : pageviews,++ pageviews_prev: prev == null ? null : prev,++ change_pct: pct,++ daily: hasDaily ? daySeries(idx, 'pageviews', 'site', id, w.days) : null,++ ...(winGet(idx, 'unique_visitors', 'site', id) != null ? { unique_visitors: winGet(idx, 'unique_visitors', 'site', id) } : {}),++ ...(winGet(idx, 'signed_in_people', 'site', id) != null ? { signed_in_people: winGet(idx, 'signed_in_people', 'site', id) } : {}),++ traffic_24h: traffic,++ });++ }++ sites.sort((a, b) => (b.pageviews || 0) - (a.pageviews || 0) || String(a.name).localeCompare(String(b.name)));++++ const networkHasDaily = dayHas(idx, 'pageviews', 'network', 'network');++ let pageviews = winGet(idx, 'pageviews', 'network', 'network') ?? daySum(idx, 'pageviews', 'network', 'network', w.currentStart, w.currentEnd);++ if (pageviews == null) pageviews = sumOrNull(sites.map((s) => s.pageviews));++ let prev = winGet(idx, 'pageviews_prev', 'network', 'network') ?? daySum(idx, 'pageviews', 'network', 'network', w.prevStart, w.prevEnd);++ if (prev == null) prev = sumOrNull(sites.map((s) => s.pageviews_prev));++ const pct = winGet(idx, 'change_pct', 'network', 'network') ?? change(pageviews, prev);++++ let pageviewsDaily = null;++ if (networkHasDaily) pageviewsDaily = daySeries(idx, 'pageviews', 'network', 'network', w.days);++ else if (sites.some((s) => s.daily)) {++ pageviewsDaily = w.days.map((d) => ({ day: d, n: sites.reduce((a, s) => a + ((s.daily && s.daily.find((x) => x.day === d)) || {}).n || 0, 0) }));++ }++++ const accountsTotal = winGet(idx, 'accounts_total', 'network', 'network');++ const accountsNew = winGet(idx, 'accounts_new', 'network', 'network') ?? daySum(idx, 'accounts_new', 'network', 'network', w.currentStart, w.currentEnd);++ const accountsPrev = winGet(idx, 'accounts_prev', 'network', 'network') ?? daySum(idx, 'accounts_new', 'network', 'network', w.prevStart, w.prevEnd);++ const accounts = (accountsTotal != null || accountsNew != null || dayHas(idx, 'accounts_new', 'network', 'network'))++ ? { total: accountsTotal, new: accountsNew, prev: accountsPrev, daily: dayHas(idx, 'accounts_new', 'network', 'network') ? daySeries(idx, 'accounts_new', 'network', 'network', w.days) : [] }++ : null;++++ // coins.earned is all time, so it is only ever the source's own window figure, never a sum of recent days.++ const coinsEarned = winGet(idx, 'coins_earned', 'network', 'network');++ const coins = (coinsEarned != null || winGet(idx, 'coins_spent', 'network', 'network') != null)++ ? {++ earned: coinsEarned,++ spent: winGet(idx, 'coins_spent', 'network', 'network'),++ holders: winGet(idx, 'coins_holders', 'network', 'network'),++ transactions: winGet(idx, 'coins_transactions', 'network', 'network'),++ earned_window: winGet(idx, 'coins_earned_window', 'network', 'network') ?? daySum(idx, 'coins_earned', 'network', 'network', w.currentStart, w.currentEnd),++ earned_prev: winGet(idx, 'coins_earned_prev', 'network', 'network') ?? daySum(idx, 'coins_earned', 'network', 'network', w.prevStart, w.prevEnd),++ daily: dayHas(idx, 'coins_earned', 'network', 'network') ? daySeries(idx, 'coins_earned', 'network', 'network', w.days) : [],++ } : null;++++ const liveKeys = ['live_now', 'watching', 'streamers', 'streams', 'accounts', 'visitors', 'messages', 'chatted_7d',++ 'vods', 'clips', 'hours_watched', 'hours_archived', 'ai_moments', 'pastes', 'follows'];++ const live = {};++ for (const k of liveKeys) { const v = winGet(idx, k, 'network', 'network'); if (v != null) live[k] = v; }++ const hasLive = Object.keys(live).length > 0;++++ const servicesUp = winGet(idx, 'services_up', 'network', 'network');++ const servicesChecked = winGet(idx, 'services_checked', 'network', 'network');++ const noteDoc = dimOf('network', 'network');++++ return {++ generated_at: new Date(nowMs).toISOString(),++ window_days: windowDays,++ note: (noteDoc && noteDoc.note) || NOTE,++ totals: {++ pageviews: pageviews == null ? null : pageviews,++ pageviews_prev: prev == null ? null : prev,++ change_pct: pct,++ sites_open: winGet(idx, 'sites_open', 'network', 'network') ?? sites.length,++ services: servicesChecked != null ? { up: servicesUp == null ? 0 : servicesUp, checked: servicesChecked } : null,++ accounts: accounts ? accounts.total : null,++ requests_24h: winGet(idx, 'requests_24h', 'network', 'network'),++ },++ pageviews_daily: pageviewsDaily,++ accounts, coins,++ ...(hasLive ? { live } : {}),++ busiest_pages_24h: idx.json.get('network|network|busiest_pages') || null,++ sites,++ };++}++++/** A creator's streaming analytics from report_streams, plus Billing's revenue read (never a computed amount). */++async function channelStats(store, channel, days = 30, { now = () => Date.now(), billing = null } = {}) {++ if (!store || !store.db) return null;++ const d = Math.min(Math.max(parseInt(days, 10) || 30, 1), 365);++ const nowMs = now();++ const since = new Date(nowMs - d * DAY).toISOString();++ const rows = await store.db.many(`SELECT * FROM report_streams WHERE (lower(channel) = lower($1) OR creator_subject = $2) AND started_at >= $3 ORDER BY started_at DESC`, [String(channel || ''), String(channel || ''), since]);++ let creator = null;++ for (const r of rows) if (r.creator_subject) { creator = r.creator_subject; break; }++ const revenue = creator && billing && typeof billing.revenue === 'function'++ ? await billing.revenue({ subject: creator, days: d, now: nowMs }).catch(() => null)++ : null;++ if (!rows.length && !revenue) return null;++++ const blank = () => ({ streams: 0, stream_seconds: 0, peak_viewers: 0, unique_chatters: 0, messages: 0, watch_minutes: 0, _avgWeighted: 0 });++ const add = (acc, r) => {++ acc.streams += 1;++ acc.stream_seconds += r.duration_seconds || 0;++ acc.peak_viewers = Math.max(acc.peak_viewers, r.peak_viewers || 0);++ acc.unique_chatters += r.unique_chatters || 0;++ acc.messages += r.messages || 0;++ acc.watch_minutes += r.watch_minutes || 0;++ acc._avgWeighted += (Number(r.avg_viewers) || 0) * (r.duration_seconds || 0);++ return acc;++ };++ const finish = (acc) => {++ acc.avg_viewers = acc.stream_seconds ? round1(acc._avgWeighted / acc.stream_seconds) : 0;++ delete acc._avgWeighted;++ return acc;++ };++ const totals = finish(rows.reduce(add, blank()));++ const byDay = new Map();++ for (const r of rows) {++ const day = String(r.started_at || '').slice(0, 10);++ if (!day) continue;++ byDay.set(day, add(byDay.get(day) || blank(), r));++ }++ const daily = [...byDay.entries()].sort((a, b) => a[0].localeCompare(b[0])).map(([day, acc]) => ({ day, ...finish(acc) }));++ const streams = rows.slice(0, 100).map((r) => {++ const out = { stream_id: r.stream_id, title: r.title, category: r.category, started_at: r.started_at, ended_at: r.ended_at, duration_seconds: r.duration_seconds };++ for (const k of ['peak_viewers', 'avg_viewers', 'unique_chatters', 'messages', 'watch_minutes']) if (r[k] != null) out[k] = Number(r[k]);++ return out;++ });++ return { creator, channel: rows[0] ? rows[0].channel : String(channel), days: d, full: true, totals, daily, streams, revenue };++}++++/** The flat report.metric@1 view: one row per number, with its day or window. */++async function metrics(store, { scope = 'network', scopeId = null, days = 30, now = () => Date.now() } = {}) {++ if (!store || !store.db) return null;++ if (!SCOPES.has(scope)) return null;++ const id = String(scopeId || (scope === 'network' ? 'network' : ''));++ if (!id) return null;++ const windowDays = normaliseDays(days);++ const w = windowOf(now(), windowDays);++ const rows = await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, value, unit, source FROM report_metric_days++ WHERE scope = $1 AND scope_id = $2 AND day >= $3 ORDER BY day, metric`, [scope, id, w.currentStart]);++ const wins = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, unit, source FROM report_metric_windows++ WHERE scope = $1 AND scope_id = $2 AND window_days = $3 ORDER BY metric`, [scope, id, windowDays]);++ const docs = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, source FROM report_metric_json++ WHERE scope = $1 AND scope_id = $2 AND window_days = $3 ORDER BY metric`, [scope, id, windowDays]);++ return {++ as_of: new Date(now()).toISOString(),++ window_days: windowDays,++ scope,++ scope_id: id,++ metrics: rows.map((r) => ({ metric: r.metric, scope: r.scope, scope_id: r.scope_id, day: r.day, value: n(r.value), unit: r.unit, source: r.source })),++ windows: wins.map((r) => ({ metric: r.metric, scope: r.scope, scope_id: r.scope_id, window_days: r.window_days, value: n(r.value), unit: r.unit, source: r.source })),++ documents: docs.map((r) => ({ metric: r.metric, scope: r.scope, scope_id: r.scope_id, window_days: r.window_days, value: r.value, source: r.source })),++ };++}++++module.exports = { reportStats, channelStats, metrics, windowOf, loadIndex, NOTE, DAY };nnew file mode 100644@@@ -0,0 +1,137 @@++'use strict';++++/**++ * The shadow comparison: read the same window from OpenVibe.Network and from Report's tables, then record every++ * number that differs in report_shadow_diffs, one run per pass (report_shadow_runs). A clean run has zero diffs.++ *++ * The two sides are normalized with the same function the ingest uses (metricRowsFromStats), so an equal answer++ * compares equal; a changed number, a missing one or an extra stale one is a difference. Billing is not in the++ * shadow: Report quotes Billing directly and never keeps a second copy of money.++ */++const crypto = require('crypto');++const { metricRowsFromStats } = require('./ingest');++const { normaliseDays } = require('../data/source');++const { windowOf } = require('./query');++++const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));++const same = (a, b) => a == null || b == null ? a == b : Math.abs(a - b) < 1e-6;++function stable(v) {++ if (Array.isArray(v)) return `[${v.map(stable).join(',')}]`;++ if (v && typeof v === 'object') return `{${Object.keys(v).sort().map((k) => `${JSON.stringify(k)}:${stable(v[k])}`).join(',')}}`;++ return JSON.stringify(v == null ? null : v);++}++const keyOf = (r) => r.kind === 'window'++ ? `w|${r.scope}|${r.scope_id}|${r.metric}|${r.window_days}`++ : `d|${r.scope}|${r.scope_id}|${r.metric}|${r.day}`;++++function runId() { return `shr_${crypto.randomBytes(10).toString('hex')}`; }++++async function reportRows(store, days, nowMs) {++ const w = windowOf(nowMs, days);++ const rows = await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, value FROM report_metric_days++ WHERE scope IN ('network', 'site') AND day >= $1`, [w.currentStart]);++ const wins = await store.db.many(`SELECT metric, scope, scope_id, window_days, value FROM report_metric_windows++ WHERE scope IN ('network', 'site') AND window_days = $1`, [days]);++ return [++ ...rows.map((r) => ({ kind: 'day', metric: r.metric, scope: r.scope, scope_id: r.scope_id, day: r.day, value: num(r.value) })),++ ...wins.map((r) => ({ kind: 'window', metric: r.metric, scope: r.scope, scope_id: r.scope_id, window_days: r.window_days, value: num(r.value) })),++ ];++}++++function createShadow({ store, network, now = () => Date.now(), log = console }) {++ if (!store || !store.db) throw new TypeError('createShadow needs an open Report store');++++ /** Compare one window. `stats` may be supplied (a fixture or a cached Network answer); otherwise Network is asked. */++ async function compare({ days = 7, stats } = {}) {++ const windowDays = normaliseDays(days);++ const id = runId();++ const startedAt = new Date(now()).toISOString();++ await store.db.prepare(`INSERT INTO report_shadow_runs (run_id, window_days, status, checked, differences, started_at)++ VALUES (?, ?, 'running', 0, 0, ?)`).run(id, windowDays, startedAt);++ try {++ let source = stats;++ if (source === undefined) source = await network.stats(windowDays);++ if (!source) {++ await store.db.prepare(`UPDATE report_shadow_runs SET status = 'network_unavailable', checked = 0, differences = 0, finished_at = ? WHERE run_id = ?`)++ .run(new Date(now()).toISOString(), id);++ return { run_id: id, status: 'network_unavailable', checked: 0, differences: 0, diffs: [] };++ }++ const networkRows = metricRowsFromStats(source);++ const stored = await reportRows(store, windowDays, now());++ const reportByKey = new Map(stored.map((r) => [keyOf(r), r]));++ const netKeys = new Set(networkRows.map(keyOf));++ const keys = new Set([...netKeys, ...stored.map(keyOf)]);++ const diffs = [];++ let checked = 0;++ const reportJson = await store.db.maybe(`SELECT value FROM report_metric_json WHERE metric = 'busiest_pages' AND scope = 'network' AND scope_id = 'network' AND window_days = $1`, [windowDays]);++ const networkJson = Array.isArray(source.busiest_pages_24h) ? source.busiest_pages_24h : null;++ if ((networkJson || reportJson) && stable(networkJson) !== stable(reportJson ? reportJson.value : null)) {++ checked++;++ diffs.push({++ metric: 'busiest_pages', scope: 'network', scope_id: 'network', day: null, window_days: windowDays,++ network_value: null, report_value: null, delta: null,++ network_json: networkJson, report_json: reportJson ? reportJson.value : null,++ });++ }++ for (const key of keys) {++ const net = networkRows.find((r) => keyOf(r) === key) || null;++ const rep = reportByKey.get(key) || null;++ if (!net && !rep) continue;++ // Only compare a stored number whose metric Network actually reports in this window; an unrelated++ // stored scope is not a shadow difference.++ if (!net && !netKeys.has(key) && !networkRows.some((r) => r.metric === (rep && rep.metric))) continue;++ checked++;++ const nv = net ? net.value : null;++ const rv = rep ? rep.value : null;++ if (same(nv, rv)) continue;++ diffs.push({++ metric: (net || rep).metric, scope: (net || rep).scope, scope_id: (net || rep).scope_id,++ day: (net || rep).kind === 'day' ? (net || rep).day : null,++ window_days: (net || rep).kind === 'window' ? (net || rep).window_days : null,++ network_value: nv, report_value: rv, delta: (rv == null ? 0 : rv) - (nv == null ? 0 : nv),++ });++ }++ for (const d of diffs) {++ await store.db.prepare(`INSERT INTO report_shadow_diffs (run_id, metric, scope, scope_id, day, window_days, network_value, report_value, delta, network_json, report_json)++ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?::jsonb, ?::jsonb)`)++ .run(id, d.metric, d.scope, d.scope_id, d.day, d.window_days, d.network_value, d.report_value, d.delta,++ d.network_json == null ? null : JSON.stringify(d.network_json), d.report_json == null ? null : JSON.stringify(d.report_json));++ }++ await store.db.prepare(`UPDATE report_shadow_runs SET status = ?, checked = ?, differences = ?, finished_at = ? WHERE run_id = ?`)++ .run(diffs.length ? 'differences' : 'equal', checked, diffs.length, new Date(now()).toISOString(), id);++ return { run_id: id, status: diffs.length ? 'differences' : 'equal', checked, differences: diffs.length, diffs };++ } catch (err) {++ const message = String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 400);++ await store.db.prepare(`UPDATE report_shadow_runs SET status = 'error', error = ?, finished_at = ? WHERE run_id = ?`)++ .run(message, new Date(now()).toISOString(), id).catch(() => {});++ if (log && log.warn) log.warn(`[OpenVibe.Report] shadow run failed: ${message}`);++ return { run_id: id, status: 'error', error: message, checked: 0, differences: 0, diffs: [] };++ }++ }++++ /** The latest run(s) and their differences, for GET /internal/shadow. */++ async function summary({ limit = 5, days = null } = {}) {++ const n = Math.min(Math.max(parseInt(limit, 10) || 5, 1), 50);++ const windowDays = days == null || days === '' ? null : normaliseDays(days);++ const runs = windowDays++ ? await store.db.many(`SELECT run_id, window_days, status, checked, differences, error, started_at, finished_at++ FROM report_shadow_runs WHERE window_days = $1 ORDER BY run_seq DESC LIMIT $2`, [windowDays, n])++ : await store.db.many(`SELECT run_id, window_days, status, checked, differences, error, started_at, finished_at++ FROM report_shadow_runs ORDER BY run_seq DESC LIMIT $1`, [n]);++ const latest = runs[0] || null;++ const diffs = latest ? await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, window_days, network_value, report_value, delta, network_json, report_json++ FROM report_shadow_diffs WHERE run_id = $1 ORDER BY id LIMIT 500`, [latest.run_id]) : [];++ return {++ as_of: new Date(now()).toISOString(),++ latest: latest ? { ...latest, checked: Number(latest.checked), differences: Number(latest.differences) } : null,++ runs: runs.map((r) => ({ ...r, checked: Number(r.checked), differences: Number(r.differences) })),++ diffs: diffs.map((d) => ({++ ...d, network_value: num(d.network_value), report_value: num(d.report_value), delta: num(d.delta),++ })),++ };++ }++++ return { compare, summary, runId };++}++++module.exports = { createShadow, reportRows };@@@ -10,8 +10,9 @@* /robots.txt, /sitemap.xml, /llms.txt crawl artifacts (http/discovery.js)* /api/health, /api/ready, /release.json, /release-metrics the plumbing (loopback scrape for /metrics)*-- * The service is stateless: no database, no sessions and no events. The two source clients are the only state-- * (their in-process caches). The plumbing is the skeleton's, modelled on OpenVibe.Help.++ * Report has its own database (server/db.js, migrations/) for the analytics it ingests, and reads Billing's money++ * through Billing's read API. The pages and GET /api/v1/stats answer from REPORT_SOURCE (network by default,++ * report after cutover); Report's tables are written by the ingest job and by OpenVibe.Events. No sessions.*/const path = require('path');const express = require('express');@@@ -21,11 +22,16 @@ const contracts = require('openvibe-contracts');const cache = require('openvibe-shared/cache-policy');const configLib = require('./config');++const { openStore } = require('./db');const { createNetworkClient } = require('./data/network');const { createLiveClient } = require('./data/live');++const { createBillingClient } = require('./data/billing');++const { createAnalytics } = require('./analytics');++const { createShadow } = require('./analytics/shadow');const { createApi } = require('./http/api');const { createPageRoutes } = require('./http/pages');const { createDiscoveryRoutes } = require('./http/discovery');++const { createInternalRoutes } = require('./http/internal');const { createServiceReadiness } = require('./observability');const views = require('./views');const { PUBLIC_DIR, assetVersion } = require('./assets');@@@ -33,18 +39,23 @@ const { PUBLIC_DIR, assetVersion } = require('./assets');const VERSION = require('../package.json').version;const PAGE_CACHE = 'public, max-age=120';--/** opts: config, fetchImpl, log, now, network, live, releaseEnv */++/** opts: config, store, fetchImpl, log, now, network, live, billing, analytics, shadow, releaseEnv */async function createApp(opts = {}) {const config = opts.config || configLib.load();const log = opts.log || console;const fetchImpl = opts.fetchImpl || globalThis.fetch;const now = opts.now || (() => Date.now());++ // `store: null` is a deliberate no-database process (most unit tests); undefined opens the configured one.++ const store = opts.store !== undefined ? opts.store : await openStore(config, { now, log });const network = opts.network || createNetworkClient({ config, fetchImpl, log, now });const live = opts.live || createLiveClient({ config, fetchImpl, log, now });++ const billing = opts.billing || createBillingClient({ config, fetchImpl, log, now });++ const analytics = opts.analytics || createAnalytics({ config, store, network, billing, now, log });++ const shadow = opts.shadow || (store ? createShadow({ store, network, now, log }) : null);views.configure({ baseUrl: config.baseUrl });-- const ctx = { config, log, fetchImpl, now, network, live };++ const ctx = { config, log, fetchImpl, now, store, network, live, billing, analytics, shadow };const app = express();app.disable('x-powered-by');app.set('trust proxy', config.trustProxy);@@@ -91,7 +102,7 @@ async function createApp(opts = {}) {// ── Machine endpoints ───────────────────────────────────app.get('/api/health', (_req, res) => res.set('Cache-Control', 'no-store').json({ status: 'ok', service: 'openvibe-report', version: VERSION }));release.mount(app, { registry: metrics.registry });-- const readiness = createServiceReadiness({ config, fetchImpl, release: release.release });++ const readiness = createServiceReadiness({ config, fetchImpl, release: release.release, store });ctx.readiness = readiness;app.get('/api/ready', readiness.handler);@@@ -108,6 +119,9 @@ async function createApp(opts = {}) {},}));++ // ── Loopback-only operator and Events routes ────────────++ app.use('/internal', createInternalRoutes({ analytics, shadow, config, now, log }));++// ── API: CORS-open, public for two minutes, 120 requests a minute per address ──app.use('/api/v1', (_req, res, next) => {res.set('Access-Control-Allow-Origin', '*');@@@ -121,7 +135,7 @@ async function createApp(opts = {}) {legacyHeaders: false,handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }),}));-- app.use('/api/v1', createApi({ network, live }));++ app.use('/api/v1', createApi({ network, live, analytics }));app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));// ── Crawl artifacts, then the pages ─────────────────────@@@ -4,19 +4,23 @@* OpenVibe.Report configuration. Every value comes from the environment (production: /etc/openvibe/report.env,* see .env.example). load(env) is pure so tests can build a config without touching process.env.*-- * Report is stateless: it has no database, no sessions and no events. It reads two loopback APIs:-- * OpenVibe.Network's open stats and OpenVibe.Live's Rundown and after-show reports.++ * Report reads Network's open stats and Live's Rundown and after-show reports over loopback. It also has its own++ * PostgreSQL database (migrations/) for the analytics it ingests, and reads revenue from Billing. REPORT_SOURCE++ * chooses which side the pages and GET /api/v1/stats answer from: `network` (today's behaviour) or `report`.*/require('dotenv').config();const trim = (s) => String(s || '').replace(/\/+$/, '');const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);++const bool = (v, def = false) => (v == null || v === '' ? def : !/^(0|false|no|off)$/i.test(String(v)));++const sourceOf = (v) => (String(v || '').toLowerCase() === 'report' ? 'report' : 'network');function load(env = process.env) {const nodeEnv = env.NODE_ENV || 'development';const isProduction = nodeEnv === 'production';const port = int(env.PORT, 5080);const baseUrl = trim(env.BASE_URL || (isProduction ? 'https://openvibe.report' : `http://localhost:${port}`));++ const networkPublicUrl = trim(env.OV_NETWORK_URL || 'https://openvibe.network');return {service: 'report',@@@ -28,11 +32,52 @@ function load(env = process.env) {baseUrl,trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 2,-- // The two sources. Both serve these read APIs on loopback; manifests/services/{network,live}.json in++ // Which source answers the pages and the stats API. `network` is the shadowing default: the pages keep++ // reading where they read today until an operator sets REPORT_SOURCE=report.++ source: sourceOf(env.REPORT_SOURCE),++++ // The two read-only sources. Both serve these APIs on loopback; manifests/services/{network,live}.json in// openvibe-contracts name the same defaults.++ // networkUrl is the loopback origin the read clients call (the manifest's internalOrigin).networkUrl: trim(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),++ networkPublicUrl,++ networkInternalUrl: trim(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),liveUrl: trim(env.OV_LIVE_INTERNAL_URL || 'http://127.0.0.1:3000'),++ // Report's own database (ADR-035). DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner++ // role); development without DATABASE_URL uses an embedded PGlite database under REPORT_PGLITE_DIR.++ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.REPORT_PGLITE_DIR || '' },++++ // OpenVibe.Billing's read API. Report holds no money and derives none: it quotes the amounts Billing++ // returns. The service token needs billing.stats.read (GET /api/v1/stats) and billing.balance.read++ // (GET /api/v1/transactions?subject=… for one channel's settled revenue and refunds).++ billing: {++ url: trim(env.OV_BILLING_INTERNAL_URL || 'http://127.0.0.1:4600'),++ token: env.REPORT_BILLING_TOKEN || '',++ timeoutMs: Math.max(300, int(env.REPORT_BILLING_TIMEOUT_MS, 5000)),++ },++ oauth: {++ tokenUrl: trim(env.OV_NETWORK_OAUTH_TOKEN_URL || `${networkPublicUrl}/oauth/token`),++ clientId: env.OV_OAUTH_CLIENT_ID || 'report',++ clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',++ },++++ // OpenVibe.Events → Report: live.stream.ended and the frame beacon. The secret signs a delivery (comma++ // separated for rotation, 32+ characters each); unset makes POST /internal/events answer 503.++ events: {++ secrets: String(env.REPORT_EVENTS_SECRET || '').split(',').map((x) => x.trim()).filter((x) => x.length >= 32),++ },++++ // The ingest and shadow jobs. REPORT_JOBS=0 turns both off (tests do this by never starting them).++ jobs: {++ enabled: bool(env.REPORT_JOBS, true),++ onBoot: bool(env.REPORT_JOBS_ON_BOOT, true),++ ingestIntervalMs: Math.max(30_000, int(env.REPORT_INGEST_INTERVAL_MS, 5 * 60_000)),++ ingestWindows: [7, 30, 90],++ shadowIntervalMs: Math.max(30_000, int(env.REPORT_SHADOW_INTERVAL_MS, 15 * 60_000)),++ shadowWindowDays: [7, 30, 90].includes(Number(env.REPORT_SHADOW_WINDOW_DAYS)) ? Number(env.REPORT_SHADOW_WINDOW_DAYS) : 7,++ },++// One caller may make this many API requests a minute (express-rate-limit, keyed by address).apiRateLimitPerMinute: Math.max(1, int(env.REPORT_API_RATE_LIMIT_PER_MIN, 120)),nnew file mode 100644@@@ -0,0 +1,129 @@++'use strict';++++/**++ * OpenVibe.Billing's read API. Report never computes money: it quotes the amounts Billing returns.++ *++ * GET /api/v1/stats?days=N billing.stats.read network-wide purchases and subscriptions++ * GET /api/v1/transactions?subject= billing.balance.read one subject's entries, for a channel's settled++ * revenue and refunds (creator_payable, vibes-bits)++ *++ * A service token is used. When OV_OAUTH_CLIENT_ID / OV_OAUTH_CLIENT_SECRET are set, Report asks Network's token++ * endpoint for the audience openvibe.billing; REPORT_BILLING_TOKEN is an explicit alternative for an operator++ * run. Unset means the revenue part is null, never zero.++ */++const { createServiceTokenClient } = require('openvibe-sdk/auth');++const { trim, warn, TIMEOUT_MS } = require('./source');++++const DEFAULTS = { url: 'http://127.0.0.1:4600', timeoutMs: 5000 };++const CAPABILITIES = ['billing.stats.read', 'billing.balance.read'];++const DAY = 86_400_000;++++const n = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));++++function createBillingClient({ config = {}, fetchImpl = globalThis.fetch, log = console, now = () => Date.now(), tokenProvider = null, timeoutMs } = {}) {++ const cfg = config.billing || {};++ const baseUrl = trim(cfg.url || config.billingUrl || DEFAULTS.url);++ const deadline = timeoutMs || cfg.timeoutMs || DEFAULTS.timeoutMs;++ let tokenClient = null;++ if (!tokenProvider && !cfg.token && config.oauth && config.oauth.clientId && config.oauth.clientSecret) {++ try {++ tokenClient = createServiceTokenClient({++ network: config.networkPublicUrl || config.networkUrl || 'https://openvibe.network',++ tokenUrl: config.oauth.tokenUrl,++ clientId: config.oauth.clientId,++ clientSecret: config.oauth.clientSecret,++ audience: 'openvibe.billing',++ scope: CAPABILITIES,++ fetch: fetchImpl,++ now,++ });++ } catch { tokenClient = null; }++ }++++ async function token() {++ if (tokenProvider) return await tokenProvider();++ if (cfg.token) return cfg.token;++ if (!tokenClient) return null;++ try { return await tokenClient.getToken({ audience: 'openvibe.billing' }); } catch (err) {++ warn(log, `billing token failed: ${(err && err.message) || 'no token'}`);++ return null;++ }++ }++++ async function get(path) {++ const auth = await token();++ if (!auth) return null;++ let res;++ try {++ res = await fetchImpl(`${baseUrl}${path}`, {++ headers: { Accept: 'application/json', Authorization: `Bearer ${auth}` },++ signal: typeof AbortSignal !== 'undefined' && typeof AbortSignal.timeout === 'function' ? AbortSignal.timeout(deadline) : undefined,++ });++ } catch (err) {++ warn(log, `billing ${path.split('?')[0]} failed: ${(err && err.message) || 'no response'}`);++ return null;++ }++ if (!res || !res.ok) {++ if (!res || res.status !== 404) warn(log, `billing ${path.split('?')[0]} failed: HTTP ${res ? res.status : 'no response'}`);++ return null;++ }++ try { return await res.json(); } catch (err) { warn(log, `billing ${path.split('?')[0]} failed: ${(err && err.message) || 'bad JSON'}`); return null; }++ }++++ /** Network-wide money counts, exactly as Billing answers them. */++ async function stats(days = 30) {++ const d = Math.min(Math.max(parseInt(days, 10) || 30, 1), 90);++ const body = await get(`/api/v1/stats?days=${d}`);++ return body ? { ...body, source: 'billing' } : null;++ }++++ /**++ * One channel's settled revenue and refunds, read from Billing's transaction history. The sums are of Billing's++ * own `creator_payable` entries; Report never derives a price, a fee or a rate.++ */++ async function revenue({ subject, days = 30, now: at = now() } = {}) {++ if (!subject) return null;++ const d = Math.min(Math.max(parseInt(days, 10) || 30, 1), 365);++ const since = at - d * DAY;++ let settledBits = 0;++ let refundedBits = 0;++ let transactions = 0;++ let cursor = null;++ for (let page = 0; page < 20; page++) {++ const suffix = cursor ? `&cursor=${encodeURIComponent(cursor)}` : '';++ const body = await get(`/api/v1/transactions?subject=${encodeURIComponent(subject)}&limit=200${suffix}`);++ if (!body || !Array.isArray(body.transactions)) return null;++ for (const txn of body.transactions) {++ if (!txn || txn.test === true) continue;++ const created = Date.parse(txn.created_at || '');++ if (!Number.isFinite(created) || created < since) continue;++ const refundLike = txn.type === 'refund' || txn.type === 'chargeback' || !!txn.reverses_txn;++ const earning = txn.type === 'donation' || txn.type === 'subscription' || txn.type === 'subscription_share';++ let seen = false;++ for (const e of Array.isArray(txn.entries) ? txn.entries : []) {++ if (!e || !e.account || e.account.kind !== 'creator_payable' || e.account.currency !== 'vibes-bits') continue;++ const amount = n(e.amount);++ if (amount == null) continue;++ seen = true;++ if (amount > 0 && earning) settledBits += amount;++ else if (amount < 0 && refundLike) refundedBits += -amount;++ }++ if (seen) transactions++;++ }++ cursor = body.next_cursor || null;++ if (!cursor) break;++ }++ return finish();++ function finish() {++ return {++ source: 'billing', capability: 'billing.balance.read', subject, days: d, currency: 'vibes-bits',++ as_of: new Date(at).toISOString(), settled_bits: settledBits, refunded_bits: refundedBits,++ net_bits: settledBits - refundedBits, transactions,++ };++ }++ }++++ return { stats, revenue, baseUrl, capabilities: CAPABILITIES, ttlMs: deadline, get };++}++++module.exports = { createBillingClient, CAPABILITIES, DEFAULTS, TIMEOUT_MS };nnew file mode 100644@@@ -0,0 +1,49 @@++'use strict';++++/**++ * OpenVibe.Report's own PostgreSQL database (ADR-035): the schema is migrations/NNNN_*.sql, applied at boot.++ *++ * Report ingests the analytics events OpenVibe.Network records (live.stream.ended, the navbar frame beacon)++ * and the daily numbers Network publishes; it keeps Billing's money untouched and reads it through Billing's++ * API. Nothing written here is a secret: a stream's public facts, counts and the ids of events already handled.++ */++const fs = require('fs');++const path = require('path');++const { createDb } = require('openvibe-sdk/db');++++const MIGRATIONS = path.join(__dirname, '..', 'migrations');++const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');++++async function openDb(config, { log = console, registry } = {}) {++ const dbConfig = (config && config.db) || {};++ if (!dbConfig.url) {++ if (config && config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh report)');++ const dir = dbConfig.pgliteDir || DEV_PGLITE;++ log.warn(`[OpenVibe.Report] DATABASE_URL unset: embedded PGlite database in ${dir} (development only, one process)`);++ fs.mkdirSync(dir, { recursive: true });++ const db = createDb({ pglite: dir, service: 'report', registry, log });++ await db.migrate({ dir: MIGRATIONS, log });++ return db;++ }++ if (!dbConfig.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');++ const owner = createDb({ url: dbConfig.directUrl, service: 'report-migrate', max: 1, log });++ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }++ return createDb({ url: dbConfig.url, service: 'report', registry, log });++}++++/** The store every module takes: the db handle, a clock (epoch ms; tests inject one) and transactions. */++function createStore(db, { now = () => Date.now() } = {}) {++ return {++ db,++ now,++ iso: () => new Date(now()).toISOString(),++ tx: async (fn) => await db.tx(fn),++ close: () => db.close(),++ };++}++++async function openStore(config, { now, log, registry } = {}) {++ return createStore(await openDb(config, { log, registry }), { now });++}++++module.exports = { openDb, openStore, createStore, MIGRATIONS };@@@ -20,7 +20,7 @@ const { normaliseDays, isSlug, isStreamId, isUsername, streamIdOf } = require('.const CACHE = 'public, max-age=120';--function createApi({ network, live }) {++function createApi({ network, live, analytics = null }) {const r = express.Router();r.use((req, res, next) => {@@@ -42,8 +42,9 @@ function createApi({ network, live }) {}r.get('/stats', async (req, res) => {-- const stats = await network.stats(normaliseDays(req.query.days));-- if (!stats) return notFound(req, res, 'report.stats_not_found', 'OpenVibe.Network did not answer with the open numbers.');++ const days = normaliseDays(req.query.days);++ const stats = analytics ? await analytics.stats(days) : await network.stats(days);++ if (!stats) return notFound(req, res, 'report.stats_not_found', 'The open numbers did not answer for this window.');return res.json(stats);});nnew file mode 100644@@@ -0,0 +1,91 @@++'use strict';++++/**++ * /internal — loopback-only operator and Events endpoints. nginx answers 404 for /internal/; the app also checks++ * that the socket really is loopback (a spoofed X-Forwarded-For cannot reach these).++ *++ * POST /internal/events OpenVibe.Events delivery (HMAC v2, REPORT_EVENTS_SECRET)++ * GET /internal/shadow?days=7 the latest shadow runs and their differences++ * POST /internal/shadow?days=7 run a comparison now++ * GET /internal/metrics?scope=&id=&days= the flat report.metric@1 rows++ * GET /internal/channels/:channel/analytics a channel's creator analytics and Billing revenue++ */++const express = require('express');++const contracts = require('openvibe-contracts');++const { parseDelivery } = require('openvibe-sdk/events');++++const LOOPBACK = new Set(['127.0.0.1', '::1', '::ffff:127.0.0.1', '::ffff:7f00:1']);++++function isLoopback(req) {++ const raw = req.socket && req.socket.remoteAddress ? String(req.socket.remoteAddress) : '';++ return LOOPBACK.has(raw) || LOOPBACK.has(raw.replace(/^::ffff:/, ''));++}++++function createInternalRoutes({ analytics = null, shadow = null, config = {}, now = () => Date.now(), log = console } = {}) {++ const r = express.Router();++ r.use((req, res, next) => {++ if (isLoopback(req)) return next();++ return contracts.http.sendProblem(res, 403, 'internal.loopback_only', { detail: 'This endpoint is for the host itself.', ctx: req.ov });++ });++++ r.get('/shadow', async (req, res) => {++ if (!shadow) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });++ try {++ const body = await shadow.summary({ limit: req.query.limit, days: req.query.days });++ res.set('Cache-Control', 'no-store').json(body);++ } catch (err) {++ log.error('[OpenVibe.Report] shadow summary failed:', err && err.message ? String(err.message).slice(0, 200) : err);++ contracts.http.sendProblem(res, 500, 'report.shadow_failed', { detail: 'The shadow summary could not be read.', ctx: req.ov });++ }++ });++++ r.post('/shadow', async (req, res) => {++ if (!shadow) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });++ try {++ const run = await shadow.compare({ days: req.query.days });++ res.set('Cache-Control', 'no-store').json(run);++ } catch (err) {++ log.error('[OpenVibe.Report] shadow run failed:', err && err.message ? String(err.message).slice(0, 200) : err);++ contracts.http.sendProblem(res, 500, 'report.shadow_failed', { detail: 'The shadow comparison could not run.', ctx: req.ov });++ }++ });++++ r.get('/metrics', async (req, res) => {++ if (!analytics) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });++ const body = await analytics.metrics({ scope: req.query.scope, scopeId: req.query.id, days: req.query.days });++ if (!body) return contracts.http.sendProblem(res, 404, 'report.metric_not_found', { detail: 'No such metric scope.', ctx: req.ov });++ res.set('Cache-Control', 'no-store').json(body);++ });++++ r.get('/channels/:channel/analytics', async (req, res) => {++ if (!analytics) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });++ const body = await analytics.channel(req.params.channel, req.query.days);++ if (!body) return contracts.http.sendProblem(res, 404, 'report.channel_not_found', { detail: 'No such channel in Report\'s tables.', ctx: req.ov });++ res.set('Cache-Control', 'no-store').json(body);++ });++++ // OpenVibe.Events delivery. The raw body is read before any JSON parser; the signature is v2 only.++ r.post('/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {++ const secrets = (config.events && config.events.secrets) || [];++ if (!secrets.length) return contracts.http.sendProblem(res, 503, 'report.events_disabled', { detail: 'REPORT_EVENTS_SECRET is not set.', ctx: req.ov });++ if (!analytics) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });++ const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0);++ let delivery = null;++ for (const secret of secrets) {++ delivery = parseDelivery(raw, req.headers, secret, { requireV2: true, now: now() });++ if (delivery) break;++ }++ if (!delivery) return contracts.http.sendProblem(res, 401, 'report.bad_signature', { detail: 'The delivery does not verify or is outside the replay window.', ctx: req.ov });++ try {++ const out = await analytics.recordEvent(delivery.event);++ return res.set('Cache-Control', 'no-store').json({ ok: true, event_id: delivery.event.event_id, ...out });++ } catch (err) {++ log.error('[OpenVibe.Report] event ingest failed:', err && err.message ? String(err.message).slice(0, 200) : err);++ return contracts.http.sendProblem(res, 500, 'report.ingest_failed', { detail: 'The event could not be stored; Events will retry.', ctx: req.ov });++ }++ });++++ return r;++}++++module.exports = { createInternalRoutes, isLoopback };@@@ -21,7 +21,7 @@ const { normaliseDays, isSlug, isStreamId, isUsername, streamIdOf } = require('.const CACHE = 'public, max-age=120';--function createPageRoutes({ network, live }) {++function createPageRoutes({ network, live, analytics = null }) {const r = express.Router();function send(res, html, status = 200) {@@@ -33,7 +33,7 @@ function createPageRoutes({ network, live }) {r.get('/', async (_req, res) => {const [stats, rundown, topics] = await Promise.all([-- network.stats(7),++ analytics ? analytics.stats(7) : network.stats(7),live.rundown(),live.topics(),]);@@@ -45,7 +45,7 @@ function createPageRoutes({ network, live }) {r.get('/stats', async (req, res) => {const days = normaliseDays(req.query.days);-- const stats = await network.stats(days);++ const stats = analytics ? await analytics.stats(days) : await network.stats(days);if (!stats) return notFound(res);return send(res, views.stats({ stats, days }));});@@@ -5,14 +5,16 @@* Listens on PORT (5080) behind nginx (deploy/).*/const { createApp } = require('./app');++const { createAnalyticsJobs } = require('./jobs/analytics');const { gracefulStop } = require('openvibe-sdk/service');/**-- * The process stop (openvibe-sdk/service): the HTTP drain runs, then anything the product added closes. Report-- * has no database, no key refresher and no subscriptions, so there is nothing else to close. Exported so a test can-- * inject `exit` and `signals: false`.++ * The process stop (openvibe-sdk/service): the HTTP drain runs, then the jobs stop and the database closes.++ * Exported so a test can inject `exit` and `signals: false`.*/--function createLifecycle({ server, exit, signals } = {}) {++function createLifecycle({ server, exit, signals, ctx } = {}) {++ const jobs = ctx && ctx.jobs;++ const store = ctx && ctx.store;return gracefulStop({name: 'OpenVibe.Report',server,@@@ -20,6 +22,8 @@ function createLifecycle({ server, exit, signals } = {}) {exit,signals,deadlineMs: 10_000,++ stop: [() => (jobs ? jobs.stop() : undefined)],++ close: [() => (store ? store.close() : undefined)],});}@@@ -27,8 +31,12 @@ async function start() {const { app, ctx } = await createApp();const { config } = ctx;++ const jobs = createAnalyticsJobs({ config, analytics: ctx.analytics, shadow: ctx.shadow, log: ctx.log });++ ctx.jobs = jobs;++ jobs.start();++const server = app.listen(config.port, config.host, () => {-- console.log(`[OpenVibe.Report] ${config.nodeEnv} on http://${config.host}:${config.port} → ${config.baseUrl} (network ${config.networkUrl}, live ${config.liveUrl})`);++ console.log(`[OpenVibe.Report] ${config.nodeEnv} on http://${config.host}:${config.port} → ${config.baseUrl} (source ${config.source}; network ${config.networkUrl}, live ${config.liveUrl}, billing ${config.billing.url})`);});server.keepAliveTimeout = 65_000;createLifecycle({ server, ctx });nnew file mode 100644@@@ -0,0 +1,73 @@++'use strict';++++/**++ * The two background jobs (server/index.js starts them; tests never do):++ *++ * ingest pull Network's open stats into Report's tables every REPORT_INGEST_INTERVAL_MS and at boot++ * shadow compare the same window from Network and Report every REPORT_SHADOW_INTERVAL_MS and at boot++ *++ * Neither job runs when REPORT_JOBS=0. A failed pass logs one short line and leaves the last good data in place;++ * the next pass retries. Overlapping passes in one process are skipped.++ */++function createAnalyticsJobs({ config = {}, analytics = null, shadow = null, log = console } = {}) {++ const cfg = config.jobs || {};++ const timers = [];++ let ingestBusy = false;++ let shadowBusy = false;++++ async function runIngest() {++ if (!analytics || ingestBusy) return null;++ ingestBusy = true;++ try {++ const out = [];++ for (const days of cfg.ingestWindows || [7, 30, 90]) out.push(await analytics.ingestNetwork({ days }));++ if (out.every((x) => x && x.ok === false && x.reason === 'network_did_not_answer')) log.warn('[OpenVibe.Report] ingest: Network did not answer; keeping the last numbers');++ return out;++ } catch (err) {++ log.warn(`[OpenVibe.Report] ingest failed: ${String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 200)}`);++ return null;++ } finally { ingestBusy = false; }++ }++++ async function runShadow() {++ if (!shadow || shadowBusy) return null;++ shadowBusy = true;++ try {++ const out = await shadow.compare({ days: cfg.shadowWindowDays || 7 });++ if (out && out.status === 'differences') log.warn(`[OpenVibe.Report] shadow: ${out.differences} difference(s) over ${cfg.shadowWindowDays || 7} days`);++ return out;++ } catch (err) {++ log.warn(`[OpenVibe.Report] shadow failed: ${String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 200)}`);++ return null;++ } finally { shadowBusy = false; }++ }++++ function every(ms, fn) {++ const t = setInterval(fn, ms);++ if (typeof t.unref === 'function') t.unref();++ timers.push(t);++ }++++ function start() {++ if (!cfg.enabled) return false;++ if (cfg.onBoot) {++ const t = setTimeout(async () => { await runIngest(); await runShadow(); }, 2_000);++ if (typeof t.unref === 'function') t.unref();++ timers.push(t);++ }++ every(cfg.ingestIntervalMs || 5 * 60_000, runIngest);++ every(cfg.shadowIntervalMs || 15 * 60_000, runShadow);++ return true;++ }++++ async function stop() {++ while (timers.length) { const t = timers.pop(); clearTimeout(t); clearInterval(t); }++ // Let an in-flight pass finish before the database closes (gracefulStop awaits this step).++ const until = Date.now() + 5_000;++ while ((ingestBusy || shadowBusy) && Date.now() < until) await new Promise((r) => setTimeout(r, 50));++ }++++ return { start, stop, runIngest, runShadow };++}++++module.exports = { createAnalyticsJobs };@@@ -3,12 +3,14 @@/*** Truthful readiness for GET /api/ready (openvibe-shared/ready, Track O).*-- * Report has no database, no sessions and no queues: its only dependencies are the two loopback read APIs.-- * Both checks are optional, so a source that does not answer makes the service degraded, never not_ready:-- * the process still serves the pages it can and the 503 would stop the deploy of a stateless front.++ * The two loopback read APIs are optional: a source that does not answer makes the service degraded, never++ * not_ready, because the process still serves the pages it can. When a PostgreSQL database is configured, its++ * check is required only after REPORT_SOURCE=report (before the cutover it is Report's shadow store, not the++ * answer to a page).** network optional OpenVibe.Network answers /api/health* live optional OpenVibe.Live answers /api/health++ * db required only when REPORT_SOURCE=report, and only when DATABASE_URL is configured*/const { createReadiness } = require('openvibe-shared/ready');@@@ -32,14 +34,26 @@ function createServiceReadiness({ config, fetchImpl = globalThis.fetch, release},});++ const checks = [++ healthOf('network', config.networkUrl),++ healthOf('live', config.liveUrl),++ ];++ if (config.db && config.db.url) {++ checks.push({++ name: 'db',++ required: config.source === 'report',++ description: 'Report\'s PostgreSQL database answers',++ async check() {++ if (!store || !store.db) return 'no database handle';++ try { await store.db.value('SELECT 1'); return true; } catch (err) { return `database did not answer: ${(err && err.message) || 'query failed'}`; }++ },++ });++ }return createReadiness({service: 'report',release,timeoutMs: config.readyTimeoutMs || 2500,-- checks: [-- healthOf('network', config.networkUrl),-- healthOf('live', config.liveUrl),-- ],++ checks,});}nnew file mode 100644@@@ -0,0 +1,58 @@++'use strict';++++/**++ * Billing's read API: network money counts (billing.stats.read) and one subject's settled creator_payable++ * entries (billing.balance.read). The stub checks the URL and the service bearer token; Report only sums the++ * amounts Billing returned, and excludes test rows.++ */++const assert = require('assert');++const { createBillingClient } = require('../server/data/billing');++const { SUBJECT, NOW, billingTransactions } = require('./fixtures/analytics');++const { check, done } = require('./helpers/boot');++++const json = (body, status = 200) => new Response(JSON.stringify(body), { status, headers: { 'content-type': 'application/json' } });++++(async () => {++ await check('a channel revenue read uses Billing\'s transactions endpoint and bearer capability', async () => {++ const calls = [];++ const client = createBillingClient({++ config: { billing: { url: 'http://billing.test', token: 'test-billing-token', timeoutMs: 1000 } },++ fetchImpl: async (url, opts) => {++ calls.push({ url: String(url), auth: opts.headers.Authorization });++ return json(billingTransactions);++ },++ now: () => NOW,++ });++ const revenue = await client.revenue({ subject: SUBJECT, days: 7, now: NOW });++ assert.deepStrictEqual(revenue, {++ source: 'billing', capability: 'billing.balance.read', subject: SUBJECT, days: 7, currency: 'vibes-bits',++ as_of: new Date(NOW).toISOString(), settled_bits: 500, refunded_bits: 100, net_bits: 400, transactions: 2,++ });++ assert.strictEqual(calls.length, 1);++ assert.strictEqual(calls[0].url, `http://billing.test/api/v1/transactions?subject=${encodeURIComponent(SUBJECT)}&limit=200`);++ assert.strictEqual(calls[0].auth, 'Bearer test-billing-token');++ assert.deepStrictEqual(client.capabilities, ['billing.stats.read', 'billing.balance.read']);++ });++++ await check('network money counts come from GET /api/v1/stats as Billing answers them', async () => {++ const body = { as_of: new Date(NOW).toISOString(), purchases: { bits_total: 1000, windows: { d: 10, w: 100, m: 200, pw: 50 }, series: [] }, subscriptions: { active: 3, windows: { d: 0, w: 1, m: 2, pw: 1 }, series: [] } };++ const urls = [];++ const client = createBillingClient({++ config: { billing: { url: 'http://billing.test', token: 't' } },++ fetchImpl: async (url) => { urls.push(String(url)); return json(body); },++ });++ const got = await client.stats(7);++ assert.deepStrictEqual(got, { ...body, source: 'billing' });++ assert.deepStrictEqual(urls, ['http://billing.test/api/v1/stats?days=7']);++ });++++ await check('no token and no OAuth client means null, and no request is made', async () => {++ let calls = 0;++ const client = createBillingClient({ config: { billing: { url: 'http://billing.test' } }, fetchImpl: async () => { calls++; return json({}); } });++ assert.strictEqual(await client.revenue({ subject: SUBJECT, days: 7, now: NOW }), null);++ assert.strictEqual(await client.stats(7), null);++ assert.strictEqual(calls, 0);++ });++++ done();++})().catch((e) => { console.error(e); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,45 @@++'use strict';++++/**++ * POST /internal/events is loopback-only and verifies the OpenVibe.Events v2 signature. A signed delivery is++ * ingested once; a replay returns duplicate and writes nothing; a forged signature is refused.++ */++const assert = require('assert');++const { signDeliveryV2 } = require('openvibe-sdk/events');++const { boot, check, done } = require('./helpers/boot');++const { streams, NOW } = require('./fixtures/analytics');++++const SECRET = 'test-report-events-secret-32-characters-long';++++(async () => {++ const t = await boot({ db: true, now: () => NOW, env: { REPORT_EVENTS_SECRET: SECRET } });++ try {++ const raw = JSON.stringify({ event: streams[0] });++ const ts = Math.floor(NOW / 1000);++ const headers = {++ 'content-type': 'application/json',++ 'x-openvibe-timestamp': String(ts),++ 'x-openvibe-signature-v2': signDeliveryV2(raw, SECRET, ts),++ };++++ await check('a signed live.stream.ended is ingested once, and a replay is a duplicate', async () => {++ const first = await t.get('/internal/events', { method: 'POST', headers, body: raw });++ assert.strictEqual(first.status, 200, first.text);++ assert.strictEqual(first.json().ok, true);++ assert.strictEqual(first.json().duplicate, false);++ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 1);++ const again = await t.get('/internal/events', { method: 'POST', headers, body: raw });++ assert.strictEqual(again.status, 200, again.text);++ assert.strictEqual(again.json().duplicate, true);++ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 1);++ });++++ await check('a forged signature is refused before any write', async () => {++ const bad = { ...headers, 'x-openvibe-signature-v2': signDeliveryV2(raw, 'another-secret-32-characters-long!!', ts) };++ const r = await t.get('/internal/events', { method: 'POST', headers: bad, body: raw });++ assert.strictEqual(r.status, 401);++ assert.strictEqual(r.json().code, 'report.bad_signature');++ });++ } finally { await t.close(); }++ done();++})().catch((e) => { console.error(e); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,123 @@++'use strict';++++/**++ * Fixed analytics facts for the tests: seven days ending 2026-10-28, two sites, two ended streams.++ * Every expected number is written down here, independent of the implementation.++ */++const NOW = Date.parse('2026-10-28T12:00:00.000Z');++const DAY = 86_400_000;++const DAYS = Array.from({ length: 7 }, (_, i) => new Date(NOW - (6 - i) * DAY).toISOString().slice(0, 10));++const SUBJECT = 'usr_01J0000000000000000000000A';++++const PAGVIEWS_DAILY = [20, 20, 20, 20, 20, 40, 60]; // 200 from the navbar site++const ACCOUNTS_DAILY = [0, 0, 1, 0, 1, 1, 0]; // 3 new++const COINS_DAILY = [10, 10, 10, 10, 10, 10, 10]; // 70 this window++++const stats = {++ generated_at: '2026-10-28T12:00:00.000Z',++ window_days: 7,++ note: 'Counts only, never about a person.',++ totals: {++ pageviews: 300, pageviews_prev: 200, change_pct: 50, sites_open: 2,++ services: { up: 1, checked: 2 }, accounts: 10, requests_24h: 1000,++ },++ pageviews_daily: DAYS.map((day, i) => ({ day, n: PAGVIEWS_DAILY[i] })),++ accounts: { total: 10, new: 3, prev: 2, daily: DAYS.map((day, i) => ({ day, n: ACCOUNTS_DAILY[i] })) },++ coins: {++ earned: 500, spent: 100, holders: 4, transactions: 20, earned_window: 70, earned_prev: 50,++ daily: DAYS.map((day, i) => ({ day, n: COINS_DAILY[i] })),++ },++ live: {++ live_now: 2, watching: 9, streamers: 5, streams: 6, visitors: 7, messages: 80, chatted_7d: 4,++ vods: 3, clips: 2, hours_watched: 11, hours_archived: 12, ai_moments: 1, pastes: 0, follows: 13,++ },++ busiest_pages_24h: [{ service: 'network', route: '/stats', requests: 100 }],++ sites: [++ {++ id: 'network', name: 'OpenVibe.Network', short: 'Network', host: 'openvibe.network', url: 'https://openvibe.network/',++ pageviews: 200, pageviews_prev: 150, change_pct: 33, daily: DAYS.map((day, i) => ({ day, n: PAGVIEWS_DAILY[i] })),++ unique_visitors: 20, traffic_24h: { requests: 800, server_errors: 2, p95_ms: 120 },++ },++ {++ id: 'live', name: 'OpenVibe.Live', short: 'Live', host: 'openvibe.live', url: 'https://openvibe.live/',++ pageviews: 100, pageviews_prev: 50, change_pct: 100, daily: null,++ unique_visitors: 30, signed_in_people: 10, traffic_24h: null,++ },++ ],++};++++const expected = {++ generated_at: new Date(NOW).toISOString(),++ window_days: 7,++ totals: {++ pageviews: 300, pageviews_prev: 200, change_pct: 50, sites_open: 2,++ services: { up: 1, checked: 2 }, accounts: 10, requests_24h: 1000,++ },++ pageviews_daily: DAYS.map((day, i) => ({ day, n: PAGVIEWS_DAILY[i] })),++ accounts: { total: 10, new: 3, prev: 2, daily: DAYS.map((day, i) => ({ day, n: ACCOUNTS_DAILY[i] })) },++ coins: {++ earned: 500, spent: 100, holders: 4, transactions: 20, earned_window: 70, earned_prev: 50,++ daily: DAYS.map((day, i) => ({ day, n: COINS_DAILY[i] })),++ },++ live: {++ live_now: 2, watching: 9, streamers: 5, streams: 6, visitors: 7, messages: 80, chatted_7d: 4,++ vods: 3, clips: 2, hours_watched: 11, hours_archived: 12, ai_moments: 1, pastes: 0, follows: 13,++ },++ busiest_pages_24h: [{ service: 'network', route: '/stats', requests: 100 }],++};++++function streamEnded(streamId, { channel = 'goosely', startedAt, endedAt, duration, peak, avg, chatters, messages, watch, subject = SUBJECT } = {}) {++ return {++ event_id: `evt_TESTSTREAM${streamId}`,++ event_type: 'live.stream.ended',++ source: 'live',++ occurred_at: endedAt,++ payload: {++ stream_id: streamId,++ channel: { username: channel, display_name: channel, url: `https://openvibe.live/@${channel}`, subject: { type: 'user', id: subject } },++ title: `Stream ${streamId}`,++ category: 'tech',++ protocol: 'webrtc',++ is_nsfw: false,++ started_at: startedAt,++ ended_at: endedAt,++ duration_seconds: duration,++ stats: { peak_viewers: peak, avg_viewers: avg, unique_chatters: chatters, messages, watch_minutes: watch },++ },++ };++}++++const streams = [++ streamEnded(101, { startedAt: '2026-10-26T10:00:00.000Z', endedAt: '2026-10-26T11:00:00.000Z', duration: 3600, peak: 50, avg: 20.5, chatters: 8, messages: 120, watch: 900 }),++ streamEnded(102, { startedAt: '2026-10-27T10:00:00.000Z', endedAt: '2026-10-27T10:30:00.000Z', duration: 1800, peak: 30, avg: 10, chatters: 4, messages: 40, watch: 300 }),++];++++const expectedChannel = {++ totals: { streams: 2, stream_seconds: 5400, peak_viewers: 50, unique_chatters: 12, messages: 160, watch_minutes: 1200, avg_viewers: 17 },++ daily: [++ { day: '2026-10-26', streams: 1, stream_seconds: 3600, peak_viewers: 50, unique_chatters: 8, messages: 120, watch_minutes: 900, avg_viewers: 20.5 },++ { day: '2026-10-27', streams: 1, stream_seconds: 1800, peak_viewers: 30, unique_chatters: 4, messages: 40, watch_minutes: 300, avg_viewers: 10 },++ ],++};++++const billingTransactions = {++ transactions: [++ {++ id: 'txn_1', type: 'donation', status: 'settled', test: false, created_at: '2026-10-27T09:00:00.000Z',++ entries: [++ { account: { kind: 'creator_payable', owner: SUBJECT, currency: 'vibes-bits' }, amount: 500 },++ { account: { kind: 'user_credit', owner: 'usr_01J0000000000000000000000B', currency: 'vibes-bits' }, amount: -500 },++ ],++ },++ {++ id: 'txn_2', type: 'refund', status: 'settled', test: false, reverses_txn: 'txn_1', created_at: '2026-10-28T09:00:00.000Z',++ entries: [{ account: { kind: 'creator_payable', owner: SUBJECT, currency: 'vibes-bits' }, amount: -100 }],++ },++ {++ id: 'txn_3', type: 'donation', status: 'settled', test: true, created_at: '2026-10-28T09:30:00.000Z',++ entries: [{ account: { kind: 'creator_payable', owner: SUBJECT, currency: 'vibes-bits' }, amount: 999 }],++ },++ ],++};++++module.exports = { NOW, DAY, DAYS, SUBJECT, stats, expected, streams, expectedChannel, billingTransactions };@@@ -6,15 +6,19 @@ process.env.RELEASE_COMMIT = process.env.RELEASE_COMMIT || '0000000';/*** Boots OpenVibe.Report on an ephemeral port with a stub fetch (no network, no running site) and returns a small-- * HTTP client. Each test file gets its own app, clients and captured log lines.++ * HTTP client. Each test file gets its own app, clients and captured log lines. `boot({ db: true })` adds a migrated++ * PGlite and `store` (the analytics tests); without it the app runs with `store: null`, as a network-only front.** const t = await boot();* t.get('/api/v1/stats?days=30') → { status, headers, text, json() }++ * t.get('/internal/events', { method: 'POST', headers, body }) → a raw-body request* t.logs()*/const http = require('http');const configLib = require('../../server/config');const { createApp } = require('../../server/app');++const { createStore } = require('../../server/db');++const { testDb } = require('./db');const NETWORK = 'http://127.0.0.1:4000';const LIVE = 'http://127.0.0.1:3000';@@@ -152,11 +156,18 @@ async function boot(opts = {}) {...(opts.env || {}),};const config = configLib.load(env);++ // Most tests never touch the database; `boot({ db: true })` gives them a migrated PGlite of their own.++ const testdatabase = (opts.db === true || opts.store) ? await testDb({ store: opts.dbStore }) : null;++ const store = opts.store || (testdatabase ? createStore(testdatabase.db, { now: opts.now }) : null);const built = await createApp({config,++ store,fetchImpl,log,now: opts.now,++ billing: opts.billing,++ analytics: opts.analytics,++ shadow: opts.shadow,releaseEnv: { ...process.env, RELEASE_COMMIT: '0000000' },});const server = await new Promise((resolve) => {@@@ -166,11 +177,11 @@ async function boot(opts = {}) {const base = `http://127.0.0.1:${server.address().port}`;async function get(pathname, o = {}) {-- const res = await fetch(base + pathname, {-- method: o.method || 'GET',-- headers: o.headers || {},-- redirect: 'manual',-- });++ const headers = { ...(o.headers || {}) };++ let body = o.body;++ if (o.json !== undefined) { body = JSON.stringify(o.json); headers['content-type'] = 'application/json'; }++ const method = o.method || (body ? 'POST' : 'GET');++ const res = await fetch(base + pathname, { method, headers, body, redirect: 'manual' });const text = await res.text();return {status: res.status,@@@ -181,11 +192,12 @@ async function boot(opts = {}) {}return {-- base, config, ctx: built.ctx, fetchImpl, get,++ base, config, ctx: built.ctx, fetchImpl, get, testdatabase,logs: () => captured.join('\n'),async close() {built.ctx.metricsStop();await new Promise((resolve) => server.close(resolve));++ if (testdatabase) await testdatabase.close().catch(() => {});},};}nnew file mode 100644@@@ -0,0 +1,12 @@++'use strict';++++/**++ * A migrated Report database for one test run (ADR-035), from openvibe-sdk/testing: PGlite by default; with++ * REPORT_TEST_STORE=pg the PostgreSQL + PgBouncer containers, with roles and a schema of this run's own.++ */++const { createTestDb, pgAvailable } = require('openvibe-sdk/testing');++const { MIGRATIONS } = require('../../server/db');++++const testDb = ({ store = process.env.REPORT_TEST_STORE || 'pglite', max = 4 } = {}) => createTestDb({ migrations: MIGRATIONS, store, service: 'report', max });++++module.exports = { testDb, pgAvailable };nnew file mode 100644@@@ -0,0 +1,68 @@++'use strict';++++/**++ * Ingest is idempotent: a repeated event id and a repeated stats snapshot change no row. The stream event writes++ * its own facts and its channel/day counts; the frame beacon is a daily total, not a page view per request.++ */++const assert = require('assert');++const { testDb } = require('./helpers/db');++const { createStore } = require('../server/db');++const { recordEvent, ingestStats } = require('../server/analytics/ingest');++const { streams, stats, NOW, DAYS } = require('./fixtures/analytics');++const { check, done } = require('./helpers/boot');++++(async () => {++ const testdatabase = await testDb();++ const store = createStore(testdatabase.db, { now: () => NOW });++ try {++ await check('a repeated live.stream.ended changes nothing: one inbox row, one stream, one stream count', async () => {++ const a = await recordEvent(store, streams[0]);++ const b = await recordEvent(store, streams[0]);++ assert.strictEqual(a.duplicate, false);++ assert.strictEqual(b.duplicate, true);++ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_inbox WHERE consumer = $1', ['report-events']), 1);++ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_events'), 1);++ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 1);++ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'streams' AND day = '2026-10-26'"), 1);++ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'messages' AND day = '2026-10-26'"), 120);++ });++++ await check('a second stream adds to the channel day but a redelivery with a new event id does not', async () => {++ await recordEvent(store, streams[1]);++ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'streams' AND day = '2026-10-27'"), 1);++ const corrected = JSON.parse(JSON.stringify(streams[1]));++ corrected.event_id = 'evt_TESTSTREAM102CORRECTED';++ const out = await recordEvent(store, corrected);++ assert.strictEqual(out.outcome, 'stream:updated');++ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'streams' AND day = '2026-10-27'"), 1, 'a corrected stream must not be counted twice');++ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 2);++ });++++ await check('a repeated stats snapshot is a no-op; the first one writes every day and window', async () => {++ const a = await ingestStats(store, stats);++ const b = await ingestStats(store, stats);++ assert.strictEqual(a.duplicate, false);++ assert.strictEqual(b.duplicate, true);++ assert.ok(a.rows > 20, `expected many metric rows, got ${a.rows}`);++ const dayRows = await store.db.value("SELECT COUNT(*)::int AS n FROM report_metric_days WHERE scope = 'network' AND metric IN ('pageviews', 'accounts_new', 'coins_earned')");++ assert.strictEqual(dayRows, DAYS.length * 3);++ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_windows WHERE scope = 'network' AND metric = 'pageviews' AND window_days = 7"), 300);++ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'site' AND scope_id = 'network' AND metric = 'pageviews' AND day = '2026-10-28'"), 60);++ assert.strictEqual(await store.db.value("SELECT SUM(hits)::int AS n FROM report_frame_days WHERE host = 'openvibe.network'"), 200);++ });++++ await check('the frame beacon stays one daily total, while a single hit adds once', async () => {++ const day = { event_id: 'evt_TESTFRAME1', event_type: 'network.frame.day', source: 'network', occurred_at: '2026-10-28T12:00:00.000Z', payload: { day: '2026-10-28', host: 'openvibe.tools', hits: 40 } };++ await recordEvent(store, day);++ await recordEvent(store, day);++ assert.strictEqual(await store.db.value("SELECT hits::int AS n FROM report_frame_days WHERE day = '2026-10-28' AND host = 'openvibe.tools'"), 40);++ const hit = { event_id: 'evt_TESTFRAME2', event_type: 'network.frame.hit', source: 'network', occurred_at: '2026-10-28T12:00:01.000Z', payload: { day: '2026-10-28', host: 'openvibe.tools' } };++ await recordEvent(store, hit);++ await recordEvent(store, hit);++ assert.strictEqual(await store.db.value("SELECT hits::int AS n FROM report_frame_days WHERE day = '2026-10-28' AND host = 'openvibe.tools'"), 41);++ });++ } finally {++ await testdatabase.close();++ }++ done();++})().catch((e) => { console.error(e); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,119 @@++'use strict';++++/**++ * Every metric the Report pages and openvibe.network/stats show, computed from a fixture and compared with the++ * fixture's own expected numbers: the network totals, the day series, every site, every channel by day, and the++ * Billing revenue read (stubbed) that Report quotes rather than calculates.++ */++const assert = require('assert');++const { testDb } = require('./helpers/db');++const { createStore } = require('../server/db');++const { recordEvent, ingestStats } = require('../server/analytics/ingest');++const { reportStats, channelStats, metrics } = require('../server/analytics/query');++const { streams, stats, expected, expectedChannel, NOW, SUBJECT, billingTransactions } = require('./fixtures/analytics');++const { check, done } = require('./helpers/boot');++++function billingStub() {++ const calls = [];++ return {++ calls,++ async revenue({ subject, days }) {++ calls.push({ subject, days });++ let settled = 0; let refunded = 0; let transactions = 0;++ for (const txn of billingTransactions.transactions) {++ if (txn.test) continue;++ let seen = false;++ for (const e of txn.entries) {++ if (e.account.kind !== 'creator_payable') continue;++ seen = true;++ if (e.amount > 0) settled += e.amount; else refunded += -e.amount;++ }++ if (seen) transactions++;++ }++ return { source: 'billing', capability: 'billing.balance.read', subject, days, currency: 'vibes-bits', settled_bits: settled, refunded_bits: refunded, net_bits: settled - refunded, transactions };++ },++ };++}++++(async () => {++ const testdatabase = await testDb();++ const store = createStore(testdatabase.db, { now: () => NOW });++ const billing = billingStub();++ try {++ await ingestStats(store, stats);++ for (const event of streams) await recordEvent(store, event);++++ await check('network stats: totals, day series, accounts, coins, live and busiest pages match the fixture', async () => {++ const got = await reportStats(store, 7, { now: () => NOW });++ assert.ok(got, 'no stats computed');++ assert.strictEqual(got.generated_at, expected.generated_at);++ assert.strictEqual(got.window_days, 7);++ assert.deepStrictEqual(got.totals, expected.totals);++ assert.deepStrictEqual(got.pageviews_daily, expected.pageviews_daily);++ assert.deepStrictEqual(got.accounts, expected.accounts);++ assert.deepStrictEqual(got.coins, expected.coins);++ assert.deepStrictEqual(got.live, expected.live);++ assert.deepStrictEqual(got.busiest_pages_24h, expected.busiest_pages_24h);++ });++++ await check('every site: page views, change, daily series, visitors and 24-hour traffic', async () => {++ const got = await reportStats(store, 7, { now: () => NOW });++ assert.strictEqual(got.sites.length, 2);++ const network = got.sites.find((s) => s.id === 'network');++ const live = got.sites.find((s) => s.id === 'live');++ assert.strictEqual(network.pageviews, 200);++ assert.strictEqual(network.pageviews_prev, 150);++ assert.strictEqual(network.change_pct, 33);++ assert.strictEqual(network.unique_visitors, 20);++ assert.deepStrictEqual(network.traffic_24h, { requests: 800, server_errors: 2, p95_ms: 120 });++ assert.strictEqual(network.daily.length, 7);++ assert.deepStrictEqual(network.daily.map((d) => d.n), [20, 20, 20, 20, 20, 40, 60]);++ assert.strictEqual(live.pageviews, 100);++ assert.strictEqual(live.pageviews_prev, 50);++ assert.strictEqual(live.unique_visitors, 30);++ assert.strictEqual(live.signed_in_people, 10);++ assert.strictEqual(live.daily, null, 'Live counts its own visits: no navbar day series');++ assert.strictEqual(live.traffic_24h, null);++ });++++ await check('a channel by day: streams, seconds, peak, chatters, messages, watch minutes and weighted average', async () => {++ const got = await channelStats(store, 'goosely', 7, { now: () => NOW, billing });++ assert.ok(got, 'no channel analytics');++ assert.strictEqual(got.creator, SUBJECT);++ assert.strictEqual(got.channel, 'goosely');++ assert.deepStrictEqual(got.totals, expectedChannel.totals);++ assert.deepStrictEqual(got.daily, expectedChannel.daily);++ assert.deepStrictEqual(got.streams.map((s) => s.stream_id), [102, 101]);++ });++++ await check('Billing revenue is quoted with its capability and never derived from an event', async () => {++ const before = billing.calls.length;++ const got = await channelStats(store, 'goosely', 7, { now: () => NOW, billing });++ assert.deepStrictEqual(got.revenue, {++ source: 'billing', capability: 'billing.balance.read', subject: SUBJECT, days: 7, currency: 'vibes-bits',++ settled_bits: 500, refunded_bits: 100, net_bits: 400, transactions: 2,++ });++ assert.strictEqual(billing.calls.length, before + 1);++ });++++ await check('the flat report.metric@1 view exposes the same numbers in windows and days', async () => {++ const got = await metrics(store, { scope: 'network', scopeId: 'network', days: 7, now: () => NOW });++ assert.strictEqual(got.window_days, 7);++ const pageviews = got.windows.find((m) => m.metric === 'pageviews');++ assert.strictEqual(pageviews.value, 300);++ assert.strictEqual(pageviews.unit, 'count');++ const daily = got.metrics.filter((m) => m.metric === 'pageviews');++ assert.strictEqual(daily.length, 7);++ assert.strictEqual(daily[daily.length - 1].value, 60);++ const channel = await metrics(store, { scope: 'channel', scopeId: 'goosely', days: 7, now: () => NOW });++ assert.strictEqual(channel.metrics.find((m) => m.metric === 'streams' && m.day === '2026-10-26').value, 1);++ });++++ await check('a window with no numbers is null, not zeroes', async () => {++ assert.strictEqual(await reportStats(store, 90, { now: () => Date.parse('2025-01-01T00:00:00Z') }), null);++ });++ } finally {++ await testdatabase.close();++ }++ done();++})().catch((e) => { console.error(e); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,58 @@++'use strict';++++/**++ * The shadow job: the same window from Network and from Report's tables. Equal numbers make one clean run with no++ * report_shadow_diffs rows; a changed number records exactly that difference, and GET /internal/shadow (loopback)++ * summarizes the latest run.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const { stats, NOW } = require('./fixtures/analytics');++++(async () => {++ const t = await boot({ db: true, now: () => NOW });++ try {++ await t.ctx.analytics.ingestStats(stats);++++ await check('no difference when Network and Report answer the same numbers', async () => {++ const run = await t.ctx.shadow.compare({ days: 7, stats });++ assert.strictEqual(run.status, 'equal');++ assert.strictEqual(run.differences, 0);++ assert.ok(run.checked > 20, `checked ${run.checked}`);++ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_shadow_diffs'), 0);++ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_shadow_runs'), 1);++ });++++ await check('a changed number is recorded once and the internal summary shows it', async () => {++ const changed = JSON.parse(JSON.stringify(stats));++ changed.totals.pageviews = 301;++ const run = await t.ctx.shadow.compare({ days: 7, stats: changed });++ assert.strictEqual(run.status, 'differences');++ assert.strictEqual(run.differences, 1);++ assert.strictEqual(run.diffs[0].metric, 'pageviews');++ assert.strictEqual(run.diffs[0].scope, 'network');++ assert.strictEqual(run.diffs[0].network_value, 301);++ assert.strictEqual(run.diffs[0].report_value, 300);++ assert.strictEqual(run.diffs[0].delta, -1);++ const r = await t.get('/internal/shadow?days=7');++ assert.strictEqual(r.status, 200);++ const body = r.json();++ assert.strictEqual(body.latest.status, 'differences');++ assert.strictEqual(body.latest.differences, 1);++ assert.strictEqual(body.diffs.length, 1);++ assert.strictEqual(body.diffs[0].metric, 'pageviews');++ });++++ await check('a Network that does not answer is a run with no differences, never a clean claim', async () => {++ const down = await t.ctx.shadow.compare({ days: 7, stats: null });++ assert.strictEqual(down.status, 'network_unavailable');++ assert.strictEqual(down.differences, 0);++ const r = await t.get('/internal/shadow');++ assert.strictEqual(r.status, 200);++ assert.strictEqual(r.json().latest.status, 'network_unavailable');++ });++ } finally {++ await t.close();++ }++ done();++})().catch((e) => { console.error(e); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,54 @@++'use strict';++++/**++ * REPORT_SOURCE switches the answer: `network` (default) keeps today's Network numbers; `report` answers the same++ * route and the same page from Report's tables. A report source with no data is a 404, not a made-up zero.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const { stats, expected, NOW } = require('./fixtures/analytics');++++(async () => {++ const network = await boot({ now: () => NOW });++ try {++ await check('the default source is network: GET /api/v1/stats and /stats stay on Network', async () => {++ assert.strictEqual(network.config.source, 'network');++ const api = await network.get('/api/v1/stats?days=7');++ assert.strictEqual(api.status, 200);++ assert.strictEqual(api.json().totals.pageviews, 77408);++ const page = await network.get('/stats?days=7');++ assert.strictEqual(page.status, 200);++ assert.ok(page.text.includes('77,408'));++ });++ } finally { await network.close(); }++++ const report = await boot({ db: true, now: () => NOW, env: { REPORT_SOURCE: 'report' } });++ try {++ await report.ctx.analytics.ingestStats(stats);++ await check('REPORT_SOURCE=report answers the stats API from Report', async () => {++ assert.strictEqual(report.config.source, 'report');++ const r = await report.get('/api/v1/stats?days=7');++ assert.strictEqual(r.status, 200);++ assert.deepStrictEqual(r.json().totals, expected.totals);++ assert.strictEqual(r.json().totals.pageviews, 300);++ assert.notStrictEqual(r.json().totals.pageviews, 77408);++ });++ await check('REPORT_SOURCE=report answers the /stats page from Report', async () => {++ const r = await report.get('/stats?days=7');++ assert.strictEqual(r.status, 200);++ assert.ok(r.text.includes('300'), 'the report page views are on the page');++ assert.ok(!r.text.includes('77,408'), 'the Network stub must not be read when the source is report');++ });++ } finally { await report.close(); }++++ const empty = await boot({ db: true, now: () => NOW, env: { REPORT_SOURCE: 'report' } });++ try {++ await check('a report source with no rows is 404, not zeroes', async () => {++ const r = await empty.get('/api/v1/stats?days=7');++ assert.strictEqual(r.status, 404);++ assert.match(r.headers.get('content-type') || '', /^application\/problem\+json/);++ });++ } finally { await empty.close(); }++++ done();++})().catch((e) => { console.error(e); process.exit(1); });
@@ -6,13 +6,43 @@ HOST=127.0.0.1
@@ -6,13 +6,43 @@ HOST=127.0.0.1
BASE_URL=https://openvibe.report
BASE_URL=https://openvibe.report
TRUST_PROXY=2
TRUST_PROXY=2
-# OpenVibe.Network's open stats (GET /api/v1/stats?days=7|30|90). Both sources serve these read APIs on loopback;
-# the defaults are the internalOrigin values in openvibe-contracts' manifests/services/{network,live}.json.
+# Report's own PostgreSQL (ADR-035). The runtime role serves through PgBouncer on DATABASE_URL; migrations run
+# on DATABASE_DIRECT_URL as the owner. Without DATABASE_URL development uses an embedded PGlite database.
+DATABASE_URL=
+DATABASE_DIRECT_URL=
+#REPORT_PGLITE_DIR=./data/pglite
+
+# Which source the pages and GET /api/v1/stats answer from: network (today's behaviour) or report (after cutover).
+REPORT_SOURCE=network
+
+# OpenVibe.Network's open stats (GET /api/v1/stats?days=7|30|90). Both sources serve their read APIs on loopback.
OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
+# The public origin, used only for the OAuth token endpoint (Report's service token to Billing).
+OV_NETWORK_URL=https://openvibe.network
# OpenVibe.Live's Rundown and after-show reports (GET /api/rundown/*, /api/recap/*).
# OpenVibe.Live's Rundown and after-show reports (GET /api/rundown/*, /api/recap/*).
OV_LIVE_INTERNAL_URL=http://127.0.0.1:3000
OV_LIVE_INTERNAL_URL=http://127.0.0.1:3000
+# OpenVibe.Billing's read API. Report quotes Billing and never computes money. The service needs the grants
+# billing.stats.read (GET /api/v1/stats) and billing.balance.read (GET /api/v1/transactions?subject=…).
+OV_BILLING_INTERNAL_URL=http://127.0.0.1:4600
+# Report's Network OAuth client (client_credentials → audience openvibe.billing). The secret stays in the env file.
+OV_OAUTH_CLIENT_ID=report
+OV_OAUTH_CLIENT_SECRET=
+# An explicit Billing bearer token for an operator run (instead of the OAuth client); never commit it.
+#REPORT_BILLING_TOKEN=
+
+# OpenVibe.Events → Report (POST /internal/events): live.stream.ended and the frame beacon. Comma-separated for
+# rotation, 32+ characters each; unset makes the route answer 503.
+REPORT_EVENTS_SECRET=
+
+# The ingest and shadow jobs (on by default; REPORT_JOBS=0 turns both off).
+#REPORT_JOBS=1
+#REPORT_JOBS_ON_BOOT=1
+#REPORT_INGEST_INTERVAL_MS=300000
+#REPORT_SHADOW_WINDOW_DAYS=7
+#REPORT_SHADOW_INTERVAL_MS=900000
+
# Optional: the per-address API limit (default 120 a minute) and the readiness probe timeout (default 2500 ms).
# Optional: the per-address API limit (default 120 a minute) and the readiness probe timeout (default 2500 ms).
#REPORT_API_RATE_LIMIT_PER_MIN=120
#REPORT_API_RATE_LIMIT_PER_MIN=120
#REPORT_READY_TIMEOUT_MS=2500
#REPORT_READY_TIMEOUT_MS=2500
@@ -8,4 +8,5 @@ node_modules/
@@ -8,4 +8,5 @@ node_modules/
*.pem
*.pem
*.key
*.key
coverage/
coverage/
+/data/
.DS_Store
.DS_Store
@@ -2,8 +2,9 @@
@@ -2,8 +2,9 @@
> Every number on OpenVibe, honest.
> Every number on OpenVibe, honest.
-**Status:** v0.1.0 service skeleton (see [STATUS.json](STATUS.json)). The routes, the API and the plumbing are
-complete; the views are plain placeholders the main session will design.
+**Status:** v0.1.0 service, now with its own analytics database and a shadow of Network's numbers (REP-2, plan
+2026-10-28 §4.8 and §5.4). The pages still read where they read today; `REPORT_SOURCE=report` switches the numbers
+when the cutover is approved.
**Domain:** `openvibe.report` · **Port:** 5080 · **Service id:** `report` · **Env prefix:** `OV_` / `REPORT_`
**Domain:** `openvibe.report` · **Port:** 5080 · **Service id:** `report` · **Env prefix:** `OV_` / `REPORT_`
**License:** AGPL-3.0 (same as every OpenVibe service).
**License:** AGPL-3.0 (same as every OpenVibe service).
@@ -12,29 +13,54 @@ about, and each stream's after-show report.
@@ -12,29 +13,54 @@ about, and each stream's after-show report.
## Purpose
## Purpose
-OpenVibe.Report is a stateless read-only front. It has no database, no sign-in and no events. It asks two loopback
-APIs for their public answers, holds each answer in this process for a few minutes, and renders it:
+OpenVibe.Report is the analytics authority in shadow:
+
+- It **ingests** the analytics OpenVibe.Network records — `live.stream.ended` for creator analytics and the shared
+ navbar's anonymous frame beacon — and Network's open-stats answer (`GET /api/v1/stats?days=7|30|90`) into its own
+ PostgreSQL tables (migrations/).
+- It **computes** the same numbers Network answers, site-wide, per site and per channel, by UTC day.
+- It **quotes money from OpenVibe.Billing** through Billing's read API; it never derives a price, fee or amount.
+- It **shadows**: a job compares the same window from Network and from Report's tables and records every difference
+ in `report_shadow_diffs`.
+
+The pages and `GET /api/v1/stats` answer from `REPORT_SOURCE`, which defaults to `network`: today's behaviour. Setting
+`REPORT_SOURCE=report` makes them answer from Report's tables. The Rundown and after-show reports still come from
+OpenVibe.Live in both modes.
| Source | Read | Used for |
| Source | Read | Used for |
|---|---|---|
|---|---|---|
-| **OpenVibe.Network** | `GET /api/v1/stats?days=7\|30\|90` | `/`, `/stats`, `GET /api/v1/stats` |
+| **OpenVibe.Network** | `GET /api/v1/stats?days=7\|30\|90` | `/`, `/stats`, `GET /api/v1/stats` while `REPORT_SOURCE=network`; also the ingest source |
+| **Report's tables** | `report_metric_days`, `report_metric_windows`, `report_streams` | the same routes while `REPORT_SOURCE=report` |
| **OpenVibe.Live** — The Rundown | `GET /api/rundown/`, `/api/rundown/topics`, `/api/rundown/t/:slug`, `/api/rundown/e/:streamId`, `/api/rundown/s/:username` | `/rundown`, `/rundown/t/:slug`, `/rundown/e/:streamId`, `/rundown/s/:username` |
| **OpenVibe.Live** — The Rundown | `GET /api/rundown/`, `/api/rundown/topics`, `/api/rundown/t/:slug`, `/api/rundown/e/:streamId`, `/api/rundown/s/:username` | `/rundown`, `/rundown/t/:slug`, `/rundown/e/:streamId`, `/rundown/s/:username` |
| **OpenVibe.Live** — after-show reports | `GET /api/recap/:streamId`, `GET /api/recap/channel/:username` | `/after-show/:streamId`, `/c/:username` |
| **OpenVibe.Live** — after-show reports | `GET /api/recap/:streamId`, `GET /api/recap/channel/:username` | `/after-show/:streamId`, `/c/:username` |
+| **OpenVibe.Billing** | `GET /api/v1/stats`, `GET /api/v1/transactions?subject=…` | the revenue read for a channel; network money counts |
A source that does not answer is null, after one short log line; the page (or API route) becomes a 404. The service
A source that does not answer is null, after one short log line; the page (or API route) becomes a 404. The service
never invents a zero and never throws a source error at the reader.
never invents a zero and never throws a source error at the reader.
+## What Report owns now
+
+- **The analytics tables** in `migrations/0001_initial.sql`: `report_inbox` (event ids already handled),
+ `report_events`, `report_streams`, `report_frame_days`, `report_metric_days`, `report_metric_windows`,
+ `report_metric_json`, `report_dimensions`, `report_shadow_runs` and `report_shadow_diffs`.
+- **The computation** in `server/analytics/`: `ingest.js` (idempotent writes), `query.js` (Network's stats shape and
+ a channel's creator analytics), `shadow.js` (the comparison).
+- **The jobs** in `server/jobs/analytics.js`: ingest every five minutes (7, 30 and 90-day windows) and shadow every
+ fifteen minutes (7 days by default).
+
## Does not own
## Does not own
-- **The numbers** — OpenVibe.Network's open stats. Report quotes them.
+- **Money** — OpenVibe.Billing. Report reads it with `billing.stats.read` and `billing.balance.read`; it holds no
+ ledger and computes no price.
- **The Rundown and the after-show reports** — OpenVibe.Live. Report quotes them, including their counts and text.
- **The Rundown and the after-show reports** — OpenVibe.Live. Report quotes them, including their counts and text.
-- **Identity, sessions and writes** — Report has none. It does not read cookies and has no POST routes.
+- **Identity, sessions and writes** — Report has no sign-in, no cookies and no public write route. `POST
+ /internal/events` is loopback-only and HMAC-verified.
- **The Shared Frame** — `openvibe-shared` (navbar, footer, theme-loader, static files), served from this site's own
- **The Shared Frame** — `openvibe-shared` (navbar, footer, theme-loader, static files), served from this site's own
pinned copy at `/shared/*`.
pinned copy at `/shared/*`.
## Pages
## Pages
-All server-rendered placeholders in `server/views/index.js`; every dynamic value goes through `esc()`.
+All server-rendered in `server/views/index.js`; every dynamic value goes through `esc()`.
| Page | What it shows |
| Page | What it shows |
|---|---|
|---|---|
@@ -52,11 +78,10 @@ status 404. Pages are `Cache-Control: public, max-age=120`.
@@ -52,11 +78,10 @@ status 404. Pages are `Cache-Control: public, max-age=120`.
## API
## API
-All under `/api/v1`, CORS `*`, `Cache-Control: public, max-age=120`, 120 requests a minute per address, and the
-source's answer as it came:
+All under `/api/v1`, CORS `*`, `Cache-Control: public, max-age=120`, 120 requests a minute per address:
```
```
-GET /api/v1/stats?days=7|30|90
+GET /api/v1/stats?days=7|30|90 REPORT_SOURCE=network|report
GET /api/v1/rundown
GET /api/v1/rundown
GET /api/v1/rundown/topics
GET /api/v1/rundown/topics
GET /api/v1/rundown/t/:slug
GET /api/v1/rundown/t/:slug
@@ -65,14 +90,32 @@ GET /api/v1/c/:username
@@ -65,14 +90,32 @@ GET /api/v1/c/:username
```
```
A bad parameter or a missing source answer is a 404 `application/problem+json` body built with
A bad parameter or a missing source answer is a 404 `application/problem+json` body built with
-`openvibe-contracts` `http.sendProblem`. The API is read-only; other methods get 405.
+`openvibe-contracts` `http.sendProblem`. The public API is read-only; other methods get 405.
-## Crawl artifacts
+### Internal (loopback only)
-- `/robots.txt` — the public pages are crawlable; `/api/` is not; the sitemap is named.
-- `/sitemap.xml` — `/`, `/stats`, `/rundown` and every `/rundown/t/<slug>` from the Rundown's topics (the three
- static pages remain when the topics source is down). `lastmod` comes from `STATUS.json`.
-- `/llms.txt` — the site, its pages and its machine-readable answers.
+```
+POST /internal/events OpenVibe.Events delivery (v2 HMAC, REPORT_EVENTS_SECRET)
+GET /internal/shadow?days=7&limit=5 the latest shadow runs and their differences
+POST /internal/shadow?days=7 run a comparison now
+GET /internal/metrics?scope=&id=&days= the flat report.metric@1 rows
+GET /internal/channels/:channel/analytics a channel's creator analytics plus Billing revenue
+```
+
+nginx refuses `/internal/` from outside; the app also checks the socket is loopback. The flat rows are proposed as
+`report.metric@1` in [docs/report.metric.v1.json](docs/report.metric.v1.json) for Contracts to own.
+
+## Ingest and idempotency
+
+- `live.stream.ended` → one `report_streams` row (same fields as Network's `creator_streams`) and the channel/day
+ counts. A redelivered event id is a no-op; a corrected copy with a new event id replaces the stream and does not
+ count it twice.
+- `network.frame.hit` / `network.frame.day` → `report_frame_days` (day, host, hits).
+- `GET /api/v1/stats` → `report_metric_days`, `report_metric_windows`, `report_metric_json` and site dimensions,
+ keyed by the answer's content hash. Polling the same numbers again is a no-op.
+- Every delivery is claimed in `report_inbox` (openvibe-sdk/events `createPgInbox`) in the same transaction as its
+ writes.
+- Billing events are deliberately not ingested: money comes from Billing's read API.
## Configuration
## Configuration
@@ -85,8 +128,22 @@ Every value comes from the environment (production: `/etc/openvibe/report.env`).
@@ -85,8 +128,22 @@ Every value comes from the environment (production: `/etc/openvibe/report.env`).
| `HOST` | `127.0.0.1` | listen address |
| `HOST` | `127.0.0.1` | listen address |
| `BASE_URL` | `https://openvibe.report` in production, otherwise `http://localhost:5080` | canonical links, sitemap and llms.txt |
| `BASE_URL` | `https://openvibe.report` in production, otherwise `http://localhost:5080` | canonical links, sitemap and llms.txt |
| `TRUST_PROXY` | `2` | Express trust proxy (the app sets no cookies, but the per-address limit uses the client address) |
| `TRUST_PROXY` | `2` | Express trust proxy (the app sets no cookies, but the per-address limit uses the client address) |
-| `OV_NETWORK_INTERNAL_URL` | `http://127.0.0.1:4000` | OpenVibe.Network's loopback origin (its manifest's `internalOrigin`) |
-| `OV_LIVE_INTERNAL_URL` | `http://127.0.0.1:3000` | OpenVibe.Live's loopback origin (its manifest's `internalOrigin`) |
+| `DATABASE_URL` | unset in development | Report's PostgreSQL through PgBouncer; without it development uses PGlite |
+| `DATABASE_DIRECT_URL` | unset | the owner connection migrations run on |
+| `REPORT_PGLITE_DIR` | `./data/pglite` | the embedded development database |
+| `REPORT_SOURCE` | `network` | `network` keeps today's behaviour; `report` answers from Report's tables |
+| `OV_NETWORK_INTERNAL_URL` | `http://127.0.0.1:4000` | Network's loopback origin (its manifest's `internalOrigin`) |
+| `OV_NETWORK_URL` | `https://openvibe.network` | the OAuth token endpoint's origin |
+| `OV_LIVE_INTERNAL_URL` | `http://127.0.0.1:3000` | Live's loopback origin |
+| `OV_BILLING_INTERNAL_URL` | `http://127.0.0.1:4600` | Billing's loopback origin |
+| `OV_OAUTH_CLIENT_ID` / `OV_OAUTH_CLIENT_SECRET` | `report` / unset | Report's client-credentials token for `openvibe.billing` |
+| `REPORT_BILLING_TOKEN` | unset | an explicit Billing bearer token for an operator run |
+| `REPORT_EVENTS_SECRET` | unset | the HMAC secret for `POST /internal/events` (32+ chars, comma list for rotation) |
+| `REPORT_JOBS` | `1` | `0` turns the ingest and shadow jobs off |
+| `REPORT_JOBS_ON_BOOT` | `1` | run both jobs once shortly after boot |
+| `REPORT_INGEST_INTERVAL_MS` | `300000` | how often the ingest job runs |
+| `REPORT_SHADOW_INTERVAL_MS` | `900000` | how often the shadow job runs |
+| `REPORT_SHADOW_WINDOW_DAYS` | `7` | the shadow window (7, 30 or 90) |
| `REPORT_API_RATE_LIMIT_PER_MIN` | `120` | per-address API requests a minute |
| `REPORT_API_RATE_LIMIT_PER_MIN` | `120` | per-address API requests a minute |
| `REPORT_READY_TIMEOUT_MS` | `2500` | how long a readiness probe may take |
| `REPORT_READY_TIMEOUT_MS` | `2500` | how long a readiness probe may take |
| `RELEASE_COMMIT` | unset | `/release.json`'s release; otherwise read from git in the checkout |
| `RELEASE_COMMIT` | unset | `/release.json`'s release; otherwise read from git in the checkout |
@@ -95,16 +152,30 @@ Every value comes from the environment (production: `/etc/openvibe/report.env`).
@@ -95,16 +152,30 @@ Every value comes from the environment (production: `/etc/openvibe/report.env`).
```sh
```sh
npm install
npm install
-npm start # http://127.0.0.1:5080
-npm test # every test/*.test.js, in its own process, with both sources stubbed
+npm start # http://127.0.0.1:5080; development uses the embedded PGlite database
+npm test # every test/*.test.js, in its own process
```
```
-The tests need no database, no network and no running OpenVibe site: `test/helpers/boot.js` supplies a URL-aware
-stub for both sources. They cover the clients' cache hit, expiry, failure and parameter rules; every page route and
-its 404s; the API shapes, CORS, cache and rate limit; robots/sitemap/llms; and health/readiness/static caching.
+Most tests stub both sources and pass `store: null`, so they need no database. The analytics tests use
+`test/helpers/db.js` (`createTestDb` from openvibe-sdk/testing) with a migrated PGlite, or the PostgreSQL containers
+with `REPORT_TEST_STORE=pg`. They cover ingest idempotency, every metric against a fixture's expected numbers, the
+shadow job's difference and clean run, `GET /internal/shadow`, the source flag, the Billing revenue read with a stub,
+and the existing pages/API/health/crawl behaviour.
`node --check` is run over `server/`, `test/` and `public/` by CI (`.github/workflows/ci.yml`, the shared
`node --check` is run over `server/`, `test/` and `public/` by CI (`.github/workflows/ci.yml`, the shared
-OpenVibe.Shared test workflow, no database service).
+OpenVibe.Shared test workflow).
+
+## Cutover
+
+1. Run the ingest and shadow jobs long enough to cover a full 90-day window, and check
+ `GET /internal/shadow` has zero differences (or only explained ones).
+2. Create Report's first-party grants on Network: `billing.stats.read` and `billing.balance.read` for the `report`
+ service (and the Network read grants when Report starts pulling creator analytics directly).
+3. Set `REPORT_SOURCE=report` in `/etc/openvibe/report.env` and restart; keep the Network ingest running.
+4. Leave Network writing during the dual-write window. Live's `ANALYTICS_SOURCE` can then move to Report; Network's
+ `/stats` becomes a 301 to `openvibe.report/stats`.
+5. After the N-1 window, a Contracts migration removes Network's analytics tables. Rollback before that is
+ `REPORT_SOURCE=network` and a restart.
## Deploy
## Deploy
@@ -114,15 +185,17 @@ The reference files are in `deploy/`:
@@ -114,15 +185,17 @@ The reference files are in `deploy/`:
`EnvironmentFile=/etc/openvibe/report.env`, `PORT=5080`, `SIGTERM` with a 10 s drain.
`EnvironmentFile=/etc/openvibe/report.env`, `PORT=5080`, `SIGTERM` with a 10 s drain.
- `deploy/nginx/openvibe.report.conf` — HTTP → HTTPS, `www` → apex 301, Let's Encrypt certificates under
- `deploy/nginx/openvibe.report.conf` — HTTP → HTTPS, `www` → apex 301, Let's Encrypt certificates under
`/etc/letsencrypt/live/openvibe.report/`, proxy to `127.0.0.1:5080`, the same security headers as Help's vhost,
`/etc/letsencrypt/live/openvibe.report/`, proxy to `127.0.0.1:5080`, the same security headers as Help's vhost,
- and `/api/` GET-only with a coarse rate limit.
+ `/api/` GET-only with a coarse rate limit, and `/internal/` refused from outside.
-`/api/ready` reports `network` and `live` as optional checks: one or both down is `status: "degraded"` with HTTP 200,
-so a stateless front is never marked not-ready just because a source is unavailable.
+`/api/ready` reports `network` and `live` as optional checks: one or both down is `status: "degraded"` with HTTP 200.
+When `DATABASE_URL` is configured, the database check is included and is required only after `REPORT_SOURCE=report`.
## Security
## Security
Every source text is escaped before it reaches HTML. Path parameters are checked against fixed patterns before any
Every source text is escaped before it reaches HTML. Path parameters are checked against fixed patterns before any
-request. Report holds no accounts, no cookies, no database and no write routes. See [SECURITY.md](SECURITY.md).
+request. Report holds no accounts and no cookies. Its database has public channel facts and counts only; event
+payloads with money or personal data are not stored. `/internal/` is loopback-only, and `POST /internal/events`
+verifies the Events v2 HMAC. See [SECURITY.md](SECURITY.md).
<!-- versions:start -->
<!-- versions:start -->
- openvibe-contracts: v0.225.0
- openvibe-contracts: v0.225.0
@@ -16,9 +16,11 @@ Every OpenVibe service and site (openvibe.live, openvibe.network, openvibe.tools
@@ -16,9 +16,11 @@ Every OpenVibe service and site (openvibe.live, openvibe.network, openvibe.tools
openvibe.games and the other openvibe.* domains) and every repository under github.com/OpenVibers. Machine-readable
openvibe.games and the other openvibe.* domains) and every repository under github.com/OpenVibers. Machine-readable
contact details are at `/.well-known/security.txt` on each site.
contact details are at `/.well-known/security.txt` on each site.
-For OpenVibe.Report specifically: the service reads two internal APIs and has no database, no accounts and no
-writes. The interesting surface is the path parameters it forwards (validated before a request), the HTML it renders
-(every source text is escaped by `server/views/index.js`) and the two source origins it is configured with.
+For OpenVibe.Report specifically: the service reads Network, Live and Billing over loopback, keeps its own
+analytics tables (counts and public channel facts only; no money payloads), and has no accounts. The interesting
+surface is the path parameters it forwards (validated before a request), the HTML it renders (every source text is
+escaped by `server/views/index.js`), the `/internal/` routes (loopback-only; `POST /internal/events` verifies the
+Events v2 HMAC) and the origins and service token it is configured with. A Billing or Events secret is never logged.
## Supported versions
## Supported versions
@@ -1,11 +1,11 @@
@@ -1,11 +1,11 @@
{
{
"repository": "OpenVibers/OpenVibe.Report",
"repository": "OpenVibers/OpenVibe.Report",
"version": "0.1.0",
"version": "0.1.0",
- "stage": "skeleton",
+ "stage": "analytics-authority-shadow",
"code": true,
"code": true,
"wave": "W21",
"wave": "W21",
- "plan": "OpenVibe.Report: a stateless front for the network's open numbers, The Rundown and every stream's after-show report",
- "runtime": "node 22, express 4, no database, no sessions, no events",
+ "plan": "REP-2 (plan 2026-10-28 §4.8 and §5.4): Report's own analytics database, ingest, shadow comparison and the Billing revenue read; no cutover",
+ "runtime": "node 22, express 4, PostgreSQL (PGlite in development), no sessions",
"port": "5080",
"port": "5080",
"domain": "openvibe.report",
"domain": "openvibe.report",
"contracts": "openvibe-contracts v0.225.0",
"contracts": "openvibe-contracts v0.225.0",
@@ -14,38 +14,51 @@
@@ -14,38 +14,51 @@
"packages": [
"packages": [
"openvibe-contracts v0.225.0",
"openvibe-contracts v0.225.0",
"openvibe-sdk v0.70.0",
"openvibe-sdk v0.70.0",
- "openvibe-shared v4.33.0"
+ "openvibe-shared v4.33.0",
+ "pg v8.23.1",
+ "pglite v0.5.13 (dev)"
],
],
- "summary": "The Report room of the OpenVibe network. A stateless front with no database, no sign-in and no events: it reads OpenVibe.Network's open stats (GET /api/v1/stats?days=7|30|90) and OpenVibe.Live's Rundown (GET /api/rundown/, /topics, /t/:slug, /e/:streamId, /s/:username) and after-show reports (GET /api/recap/:streamId, /api/recap/channel/:username) over loopback, caches each answer for 2 to 10 minutes in this process, and shows it as the source answered it. Network and Live failures answer null and a page or API 404, never a made-up zero. The pages are plain placeholders the main session will design; the routes, the JSON API, the crawl artifacts, readiness (degraded, never 503, when a source is down) and the deploy files are complete.",
+ "summary": "OpenVibe.Report keeps its own PostgreSQL analytics store (migrations/), ingests Network's live.stream.ended and frame beacon plus its open-stats answer idempotently (report_inbox), computes the same site-wide, per-site and per-channel numbers by day, quotes Billing's revenue through Billing's read API, and runs a shadow job that records every difference from Network in report_shadow_diffs with GET /internal/shadow on loopback. REPORT_SOURCE=network (default) keeps the pages on today's Network numbers; REPORT_SOURCE=report answers them from Report's tables. The Rundown and after-show pages still read OpenVibe.Live.",
"works": [
"works": [
- "The three sources read through two clients with injectable fetch, 8 s timeouts and an in-memory cache (stats 5 min, Rundown home 2 min, topics 5 min, a topic 5 min, an episode 5 min, an after-show report 10 min, a channel's reports 5 min; 1000 entries, oldest out)",
- "Path parameters validated before a request: slug ^[a-z0-9-]{1,80}$, a positive integer stream id, username ^[A-Za-z0-9_]{1,32}$",
- "Pages: /, /stats?days=7|30|90, /rundown, /rundown/t/:slug, /rundown/e/:streamId, /rundown/s/:username, /after-show/:streamId, /c/:username; a bad parameter or a missing source answer is the 404 view",
- "JSON API: /api/v1/stats, /rundown, /rundown/topics, /rundown/t/:slug, /after-show/:streamId, /c/:username — the source's answer, CORS *, public for two minutes, 120 requests a minute per address, problem+json 404s",
- "Plumbing: helmet with Help's CSP, /api/health, /api/ready (Network and Live optional → degraded, never 503), /release.json, /release-metrics, /metrics on loopback, /shared/*, /css and /js (immutable only with the matching ?v=), robots.txt, sitemap.xml, llms.txt, graceful stop",
- "Deploy files: systemd unit openvibe-report on 127.0.0.1:5080 and an nginx vhost for openvibe.report + www (apex redirect, certs, proxy, GET-only API)",
- "Tests: the clients' cache, expiry, failure and validation; every page 200 and its 404s; the API shapes, CORS and cache; robots/sitemap/llms; health, readiness and static cache"
+ "Database: server/db.js (openvibe-sdk/db, migrations at boot; PostgreSQL through PgBouncer, PGlite in development) and test/helpers/db.js (createTestDb)",
+ "Tables: report_inbox, report_events, report_streams, report_frame_days, report_metric_days, report_metric_windows, report_metric_json, report_dimensions, report_shadow_runs, report_shadow_diffs",
+ "Ingest: live.stream.ended → report_streams and channel/day counts; network.frame.hit/day → report_frame_days; GET /api/v1/stats → day and window metrics, JSON lists and site dimensions; every write claimed in report_inbox in the same transaction",
+ "Read: reportStats() reproduces GET /api/v1/stats (totals, sites, pageviews_daily, accounts, coins, live, busiest pages); channelStats() reproduces creator analytics from report_streams and adds Billing revenue; metrics() returns report.metric@1 rows",
+ "Billing: GET /api/v1/stats with billing.stats.read; GET /api/v1/transactions?subject= with billing.balance.read for settled creator_payable revenue/refunds; service token via Network client-credentials or REPORT_BILLING_TOKEN; never derives a price",
+ "Shadow: server/analytics/shadow.js compares the same window from Network and Report, records report_shadow_runs + report_shadow_diffs; GET/POST /internal/shadow (loopback)",
+ "Jobs: ingest 7/30/90-day windows every 5 minutes and shadow every 15 minutes (server/jobs/analytics.js), on by default; REPORT_JOBS=0 turns them off",
+ "Events: POST /internal/events with Events v2 HMAC, loopback only",
+ "Source switch: REPORT_SOURCE=network|report (default network) for /, /stats and GET /api/v1/stats",
+ "Tests: ingest idempotency, every metric from a fixture, shadow difference and clean run, source flag, Billing stub, signed Events delivery, and the existing pages/API/health/crawl tests"
],
],
"notYet": [
"notYet": [
- "The real page designs: v1 ships plain placeholder views, as the task asked; the main session designs the room itself",
- "A report manifest in openvibe-contracts: this skeleton does not register one, and CI skips the contracts-service check until it lands",
- "Any write path: Report has no accounts, no reactions and no saved views; it is a read-only front"
+ "Cutover: REPORT_SOURCE stays network; Network and Live keep answering until the shadow is clean for a full window",
+ "Report's first-party grants on Network (billing.stats.read, billing.balance.read, and the Network analytics grant when creator analytics is pulled directly) are not created by this change",
+ "No report service manifest or capabilities are registered in openvibe-contracts yet; report.metric@1 is proposed in docs/report.metric.v1.json",
+ "The ingest job reads Network's open-stats answer; it does not call Network's creator analytics endpoint (channels come from live.stream.ended)",
+ "The Rundown and after-show pages remain Live reads until Report takes those products"
],
],
+ "capabilitiesUsed": ["billing.stats.read", "billing.balance.read"],
"capabilitiesRegistered": false,
"capabilitiesRegistered": false,
"serviceManifestRegistered": false,
"serviceManifestRegistered": false,
"deployed": false,
"deployed": false,
"placeholderInSites": false,
"placeholderInSites": false,
"features": [
"features": [
+ "analytics ingest with an event-id inbox",
+ "site and channel metrics by day",
+ "Billing revenue read",
+ "shadow comparison and /internal/shadow",
+ "REPORT_SOURCE switch",
+ "report.metric@1",
"open network stats",
"open network stats",
"The Rundown",
"The Rundown",
"after-show reports",
"after-show reports",
- "source clients with caches",
- "server-rendered placeholder pages",
+ "server-rendered pages",
"JSON API",
"JSON API",
"crawl artifacts",
"crawl artifacts",
"readiness",
"readiness",
"deploy files"
"deploy files"
],
],
- "deployment": "planned: openvibe-ovh, unit openvibe-report.service on 127.0.0.1:5080 (git checkout /opt/openvibe.report, env /etc/openvibe/report.env), public at https://openvibe.report",
+ "deployment": "planned: openvibe-ovh, unit openvibe-report.service on 127.0.0.1:5080 (git checkout /opt/openvibe.report, env /etc/openvibe/report.env), public at https://openvibe.report; DATABASE_URL/DATABASE_DIRECT_URL from OpenVibe.Host roles/data add-service.sh report",
"updated": "2026-10-28"
"updated": "2026-10-28"
}
}
@@ -77,7 +77,7 @@ server {
@@ -77,7 +77,7 @@ server {
# ── Prometheus metrics: host-local scrapers only (http://127.0.0.1:5080/metrics) ──
# ── Prometheus metrics: host-local scrapers only (http://127.0.0.1:5080/metrics) ──
location ~* ^/metrics(/|$) { return 404; }
location ~* ^/metrics(/|$) { return 404; }
- # ── Internal delivery would be loopback-only; Report has none, so refuse it from outside ──
+ # ── Internal delivery is loopback-only (Events, shadow); refuse it from outside ──
location ~* ^/internal(/|$) { return 404; }
location ~* ^/internal(/|$) { return 404; }
# ── Pages, crawl artifacts, static assets, release metadata ──
# ── Pages, crawl artifacts, static assets, release metadata ──
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
[Unit]
[Unit]
Description=OpenVibe.Report — Every number on OpenVibe, honest.
Description=OpenVibe.Report — Every number on OpenVibe, honest.
-# Network and Live should be up first, but Report degrades honestly without either: the pages and the API answer
-# what they can and /api/ready says degraded (never 503), because neither source is required to serve a page.
+# Network, Live and Billing should be up first, but Report degrades honestly without either read API. The database
+# is required after REPORT_SOURCE=report; before that it is the shadow store.
After=network.target openvibe-network.service openvibe-live.service
After=network.target openvibe-network.service openvibe-live.service
Wants=openvibe-network.service openvibe-live.service
Wants=openvibe-network.service openvibe-live.service
new file mode 100644
new file mode 100644
@@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://openvibe.network/contracts/report/metric.v1.json",
+ "title": "ReportMetric",
+ "description": "report.metric@1: one number OpenVibe.Report computed or ingested, with its scope and either its UTC day (a day series) or its window. Counts only. Money values are Billing's, read through Billing's API; Report never derives them.",
+ "type": "object",
+ "required": ["metric", "scope", "scope_id", "value", "unit", "source"],
+ "additionalProperties": false,
+ "properties": {
+ "metric": {
+ "type": "string",
+ "pattern": "^[a-z][a-z0-9_]{1,60}$",
+ "description": "The number's name: pageviews, accounts_new, coins_earned, live_now, streams, stream_seconds, messages, watch_minutes, traffic_p95_ms, revenue_bits, refund_bits, ..."
+ },
+ "scope": { "enum": ["network", "site", "channel", "stream"] },
+ "scope_id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 200,
+ "description": "network for the network scope; a site id (network, live, ...), a channel username, or a stream id."
+ },
+ "day": {
+ "type": "string",
+ "format": "date",
+ "description": "The UTC day this number belongs to. Present on a day row."
+ },
+ "window_days": {
+ "type": "integer",
+ "enum": [7, 30, 90],
+ "description": "The window this number is for. Present on a window row."
+ },
+ "value": { "type": "number" },
+ "unit": { "enum": ["count", "seconds", "minutes", "bits", "cents", "milliseconds", "percent"] },
+ "source": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 40,
+ "description": "network-stats, live, billing, report, or the job that wrote it."
+ }
+ },
+ "oneOf": [
+ { "required": ["day"], "not": { "required": ["window_days"] } },
+ { "required": ["window_days"], "not": { "required": ["day"] } }
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,138 @@
@@ -0,0 +1,138 @@
+-- phase: expand
+-- OpenVibe.Report's analytics authority (REP-2, plan 2026-10-28 §4.8 and §5.4).
+-- Counts and public channel facts only. Money is never stored here: it is read from OpenVibe.Billing.
+
+-- The inbox of event ids already handled. openvibe-sdk/events createPgInbox() uses
+-- (consumer, event_id, processed_at); the extra columns are audit context, never a secret.
+CREATE TABLE report_inbox (
+ consumer text NOT NULL,
+ event_id text NOT NULL,
+ processed_at bigint NOT NULL,
+ source text,
+ topic text,
+ payload_hash text,
+ received_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (consumer, event_id)
+);
+CREATE INDEX report_inbox_received ON report_inbox (received_at);
+
+-- The raw event audit trail (public facts only: no viewer ids, no IP addresses, no billing payloads).
+CREATE TABLE report_events (
+ event_id text COLLATE "C" PRIMARY KEY,
+ source text COLLATE "C" NOT NULL,
+ topic text COLLATE "C" NOT NULL,
+ occurred_at timestamptz,
+ payload jsonb NOT NULL DEFAULT '{}'::jsonb,
+ received_at timestamptz NOT NULL DEFAULT now()
+);
+CREATE INDEX report_events_topic_day ON report_events (topic, occurred_at);
+
+-- One row per ended stream; the same fields Network's creator_streams keeps (from live.stream.ended).
+CREATE TABLE report_streams (
+ stream_id bigint PRIMARY KEY,
+ creator_subject text COLLATE "C",
+ channel text COLLATE "C" NOT NULL,
+ title text,
+ category text,
+ started_at timestamptz,
+ ended_at timestamptz,
+ duration_seconds bigint NOT NULL DEFAULT 0,
+ peak_viewers bigint,
+ avg_viewers numeric,
+ unique_chatters bigint,
+ messages bigint,
+ watch_minutes bigint,
+ payload jsonb NOT NULL DEFAULT '{}'::jsonb,
+ received_at timestamptz NOT NULL DEFAULT now()
+);
+CREATE INDEX report_streams_channel_day ON report_streams (lower(channel), started_at DESC);
+CREATE INDEX report_streams_creator_day ON report_streams (creator_subject, started_at DESC);
+
+-- The navbar's anonymous beacon, aggregated by day and host.
+CREATE TABLE report_frame_days (
+ day date NOT NULL,
+ host text COLLATE "C" NOT NULL,
+ hits bigint NOT NULL DEFAULT 0,
+ source text COLLATE "C" NOT NULL DEFAULT 'frame',
+ updated_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (day, host)
+);
+
+-- A number by day for one scope: scope = network | site | channel | stream.
+CREATE TABLE report_metric_days (
+ metric text COLLATE "C" NOT NULL,
+ scope text COLLATE "C" NOT NULL,
+ scope_id text COLLATE "C" NOT NULL,
+ day date NOT NULL,
+ value numeric NOT NULL,
+ unit text COLLATE "C" NOT NULL DEFAULT 'count',
+ source text COLLATE "C" NOT NULL,
+ updated_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (metric, scope, scope_id, day)
+);
+CREATE INDEX report_metric_days_scope_day ON report_metric_days (scope, scope_id, day);
+
+-- A number for one window (7, 30 or 90 days) when the source published a window total.
+CREATE TABLE report_metric_windows (
+ metric text COLLATE "C" NOT NULL,
+ scope text COLLATE "C" NOT NULL,
+ scope_id text COLLATE "C" NOT NULL,
+ window_days integer NOT NULL,
+ value numeric NOT NULL,
+ unit text COLLATE "C" NOT NULL DEFAULT 'count',
+ source text COLLATE "C" NOT NULL,
+ updated_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (metric, scope, scope_id, window_days)
+);
+
+-- Lists that are not one number (the busiest route templates, for example).
+CREATE TABLE report_metric_json (
+ metric text COLLATE "C" NOT NULL,
+ scope text COLLATE "C" NOT NULL,
+ scope_id text COLLATE "C" NOT NULL,
+ window_days integer NOT NULL,
+ value jsonb NOT NULL,
+ source text COLLATE "C" NOT NULL,
+ updated_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (metric, scope, scope_id, window_days)
+);
+
+-- Public labels for a scope: a site's name, host and URL, a channel's display name.
+CREATE TABLE report_dimensions (
+ kind text COLLATE "C" NOT NULL,
+ id text COLLATE "C" NOT NULL,
+ doc jsonb NOT NULL,
+ updated_at timestamptz NOT NULL DEFAULT now(),
+ PRIMARY KEY (kind, id)
+);
+
+-- One shadow comparison run and the differences it found. A run with zero differences is a clean run.
+CREATE TABLE report_shadow_runs (
+ run_id text COLLATE "C" PRIMARY KEY,
+ run_seq bigint GENERATED BY DEFAULT AS IDENTITY UNIQUE,
+ window_days integer NOT NULL,
+ status text COLLATE "C" NOT NULL,
+ checked integer NOT NULL DEFAULT 0,
+ differences integer NOT NULL DEFAULT 0,
+ error text,
+ started_at timestamptz NOT NULL,
+ finished_at timestamptz
+);
+CREATE INDEX report_shadow_runs_finished ON report_shadow_runs (finished_at DESC);
+
+CREATE TABLE report_shadow_diffs (
+ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+ run_id text COLLATE "C" NOT NULL REFERENCES report_shadow_runs(run_id) ON DELETE CASCADE,
+ metric text COLLATE "C" NOT NULL,
+ scope text COLLATE "C" NOT NULL,
+ scope_id text COLLATE "C" NOT NULL,
+ day date,
+ window_days integer,
+ network_value numeric,
+ report_value numeric,
+ delta numeric,
+ network_json jsonb,
+ report_json jsonb,
+ created_at timestamptz NOT NULL DEFAULT now()
+);
+CREATE INDEX report_shadow_diffs_run ON report_shadow_diffs (run_id, metric, scope, scope_id);
@@ -15,12 +15,23 @@
@@ -15,12 +15,23 @@
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0"
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0",
+ "pg": "^8.23.1"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.13"
},
},
"engines": {
"engines": {
"node": ">=22.0.0"
"node": ">=22.0.0"
}
}
},
},
+ "node_modules/@electric-sql/pglite": {
+ "version": "0.5.13",
+ "resolved": "https://registry.npmjs.org/@electric-sql/pglite/-/pglite-0.5.13.tgz",
+ "integrity": "sha512-4rZuxK1k07w6sSpJzjMri0B9+HuXuC7V9pxVB51pR0T9KPqXg6TD0mdaEwBA9rkcL4jdht4tGDf/wYzAiGhT7g==",
+ "devOptional": true,
+ "license": "Apache-2.0"
+ },
"node_modules/accepts": {
"node_modules/accepts": {
"version": "1.3.8",
"version": "1.3.8",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
"resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz",
@@ -840,6 +851,134 @@
@@ -840,6 +851,134 @@
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/pg": {
+ "version": "8.23.1",
+ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.1.tgz",
+ "integrity": "sha512-aL96AHANtWjPLDOLqnhx+ngp9+UK7ETEU8VJrDCGvsSSi/mGLcWYsS6Herg7lmaBJe4uwrfqsa7gTEFaSizDoQ==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-connection-string": "^2.14.1",
+ "pg-pool": "^3.14.0",
+ "pg-protocol": "^1.16.1",
+ "pg-types": "2.2.0",
+ "pgpass": "1.0.5"
+ },
+ "engines": {
+ "node": ">= 16.0.0"
+ },
+ "optionalDependencies": {
+ "pg-cloudflare": "^1.4.1"
+ },
+ "peerDependencies": {
+ "pg-native": ">=3.0.1"
+ },
+ "peerDependenciesMeta": {
+ "pg-native": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/pg-cloudflare": {
+ "version": "1.4.1",
+ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.1.tgz",
+ "integrity": "sha512-6PQbsFWZcp9EmJEwy5cGQ2La+AMWpP46lgbb8X+U/XsHIUweYDNCpeuKck5RxL2MdVFi7krbbEi5nX4Zh7JhrQ==",
+ "license": "MIT",
+ "optional": true
+ },
+ "node_modules/pg-connection-string": {
+ "version": "2.14.1",
+ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.1.tgz",
+ "integrity": "sha512-qR3kGNPBLpCNtz0evbKA0Y/MRFXwSSdT+pTJvYp/bXTcReZbvX1kzF0IyTc1QnxqF7AZbOeBhNL8R5mYQZV/MA==",
+ "license": "MIT"
+ },
+ "node_modules/pg-int8": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+ "license": "ISC",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/pg-pool": {
+ "version": "3.14.0",
+ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+ "license": "MIT",
+ "peerDependencies": {
+ "pg": ">=8.0"
+ }
+ },
+ "node_modules/pg-protocol": {
+ "version": "1.16.1",
+ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.1.tgz",
+ "integrity": "sha512-p9VOFMiHB/ZbJATetbg+99PxssTVSQRnyuPSQ67mN1+1KBOjZaZ83ZQzltnxPhJwSsC3nwVjJ10DVJlerbFzLg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-int8": "1.0.1",
+ "postgres-array": "~2.0.0",
+ "postgres-bytea": "~1.0.0",
+ "postgres-date": "~1.0.4",
+ "postgres-interval": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/pgpass": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+ "license": "MIT",
+ "dependencies": {
+ "split2": "^4.1.0"
+ }
+ },
+ "node_modules/postgres-array": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/postgres-bytea": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-date": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-interval": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "xtend": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/proxy-addr": {
"node_modules/proxy-addr": {
"version": "2.0.8",
"version": "2.0.8",
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.8.tgz",
@@ -1067,6 +1206,15 @@
@@ -1067,6 +1206,15 @@
"url": "https://github.com/sponsors/ljharb"
"url": "https://github.com/sponsors/ljharb"
}
}
},
},
+ "node_modules/split2": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+ "license": "ISC",
+ "engines": {
+ "node": ">= 10.x"
+ }
+ },
"node_modules/statuses": {
"node_modules/statuses": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@@ -1124,6 +1272,15 @@
@@ -1124,6 +1272,15 @@
"engines": {
"engines": {
"node": ">= 0.8"
"node": ">= 0.8"
}
}
+ },
+ "node_modules/xtend": {
+ "version": "4.0.2",
+ "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
+ "integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.4"
+ }
}
}
}
}
}
}
@@ -21,9 +21,13 @@
@@ -21,9 +21,13 @@
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.70.0",
- "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0"
+ "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.33.0",
+ "pg": "^8.23.1"
},
},
"engines": {
"engines": {
"node": ">=22.0.0"
"node": ">=22.0.0"
+ },
+ "devDependencies": {
+ "@electric-sql/pglite": "^0.5.13"
}
}
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+'use strict';
+
+/**
+ * Report's analytics authority: the source switch and the one object pages, the API, the ingest job and the shadow
+ * job share.
+ *
+ * stats(days) REPORT_SOURCE=network → Network's answer (today's behaviour); report → computed from Report's tables.
+ * channel always computed from Report's own report_streams, plus Billing's revenue read.
+ * metrics the flat report.metric@1 rows.
+ *
+ * The default is `network`: the pages keep reading where they read today while Report shadows. The cutover is one
+ * environment value, REPORT_SOURCE=report.
+ */
+const query = require('./query');
+const ingest = require('./ingest');
+
+function createAnalytics({ config = {}, store = null, network = null, billing = null, now = () => Date.now(), log = console } = {}) {
+ const source = config.source === 'report' ? 'report' : 'network';
+
+ function warn(what, err) {
+ try { (log.warn || log.error || console.warn).call(log, `[OpenVibe.Report] ${what} failed: ${String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 200)}`); } catch { /* logging must never fail a page */ }
+ }
+
+ async function stats(days) {
+ try {
+ if (source === 'report') return store ? await query.reportStats(store, days, { now }) : null;
+ return network && typeof network.stats === 'function' ? await network.stats(days) : null;
+ } catch (err) { warn('stats', err); return null; }
+ }
+
+ async function channel(name, days, opts = {}) {
+ try { return store ? await query.channelStats(store, name, days, { now, billing, ...opts }) : null; } catch (err) { warn('channel analytics', err); return null; }
+ }
+
+ async function metrics(opts = {}) {
+ try { return store ? await query.metrics(store, { now, ...opts }) : null; } catch (err) { warn('metrics', err); return null; }
+ }
+
+ return {
+ source, store, billing, network,
+ stats, channel, metrics,
+ reportStats: (days, opts = {}) => (store ? query.reportStats(store, days, { now, ...opts }) : null),
+ recordEvent: (event, opts = {}) => (store ? ingest.recordEvent(store, event, { now, ...opts }) : Promise.resolve({ duplicate: false, outcome: 'no_database' })),
+ ingestStats: (payload, opts = {}) => (store ? ingest.ingestStats(store, payload, { now, ...opts }) : Promise.resolve({ ok: false, reason: 'no_database' })),
+ ingestNetwork: (opts = {}) => (store && network ? ingest.ingestNetwork(store, network, { now, ...opts }) : Promise.resolve({ ok: false, reason: 'no_database' })),
+ log,
+ };
+}
+
+module.exports = { createAnalytics };
new file mode 100644
new file mode 100644
@@ -0,0 +1,339 @@
@@ -0,0 +1,339 @@
+'use strict';
+
+/**
+ * Report's ingest: the analytics facts OpenVibe.Network records today, written to Report's own tables.
+ *
+ * live.stream.ended one report_streams row and its channel/day counts (Network's creator-analytics input)
+ * the frame beacon report_frame_days (day, host, hits) from a daily snapshot or a single hit
+ * Network's open stats the daily page views, accounts and OpenCoins, and the window totals, kept as numbers
+ *
+ * Every delivery goes through report_inbox (openvibe-sdk/events createPgInbox) in the same transaction as the
+ * writes: a redelivered event id changes nothing. A stats snapshot is keyed by its own content hash, so polling
+ * the same numbers again is a no-op while new numbers replace the stored ones.
+ *
+ * Billing events are consciously not ingested: Report reads money from Billing's API, never from an event.
+ */
+const crypto = require('crypto');
+const { createPgInbox } = require('openvibe-sdk/events');
+const { normaliseDays } = require('../data/source');
+
+const INBOX = 'report-events';
+const STATS_INBOX = 'network-stats';
+const DAY = 86_400_000;
+const ALLOWED_TOPICS = new Set(['live.stream.ended', 'live.stream.started', 'network.frame.hit', 'network.frame.day', 'frame.hit']);
+
+const hashOf = (value) => crypto.createHash('sha256').update(JSON.stringify(value)).digest('hex');
+const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
+const int = (v) => (Number.isFinite(Number(v)) && Number(v) >= 0 ? Math.round(Number(v)) : null);
+const dayOf = (value) => {
+ if (value == null || value === '') return null;
+ if (typeof value === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(value)) return value;
+ const t = Date.parse(String(value));
+ return Number.isFinite(t) ? new Date(t).toISOString().slice(0, 10) : null;
+};
+const isChannel = (v) => typeof v === 'string' && /^[A-Za-z0-9_]{1,80}$/.test(v);
+
+function unitOf(metric) {
+ if (metric.endsWith('_ms')) return 'milliseconds';
+ if (metric === 'change_pct' || metric.endsWith('_pct')) return 'percent';
+ if (metric === 'stream_seconds' || metric === 'duration_seconds' || metric.endsWith('_seconds')) return 'seconds';
+ if (metric === 'watch_minutes' || metric.endsWith('_minutes')) return 'minutes';
+ if (metric.includes('revenue') || metric.includes('refund') || metric.endsWith('_bits')) return 'bits';
+ if (metric.includes('cents')) return 'cents';
+ return 'count';
+}
+
+function seriesOf(value) {
+ if (!Array.isArray(value)) return [];
+ const out = [];
+ for (const p of value) {
+ if (Array.isArray(p)) { const d = dayOf(p[0]); const n = num(p[1]); if (d && n != null) out.push({ day: d, n }); continue; }
+ if (!p || typeof p !== 'object') continue;
+ const d = dayOf(p.day || p.date);
+ const n = num(p.n != null ? p.n : p.value);
+ if (d && n != null) out.push({ day: d, n });
+ }
+ return out;
+}
+
+/**
+ * The same numbers in one flat shape, used both to write the tables and by the shadow job to compare them.
+ * A row is a day number or a window number; nothing is invented for a field the source did not send.
+ */
+function metricRowsFromStats(stats) {
+ const rows = [];
+ const seen = new Map();
+ function add(kind, metric, scope, scopeId, value, unit = unitOf(metric), day = null, windowDays = null) {
+ const n = num(value);
+ if (n == null || !scopeId) return;
+ const row = { kind, metric, scope, scope_id: String(scopeId), value: n, unit, day: kind === 'day' ? day : null, window_days: kind === 'window' ? windowDays : null };
+ if (kind === 'day' && !row.day) return;
+ if (kind === 'window' && !row.window_days) return;
+ seen.set(`${kind}|${scope}|${row.scope_id}|${metric}|${row.day || ''}|${row.window_days || ''}`, row);
+ }
+ const win = (metric, scope, id, value) => add('window', metric, scope, id, value, unitOf(metric), null, days);
+ const day = (metric, scope, id, d, value) => add('day', metric, scope, id, value, unitOf(metric), d, null);
+
+ const s = stats && typeof stats === 'object' ? stats : {};
+ const days = normaliseDays(s.window_days);
+ const T = s.totals && typeof s.totals === 'object' ? s.totals : {};
+ const A = s.accounts && typeof s.accounts === 'object' ? s.accounts : null;
+ const C = s.coins && typeof s.coins === 'object' ? s.coins : null;
+ const L = s.live && typeof s.live === 'object' ? s.live : null;
+
+ win('pageviews', 'network', 'network', T.pageviews);
+ win('pageviews_prev', 'network', 'network', T.pageviews_prev);
+ win('change_pct', 'network', 'network', T.change_pct);
+ win('sites_open', 'network', 'network', T.sites_open);
+ win('accounts_total', 'network', 'network', T.accounts);
+ win('requests_24h', 'network', 'network', T.requests_24h);
+ if (T.services && typeof T.services === 'object') {
+ win('services_up', 'network', 'network', T.services.up);
+ win('services_checked', 'network', 'network', T.services.checked);
+ }
+ for (const p of seriesOf(s.pageviews_daily)) day('pageviews', 'network', 'network', p.day, p.n);
+
+ if (A) {
+ win('accounts_total', 'network', 'network', A.total);
+ win('accounts_new', 'network', 'network', A.new);
+ win('accounts_prev', 'network', 'network', A.prev);
+ for (const p of seriesOf(A.daily)) day('accounts_new', 'network', 'network', p.day, p.n);
+ }
+ if (C) {
+ win('coins_earned', 'network', 'network', C.earned);
+ win('coins_spent', 'network', 'network', C.spent);
+ win('coins_holders', 'network', 'network', C.holders);
+ win('coins_transactions', 'network', 'network', C.transactions);
+ win('coins_earned_window', 'network', 'network', C.earned_window);
+ win('coins_earned_prev', 'network', 'network', C.earned_prev);
+ for (const p of seriesOf(C.daily)) day('coins_earned', 'network', 'network', p.day, p.n);
+ }
+ if (L) {
+ for (const k of ['live_now', 'watching', 'streamers', 'streams', 'accounts', 'visitors', 'messages', 'chatted_7d',
+ 'vods', 'clips', 'hours_watched', 'hours_archived', 'ai_moments', 'pastes', 'follows']) {
+ win(k, 'network', 'network', L[k]);
+ }
+ }
+
+ for (const site of Array.isArray(s.sites) ? s.sites : []) {
+ if (!site || typeof site !== 'object') continue;
+ const id = site.id || site.host || site.short;
+ if (!id) continue;
+ win('pageviews', 'site', id, site.pageviews);
+ win('pageviews_prev', 'site', id, site.pageviews_prev);
+ win('change_pct', 'site', id, site.change_pct);
+ win('unique_visitors', 'site', id, site.unique_visitors);
+ win('signed_in_people', 'site', id, site.signed_in_people);
+ for (const p of seriesOf(site.daily)) day('pageviews', 'site', id, p.day, p.n);
+ const t = site.traffic_24h && typeof site.traffic_24h === 'object' ? site.traffic_24h : null;
+ if (t) {
+ win('traffic_requests', 'site', id, t.requests);
+ win('traffic_server_errors', 'site', id, t.server_errors);
+ win('traffic_p95_ms', 'site', id, t.p95_ms);
+ }
+ }
+ return [...seen.values()];
+}
+
+function jsonRowsFromStats(stats) {
+ const rows = [];
+ const s = stats && typeof stats === 'object' ? stats : {};
+ if (Array.isArray(s.busiest_pages_24h)) {
+ rows.push({ metric: 'busiest_pages', scope: 'network', scope_id: 'network', window_days: normaliseDays(s.window_days), value: s.busiest_pages_24h, source: 'network' });
+ }
+ return rows;
+}
+
+function frameRowsFromStats(stats) {
+ const rows = [];
+ const s = stats && typeof stats === 'object' ? stats : {};
+ for (const site of Array.isArray(s.sites) ? s.sites : []) {
+ if (!site || !site.host || !Array.isArray(site.daily)) continue;
+ for (const p of seriesOf(site.daily)) rows.push({ day: p.day, host: String(site.host).toLowerCase(), hits: p.n });
+ }
+ return rows;
+}
+
+function dimensionsFromStats(stats) {
+ const docs = [];
+ const s = stats && typeof stats === 'object' ? stats : {};
+ if (typeof s.note === 'string' && s.note.trim()) docs.push({ kind: 'network', id: 'network', doc: { note: s.note.trim().slice(0, 600) } });
+ for (const site of Array.isArray(s.sites) ? s.sites : []) {
+ if (!site || typeof site !== 'object') continue;
+ const id = site.id || site.host || site.short;
+ if (!id) continue;
+ docs.push({
+ kind: 'site', id,
+ doc: {
+ id, name: site.name || `OpenVibe.${site.short || id}`, short: site.short || id, host: site.host || null,
+ url: site.url || (site.host ? `https://${site.host}/` : null), tagline: site.tagline || '', icon: site.icon || null,
+ },
+ });
+ }
+ return docs;
+}
+
+// ── Writes ──────────────────────────────────────────────────
+async function setDay(t, row, source) {
+ await t.prepare(`INSERT INTO report_metric_days (metric, scope, scope_id, day, value, unit, source)
+ VALUES (?, ?, ?, ?, ?, ?, ?)
+ ON CONFLICT(metric, scope, scope_id, day) DO UPDATE SET value = excluded.value, unit = excluded.unit, source = excluded.source, updated_at = now()`)
+ .run(row.metric, row.scope, row.scope_id, row.day, row.value, row.unit, source);
+}
+async function addDay(t, metric, scope, id, day, value, unit = unitOf(metric), source = 'live') {
+ await t.prepare(`INSERT INTO report_metric_days (metric, scope, scope_id, day, value, unit, source)
+ VALUES (?, ?, ?, ?, ?, ?, ?)
+ ON CONFLICT(metric, scope, scope_id, day) DO UPDATE SET value = report_metric_days.value + excluded.value, updated_at = now()`)
+ .run(metric, scope, id, day, value, unit, source);
+}
+async function maxDay(t, metric, scope, id, day, value, unit = unitOf(metric), source = 'live') {
+ await t.prepare(`INSERT INTO report_metric_days (metric, scope, scope_id, day, value, unit, source)
+ VALUES (?, ?, ?, ?, ?, ?, ?)
+ ON CONFLICT(metric, scope, scope_id, day) DO UPDATE SET value = GREATEST(report_metric_days.value, excluded.value), updated_at = now()`)
+ .run(metric, scope, id, day, value, unit, source);
+}
+async function setWindow(t, row, source) {
+ await t.prepare(`INSERT INTO report_metric_windows (metric, scope, scope_id, window_days, value, unit, source)
+ VALUES (?, ?, ?, ?, ?, ?, ?)
+ ON CONFLICT(metric, scope, scope_id, window_days) DO UPDATE SET value = excluded.value, unit = excluded.unit, source = excluded.source, updated_at = now()`)
+ .run(row.metric, row.scope, row.scope_id, row.window_days, row.value, row.unit, source);
+}
+async function setJson(t, row) {
+ await t.prepare(`INSERT INTO report_metric_json (metric, scope, scope_id, window_days, value, source)
+ VALUES (?, ?, ?, ?, ?::jsonb, ?)
+ ON CONFLICT(metric, scope, scope_id, window_days) DO UPDATE SET value = excluded.value, source = excluded.source, updated_at = now()`)
+ .run(row.metric, row.scope, row.scope_id, row.window_days, JSON.stringify(row.value), row.source);
+}
+async function setFrame(t, row) {
+ await t.prepare(`INSERT INTO report_frame_days (day, host, hits, source) VALUES (?, ?, ?, 'network-stats')
+ ON CONFLICT(day, host) DO UPDATE SET hits = excluded.hits, source = excluded.source, updated_at = now()`)
+ .run(row.day, row.host, row.hits);
+}
+async function setDimension(t, row) {
+ await t.prepare(`INSERT INTO report_dimensions (kind, id, doc) VALUES (?, ?, ?::jsonb)
+ ON CONFLICT(kind, id) DO UPDATE SET doc = excluded.doc, updated_at = now()`)
+ .run(row.kind, row.id, JSON.stringify(row.doc));
+}
+
+async function recordFrame(t, topic, payload, occurredAt, source) {
+ const p = payload && typeof payload === 'object' ? payload : {};
+ const day = dayOf(p.day || occurredAt);
+ const host = String(p.host || p.site || '').toLowerCase();
+ if (!day || !/^[a-z0-9.-]{1,253}$/.test(host)) return 'ignored:payload';
+ const hits = p.hits != null ? int(p.hits) : 1;
+ if (hits == null) return 'ignored:payload';
+ const increment = (topic === 'network.frame.hit' || topic === 'frame.hit') && p.hits == null;
+ const sql = increment
+ ? `INSERT INTO report_frame_days (day, host, hits, source) VALUES (?, ?, ?, ?)
+ ON CONFLICT(day, host) DO UPDATE SET hits = report_frame_days.hits + excluded.hits, source = excluded.source, updated_at = now()`
+ : `INSERT INTO report_frame_days (day, host, hits, source) VALUES (?, ?, ?, ?)
+ ON CONFLICT(day, host) DO UPDATE SET hits = excluded.hits, source = excluded.source, updated_at = now()`;
+ await t.prepare(sql).run(day, host, hits, source || (increment ? 'frame-hit' : 'frame'));
+ return 'frame:recorded';
+}
+
+async function recordStream(t, event, payload, occurredAt, source) {
+ const p = payload && typeof payload === 'object' ? payload : {};
+ const streamId = Number(p.stream_id);
+ const channel = p.channel && typeof p.channel.username === 'string' ? p.channel.username : null;
+ if (!Number.isSafeInteger(streamId) || streamId < 1 || !isChannel(channel)) return 'ignored:payload';
+ const creator = p.channel && p.channel.subject && typeof p.channel.subject.id === 'string' ? p.channel.subject.id : null;
+ const startedMs = Date.parse(p.started_at || occurredAt || '');
+ const endedMs = Date.parse(p.ended_at || occurredAt || '');
+ if (!Number.isFinite(startedMs) || !Number.isFinite(endedMs) || endedMs < startedMs) return 'ignored:times';
+ const stats = p.stats && typeof p.stats === 'object' ? p.stats : {};
+ const duration = int(p.duration_seconds) != null ? int(p.duration_seconds) : Math.round((endedMs - startedMs) / 1000);
+ const existing = await t.prepare('SELECT stream_id FROM report_streams WHERE stream_id = ?').get(streamId);
+ await t.prepare(`INSERT INTO report_streams (stream_id, creator_subject, channel, title, category, started_at, ended_at, duration_seconds,
+ peak_viewers, avg_viewers, unique_chatters, messages, watch_minutes, payload)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?::jsonb)
+ ON CONFLICT(stream_id) DO UPDATE SET creator_subject = excluded.creator_subject, channel = excluded.channel, title = excluded.title,
+ category = excluded.category, started_at = excluded.started_at, ended_at = excluded.ended_at, duration_seconds = excluded.duration_seconds,
+ peak_viewers = excluded.peak_viewers, avg_viewers = excluded.avg_viewers, unique_chatters = excluded.unique_chatters,
+ messages = excluded.messages, watch_minutes = excluded.watch_minutes, payload = excluded.payload, received_at = now()`)
+ .run(streamId, creator, channel, p.title == null ? null : String(p.title).slice(0, 300), p.category == null ? null : String(p.category).slice(0, 100),
+ new Date(startedMs).toISOString(), new Date(endedMs).toISOString(), duration,
+ int(stats.peak_viewers), num(stats.avg_viewers), int(stats.unique_chatters), int(stats.messages), int(stats.watch_minutes),
+ JSON.stringify({ stream_id: streamId, channel: p.channel, title: p.title || null, category: p.category || null, started_at: p.started_at || null, ended_at: p.ended_at || null, duration_seconds: duration, stats }));
+ if (existing) return 'stream:updated'; // a corrected copy: replace the facts, never count the stream twice
+ const day = new Date(startedMs).toISOString().slice(0, 10);
+ await addDay(t, 'streams', 'channel', channel, day, 1, 'count', source || 'live');
+ await addDay(t, 'stream_seconds', 'channel', channel, day, duration, 'seconds', source || 'live');
+ await addDay(t, 'unique_chatters', 'channel', channel, day, int(stats.unique_chatters) || 0, 'count', source || 'live');
+ await addDay(t, 'messages', 'channel', channel, day, int(stats.messages) || 0, 'count', source || 'live');
+ await addDay(t, 'watch_minutes', 'channel', channel, day, int(stats.watch_minutes) || 0, 'minutes', source || 'live');
+ if (num(stats.avg_viewers) != null) await addDay(t, 'avg_viewers_weighted', 'channel', channel, day, num(stats.avg_viewers) * duration, 'count', source || 'live');
+ if (int(stats.peak_viewers) != null) await maxDay(t, 'peak_viewers', 'channel', channel, day, int(stats.peak_viewers), 'count', source || 'live');
+ return 'stream:recorded';
+}
+
+/** One Events envelope → the tables. Returns { duplicate, outcome }. */
+async function recordEvent(store, event, { now = () => Date.now() } = {}) {
+ if (!store || !store.db) throw new TypeError('recordEvent needs an open Report store');
+ const e = event && typeof event === 'object' ? event : {};
+ const eventId = String(e.event_id || e.id || '');
+ const topic = String(e.event_type || e.topic || e.type || '');
+ if (!eventId) throw new TypeError('an event_id is required');
+ if (!topic) return { duplicate: false, outcome: 'ignored:topic' };
+ const source = String(e.source || topic.split('.')[0] || 'events');
+ const rawOccurred = e.occurred_at || e.timestamp || (e.payload && (e.payload.ended_at || e.payload.at)) || new Date(now()).toISOString();
+ const occurredMs = Date.parse(String(rawOccurred));
+ const occurredAt = Number.isFinite(occurredMs) ? new Date(occurredMs).toISOString() : null;
+ if (!ALLOWED_TOPICS.has(topic)) {
+ // Today's billing and traffic events are not analytics facts Report may derive: a seen id changes nothing.
+ const inbox = createPgInbox(store.db, { table: 'report_inbox', now });
+ const r = await inbox.once(INBOX, eventId, async () => 'ignored:topic');
+ return { duplicate: r.duplicate, outcome: r.result || 'ignored:topic' };
+ }
+ const payload = e.payload && typeof e.payload === 'object' ? e.payload : {};
+ const inbox = createPgInbox(store.db, { table: 'report_inbox', now });
+ const r = await inbox.once(INBOX, eventId, async (t) => {
+ await t.prepare(`INSERT INTO report_events (event_id, source, topic, occurred_at, payload) VALUES (?, ?, ?, ?, ?::jsonb) ON CONFLICT (event_id) DO NOTHING`)
+ .run(eventId, source, topic, occurredAt, JSON.stringify(payload));
+ await t.prepare('UPDATE report_inbox SET source = ?, topic = ?, payload_hash = ? WHERE consumer = ? AND event_id = ?')
+ .run(source, topic, hashOf(payload), INBOX, eventId);
+ if (topic === 'live.stream.ended') return await recordStream(t, e, payload, occurredAt, source);
+ if (topic === 'network.frame.hit' || topic === 'network.frame.day' || topic === 'frame.hit') return await recordFrame(t, topic, payload, occurredAt, source);
+ return 'event:recorded';
+ });
+ return { duplicate: r.duplicate, outcome: r.result || 'event:recorded' };
+}
+
+/** Ingest one GET /api/v1/stats answer. Idempotent by its content hash. */
+async function ingestStats(store, stats, { now = () => Date.now(), source = 'network-stats' } = {}) {
+ if (!store || !store.db) throw new TypeError('ingestStats needs an open Report store');
+ if (!stats || typeof stats !== 'object') return { ok: false, reason: 'no stats' };
+ const windowDays = normaliseDays(stats.window_days);
+ const rows = metricRowsFromStats(stats);
+ const docs = jsonRowsFromStats(stats);
+ const frames = frameRowsFromStats(stats);
+ const dims = dimensionsFromStats(stats);
+ const eventId = `stats:${windowDays}:${hashOf({ stats })}`;
+ const inbox = createPgInbox(store.db, { table: 'report_inbox', now });
+ const r = await inbox.once(STATS_INBOX, eventId, async (t) => {
+ for (const row of rows) {
+ if (row.kind === 'window') await setWindow(t, row, source);
+ else if (row.kind === 'day') await setDay(t, row, source);
+ }
+ for (const doc of docs) await setJson(t, doc);
+ for (const frame of frames) await setFrame(t, frame);
+ for (const dim of dims) await setDimension(t, dim);
+ await t.prepare('UPDATE report_inbox SET source = ?, topic = ?, payload_hash = ? WHERE consumer = ? AND event_id = ?')
+ .run(source, 'network.stats', hashOf(stats), STATS_INBOX, eventId);
+ return `stats:${rows.length}`;
+ });
+ return { ok: true, duplicate: r.duplicate, outcome: r.result, window_days: windowDays, rows: rows.length, event_id: eventId };
+}
+
+/** Pull Network's open stats once and ingest them. Null from Network is "no answer", never zeroes. */
+async function ingestNetwork(store, network, { days = 90, now = () => Date.now() } = {}) {
+ if (!network || typeof network.stats !== 'function') throw new TypeError('ingestNetwork needs a Network client');
+ const stats = await network.stats(days);
+ if (!stats) return { ok: false, reason: 'network_did_not_answer' };
+ return await ingestStats(store, stats, { now });
+}
+
+module.exports = {
+ recordEvent, ingestStats, ingestNetwork, metricRowsFromStats, jsonRowsFromStats, frameRowsFromStats, dimensionsFromStats,
+ setDay, addDay, maxDay, setWindow, setJson, setFrame, setDimension, hashOf, dayOf, unitOf, ALLOWED_TOPICS, INBOX, STATS_INBOX, DAY,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,272 @@
@@ -0,0 +1,272 @@
+'use strict';
+
+/**
+ * Report's read side: the same numbers Network answers, computed from Report's own tables.
+ *
+ * reportStats(days) GET /api/v1/stats' shape: network totals, sites, accounts, OpenCoins, Live and day series
+ * channelStats(channel) creator analytics from the live.stream.ended events, plus Billing's revenue read
+ * metrics(scope,id) the flat report.metric@1 rows the shadow job and operators inspect
+ *
+ * Window totals come from the source's own window answer when it published one (report_metric_windows); a day
+ * series is summed from report_metric_days; a channel is aggregated from report_streams. A number that is not
+ * present is null / left out, never a made-up zero.
+ */
+const { normaliseDays } = require('../data/source');
+
+const DAY = 86_400_000;
+const NOTE = 'Counts only, never about a person. Page views come from the shared navbar\'s anonymous counter (day and site) and OpenVibe.Live\'s own analytics; traffic is over 24 hours.';
+const SCOPES = new Set(['network', 'site', 'channel', 'stream']);
+
+const n = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
+const dayOf = (ms) => new Date(ms).toISOString().slice(0, 10);
+const round1 = (v) => Math.round(v * 10) / 10;
+const change = (now, before) => (before > 0 && now != null ? Math.round(((now - before) / before) * 100) : null);
+const sumOrNull = (values) => { let any = false; let total = 0; for (const v of values) if (v != null) { any = true; total += v; } return any ? total : null; };
+
+function windowOf(nowMs, days) {
+ return {
+ currentStart: dayOf(nowMs - (days - 1) * DAY),
+ currentEnd: dayOf(nowMs),
+ prevStart: dayOf(nowMs - (2 * days - 1) * DAY),
+ prevEnd: dayOf(nowMs - days * DAY),
+ days: Array.from({ length: days }, (_, i) => dayOf(nowMs - (days - 1 - i) * DAY)),
+ };
+}
+
+function indexRows(rows, winRows, jsonRows) {
+ const day = new Map();
+ for (const r of rows) {
+ const key = `${r.scope}|${r.scope_id}|${r.metric}`;
+ let m = day.get(key);
+ if (!m) { m = new Map(); day.set(key, m); }
+ m.set(r.day, { value: n(r.value), unit: r.unit, source: r.source });
+ }
+ const win = new Map();
+ for (const r of winRows) win.set(`${r.scope}|${r.scope_id}|${r.metric}`, { value: n(r.value), unit: r.unit, source: r.source });
+ const json = new Map();
+ for (const r of jsonRows) json.set(`${r.scope}|${r.scope_id}|${r.metric}`, r.value);
+ return { day, win, json };
+}
+
+async function loadIndex(store, days, nowMs) {
+ const w = windowOf(nowMs, days);
+ const rows = await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, value, unit, source FROM report_metric_days WHERE day >= $1`, [w.prevStart]);
+ const winRows = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, unit, source FROM report_metric_windows WHERE window_days = $1`, [days]);
+ const jsonRows = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, source FROM report_metric_json WHERE window_days = $1`, [days]);
+ const dims = await store.db.many(`SELECT kind, id, doc FROM report_dimensions`);
+ return { rows, winRows, jsonRows, dims, w, ...indexRows(rows, winRows, jsonRows) };
+}
+
+function dayGet(idx, metric, scope, id, day) {
+ const m = idx.day.get(`${scope}|${id}|${metric}`);
+ return m && m.get(day) ? m.get(day).value : null;
+}
+function dayHas(idx, metric, scope, id) {
+ const m = idx.day.get(`${scope}|${id}|${metric}`);
+ return !!m && m.size > 0;
+}
+function daySum(idx, metric, scope, id, from, to) {
+ const m = idx.day.get(`${scope}|${id}|${metric}`);
+ if (!m) return null;
+ let total = 0;
+ let seen = false;
+ for (const [day, v] of m) {
+ if (day >= from && day <= to) { total += v.value || 0; seen = true; }
+ }
+ return seen ? total : null;
+}
+function daySeries(idx, metric, scope, id, days) {
+ return days.map((day) => ({ day, n: dayGet(idx, metric, scope, id, day) || 0 }));
+}
+function winGet(idx, metric, scope, id) {
+ const r = idx.win.get(`${scope}|${id}|${metric}`);
+ return r ? r.value : null;
+}
+function winHas(idx, scope, id) {
+ for (const key of idx.win.keys()) if (key.startsWith(`${scope}|${id}|`)) return true;
+ return false;
+}
+
+/** GET /api/v1/stats' shape, from Report's tables. Null when no number exists for the window. */
+async function reportStats(store, days = 30, { now = () => Date.now() } = {}) {
+ if (!store || !store.db) return null;
+ const windowDays = normaliseDays(days);
+ const nowMs = now();
+ const idx = await loadIndex(store, windowDays, nowMs);
+ const w = idx.w;
+ const anyCurrentDay = idx.rows.some((r) => r.day >= w.currentStart && r.day <= w.currentEnd);
+ if (!anyCurrentDay && !idx.winRows.length && !idx.jsonRows.length) return null;
+
+ const dimOf = (kind, id) => {
+ const d = idx.dims.find((x) => x.kind === kind && x.id === id);
+ return d && d.doc ? d.doc : null;
+ };
+ const siteIds = new Set();
+ for (const r of idx.rows) if (r.scope === 'site' && r.day >= w.currentStart) siteIds.add(r.scope_id);
+ for (const r of idx.winRows) if (r.scope === 'site') siteIds.add(r.scope_id);
+ for (const d of idx.dims) if (d.kind === 'site') siteIds.add(d.id);
+
+ const sites = [];
+ for (const id of siteIds) {
+ const meta = dimOf('site', id) || { id, name: id, short: id };
+ const hasDaily = dayHas(idx, 'pageviews', 'site', id);
+ const hasAny = hasDaily || winHas(idx, 'site', id);
+ if (!hasAny) continue;
+ const pageviews = winGet(idx, 'pageviews', 'site', id) ?? daySum(idx, 'pageviews', 'site', id, w.currentStart, w.currentEnd);
+ const prev = winGet(idx, 'pageviews_prev', 'site', id) ?? daySum(idx, 'pageviews', 'site', id, w.prevStart, w.prevEnd);
+ const pct = winGet(idx, 'change_pct', 'site', id) ?? change(pageviews, prev);
+ const traffic = (winGet(idx, 'traffic_requests', 'site', id) != null || winGet(idx, 'traffic_p95_ms', 'site', id) != null)
+ ? {
+ requests: winGet(idx, 'traffic_requests', 'site', id),
+ server_errors: winGet(idx, 'traffic_server_errors', 'site', id),
+ p95_ms: winGet(idx, 'traffic_p95_ms', 'site', id),
+ } : null;
+ sites.push({
+ ...meta,
+ pageviews: pageviews == null ? null : pageviews,
+ pageviews_prev: prev == null ? null : prev,
+ change_pct: pct,
+ daily: hasDaily ? daySeries(idx, 'pageviews', 'site', id, w.days) : null,
+ ...(winGet(idx, 'unique_visitors', 'site', id) != null ? { unique_visitors: winGet(idx, 'unique_visitors', 'site', id) } : {}),
+ ...(winGet(idx, 'signed_in_people', 'site', id) != null ? { signed_in_people: winGet(idx, 'signed_in_people', 'site', id) } : {}),
+ traffic_24h: traffic,
+ });
+ }
+ sites.sort((a, b) => (b.pageviews || 0) - (a.pageviews || 0) || String(a.name).localeCompare(String(b.name)));
+
+ const networkHasDaily = dayHas(idx, 'pageviews', 'network', 'network');
+ let pageviews = winGet(idx, 'pageviews', 'network', 'network') ?? daySum(idx, 'pageviews', 'network', 'network', w.currentStart, w.currentEnd);
+ if (pageviews == null) pageviews = sumOrNull(sites.map((s) => s.pageviews));
+ let prev = winGet(idx, 'pageviews_prev', 'network', 'network') ?? daySum(idx, 'pageviews', 'network', 'network', w.prevStart, w.prevEnd);
+ if (prev == null) prev = sumOrNull(sites.map((s) => s.pageviews_prev));
+ const pct = winGet(idx, 'change_pct', 'network', 'network') ?? change(pageviews, prev);
+
+ let pageviewsDaily = null;
+ if (networkHasDaily) pageviewsDaily = daySeries(idx, 'pageviews', 'network', 'network', w.days);
+ else if (sites.some((s) => s.daily)) {
+ pageviewsDaily = w.days.map((d) => ({ day: d, n: sites.reduce((a, s) => a + ((s.daily && s.daily.find((x) => x.day === d)) || {}).n || 0, 0) }));
+ }
+
+ const accountsTotal = winGet(idx, 'accounts_total', 'network', 'network');
+ const accountsNew = winGet(idx, 'accounts_new', 'network', 'network') ?? daySum(idx, 'accounts_new', 'network', 'network', w.currentStart, w.currentEnd);
+ const accountsPrev = winGet(idx, 'accounts_prev', 'network', 'network') ?? daySum(idx, 'accounts_new', 'network', 'network', w.prevStart, w.prevEnd);
+ const accounts = (accountsTotal != null || accountsNew != null || dayHas(idx, 'accounts_new', 'network', 'network'))
+ ? { total: accountsTotal, new: accountsNew, prev: accountsPrev, daily: dayHas(idx, 'accounts_new', 'network', 'network') ? daySeries(idx, 'accounts_new', 'network', 'network', w.days) : [] }
+ : null;
+
+ // coins.earned is all time, so it is only ever the source's own window figure, never a sum of recent days.
+ const coinsEarned = winGet(idx, 'coins_earned', 'network', 'network');
+ const coins = (coinsEarned != null || winGet(idx, 'coins_spent', 'network', 'network') != null)
+ ? {
+ earned: coinsEarned,
+ spent: winGet(idx, 'coins_spent', 'network', 'network'),
+ holders: winGet(idx, 'coins_holders', 'network', 'network'),
+ transactions: winGet(idx, 'coins_transactions', 'network', 'network'),
+ earned_window: winGet(idx, 'coins_earned_window', 'network', 'network') ?? daySum(idx, 'coins_earned', 'network', 'network', w.currentStart, w.currentEnd),
+ earned_prev: winGet(idx, 'coins_earned_prev', 'network', 'network') ?? daySum(idx, 'coins_earned', 'network', 'network', w.prevStart, w.prevEnd),
+ daily: dayHas(idx, 'coins_earned', 'network', 'network') ? daySeries(idx, 'coins_earned', 'network', 'network', w.days) : [],
+ } : null;
+
+ const liveKeys = ['live_now', 'watching', 'streamers', 'streams', 'accounts', 'visitors', 'messages', 'chatted_7d',
+ 'vods', 'clips', 'hours_watched', 'hours_archived', 'ai_moments', 'pastes', 'follows'];
+ const live = {};
+ for (const k of liveKeys) { const v = winGet(idx, k, 'network', 'network'); if (v != null) live[k] = v; }
+ const hasLive = Object.keys(live).length > 0;
+
+ const servicesUp = winGet(idx, 'services_up', 'network', 'network');
+ const servicesChecked = winGet(idx, 'services_checked', 'network', 'network');
+ const noteDoc = dimOf('network', 'network');
+
+ return {
+ generated_at: new Date(nowMs).toISOString(),
+ window_days: windowDays,
+ note: (noteDoc && noteDoc.note) || NOTE,
+ totals: {
+ pageviews: pageviews == null ? null : pageviews,
+ pageviews_prev: prev == null ? null : prev,
+ change_pct: pct,
+ sites_open: winGet(idx, 'sites_open', 'network', 'network') ?? sites.length,
+ services: servicesChecked != null ? { up: servicesUp == null ? 0 : servicesUp, checked: servicesChecked } : null,
+ accounts: accounts ? accounts.total : null,
+ requests_24h: winGet(idx, 'requests_24h', 'network', 'network'),
+ },
+ pageviews_daily: pageviewsDaily,
+ accounts, coins,
+ ...(hasLive ? { live } : {}),
+ busiest_pages_24h: idx.json.get('network|network|busiest_pages') || null,
+ sites,
+ };
+}
+
+/** A creator's streaming analytics from report_streams, plus Billing's revenue read (never a computed amount). */
+async function channelStats(store, channel, days = 30, { now = () => Date.now(), billing = null } = {}) {
+ if (!store || !store.db) return null;
+ const d = Math.min(Math.max(parseInt(days, 10) || 30, 1), 365);
+ const nowMs = now();
+ const since = new Date(nowMs - d * DAY).toISOString();
+ const rows = await store.db.many(`SELECT * FROM report_streams WHERE (lower(channel) = lower($1) OR creator_subject = $2) AND started_at >= $3 ORDER BY started_at DESC`, [String(channel || ''), String(channel || ''), since]);
+ let creator = null;
+ for (const r of rows) if (r.creator_subject) { creator = r.creator_subject; break; }
+ const revenue = creator && billing && typeof billing.revenue === 'function'
+ ? await billing.revenue({ subject: creator, days: d, now: nowMs }).catch(() => null)
+ : null;
+ if (!rows.length && !revenue) return null;
+
+ const blank = () => ({ streams: 0, stream_seconds: 0, peak_viewers: 0, unique_chatters: 0, messages: 0, watch_minutes: 0, _avgWeighted: 0 });
+ const add = (acc, r) => {
+ acc.streams += 1;
+ acc.stream_seconds += r.duration_seconds || 0;
+ acc.peak_viewers = Math.max(acc.peak_viewers, r.peak_viewers || 0);
+ acc.unique_chatters += r.unique_chatters || 0;
+ acc.messages += r.messages || 0;
+ acc.watch_minutes += r.watch_minutes || 0;
+ acc._avgWeighted += (Number(r.avg_viewers) || 0) * (r.duration_seconds || 0);
+ return acc;
+ };
+ const finish = (acc) => {
+ acc.avg_viewers = acc.stream_seconds ? round1(acc._avgWeighted / acc.stream_seconds) : 0;
+ delete acc._avgWeighted;
+ return acc;
+ };
+ const totals = finish(rows.reduce(add, blank()));
+ const byDay = new Map();
+ for (const r of rows) {
+ const day = String(r.started_at || '').slice(0, 10);
+ if (!day) continue;
+ byDay.set(day, add(byDay.get(day) || blank(), r));
+ }
+ const daily = [...byDay.entries()].sort((a, b) => a[0].localeCompare(b[0])).map(([day, acc]) => ({ day, ...finish(acc) }));
+ const streams = rows.slice(0, 100).map((r) => {
+ const out = { stream_id: r.stream_id, title: r.title, category: r.category, started_at: r.started_at, ended_at: r.ended_at, duration_seconds: r.duration_seconds };
+ for (const k of ['peak_viewers', 'avg_viewers', 'unique_chatters', 'messages', 'watch_minutes']) if (r[k] != null) out[k] = Number(r[k]);
+ return out;
+ });
+ return { creator, channel: rows[0] ? rows[0].channel : String(channel), days: d, full: true, totals, daily, streams, revenue };
+}
+
+/** The flat report.metric@1 view: one row per number, with its day or window. */
+async function metrics(store, { scope = 'network', scopeId = null, days = 30, now = () => Date.now() } = {}) {
+ if (!store || !store.db) return null;
+ if (!SCOPES.has(scope)) return null;
+ const id = String(scopeId || (scope === 'network' ? 'network' : ''));
+ if (!id) return null;
+ const windowDays = normaliseDays(days);
+ const w = windowOf(now(), windowDays);
+ const rows = await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, value, unit, source FROM report_metric_days
+ WHERE scope = $1 AND scope_id = $2 AND day >= $3 ORDER BY day, metric`, [scope, id, w.currentStart]);
+ const wins = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, unit, source FROM report_metric_windows
+ WHERE scope = $1 AND scope_id = $2 AND window_days = $3 ORDER BY metric`, [scope, id, windowDays]);
+ const docs = await store.db.many(`SELECT metric, scope, scope_id, window_days, value, source FROM report_metric_json
+ WHERE scope = $1 AND scope_id = $2 AND window_days = $3 ORDER BY metric`, [scope, id, windowDays]);
+ return {
+ as_of: new Date(now()).toISOString(),
+ window_days: windowDays,
+ scope,
+ scope_id: id,
+ metrics: rows.map((r) => ({ metric: r.metric, scope: r.scope, scope_id: r.scope_id, day: r.day, value: n(r.value), unit: r.unit, source: r.source })),
+ windows: wins.map((r) => ({ metric: r.metric, scope: r.scope, scope_id: r.scope_id, window_days: r.window_days, value: n(r.value), unit: r.unit, source: r.source })),
+ documents: docs.map((r) => ({ metric: r.metric, scope: r.scope, scope_id: r.scope_id, window_days: r.window_days, value: r.value, source: r.source })),
+ };
+}
+
+module.exports = { reportStats, channelStats, metrics, windowOf, loadIndex, NOTE, DAY };
new file mode 100644
new file mode 100644
@@ -0,0 +1,137 @@
@@ -0,0 +1,137 @@
+'use strict';
+
+/**
+ * The shadow comparison: read the same window from OpenVibe.Network and from Report's tables, then record every
+ * number that differs in report_shadow_diffs, one run per pass (report_shadow_runs). A clean run has zero diffs.
+ *
+ * The two sides are normalized with the same function the ingest uses (metricRowsFromStats), so an equal answer
+ * compares equal; a changed number, a missing one or an extra stale one is a difference. Billing is not in the
+ * shadow: Report quotes Billing directly and never keeps a second copy of money.
+ */
+const crypto = require('crypto');
+const { metricRowsFromStats } = require('./ingest');
+const { normaliseDays } = require('../data/source');
+const { windowOf } = require('./query');
+
+const num = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
+const same = (a, b) => a == null || b == null ? a == b : Math.abs(a - b) < 1e-6;
+function stable(v) {
+ if (Array.isArray(v)) return `[${v.map(stable).join(',')}]`;
+ if (v && typeof v === 'object') return `{${Object.keys(v).sort().map((k) => `${JSON.stringify(k)}:${stable(v[k])}`).join(',')}}`;
+ return JSON.stringify(v == null ? null : v);
+}
+const keyOf = (r) => r.kind === 'window'
+ ? `w|${r.scope}|${r.scope_id}|${r.metric}|${r.window_days}`
+ : `d|${r.scope}|${r.scope_id}|${r.metric}|${r.day}`;
+
+function runId() { return `shr_${crypto.randomBytes(10).toString('hex')}`; }
+
+async function reportRows(store, days, nowMs) {
+ const w = windowOf(nowMs, days);
+ const rows = await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, value FROM report_metric_days
+ WHERE scope IN ('network', 'site') AND day >= $1`, [w.currentStart]);
+ const wins = await store.db.many(`SELECT metric, scope, scope_id, window_days, value FROM report_metric_windows
+ WHERE scope IN ('network', 'site') AND window_days = $1`, [days]);
+ return [
+ ...rows.map((r) => ({ kind: 'day', metric: r.metric, scope: r.scope, scope_id: r.scope_id, day: r.day, value: num(r.value) })),
+ ...wins.map((r) => ({ kind: 'window', metric: r.metric, scope: r.scope, scope_id: r.scope_id, window_days: r.window_days, value: num(r.value) })),
+ ];
+}
+
+function createShadow({ store, network, now = () => Date.now(), log = console }) {
+ if (!store || !store.db) throw new TypeError('createShadow needs an open Report store');
+
+ /** Compare one window. `stats` may be supplied (a fixture or a cached Network answer); otherwise Network is asked. */
+ async function compare({ days = 7, stats } = {}) {
+ const windowDays = normaliseDays(days);
+ const id = runId();
+ const startedAt = new Date(now()).toISOString();
+ await store.db.prepare(`INSERT INTO report_shadow_runs (run_id, window_days, status, checked, differences, started_at)
+ VALUES (?, ?, 'running', 0, 0, ?)`).run(id, windowDays, startedAt);
+ try {
+ let source = stats;
+ if (source === undefined) source = await network.stats(windowDays);
+ if (!source) {
+ await store.db.prepare(`UPDATE report_shadow_runs SET status = 'network_unavailable', checked = 0, differences = 0, finished_at = ? WHERE run_id = ?`)
+ .run(new Date(now()).toISOString(), id);
+ return { run_id: id, status: 'network_unavailable', checked: 0, differences: 0, diffs: [] };
+ }
+ const networkRows = metricRowsFromStats(source);
+ const stored = await reportRows(store, windowDays, now());
+ const reportByKey = new Map(stored.map((r) => [keyOf(r), r]));
+ const netKeys = new Set(networkRows.map(keyOf));
+ const keys = new Set([...netKeys, ...stored.map(keyOf)]);
+ const diffs = [];
+ let checked = 0;
+ const reportJson = await store.db.maybe(`SELECT value FROM report_metric_json WHERE metric = 'busiest_pages' AND scope = 'network' AND scope_id = 'network' AND window_days = $1`, [windowDays]);
+ const networkJson = Array.isArray(source.busiest_pages_24h) ? source.busiest_pages_24h : null;
+ if ((networkJson || reportJson) && stable(networkJson) !== stable(reportJson ? reportJson.value : null)) {
+ checked++;
+ diffs.push({
+ metric: 'busiest_pages', scope: 'network', scope_id: 'network', day: null, window_days: windowDays,
+ network_value: null, report_value: null, delta: null,
+ network_json: networkJson, report_json: reportJson ? reportJson.value : null,
+ });
+ }
+ for (const key of keys) {
+ const net = networkRows.find((r) => keyOf(r) === key) || null;
+ const rep = reportByKey.get(key) || null;
+ if (!net && !rep) continue;
+ // Only compare a stored number whose metric Network actually reports in this window; an unrelated
+ // stored scope is not a shadow difference.
+ if (!net && !netKeys.has(key) && !networkRows.some((r) => r.metric === (rep && rep.metric))) continue;
+ checked++;
+ const nv = net ? net.value : null;
+ const rv = rep ? rep.value : null;
+ if (same(nv, rv)) continue;
+ diffs.push({
+ metric: (net || rep).metric, scope: (net || rep).scope, scope_id: (net || rep).scope_id,
+ day: (net || rep).kind === 'day' ? (net || rep).day : null,
+ window_days: (net || rep).kind === 'window' ? (net || rep).window_days : null,
+ network_value: nv, report_value: rv, delta: (rv == null ? 0 : rv) - (nv == null ? 0 : nv),
+ });
+ }
+ for (const d of diffs) {
+ await store.db.prepare(`INSERT INTO report_shadow_diffs (run_id, metric, scope, scope_id, day, window_days, network_value, report_value, delta, network_json, report_json)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?::jsonb, ?::jsonb)`)
+ .run(id, d.metric, d.scope, d.scope_id, d.day, d.window_days, d.network_value, d.report_value, d.delta,
+ d.network_json == null ? null : JSON.stringify(d.network_json), d.report_json == null ? null : JSON.stringify(d.report_json));
+ }
+ await store.db.prepare(`UPDATE report_shadow_runs SET status = ?, checked = ?, differences = ?, finished_at = ? WHERE run_id = ?`)
+ .run(diffs.length ? 'differences' : 'equal', checked, diffs.length, new Date(now()).toISOString(), id);
+ return { run_id: id, status: diffs.length ? 'differences' : 'equal', checked, differences: diffs.length, diffs };
+ } catch (err) {
+ const message = String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 400);
+ await store.db.prepare(`UPDATE report_shadow_runs SET status = 'error', error = ?, finished_at = ? WHERE run_id = ?`)
+ .run(message, new Date(now()).toISOString(), id).catch(() => {});
+ if (log && log.warn) log.warn(`[OpenVibe.Report] shadow run failed: ${message}`);
+ return { run_id: id, status: 'error', error: message, checked: 0, differences: 0, diffs: [] };
+ }
+ }
+
+ /** The latest run(s) and their differences, for GET /internal/shadow. */
+ async function summary({ limit = 5, days = null } = {}) {
+ const n = Math.min(Math.max(parseInt(limit, 10) || 5, 1), 50);
+ const windowDays = days == null || days === '' ? null : normaliseDays(days);
+ const runs = windowDays
+ ? await store.db.many(`SELECT run_id, window_days, status, checked, differences, error, started_at, finished_at
+ FROM report_shadow_runs WHERE window_days = $1 ORDER BY run_seq DESC LIMIT $2`, [windowDays, n])
+ : await store.db.many(`SELECT run_id, window_days, status, checked, differences, error, started_at, finished_at
+ FROM report_shadow_runs ORDER BY run_seq DESC LIMIT $1`, [n]);
+ const latest = runs[0] || null;
+ const diffs = latest ? await store.db.many(`SELECT metric, scope, scope_id, day::text AS day, window_days, network_value, report_value, delta, network_json, report_json
+ FROM report_shadow_diffs WHERE run_id = $1 ORDER BY id LIMIT 500`, [latest.run_id]) : [];
+ return {
+ as_of: new Date(now()).toISOString(),
+ latest: latest ? { ...latest, checked: Number(latest.checked), differences: Number(latest.differences) } : null,
+ runs: runs.map((r) => ({ ...r, checked: Number(r.checked), differences: Number(r.differences) })),
+ diffs: diffs.map((d) => ({
+ ...d, network_value: num(d.network_value), report_value: num(d.report_value), delta: num(d.delta),
+ })),
+ };
+ }
+
+ return { compare, summary, runId };
+}
+
+module.exports = { createShadow, reportRows };
@@ -10,8 +10,9 @@
@@ -10,8 +10,9 @@
* /robots.txt, /sitemap.xml, /llms.txt crawl artifacts (http/discovery.js)
* /robots.txt, /sitemap.xml, /llms.txt crawl artifacts (http/discovery.js)
* /api/health, /api/ready, /release.json, /release-metrics the plumbing (loopback scrape for /metrics)
* /api/health, /api/ready, /release.json, /release-metrics the plumbing (loopback scrape for /metrics)
*
*
- * The service is stateless: no database, no sessions and no events. The two source clients are the only state
- * (their in-process caches). The plumbing is the skeleton's, modelled on OpenVibe.Help.
+ * Report has its own database (server/db.js, migrations/) for the analytics it ingests, and reads Billing's money
+ * through Billing's read API. The pages and GET /api/v1/stats answer from REPORT_SOURCE (network by default,
+ * report after cutover); Report's tables are written by the ingest job and by OpenVibe.Events. No sessions.
*/
*/
const path = require('path');
const path = require('path');
const express = require('express');
const express = require('express');
@@ -21,11 +22,16 @@ const contracts = require('openvibe-contracts');
@@ -21,11 +22,16 @@ const contracts = require('openvibe-contracts');
const cache = require('openvibe-shared/cache-policy');
const cache = require('openvibe-shared/cache-policy');
const configLib = require('./config');
const configLib = require('./config');
+const { openStore } = require('./db');
const { createNetworkClient } = require('./data/network');
const { createNetworkClient } = require('./data/network');
const { createLiveClient } = require('./data/live');
const { createLiveClient } = require('./data/live');
+const { createBillingClient } = require('./data/billing');
+const { createAnalytics } = require('./analytics');
+const { createShadow } = require('./analytics/shadow');
const { createApi } = require('./http/api');
const { createApi } = require('./http/api');
const { createPageRoutes } = require('./http/pages');
const { createPageRoutes } = require('./http/pages');
const { createDiscoveryRoutes } = require('./http/discovery');
const { createDiscoveryRoutes } = require('./http/discovery');
+const { createInternalRoutes } = require('./http/internal');
const { createServiceReadiness } = require('./observability');
const { createServiceReadiness } = require('./observability');
const views = require('./views');
const views = require('./views');
const { PUBLIC_DIR, assetVersion } = require('./assets');
const { PUBLIC_DIR, assetVersion } = require('./assets');
@@ -33,18 +39,23 @@ const { PUBLIC_DIR, assetVersion } = require('./assets');
@@ -33,18 +39,23 @@ const { PUBLIC_DIR, assetVersion } = require('./assets');
const VERSION = require('../package.json').version;
const VERSION = require('../package.json').version;
const PAGE_CACHE = 'public, max-age=120';
const PAGE_CACHE = 'public, max-age=120';
-/** opts: config, fetchImpl, log, now, network, live, releaseEnv */
+/** opts: config, store, fetchImpl, log, now, network, live, billing, analytics, shadow, releaseEnv */
async function createApp(opts = {}) {
async function createApp(opts = {}) {
const config = opts.config || configLib.load();
const config = opts.config || configLib.load();
const log = opts.log || console;
const log = opts.log || console;
const fetchImpl = opts.fetchImpl || globalThis.fetch;
const fetchImpl = opts.fetchImpl || globalThis.fetch;
const now = opts.now || (() => Date.now());
const now = opts.now || (() => Date.now());
+ // `store: null` is a deliberate no-database process (most unit tests); undefined opens the configured one.
+ const store = opts.store !== undefined ? opts.store : await openStore(config, { now, log });
const network = opts.network || createNetworkClient({ config, fetchImpl, log, now });
const network = opts.network || createNetworkClient({ config, fetchImpl, log, now });
const live = opts.live || createLiveClient({ config, fetchImpl, log, now });
const live = opts.live || createLiveClient({ config, fetchImpl, log, now });
+ const billing = opts.billing || createBillingClient({ config, fetchImpl, log, now });
+ const analytics = opts.analytics || createAnalytics({ config, store, network, billing, now, log });
+ const shadow = opts.shadow || (store ? createShadow({ store, network, now, log }) : null);
views.configure({ baseUrl: config.baseUrl });
views.configure({ baseUrl: config.baseUrl });
- const ctx = { config, log, fetchImpl, now, network, live };
+ const ctx = { config, log, fetchImpl, now, store, network, live, billing, analytics, shadow };
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
app.set('trust proxy', config.trustProxy);
app.set('trust proxy', config.trustProxy);
@@ -91,7 +102,7 @@ async function createApp(opts = {}) {
@@ -91,7 +102,7 @@ async function createApp(opts = {}) {
// ── Machine endpoints ───────────────────────────────────
// ── Machine endpoints ───────────────────────────────────
app.get('/api/health', (_req, res) => res.set('Cache-Control', 'no-store').json({ status: 'ok', service: 'openvibe-report', version: VERSION }));
app.get('/api/health', (_req, res) => res.set('Cache-Control', 'no-store').json({ status: 'ok', service: 'openvibe-report', version: VERSION }));
release.mount(app, { registry: metrics.registry });
release.mount(app, { registry: metrics.registry });
- const readiness = createServiceReadiness({ config, fetchImpl, release: release.release });
+ const readiness = createServiceReadiness({ config, fetchImpl, release: release.release, store });
ctx.readiness = readiness;
ctx.readiness = readiness;
app.get('/api/ready', readiness.handler);
app.get('/api/ready', readiness.handler);
@@ -108,6 +119,9 @@ async function createApp(opts = {}) {
@@ -108,6 +119,9 @@ async function createApp(opts = {}) {
},
},
}));
}));
+ // ── Loopback-only operator and Events routes ────────────
+ app.use('/internal', createInternalRoutes({ analytics, shadow, config, now, log }));
+
// ── API: CORS-open, public for two minutes, 120 requests a minute per address ──
// ── API: CORS-open, public for two minutes, 120 requests a minute per address ──
app.use('/api/v1', (_req, res, next) => {
app.use('/api/v1', (_req, res, next) => {
res.set('Access-Control-Allow-Origin', '*');
res.set('Access-Control-Allow-Origin', '*');
@@ -121,7 +135,7 @@ async function createApp(opts = {}) {
@@ -121,7 +135,7 @@ async function createApp(opts = {}) {
legacyHeaders: false,
legacyHeaders: false,
handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }),
handler: (req, res) => contracts.http.sendProblem(res, 429, 'rate_limited', { detail: 'too many requests from this address; retry shortly', ctx: req.ov }),
}));
}));
- app.use('/api/v1', createApi({ network, live }));
+ app.use('/api/v1', createApi({ network, live, analytics }));
app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));
app.use('/api', (req, res) => contracts.http.sendProblem(res, 404, 'route.not_found', { detail: 'No such API route', ctx: req.ov }));
// ── Crawl artifacts, then the pages ─────────────────────
// ── Crawl artifacts, then the pages ─────────────────────
@@ -4,19 +4,23 @@
@@ -4,19 +4,23 @@
* OpenVibe.Report configuration. Every value comes from the environment (production: /etc/openvibe/report.env,
* OpenVibe.Report configuration. Every value comes from the environment (production: /etc/openvibe/report.env,
* see .env.example). load(env) is pure so tests can build a config without touching process.env.
* see .env.example). load(env) is pure so tests can build a config without touching process.env.
*
*
- * Report is stateless: it has no database, no sessions and no events. It reads two loopback APIs:
- * OpenVibe.Network's open stats and OpenVibe.Live's Rundown and after-show reports.
+ * Report reads Network's open stats and Live's Rundown and after-show reports over loopback. It also has its own
+ * PostgreSQL database (migrations/) for the analytics it ingests, and reads revenue from Billing. REPORT_SOURCE
+ * chooses which side the pages and GET /api/v1/stats answer from: `network` (today's behaviour) or `report`.
*/
*/
require('dotenv').config();
require('dotenv').config();
const trim = (s) => String(s || '').replace(/\/+$/, '');
const trim = (s) => String(s || '').replace(/\/+$/, '');
const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);
const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);
+const bool = (v, def = false) => (v == null || v === '' ? def : !/^(0|false|no|off)$/i.test(String(v)));
+const sourceOf = (v) => (String(v || '').toLowerCase() === 'report' ? 'report' : 'network');
function load(env = process.env) {
function load(env = process.env) {
const nodeEnv = env.NODE_ENV || 'development';
const nodeEnv = env.NODE_ENV || 'development';
const isProduction = nodeEnv === 'production';
const isProduction = nodeEnv === 'production';
const port = int(env.PORT, 5080);
const port = int(env.PORT, 5080);
const baseUrl = trim(env.BASE_URL || (isProduction ? 'https://openvibe.report' : `http://localhost:${port}`));
const baseUrl = trim(env.BASE_URL || (isProduction ? 'https://openvibe.report' : `http://localhost:${port}`));
+ const networkPublicUrl = trim(env.OV_NETWORK_URL || 'https://openvibe.network');
return {
return {
service: 'report',
service: 'report',
@@ -28,11 +32,52 @@ function load(env = process.env) {
@@ -28,11 +32,52 @@ function load(env = process.env) {
baseUrl,
baseUrl,
trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 2,
trustProxy: env.TRUST_PROXY != null ? Number(env.TRUST_PROXY) : 2,
- // The two sources. Both serve these read APIs on loopback; manifests/services/{network,live}.json in
+ // Which source answers the pages and the stats API. `network` is the shadowing default: the pages keep
+ // reading where they read today until an operator sets REPORT_SOURCE=report.
+ source: sourceOf(env.REPORT_SOURCE),
+
+ // The two read-only sources. Both serve these APIs on loopback; manifests/services/{network,live}.json in
// openvibe-contracts name the same defaults.
// openvibe-contracts name the same defaults.
+ // networkUrl is the loopback origin the read clients call (the manifest's internalOrigin).
networkUrl: trim(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),
networkUrl: trim(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),
+ networkPublicUrl,
+ networkInternalUrl: trim(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),
liveUrl: trim(env.OV_LIVE_INTERNAL_URL || 'http://127.0.0.1:3000'),
liveUrl: trim(env.OV_LIVE_INTERNAL_URL || 'http://127.0.0.1:3000'),
+ // Report's own database (ADR-035). DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner
+ // role); development without DATABASE_URL uses an embedded PGlite database under REPORT_PGLITE_DIR.
+ db: { url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '', pgliteDir: env.REPORT_PGLITE_DIR || '' },
+
+ // OpenVibe.Billing's read API. Report holds no money and derives none: it quotes the amounts Billing
+ // returns. The service token needs billing.stats.read (GET /api/v1/stats) and billing.balance.read
+ // (GET /api/v1/transactions?subject=… for one channel's settled revenue and refunds).
+ billing: {
+ url: trim(env.OV_BILLING_INTERNAL_URL || 'http://127.0.0.1:4600'),
+ token: env.REPORT_BILLING_TOKEN || '',
+ timeoutMs: Math.max(300, int(env.REPORT_BILLING_TIMEOUT_MS, 5000)),
+ },
+ oauth: {
+ tokenUrl: trim(env.OV_NETWORK_OAUTH_TOKEN_URL || `${networkPublicUrl}/oauth/token`),
+ clientId: env.OV_OAUTH_CLIENT_ID || 'report',
+ clientSecret: env.OV_OAUTH_CLIENT_SECRET || '',
+ },
+
+ // OpenVibe.Events → Report: live.stream.ended and the frame beacon. The secret signs a delivery (comma
+ // separated for rotation, 32+ characters each); unset makes POST /internal/events answer 503.
+ events: {
+ secrets: String(env.REPORT_EVENTS_SECRET || '').split(',').map((x) => x.trim()).filter((x) => x.length >= 32),
+ },
+
+ // The ingest and shadow jobs. REPORT_JOBS=0 turns both off (tests do this by never starting them).
+ jobs: {
+ enabled: bool(env.REPORT_JOBS, true),
+ onBoot: bool(env.REPORT_JOBS_ON_BOOT, true),
+ ingestIntervalMs: Math.max(30_000, int(env.REPORT_INGEST_INTERVAL_MS, 5 * 60_000)),
+ ingestWindows: [7, 30, 90],
+ shadowIntervalMs: Math.max(30_000, int(env.REPORT_SHADOW_INTERVAL_MS, 15 * 60_000)),
+ shadowWindowDays: [7, 30, 90].includes(Number(env.REPORT_SHADOW_WINDOW_DAYS)) ? Number(env.REPORT_SHADOW_WINDOW_DAYS) : 7,
+ },
+
// One caller may make this many API requests a minute (express-rate-limit, keyed by address).
// One caller may make this many API requests a minute (express-rate-limit, keyed by address).
apiRateLimitPerMinute: Math.max(1, int(env.REPORT_API_RATE_LIMIT_PER_MIN, 120)),
apiRateLimitPerMinute: Math.max(1, int(env.REPORT_API_RATE_LIMIT_PER_MIN, 120)),
new file mode 100644
new file mode 100644
@@ -0,0 +1,129 @@
@@ -0,0 +1,129 @@
+'use strict';
+
+/**
+ * OpenVibe.Billing's read API. Report never computes money: it quotes the amounts Billing returns.
+ *
+ * GET /api/v1/stats?days=N billing.stats.read network-wide purchases and subscriptions
+ * GET /api/v1/transactions?subject= billing.balance.read one subject's entries, for a channel's settled
+ * revenue and refunds (creator_payable, vibes-bits)
+ *
+ * A service token is used. When OV_OAUTH_CLIENT_ID / OV_OAUTH_CLIENT_SECRET are set, Report asks Network's token
+ * endpoint for the audience openvibe.billing; REPORT_BILLING_TOKEN is an explicit alternative for an operator
+ * run. Unset means the revenue part is null, never zero.
+ */
+const { createServiceTokenClient } = require('openvibe-sdk/auth');
+const { trim, warn, TIMEOUT_MS } = require('./source');
+
+const DEFAULTS = { url: 'http://127.0.0.1:4600', timeoutMs: 5000 };
+const CAPABILITIES = ['billing.stats.read', 'billing.balance.read'];
+const DAY = 86_400_000;
+
+const n = (v) => (v == null || v === '' || !Number.isFinite(Number(v)) ? null : Number(v));
+
+function createBillingClient({ config = {}, fetchImpl = globalThis.fetch, log = console, now = () => Date.now(), tokenProvider = null, timeoutMs } = {}) {
+ const cfg = config.billing || {};
+ const baseUrl = trim(cfg.url || config.billingUrl || DEFAULTS.url);
+ const deadline = timeoutMs || cfg.timeoutMs || DEFAULTS.timeoutMs;
+ let tokenClient = null;
+ if (!tokenProvider && !cfg.token && config.oauth && config.oauth.clientId && config.oauth.clientSecret) {
+ try {
+ tokenClient = createServiceTokenClient({
+ network: config.networkPublicUrl || config.networkUrl || 'https://openvibe.network',
+ tokenUrl: config.oauth.tokenUrl,
+ clientId: config.oauth.clientId,
+ clientSecret: config.oauth.clientSecret,
+ audience: 'openvibe.billing',
+ scope: CAPABILITIES,
+ fetch: fetchImpl,
+ now,
+ });
+ } catch { tokenClient = null; }
+ }
+
+ async function token() {
+ if (tokenProvider) return await tokenProvider();
+ if (cfg.token) return cfg.token;
+ if (!tokenClient) return null;
+ try { return await tokenClient.getToken({ audience: 'openvibe.billing' }); } catch (err) {
+ warn(log, `billing token failed: ${(err && err.message) || 'no token'}`);
+ return null;
+ }
+ }
+
+ async function get(path) {
+ const auth = await token();
+ if (!auth) return null;
+ let res;
+ try {
+ res = await fetchImpl(`${baseUrl}${path}`, {
+ headers: { Accept: 'application/json', Authorization: `Bearer ${auth}` },
+ signal: typeof AbortSignal !== 'undefined' && typeof AbortSignal.timeout === 'function' ? AbortSignal.timeout(deadline) : undefined,
+ });
+ } catch (err) {
+ warn(log, `billing ${path.split('?')[0]} failed: ${(err && err.message) || 'no response'}`);
+ return null;
+ }
+ if (!res || !res.ok) {
+ if (!res || res.status !== 404) warn(log, `billing ${path.split('?')[0]} failed: HTTP ${res ? res.status : 'no response'}`);
+ return null;
+ }
+ try { return await res.json(); } catch (err) { warn(log, `billing ${path.split('?')[0]} failed: ${(err && err.message) || 'bad JSON'}`); return null; }
+ }
+
+ /** Network-wide money counts, exactly as Billing answers them. */
+ async function stats(days = 30) {
+ const d = Math.min(Math.max(parseInt(days, 10) || 30, 1), 90);
+ const body = await get(`/api/v1/stats?days=${d}`);
+ return body ? { ...body, source: 'billing' } : null;
+ }
+
+ /**
+ * One channel's settled revenue and refunds, read from Billing's transaction history. The sums are of Billing's
+ * own `creator_payable` entries; Report never derives a price, a fee or a rate.
+ */
+ async function revenue({ subject, days = 30, now: at = now() } = {}) {
+ if (!subject) return null;
+ const d = Math.min(Math.max(parseInt(days, 10) || 30, 1), 365);
+ const since = at - d * DAY;
+ let settledBits = 0;
+ let refundedBits = 0;
+ let transactions = 0;
+ let cursor = null;
+ for (let page = 0; page < 20; page++) {
+ const suffix = cursor ? `&cursor=${encodeURIComponent(cursor)}` : '';
+ const body = await get(`/api/v1/transactions?subject=${encodeURIComponent(subject)}&limit=200${suffix}`);
+ if (!body || !Array.isArray(body.transactions)) return null;
+ for (const txn of body.transactions) {
+ if (!txn || txn.test === true) continue;
+ const created = Date.parse(txn.created_at || '');
+ if (!Number.isFinite(created) || created < since) continue;
+ const refundLike = txn.type === 'refund' || txn.type === 'chargeback' || !!txn.reverses_txn;
+ const earning = txn.type === 'donation' || txn.type === 'subscription' || txn.type === 'subscription_share';
+ let seen = false;
+ for (const e of Array.isArray(txn.entries) ? txn.entries : []) {
+ if (!e || !e.account || e.account.kind !== 'creator_payable' || e.account.currency !== 'vibes-bits') continue;
+ const amount = n(e.amount);
+ if (amount == null) continue;
+ seen = true;
+ if (amount > 0 && earning) settledBits += amount;
+ else if (amount < 0 && refundLike) refundedBits += -amount;
+ }
+ if (seen) transactions++;
+ }
+ cursor = body.next_cursor || null;
+ if (!cursor) break;
+ }
+ return finish();
+ function finish() {
+ return {
+ source: 'billing', capability: 'billing.balance.read', subject, days: d, currency: 'vibes-bits',
+ as_of: new Date(at).toISOString(), settled_bits: settledBits, refunded_bits: refundedBits,
+ net_bits: settledBits - refundedBits, transactions,
+ };
+ }
+ }
+
+ return { stats, revenue, baseUrl, capabilities: CAPABILITIES, ttlMs: deadline, get };
+}
+
+module.exports = { createBillingClient, CAPABILITIES, DEFAULTS, TIMEOUT_MS };
new file mode 100644
new file mode 100644
@@ -0,0 +1,49 @@
@@ -0,0 +1,49 @@
+'use strict';
+
+/**
+ * OpenVibe.Report's own PostgreSQL database (ADR-035): the schema is migrations/NNNN_*.sql, applied at boot.
+ *
+ * Report ingests the analytics events OpenVibe.Network records (live.stream.ended, the navbar frame beacon)
+ * and the daily numbers Network publishes; it keeps Billing's money untouched and reads it through Billing's
+ * API. Nothing written here is a secret: a stream's public facts, counts and the ids of events already handled.
+ */
+const fs = require('fs');
+const path = require('path');
+const { createDb } = require('openvibe-sdk/db');
+
+const MIGRATIONS = path.join(__dirname, '..', 'migrations');
+const DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');
+
+async function openDb(config, { log = console, registry } = {}) {
+ const dbConfig = (config && config.db) || {};
+ if (!dbConfig.url) {
+ if (config && config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh report)');
+ const dir = dbConfig.pgliteDir || DEV_PGLITE;
+ log.warn(`[OpenVibe.Report] DATABASE_URL unset: embedded PGlite database in ${dir} (development only, one process)`);
+ fs.mkdirSync(dir, { recursive: true });
+ const db = createDb({ pglite: dir, service: 'report', registry, log });
+ await db.migrate({ dir: MIGRATIONS, log });
+ return db;
+ }
+ if (!dbConfig.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');
+ const owner = createDb({ url: dbConfig.directUrl, service: 'report-migrate', max: 1, log });
+ try { await owner.migrate({ dir: MIGRATIONS, log }); } finally { await owner.close(); }
+ return createDb({ url: dbConfig.url, service: 'report', registry, log });
+}
+
+/** The store every module takes: the db handle, a clock (epoch ms; tests inject one) and transactions. */
+function createStore(db, { now = () => Date.now() } = {}) {
+ return {
+ db,
+ now,
+ iso: () => new Date(now()).toISOString(),
+ tx: async (fn) => await db.tx(fn),
+ close: () => db.close(),
+ };
+}
+
+async function openStore(config, { now, log, registry } = {}) {
+ return createStore(await openDb(config, { log, registry }), { now });
+}
+
+module.exports = { openDb, openStore, createStore, MIGRATIONS };
@@ -20,7 +20,7 @@ const { normaliseDays, isSlug, isStreamId, isUsername, streamIdOf } = require('.
@@ -20,7 +20,7 @@ const { normaliseDays, isSlug, isStreamId, isUsername, streamIdOf } = require('.
const CACHE = 'public, max-age=120';
const CACHE = 'public, max-age=120';
-function createApi({ network, live }) {
+function createApi({ network, live, analytics = null }) {
const r = express.Router();
const r = express.Router();
r.use((req, res, next) => {
r.use((req, res, next) => {
@@ -42,8 +42,9 @@ function createApi({ network, live }) {
@@ -42,8 +42,9 @@ function createApi({ network, live }) {
}
}
r.get('/stats', async (req, res) => {
r.get('/stats', async (req, res) => {
- const stats = await network.stats(normaliseDays(req.query.days));
- if (!stats) return notFound(req, res, 'report.stats_not_found', 'OpenVibe.Network did not answer with the open numbers.');
+ const days = normaliseDays(req.query.days);
+ const stats = analytics ? await analytics.stats(days) : await network.stats(days);
+ if (!stats) return notFound(req, res, 'report.stats_not_found', 'The open numbers did not answer for this window.');
return res.json(stats);
return res.json(stats);
});
});
new file mode 100644
new file mode 100644
@@ -0,0 +1,91 @@
@@ -0,0 +1,91 @@
+'use strict';
+
+/**
+ * /internal — loopback-only operator and Events endpoints. nginx answers 404 for /internal/; the app also checks
+ * that the socket really is loopback (a spoofed X-Forwarded-For cannot reach these).
+ *
+ * POST /internal/events OpenVibe.Events delivery (HMAC v2, REPORT_EVENTS_SECRET)
+ * GET /internal/shadow?days=7 the latest shadow runs and their differences
+ * POST /internal/shadow?days=7 run a comparison now
+ * GET /internal/metrics?scope=&id=&days= the flat report.metric@1 rows
+ * GET /internal/channels/:channel/analytics a channel's creator analytics and Billing revenue
+ */
+const express = require('express');
+const contracts = require('openvibe-contracts');
+const { parseDelivery } = require('openvibe-sdk/events');
+
+const LOOPBACK = new Set(['127.0.0.1', '::1', '::ffff:127.0.0.1', '::ffff:7f00:1']);
+
+function isLoopback(req) {
+ const raw = req.socket && req.socket.remoteAddress ? String(req.socket.remoteAddress) : '';
+ return LOOPBACK.has(raw) || LOOPBACK.has(raw.replace(/^::ffff:/, ''));
+}
+
+function createInternalRoutes({ analytics = null, shadow = null, config = {}, now = () => Date.now(), log = console } = {}) {
+ const r = express.Router();
+ r.use((req, res, next) => {
+ if (isLoopback(req)) return next();
+ return contracts.http.sendProblem(res, 403, 'internal.loopback_only', { detail: 'This endpoint is for the host itself.', ctx: req.ov });
+ });
+
+ r.get('/shadow', async (req, res) => {
+ if (!shadow) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });
+ try {
+ const body = await shadow.summary({ limit: req.query.limit, days: req.query.days });
+ res.set('Cache-Control', 'no-store').json(body);
+ } catch (err) {
+ log.error('[OpenVibe.Report] shadow summary failed:', err && err.message ? String(err.message).slice(0, 200) : err);
+ contracts.http.sendProblem(res, 500, 'report.shadow_failed', { detail: 'The shadow summary could not be read.', ctx: req.ov });
+ }
+ });
+
+ r.post('/shadow', async (req, res) => {
+ if (!shadow) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });
+ try {
+ const run = await shadow.compare({ days: req.query.days });
+ res.set('Cache-Control', 'no-store').json(run);
+ } catch (err) {
+ log.error('[OpenVibe.Report] shadow run failed:', err && err.message ? String(err.message).slice(0, 200) : err);
+ contracts.http.sendProblem(res, 500, 'report.shadow_failed', { detail: 'The shadow comparison could not run.', ctx: req.ov });
+ }
+ });
+
+ r.get('/metrics', async (req, res) => {
+ if (!analytics) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });
+ const body = await analytics.metrics({ scope: req.query.scope, scopeId: req.query.id, days: req.query.days });
+ if (!body) return contracts.http.sendProblem(res, 404, 'report.metric_not_found', { detail: 'No such metric scope.', ctx: req.ov });
+ res.set('Cache-Control', 'no-store').json(body);
+ });
+
+ r.get('/channels/:channel/analytics', async (req, res) => {
+ if (!analytics) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });
+ const body = await analytics.channel(req.params.channel, req.query.days);
+ if (!body) return contracts.http.sendProblem(res, 404, 'report.channel_not_found', { detail: 'No such channel in Report\'s tables.', ctx: req.ov });
+ res.set('Cache-Control', 'no-store').json(body);
+ });
+
+ // OpenVibe.Events delivery. The raw body is read before any JSON parser; the signature is v2 only.
+ r.post('/events', express.raw({ type: () => true, limit: '256kb' }), async (req, res) => {
+ const secrets = (config.events && config.events.secrets) || [];
+ if (!secrets.length) return contracts.http.sendProblem(res, 503, 'report.events_disabled', { detail: 'REPORT_EVENTS_SECRET is not set.', ctx: req.ov });
+ if (!analytics) return contracts.http.sendProblem(res, 503, 'report.no_database', { detail: 'Report has no database in this process.', ctx: req.ov });
+ const raw = Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0);
+ let delivery = null;
+ for (const secret of secrets) {
+ delivery = parseDelivery(raw, req.headers, secret, { requireV2: true, now: now() });
+ if (delivery) break;
+ }
+ if (!delivery) return contracts.http.sendProblem(res, 401, 'report.bad_signature', { detail: 'The delivery does not verify or is outside the replay window.', ctx: req.ov });
+ try {
+ const out = await analytics.recordEvent(delivery.event);
+ return res.set('Cache-Control', 'no-store').json({ ok: true, event_id: delivery.event.event_id, ...out });
+ } catch (err) {
+ log.error('[OpenVibe.Report] event ingest failed:', err && err.message ? String(err.message).slice(0, 200) : err);
+ return contracts.http.sendProblem(res, 500, 'report.ingest_failed', { detail: 'The event could not be stored; Events will retry.', ctx: req.ov });
+ }
+ });
+
+ return r;
+}
+
+module.exports = { createInternalRoutes, isLoopback };
@@ -21,7 +21,7 @@ const { normaliseDays, isSlug, isStreamId, isUsername, streamIdOf } = require('.
@@ -21,7 +21,7 @@ const { normaliseDays, isSlug, isStreamId, isUsername, streamIdOf } = require('.
const CACHE = 'public, max-age=120';
const CACHE = 'public, max-age=120';
-function createPageRoutes({ network, live }) {
+function createPageRoutes({ network, live, analytics = null }) {
const r = express.Router();
const r = express.Router();
function send(res, html, status = 200) {
function send(res, html, status = 200) {
@@ -33,7 +33,7 @@ function createPageRoutes({ network, live }) {
@@ -33,7 +33,7 @@ function createPageRoutes({ network, live }) {
r.get('/', async (_req, res) => {
r.get('/', async (_req, res) => {
const [stats, rundown, topics] = await Promise.all([
const [stats, rundown, topics] = await Promise.all([
- network.stats(7),
+ analytics ? analytics.stats(7) : network.stats(7),
live.rundown(),
live.rundown(),
live.topics(),
live.topics(),
]);
]);
@@ -45,7 +45,7 @@ function createPageRoutes({ network, live }) {
@@ -45,7 +45,7 @@ function createPageRoutes({ network, live }) {
r.get('/stats', async (req, res) => {
r.get('/stats', async (req, res) => {
const days = normaliseDays(req.query.days);
const days = normaliseDays(req.query.days);
- const stats = await network.stats(days);
+ const stats = analytics ? await analytics.stats(days) : await network.stats(days);
if (!stats) return notFound(res);
if (!stats) return notFound(res);
return send(res, views.stats({ stats, days }));
return send(res, views.stats({ stats, days }));
});
});
@@ -5,14 +5,16 @@
@@ -5,14 +5,16 @@
* Listens on PORT (5080) behind nginx (deploy/).
* Listens on PORT (5080) behind nginx (deploy/).
*/
*/
const { createApp } = require('./app');
const { createApp } = require('./app');
+const { createAnalyticsJobs } = require('./jobs/analytics');
const { gracefulStop } = require('openvibe-sdk/service');
const { gracefulStop } = require('openvibe-sdk/service');
/**
/**
- * The process stop (openvibe-sdk/service): the HTTP drain runs, then anything the product added closes. Report
- * has no database, no key refresher and no subscriptions, so there is nothing else to close. Exported so a test can
- * inject `exit` and `signals: false`.
+ * The process stop (openvibe-sdk/service): the HTTP drain runs, then the jobs stop and the database closes.
+ * Exported so a test can inject `exit` and `signals: false`.
*/
*/
-function createLifecycle({ server, exit, signals } = {}) {
+function createLifecycle({ server, exit, signals, ctx } = {}) {
+ const jobs = ctx && ctx.jobs;
+ const store = ctx && ctx.store;
return gracefulStop({
return gracefulStop({
name: 'OpenVibe.Report',
name: 'OpenVibe.Report',
server,
server,
@@ -20,6 +22,8 @@ function createLifecycle({ server, exit, signals } = {}) {
@@ -20,6 +22,8 @@ function createLifecycle({ server, exit, signals } = {}) {
exit,
exit,
signals,
signals,
deadlineMs: 10_000,
deadlineMs: 10_000,
+ stop: [() => (jobs ? jobs.stop() : undefined)],
+ close: [() => (store ? store.close() : undefined)],
});
});
}
}
@@ -27,8 +31,12 @@ async function start() {
@@ -27,8 +31,12 @@ async function start() {
const { app, ctx } = await createApp();
const { app, ctx } = await createApp();
const { config } = ctx;
const { config } = ctx;
+ const jobs = createAnalyticsJobs({ config, analytics: ctx.analytics, shadow: ctx.shadow, log: ctx.log });
+ ctx.jobs = jobs;
+ jobs.start();
+
const server = app.listen(config.port, config.host, () => {
const server = app.listen(config.port, config.host, () => {
- console.log(`[OpenVibe.Report] ${config.nodeEnv} on http://${config.host}:${config.port} → ${config.baseUrl} (network ${config.networkUrl}, live ${config.liveUrl})`);
+ console.log(`[OpenVibe.Report] ${config.nodeEnv} on http://${config.host}:${config.port} → ${config.baseUrl} (source ${config.source}; network ${config.networkUrl}, live ${config.liveUrl}, billing ${config.billing.url})`);
});
});
server.keepAliveTimeout = 65_000;
server.keepAliveTimeout = 65_000;
createLifecycle({ server, ctx });
createLifecycle({ server, ctx });
new file mode 100644
new file mode 100644
@@ -0,0 +1,73 @@
@@ -0,0 +1,73 @@
+'use strict';
+
+/**
+ * The two background jobs (server/index.js starts them; tests never do):
+ *
+ * ingest pull Network's open stats into Report's tables every REPORT_INGEST_INTERVAL_MS and at boot
+ * shadow compare the same window from Network and Report every REPORT_SHADOW_INTERVAL_MS and at boot
+ *
+ * Neither job runs when REPORT_JOBS=0. A failed pass logs one short line and leaves the last good data in place;
+ * the next pass retries. Overlapping passes in one process are skipped.
+ */
+function createAnalyticsJobs({ config = {}, analytics = null, shadow = null, log = console } = {}) {
+ const cfg = config.jobs || {};
+ const timers = [];
+ let ingestBusy = false;
+ let shadowBusy = false;
+
+ async function runIngest() {
+ if (!analytics || ingestBusy) return null;
+ ingestBusy = true;
+ try {
+ const out = [];
+ for (const days of cfg.ingestWindows || [7, 30, 90]) out.push(await analytics.ingestNetwork({ days }));
+ if (out.every((x) => x && x.ok === false && x.reason === 'network_did_not_answer')) log.warn('[OpenVibe.Report] ingest: Network did not answer; keeping the last numbers');
+ return out;
+ } catch (err) {
+ log.warn(`[OpenVibe.Report] ingest failed: ${String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 200)}`);
+ return null;
+ } finally { ingestBusy = false; }
+ }
+
+ async function runShadow() {
+ if (!shadow || shadowBusy) return null;
+ shadowBusy = true;
+ try {
+ const out = await shadow.compare({ days: cfg.shadowWindowDays || 7 });
+ if (out && out.status === 'differences') log.warn(`[OpenVibe.Report] shadow: ${out.differences} difference(s) over ${cfg.shadowWindowDays || 7} days`);
+ return out;
+ } catch (err) {
+ log.warn(`[OpenVibe.Report] shadow failed: ${String((err && err.message) || err).replace(/\s+/g, ' ').slice(0, 200)}`);
+ return null;
+ } finally { shadowBusy = false; }
+ }
+
+ function every(ms, fn) {
+ const t = setInterval(fn, ms);
+ if (typeof t.unref === 'function') t.unref();
+ timers.push(t);
+ }
+
+ function start() {
+ if (!cfg.enabled) return false;
+ if (cfg.onBoot) {
+ const t = setTimeout(async () => { await runIngest(); await runShadow(); }, 2_000);
+ if (typeof t.unref === 'function') t.unref();
+ timers.push(t);
+ }
+ every(cfg.ingestIntervalMs || 5 * 60_000, runIngest);
+ every(cfg.shadowIntervalMs || 15 * 60_000, runShadow);
+ return true;
+ }
+
+ async function stop() {
+ while (timers.length) { const t = timers.pop(); clearTimeout(t); clearInterval(t); }
+ // Let an in-flight pass finish before the database closes (gracefulStop awaits this step).
+ const until = Date.now() + 5_000;
+ while ((ingestBusy || shadowBusy) && Date.now() < until) await new Promise((r) => setTimeout(r, 50));
+ }
+
+ return { start, stop, runIngest, runShadow };
+}
+
+module.exports = { createAnalyticsJobs };
@@ -3,12 +3,14 @@
@@ -3,12 +3,14 @@
/**
/**
* Truthful readiness for GET /api/ready (openvibe-shared/ready, Track O).
* Truthful readiness for GET /api/ready (openvibe-shared/ready, Track O).
*
*
- * Report has no database, no sessions and no queues: its only dependencies are the two loopback read APIs.
- * Both checks are optional, so a source that does not answer makes the service degraded, never not_ready:
- * the process still serves the pages it can and the 503 would stop the deploy of a stateless front.
+ * The two loopback read APIs are optional: a source that does not answer makes the service degraded, never
+ * not_ready, because the process still serves the pages it can. When a PostgreSQL database is configured, its
+ * check is required only after REPORT_SOURCE=report (before the cutover it is Report's shadow store, not the
+ * answer to a page).
*
*
* network optional OpenVibe.Network answers /api/health
* network optional OpenVibe.Network answers /api/health
* live optional OpenVibe.Live answers /api/health
* live optional OpenVibe.Live answers /api/health
+ * db required only when REPORT_SOURCE=report, and only when DATABASE_URL is configured
*/
*/
const { createReadiness } = require('openvibe-shared/ready');
const { createReadiness } = require('openvibe-shared/ready');
@@ -32,14 +34,26 @@ function createServiceReadiness({ config, fetchImpl = globalThis.fetch, release
@@ -32,14 +34,26 @@ function createServiceReadiness({ config, fetchImpl = globalThis.fetch, release
},
},
});
});
+ const checks = [
+ healthOf('network', config.networkUrl),
+ healthOf('live', config.liveUrl),
+ ];
+ if (config.db && config.db.url) {
+ checks.push({
+ name: 'db',
+ required: config.source === 'report',
+ description: 'Report\'s PostgreSQL database answers',
+ async check() {
+ if (!store || !store.db) return 'no database handle';
+ try { await store.db.value('SELECT 1'); return true; } catch (err) { return `database did not answer: ${(err && err.message) || 'query failed'}`; }
+ },
+ });
+ }
return createReadiness({
return createReadiness({
service: 'report',
service: 'report',
release,
release,
timeoutMs: config.readyTimeoutMs || 2500,
timeoutMs: config.readyTimeoutMs || 2500,
- checks: [
- healthOf('network', config.networkUrl),
- healthOf('live', config.liveUrl),
- ],
+ checks,
});
});
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,58 @@
@@ -0,0 +1,58 @@
+'use strict';
+
+/**
+ * Billing's read API: network money counts (billing.stats.read) and one subject's settled creator_payable
+ * entries (billing.balance.read). The stub checks the URL and the service bearer token; Report only sums the
+ * amounts Billing returned, and excludes test rows.
+ */
+const assert = require('assert');
+const { createBillingClient } = require('../server/data/billing');
+const { SUBJECT, NOW, billingTransactions } = require('./fixtures/analytics');
+const { check, done } = require('./helpers/boot');
+
+const json = (body, status = 200) => new Response(JSON.stringify(body), { status, headers: { 'content-type': 'application/json' } });
+
+(async () => {
+ await check('a channel revenue read uses Billing\'s transactions endpoint and bearer capability', async () => {
+ const calls = [];
+ const client = createBillingClient({
+ config: { billing: { url: 'http://billing.test', token: 'test-billing-token', timeoutMs: 1000 } },
+ fetchImpl: async (url, opts) => {
+ calls.push({ url: String(url), auth: opts.headers.Authorization });
+ return json(billingTransactions);
+ },
+ now: () => NOW,
+ });
+ const revenue = await client.revenue({ subject: SUBJECT, days: 7, now: NOW });
+ assert.deepStrictEqual(revenue, {
+ source: 'billing', capability: 'billing.balance.read', subject: SUBJECT, days: 7, currency: 'vibes-bits',
+ as_of: new Date(NOW).toISOString(), settled_bits: 500, refunded_bits: 100, net_bits: 400, transactions: 2,
+ });
+ assert.strictEqual(calls.length, 1);
+ assert.strictEqual(calls[0].url, `http://billing.test/api/v1/transactions?subject=${encodeURIComponent(SUBJECT)}&limit=200`);
+ assert.strictEqual(calls[0].auth, 'Bearer test-billing-token');
+ assert.deepStrictEqual(client.capabilities, ['billing.stats.read', 'billing.balance.read']);
+ });
+
+ await check('network money counts come from GET /api/v1/stats as Billing answers them', async () => {
+ const body = { as_of: new Date(NOW).toISOString(), purchases: { bits_total: 1000, windows: { d: 10, w: 100, m: 200, pw: 50 }, series: [] }, subscriptions: { active: 3, windows: { d: 0, w: 1, m: 2, pw: 1 }, series: [] } };
+ const urls = [];
+ const client = createBillingClient({
+ config: { billing: { url: 'http://billing.test', token: 't' } },
+ fetchImpl: async (url) => { urls.push(String(url)); return json(body); },
+ });
+ const got = await client.stats(7);
+ assert.deepStrictEqual(got, { ...body, source: 'billing' });
+ assert.deepStrictEqual(urls, ['http://billing.test/api/v1/stats?days=7']);
+ });
+
+ await check('no token and no OAuth client means null, and no request is made', async () => {
+ let calls = 0;
+ const client = createBillingClient({ config: { billing: { url: 'http://billing.test' } }, fetchImpl: async () => { calls++; return json({}); } });
+ assert.strictEqual(await client.revenue({ subject: SUBJECT, days: 7, now: NOW }), null);
+ assert.strictEqual(await client.stats(7), null);
+ assert.strictEqual(calls, 0);
+ });
+
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
+'use strict';
+
+/**
+ * POST /internal/events is loopback-only and verifies the OpenVibe.Events v2 signature. A signed delivery is
+ * ingested once; a replay returns duplicate and writes nothing; a forged signature is refused.
+ */
+const assert = require('assert');
+const { signDeliveryV2 } = require('openvibe-sdk/events');
+const { boot, check, done } = require('./helpers/boot');
+const { streams, NOW } = require('./fixtures/analytics');
+
+const SECRET = 'test-report-events-secret-32-characters-long';
+
+(async () => {
+ const t = await boot({ db: true, now: () => NOW, env: { REPORT_EVENTS_SECRET: SECRET } });
+ try {
+ const raw = JSON.stringify({ event: streams[0] });
+ const ts = Math.floor(NOW / 1000);
+ const headers = {
+ 'content-type': 'application/json',
+ 'x-openvibe-timestamp': String(ts),
+ 'x-openvibe-signature-v2': signDeliveryV2(raw, SECRET, ts),
+ };
+
+ await check('a signed live.stream.ended is ingested once, and a replay is a duplicate', async () => {
+ const first = await t.get('/internal/events', { method: 'POST', headers, body: raw });
+ assert.strictEqual(first.status, 200, first.text);
+ assert.strictEqual(first.json().ok, true);
+ assert.strictEqual(first.json().duplicate, false);
+ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 1);
+ const again = await t.get('/internal/events', { method: 'POST', headers, body: raw });
+ assert.strictEqual(again.status, 200, again.text);
+ assert.strictEqual(again.json().duplicate, true);
+ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 1);
+ });
+
+ await check('a forged signature is refused before any write', async () => {
+ const bad = { ...headers, 'x-openvibe-signature-v2': signDeliveryV2(raw, 'another-secret-32-characters-long!!', ts) };
+ const r = await t.get('/internal/events', { method: 'POST', headers: bad, body: raw });
+ assert.strictEqual(r.status, 401);
+ assert.strictEqual(r.json().code, 'report.bad_signature');
+ });
+ } finally { await t.close(); }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,123 @@
@@ -0,0 +1,123 @@
+'use strict';
+
+/**
+ * Fixed analytics facts for the tests: seven days ending 2026-10-28, two sites, two ended streams.
+ * Every expected number is written down here, independent of the implementation.
+ */
+const NOW = Date.parse('2026-10-28T12:00:00.000Z');
+const DAY = 86_400_000;
+const DAYS = Array.from({ length: 7 }, (_, i) => new Date(NOW - (6 - i) * DAY).toISOString().slice(0, 10));
+const SUBJECT = 'usr_01J0000000000000000000000A';
+
+const PAGVIEWS_DAILY = [20, 20, 20, 20, 20, 40, 60]; // 200 from the navbar site
+const ACCOUNTS_DAILY = [0, 0, 1, 0, 1, 1, 0]; // 3 new
+const COINS_DAILY = [10, 10, 10, 10, 10, 10, 10]; // 70 this window
+
+const stats = {
+ generated_at: '2026-10-28T12:00:00.000Z',
+ window_days: 7,
+ note: 'Counts only, never about a person.',
+ totals: {
+ pageviews: 300, pageviews_prev: 200, change_pct: 50, sites_open: 2,
+ services: { up: 1, checked: 2 }, accounts: 10, requests_24h: 1000,
+ },
+ pageviews_daily: DAYS.map((day, i) => ({ day, n: PAGVIEWS_DAILY[i] })),
+ accounts: { total: 10, new: 3, prev: 2, daily: DAYS.map((day, i) => ({ day, n: ACCOUNTS_DAILY[i] })) },
+ coins: {
+ earned: 500, spent: 100, holders: 4, transactions: 20, earned_window: 70, earned_prev: 50,
+ daily: DAYS.map((day, i) => ({ day, n: COINS_DAILY[i] })),
+ },
+ live: {
+ live_now: 2, watching: 9, streamers: 5, streams: 6, visitors: 7, messages: 80, chatted_7d: 4,
+ vods: 3, clips: 2, hours_watched: 11, hours_archived: 12, ai_moments: 1, pastes: 0, follows: 13,
+ },
+ busiest_pages_24h: [{ service: 'network', route: '/stats', requests: 100 }],
+ sites: [
+ {
+ id: 'network', name: 'OpenVibe.Network', short: 'Network', host: 'openvibe.network', url: 'https://openvibe.network/',
+ pageviews: 200, pageviews_prev: 150, change_pct: 33, daily: DAYS.map((day, i) => ({ day, n: PAGVIEWS_DAILY[i] })),
+ unique_visitors: 20, traffic_24h: { requests: 800, server_errors: 2, p95_ms: 120 },
+ },
+ {
+ id: 'live', name: 'OpenVibe.Live', short: 'Live', host: 'openvibe.live', url: 'https://openvibe.live/',
+ pageviews: 100, pageviews_prev: 50, change_pct: 100, daily: null,
+ unique_visitors: 30, signed_in_people: 10, traffic_24h: null,
+ },
+ ],
+};
+
+const expected = {
+ generated_at: new Date(NOW).toISOString(),
+ window_days: 7,
+ totals: {
+ pageviews: 300, pageviews_prev: 200, change_pct: 50, sites_open: 2,
+ services: { up: 1, checked: 2 }, accounts: 10, requests_24h: 1000,
+ },
+ pageviews_daily: DAYS.map((day, i) => ({ day, n: PAGVIEWS_DAILY[i] })),
+ accounts: { total: 10, new: 3, prev: 2, daily: DAYS.map((day, i) => ({ day, n: ACCOUNTS_DAILY[i] })) },
+ coins: {
+ earned: 500, spent: 100, holders: 4, transactions: 20, earned_window: 70, earned_prev: 50,
+ daily: DAYS.map((day, i) => ({ day, n: COINS_DAILY[i] })),
+ },
+ live: {
+ live_now: 2, watching: 9, streamers: 5, streams: 6, visitors: 7, messages: 80, chatted_7d: 4,
+ vods: 3, clips: 2, hours_watched: 11, hours_archived: 12, ai_moments: 1, pastes: 0, follows: 13,
+ },
+ busiest_pages_24h: [{ service: 'network', route: '/stats', requests: 100 }],
+};
+
+function streamEnded(streamId, { channel = 'goosely', startedAt, endedAt, duration, peak, avg, chatters, messages, watch, subject = SUBJECT } = {}) {
+ return {
+ event_id: `evt_TESTSTREAM${streamId}`,
+ event_type: 'live.stream.ended',
+ source: 'live',
+ occurred_at: endedAt,
+ payload: {
+ stream_id: streamId,
+ channel: { username: channel, display_name: channel, url: `https://openvibe.live/@${channel}`, subject: { type: 'user', id: subject } },
+ title: `Stream ${streamId}`,
+ category: 'tech',
+ protocol: 'webrtc',
+ is_nsfw: false,
+ started_at: startedAt,
+ ended_at: endedAt,
+ duration_seconds: duration,
+ stats: { peak_viewers: peak, avg_viewers: avg, unique_chatters: chatters, messages, watch_minutes: watch },
+ },
+ };
+}
+
+const streams = [
+ streamEnded(101, { startedAt: '2026-10-26T10:00:00.000Z', endedAt: '2026-10-26T11:00:00.000Z', duration: 3600, peak: 50, avg: 20.5, chatters: 8, messages: 120, watch: 900 }),
+ streamEnded(102, { startedAt: '2026-10-27T10:00:00.000Z', endedAt: '2026-10-27T10:30:00.000Z', duration: 1800, peak: 30, avg: 10, chatters: 4, messages: 40, watch: 300 }),
+];
+
+const expectedChannel = {
+ totals: { streams: 2, stream_seconds: 5400, peak_viewers: 50, unique_chatters: 12, messages: 160, watch_minutes: 1200, avg_viewers: 17 },
+ daily: [
+ { day: '2026-10-26', streams: 1, stream_seconds: 3600, peak_viewers: 50, unique_chatters: 8, messages: 120, watch_minutes: 900, avg_viewers: 20.5 },
+ { day: '2026-10-27', streams: 1, stream_seconds: 1800, peak_viewers: 30, unique_chatters: 4, messages: 40, watch_minutes: 300, avg_viewers: 10 },
+ ],
+};
+
+const billingTransactions = {
+ transactions: [
+ {
+ id: 'txn_1', type: 'donation', status: 'settled', test: false, created_at: '2026-10-27T09:00:00.000Z',
+ entries: [
+ { account: { kind: 'creator_payable', owner: SUBJECT, currency: 'vibes-bits' }, amount: 500 },
+ { account: { kind: 'user_credit', owner: 'usr_01J0000000000000000000000B', currency: 'vibes-bits' }, amount: -500 },
+ ],
+ },
+ {
+ id: 'txn_2', type: 'refund', status: 'settled', test: false, reverses_txn: 'txn_1', created_at: '2026-10-28T09:00:00.000Z',
+ entries: [{ account: { kind: 'creator_payable', owner: SUBJECT, currency: 'vibes-bits' }, amount: -100 }],
+ },
+ {
+ id: 'txn_3', type: 'donation', status: 'settled', test: true, created_at: '2026-10-28T09:30:00.000Z',
+ entries: [{ account: { kind: 'creator_payable', owner: SUBJECT, currency: 'vibes-bits' }, amount: 999 }],
+ },
+ ],
+};
+
+module.exports = { NOW, DAY, DAYS, SUBJECT, stats, expected, streams, expectedChannel, billingTransactions };
@@ -6,15 +6,19 @@ process.env.RELEASE_COMMIT = process.env.RELEASE_COMMIT || '0000000';
@@ -6,15 +6,19 @@ process.env.RELEASE_COMMIT = process.env.RELEASE_COMMIT || '0000000';
/**
/**
* Boots OpenVibe.Report on an ephemeral port with a stub fetch (no network, no running site) and returns a small
* Boots OpenVibe.Report on an ephemeral port with a stub fetch (no network, no running site) and returns a small
- * HTTP client. Each test file gets its own app, clients and captured log lines.
+ * HTTP client. Each test file gets its own app, clients and captured log lines. `boot({ db: true })` adds a migrated
+ * PGlite and `store` (the analytics tests); without it the app runs with `store: null`, as a network-only front.
*
*
* const t = await boot();
* const t = await boot();
* t.get('/api/v1/stats?days=30') → { status, headers, text, json() }
* t.get('/api/v1/stats?days=30') → { status, headers, text, json() }
+ * t.get('/internal/events', { method: 'POST', headers, body }) → a raw-body request
* t.logs()
* t.logs()
*/
*/
const http = require('http');
const http = require('http');
const configLib = require('../../server/config');
const configLib = require('../../server/config');
const { createApp } = require('../../server/app');
const { createApp } = require('../../server/app');
+const { createStore } = require('../../server/db');
+const { testDb } = require('./db');
const NETWORK = 'http://127.0.0.1:4000';
const NETWORK = 'http://127.0.0.1:4000';
const LIVE = 'http://127.0.0.1:3000';
const LIVE = 'http://127.0.0.1:3000';
@@ -152,11 +156,18 @@ async function boot(opts = {}) {
@@ -152,11 +156,18 @@ async function boot(opts = {}) {
...(opts.env || {}),
...(opts.env || {}),
};
};
const config = configLib.load(env);
const config = configLib.load(env);
+ // Most tests never touch the database; `boot({ db: true })` gives them a migrated PGlite of their own.
+ const testdatabase = (opts.db === true || opts.store) ? await testDb({ store: opts.dbStore }) : null;
+ const store = opts.store || (testdatabase ? createStore(testdatabase.db, { now: opts.now }) : null);
const built = await createApp({
const built = await createApp({
config,
config,
+ store,
fetchImpl,
fetchImpl,
log,
log,
now: opts.now,
now: opts.now,
+ billing: opts.billing,
+ analytics: opts.analytics,
+ shadow: opts.shadow,
releaseEnv: { ...process.env, RELEASE_COMMIT: '0000000' },
releaseEnv: { ...process.env, RELEASE_COMMIT: '0000000' },
});
});
const server = await new Promise((resolve) => {
const server = await new Promise((resolve) => {
@@ -166,11 +177,11 @@ async function boot(opts = {}) {
@@ -166,11 +177,11 @@ async function boot(opts = {}) {
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
async function get(pathname, o = {}) {
async function get(pathname, o = {}) {
- const res = await fetch(base + pathname, {
- method: o.method || 'GET',
- headers: o.headers || {},
- redirect: 'manual',
- });
+ const headers = { ...(o.headers || {}) };
+ let body = o.body;
+ if (o.json !== undefined) { body = JSON.stringify(o.json); headers['content-type'] = 'application/json'; }
+ const method = o.method || (body ? 'POST' : 'GET');
+ const res = await fetch(base + pathname, { method, headers, body, redirect: 'manual' });
const text = await res.text();
const text = await res.text();
return {
return {
status: res.status,
status: res.status,
@@ -181,11 +192,12 @@ async function boot(opts = {}) {
@@ -181,11 +192,12 @@ async function boot(opts = {}) {
}
}
return {
return {
- base, config, ctx: built.ctx, fetchImpl, get,
+ base, config, ctx: built.ctx, fetchImpl, get, testdatabase,
logs: () => captured.join('\n'),
logs: () => captured.join('\n'),
async close() {
async close() {
built.ctx.metricsStop();
built.ctx.metricsStop();
await new Promise((resolve) => server.close(resolve));
await new Promise((resolve) => server.close(resolve));
+ if (testdatabase) await testdatabase.close().catch(() => {});
},
},
};
};
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,12 @@
@@ -0,0 +1,12 @@
+'use strict';
+
+/**
+ * A migrated Report database for one test run (ADR-035), from openvibe-sdk/testing: PGlite by default; with
+ * REPORT_TEST_STORE=pg the PostgreSQL + PgBouncer containers, with roles and a schema of this run's own.
+ */
+const { createTestDb, pgAvailable } = require('openvibe-sdk/testing');
+const { MIGRATIONS } = require('../../server/db');
+
+const testDb = ({ store = process.env.REPORT_TEST_STORE || 'pglite', max = 4 } = {}) => createTestDb({ migrations: MIGRATIONS, store, service: 'report', max });
+
+module.exports = { testDb, pgAvailable };
new file mode 100644
new file mode 100644
@@ -0,0 +1,68 @@
@@ -0,0 +1,68 @@
+'use strict';
+
+/**
+ * Ingest is idempotent: a repeated event id and a repeated stats snapshot change no row. The stream event writes
+ * its own facts and its channel/day counts; the frame beacon is a daily total, not a page view per request.
+ */
+const assert = require('assert');
+const { testDb } = require('./helpers/db');
+const { createStore } = require('../server/db');
+const { recordEvent, ingestStats } = require('../server/analytics/ingest');
+const { streams, stats, NOW, DAYS } = require('./fixtures/analytics');
+const { check, done } = require('./helpers/boot');
+
+(async () => {
+ const testdatabase = await testDb();
+ const store = createStore(testdatabase.db, { now: () => NOW });
+ try {
+ await check('a repeated live.stream.ended changes nothing: one inbox row, one stream, one stream count', async () => {
+ const a = await recordEvent(store, streams[0]);
+ const b = await recordEvent(store, streams[0]);
+ assert.strictEqual(a.duplicate, false);
+ assert.strictEqual(b.duplicate, true);
+ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_inbox WHERE consumer = $1', ['report-events']), 1);
+ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_events'), 1);
+ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 1);
+ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'streams' AND day = '2026-10-26'"), 1);
+ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'messages' AND day = '2026-10-26'"), 120);
+ });
+
+ await check('a second stream adds to the channel day but a redelivery with a new event id does not', async () => {
+ await recordEvent(store, streams[1]);
+ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'streams' AND day = '2026-10-27'"), 1);
+ const corrected = JSON.parse(JSON.stringify(streams[1]));
+ corrected.event_id = 'evt_TESTSTREAM102CORRECTED';
+ const out = await recordEvent(store, corrected);
+ assert.strictEqual(out.outcome, 'stream:updated');
+ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'channel' AND scope_id = 'goosely' AND metric = 'streams' AND day = '2026-10-27'"), 1, 'a corrected stream must not be counted twice');
+ assert.strictEqual(await store.db.value('SELECT COUNT(*)::int AS n FROM report_streams'), 2);
+ });
+
+ await check('a repeated stats snapshot is a no-op; the first one writes every day and window', async () => {
+ const a = await ingestStats(store, stats);
+ const b = await ingestStats(store, stats);
+ assert.strictEqual(a.duplicate, false);
+ assert.strictEqual(b.duplicate, true);
+ assert.ok(a.rows > 20, `expected many metric rows, got ${a.rows}`);
+ const dayRows = await store.db.value("SELECT COUNT(*)::int AS n FROM report_metric_days WHERE scope = 'network' AND metric IN ('pageviews', 'accounts_new', 'coins_earned')");
+ assert.strictEqual(dayRows, DAYS.length * 3);
+ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_windows WHERE scope = 'network' AND metric = 'pageviews' AND window_days = 7"), 300);
+ assert.strictEqual(await store.db.value("SELECT value::int AS n FROM report_metric_days WHERE scope = 'site' AND scope_id = 'network' AND metric = 'pageviews' AND day = '2026-10-28'"), 60);
+ assert.strictEqual(await store.db.value("SELECT SUM(hits)::int AS n FROM report_frame_days WHERE host = 'openvibe.network'"), 200);
+ });
+
+ await check('the frame beacon stays one daily total, while a single hit adds once', async () => {
+ const day = { event_id: 'evt_TESTFRAME1', event_type: 'network.frame.day', source: 'network', occurred_at: '2026-10-28T12:00:00.000Z', payload: { day: '2026-10-28', host: 'openvibe.tools', hits: 40 } };
+ await recordEvent(store, day);
+ await recordEvent(store, day);
+ assert.strictEqual(await store.db.value("SELECT hits::int AS n FROM report_frame_days WHERE day = '2026-10-28' AND host = 'openvibe.tools'"), 40);
+ const hit = { event_id: 'evt_TESTFRAME2', event_type: 'network.frame.hit', source: 'network', occurred_at: '2026-10-28T12:00:01.000Z', payload: { day: '2026-10-28', host: 'openvibe.tools' } };
+ await recordEvent(store, hit);
+ await recordEvent(store, hit);
+ assert.strictEqual(await store.db.value("SELECT hits::int AS n FROM report_frame_days WHERE day = '2026-10-28' AND host = 'openvibe.tools'"), 41);
+ });
+ } finally {
+ await testdatabase.close();
+ }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,119 @@
@@ -0,0 +1,119 @@
+'use strict';
+
+/**
+ * Every metric the Report pages and openvibe.network/stats show, computed from a fixture and compared with the
+ * fixture's own expected numbers: the network totals, the day series, every site, every channel by day, and the
+ * Billing revenue read (stubbed) that Report quotes rather than calculates.
+ */
+const assert = require('assert');
+const { testDb } = require('./helpers/db');
+const { createStore } = require('../server/db');
+const { recordEvent, ingestStats } = require('../server/analytics/ingest');
+const { reportStats, channelStats, metrics } = require('../server/analytics/query');
+const { streams, stats, expected, expectedChannel, NOW, SUBJECT, billingTransactions } = require('./fixtures/analytics');
+const { check, done } = require('./helpers/boot');
+
+function billingStub() {
+ const calls = [];
+ return {
+ calls,
+ async revenue({ subject, days }) {
+ calls.push({ subject, days });
+ let settled = 0; let refunded = 0; let transactions = 0;
+ for (const txn of billingTransactions.transactions) {
+ if (txn.test) continue;
+ let seen = false;
+ for (const e of txn.entries) {
+ if (e.account.kind !== 'creator_payable') continue;
+ seen = true;
+ if (e.amount > 0) settled += e.amount; else refunded += -e.amount;
+ }
+ if (seen) transactions++;
+ }
+ return { source: 'billing', capability: 'billing.balance.read', subject, days, currency: 'vibes-bits', settled_bits: settled, refunded_bits: refunded, net_bits: settled - refunded, transactions };
+ },
+ };
+}
+
+(async () => {
+ const testdatabase = await testDb();
+ const store = createStore(testdatabase.db, { now: () => NOW });
+ const billing = billingStub();
+ try {
+ await ingestStats(store, stats);
+ for (const event of streams) await recordEvent(store, event);
+
+ await check('network stats: totals, day series, accounts, coins, live and busiest pages match the fixture', async () => {
+ const got = await reportStats(store, 7, { now: () => NOW });
+ assert.ok(got, 'no stats computed');
+ assert.strictEqual(got.generated_at, expected.generated_at);
+ assert.strictEqual(got.window_days, 7);
+ assert.deepStrictEqual(got.totals, expected.totals);
+ assert.deepStrictEqual(got.pageviews_daily, expected.pageviews_daily);
+ assert.deepStrictEqual(got.accounts, expected.accounts);
+ assert.deepStrictEqual(got.coins, expected.coins);
+ assert.deepStrictEqual(got.live, expected.live);
+ assert.deepStrictEqual(got.busiest_pages_24h, expected.busiest_pages_24h);
+ });
+
+ await check('every site: page views, change, daily series, visitors and 24-hour traffic', async () => {
+ const got = await reportStats(store, 7, { now: () => NOW });
+ assert.strictEqual(got.sites.length, 2);
+ const network = got.sites.find((s) => s.id === 'network');
+ const live = got.sites.find((s) => s.id === 'live');
+ assert.strictEqual(network.pageviews, 200);
+ assert.strictEqual(network.pageviews_prev, 150);
+ assert.strictEqual(network.change_pct, 33);
+ assert.strictEqual(network.unique_visitors, 20);
+ assert.deepStrictEqual(network.traffic_24h, { requests: 800, server_errors: 2, p95_ms: 120 });
+ assert.strictEqual(network.daily.length, 7);
+ assert.deepStrictEqual(network.daily.map((d) => d.n), [20, 20, 20, 20, 20, 40, 60]);
+ assert.strictEqual(live.pageviews, 100);
+ assert.strictEqual(live.pageviews_prev, 50);
+ assert.strictEqual(live.unique_visitors, 30);
+ assert.strictEqual(live.signed_in_people, 10);
+ assert.strictEqual(live.daily, null, 'Live counts its own visits: no navbar day series');
+ assert.strictEqual(live.traffic_24h, null);
+ });
+
+ await check('a channel by day: streams, seconds, peak, chatters, messages, watch minutes and weighted average', async () => {
+ const got = await channelStats(store, 'goosely', 7, { now: () => NOW, billing });
+ assert.ok(got, 'no channel analytics');
+ assert.strictEqual(got.creator, SUBJECT);
+ assert.strictEqual(got.channel, 'goosely');
+ assert.deepStrictEqual(got.totals, expectedChannel.totals);
+ assert.deepStrictEqual(got.daily, expectedChannel.daily);
+ assert.deepStrictEqual(got.streams.map((s) => s.stream_id), [102, 101]);
+ });
+
+ await check('Billing revenue is quoted with its capability and never derived from an event', async () => {
+ const before = billing.calls.length;
+ const got = await channelStats(store, 'goosely', 7, { now: () => NOW, billing });
+ assert.deepStrictEqual(got.revenue, {
+ source: 'billing', capability: 'billing.balance.read', subject: SUBJECT, days: 7, currency: 'vibes-bits',
+ settled_bits: 500, refunded_bits: 100, net_bits: 400, transactions: 2,
+ });
+ assert.strictEqual(billing.calls.length, before + 1);
+ });
+
+ await check('the flat report.metric@1 view exposes the same numbers in windows and days', async () => {
+ const got = await metrics(store, { scope: 'network', scopeId: 'network', days: 7, now: () => NOW });
+ assert.strictEqual(got.window_days, 7);
+ const pageviews = got.windows.find((m) => m.metric === 'pageviews');
+ assert.strictEqual(pageviews.value, 300);
+ assert.strictEqual(pageviews.unit, 'count');
+ const daily = got.metrics.filter((m) => m.metric === 'pageviews');
+ assert.strictEqual(daily.length, 7);
+ assert.strictEqual(daily[daily.length - 1].value, 60);
+ const channel = await metrics(store, { scope: 'channel', scopeId: 'goosely', days: 7, now: () => NOW });
+ assert.strictEqual(channel.metrics.find((m) => m.metric === 'streams' && m.day === '2026-10-26').value, 1);
+ });
+
+ await check('a window with no numbers is null, not zeroes', async () => {
+ assert.strictEqual(await reportStats(store, 90, { now: () => Date.parse('2025-01-01T00:00:00Z') }), null);
+ });
+ } finally {
+ await testdatabase.close();
+ }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,58 @@
@@ -0,0 +1,58 @@
+'use strict';
+
+/**
+ * The shadow job: the same window from Network and from Report's tables. Equal numbers make one clean run with no
+ * report_shadow_diffs rows; a changed number records exactly that difference, and GET /internal/shadow (loopback)
+ * summarizes the latest run.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const { stats, NOW } = require('./fixtures/analytics');
+
+(async () => {
+ const t = await boot({ db: true, now: () => NOW });
+ try {
+ await t.ctx.analytics.ingestStats(stats);
+
+ await check('no difference when Network and Report answer the same numbers', async () => {
+ const run = await t.ctx.shadow.compare({ days: 7, stats });
+ assert.strictEqual(run.status, 'equal');
+ assert.strictEqual(run.differences, 0);
+ assert.ok(run.checked > 20, `checked ${run.checked}`);
+ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_shadow_diffs'), 0);
+ assert.strictEqual(await t.ctx.store.db.value('SELECT COUNT(*)::int AS n FROM report_shadow_runs'), 1);
+ });
+
+ await check('a changed number is recorded once and the internal summary shows it', async () => {
+ const changed = JSON.parse(JSON.stringify(stats));
+ changed.totals.pageviews = 301;
+ const run = await t.ctx.shadow.compare({ days: 7, stats: changed });
+ assert.strictEqual(run.status, 'differences');
+ assert.strictEqual(run.differences, 1);
+ assert.strictEqual(run.diffs[0].metric, 'pageviews');
+ assert.strictEqual(run.diffs[0].scope, 'network');
+ assert.strictEqual(run.diffs[0].network_value, 301);
+ assert.strictEqual(run.diffs[0].report_value, 300);
+ assert.strictEqual(run.diffs[0].delta, -1);
+ const r = await t.get('/internal/shadow?days=7');
+ assert.strictEqual(r.status, 200);
+ const body = r.json();
+ assert.strictEqual(body.latest.status, 'differences');
+ assert.strictEqual(body.latest.differences, 1);
+ assert.strictEqual(body.diffs.length, 1);
+ assert.strictEqual(body.diffs[0].metric, 'pageviews');
+ });
+
+ await check('a Network that does not answer is a run with no differences, never a clean claim', async () => {
+ const down = await t.ctx.shadow.compare({ days: 7, stats: null });
+ assert.strictEqual(down.status, 'network_unavailable');
+ assert.strictEqual(down.differences, 0);
+ const r = await t.get('/internal/shadow');
+ assert.strictEqual(r.status, 200);
+ assert.strictEqual(r.json().latest.status, 'network_unavailable');
+ });
+ } finally {
+ await t.close();
+ }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,54 @@
@@ -0,0 +1,54 @@
+'use strict';
+
+/**
+ * REPORT_SOURCE switches the answer: `network` (default) keeps today's Network numbers; `report` answers the same
+ * route and the same page from Report's tables. A report source with no data is a 404, not a made-up zero.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const { stats, expected, NOW } = require('./fixtures/analytics');
+
+(async () => {
+ const network = await boot({ now: () => NOW });
+ try {
+ await check('the default source is network: GET /api/v1/stats and /stats stay on Network', async () => {
+ assert.strictEqual(network.config.source, 'network');
+ const api = await network.get('/api/v1/stats?days=7');
+ assert.strictEqual(api.status, 200);
+ assert.strictEqual(api.json().totals.pageviews, 77408);
+ const page = await network.get('/stats?days=7');
+ assert.strictEqual(page.status, 200);
+ assert.ok(page.text.includes('77,408'));
+ });
+ } finally { await network.close(); }
+
+ const report = await boot({ db: true, now: () => NOW, env: { REPORT_SOURCE: 'report' } });
+ try {
+ await report.ctx.analytics.ingestStats(stats);
+ await check('REPORT_SOURCE=report answers the stats API from Report', async () => {
+ assert.strictEqual(report.config.source, 'report');
+ const r = await report.get('/api/v1/stats?days=7');
+ assert.strictEqual(r.status, 200);
+ assert.deepStrictEqual(r.json().totals, expected.totals);
+ assert.strictEqual(r.json().totals.pageviews, 300);
+ assert.notStrictEqual(r.json().totals.pageviews, 77408);
+ });
+ await check('REPORT_SOURCE=report answers the /stats page from Report', async () => {
+ const r = await report.get('/stats?days=7');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.text.includes('300'), 'the report page views are on the page');
+ assert.ok(!r.text.includes('77,408'), 'the Network stub must not be read when the source is report');
+ });
+ } finally { await report.close(); }
+
+ const empty = await boot({ db: true, now: () => NOW, env: { REPORT_SOURCE: 'report' } });
+ try {
+ await check('a report source with no rows is 404, not zeroes', async () => {
+ const r = await empty.get('/api/v1/stats?days=7');
+ assert.strictEqual(r.status, 404);
+ assert.match(r.headers.get('content-type') || '', /^application\/problem\+json/);
+ });
+ } finally { await empty.close(); }
+
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });