openvibers/openvibe.live Commits
Add gated PostgreSQL page analytics slice
Changes
@@@ -37,6 +37,10 @@ JWT_SECRET=change-me-to-a-random-secret-at-least-32-chars# DatabaseDB_PATH=./data/live.db++# Page analytics only. Defaults to SQLite until its PostgreSQL schema is migrated.++# ANALYTICS_DRIVER=postgres++# DATABASE_URL=postgres://live_runtime:password@localhost:5432/ov_live++# DATABASE_DIRECT_URL=postgres://live_owner:password@localhost:5432/ov_live# Every file Live writes lives under one data directory (default ./data, relative to the working# directory). The *_PATH variables below still move single locations (server/paths.js).# DATA_DIR=./datannew file mode 100644@@@ -0,0 +1,21 @@++# Live PostgreSQL migration: slice 1++++Status: implementation slice, 2026-09-30. The default remains SQLite.++++## Decision++++Move page analytics first, behind `ANALYTICS_DRIVER=postgres`. It is Live's second SQLite database and already has a PostgreSQL implementation in the pinned `openvibe-shared` release. Its request-facing `record()` stays synchronous while the PostgreSQL tracker buffers writes; only the three analytics read routes and the retention job need to await database work. This makes the slice deployable independently of the 2,000-plus synchronous calls to Live's primary database.++++`ANALYTICS_DRIVER` defaults to `sqlite`. With `postgres`, `DATABASE_URL` connects the runtime tracker to PostgreSQL. Run `DATABASE_DIRECT_URL=... npm run migrate:analytics-pg` with the schema owner before enabling it. The migration is versioned and checked by the SDK. Startup checks that `analytics_events` exists; the runtime role never creates tables. `LIVE_DRILL` uses its isolated SQLite analytics file regardless of the driver setting and starts no analytics timers or outbound calls. Existing SQLite analytics data is not imported by this slice; preserve the file and import its history before a production analytics switch if historical dashboards must be continuous.++++## Target architecture++++The primary Live database will use `openvibe-sdk/db` with `DATABASE_URL`, versioned PostgreSQL migrations on `DATABASE_DIRECT_URL`, and async methods retaining the current exported names. Generate its schema from a booted SQLite database, omit the frozen tables, and move all boot-time DDL into migrations. PostgreSQL analytics then shares that database and migration ledger. Live's outbox and inbox use SDK PostgreSQL implementations; page analytics uses `openvibe-shared/analytics/pg`. The cutover imports and verifies data through the Host switch procedure. The SQLite file remains the rollback copy for the agreed window.++++## Next slices++++1. Land the remaining Chat table retirement and settle money ownership before generating the primary schema, so deleted tables are never imported.++2. Add the primary database's async driver and generated schema in a separately tested slice; convert one bounded caller group at a time, including its transactions and route handlers. Keep SQLite as the default until every caller is async.++3. Convert the outbox/inbox, analytics history, drill guard, readiness, remaining callers, and tests; rehearse the import with row counts and checksums before the Host cutover.++++The source survey is `openvibe/agents/jobs/brief-t4-live-pg.out.md`. The PostgreSQL analytics test runs against `OV_TEST_PG_DIRECT_URL`; it skips when the disposable service is unavailable.nnew file mode 100644@@@ -0,0 +1,57 @@++-- phase: expand++-- Page analytics schema from openvibe-shared v1.30.1 analytics/pg.js.++-- Run with DATABASE_DIRECT_URL before ANALYTICS_DRIVER=postgres is enabled.++++CREATE TABLE IF NOT EXISTS analytics_events (++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,++ service text NOT NULL,++ event_type text NOT NULL DEFAULT 'pageview',++ path text,++ method text DEFAULT 'GET',++ status_code integer,++ response_time_ms integer,++ user_id bigint,++ session_id text,++ ip text,++ country text,++ city text,++ user_agent text,++ referer text,++ is_bot integer DEFAULT 0,++ bot_type text,++ device_type text,++ browser text,++ os text,++ authenticated integer NOT NULL DEFAULT 0,++ created_at text NOT NULL++);++CREATE TABLE IF NOT EXISTS analytics_hourly (++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,++ service text NOT NULL, hour text NOT NULL,++ pageviews bigint DEFAULT 0, api_calls bigint DEFAULT 0, unique_visitors bigint DEFAULT 0, unique_users bigint DEFAULT 0,++ bot_hits bigint DEFAULT 0, avg_response_ms integer DEFAULT 0, error_count bigint DEFAULT 0, bandwidth_bytes bigint DEFAULT 0,++ top_paths text, top_referers text,++ UNIQUE (service, hour)++);++CREATE TABLE IF NOT EXISTS analytics_daily (++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,++ service text NOT NULL, date text NOT NULL,++ pageviews bigint DEFAULT 0, api_calls bigint DEFAULT 0, unique_visitors bigint DEFAULT 0, unique_users bigint DEFAULT 0,++ new_users bigint DEFAULT 0, bot_hits bigint DEFAULT 0, avg_response_ms integer DEFAULT 0, error_count bigint DEFAULT 0,++ top_paths text, top_referers text, top_countries text, device_breakdown text, browser_breakdown text,++ UNIQUE (service, date)++);++CREATE TABLE IF NOT EXISTS analytics_visitor_days (++ service text NOT NULL, day text NOT NULL, hour text NOT NULL, vhash text NOT NULL,++ authenticated integer NOT NULL DEFAULT 0,++ PRIMARY KEY (service, hour, vhash)++);++CREATE TABLE IF NOT EXISTS analytics_day_salts (++ service text NOT NULL, day text NOT NULL, salt text NOT NULL,++ PRIMARY KEY (service, day)++);++CREATE INDEX IF NOT EXISTS idx_analytics_events_created ON analytics_events (created_at);++CREATE INDEX IF NOT EXISTS idx_analytics_events_service ON analytics_events (service, created_at);++CREATE INDEX IF NOT EXISTS idx_analytics_events_path ON analytics_events (service, path, created_at);++CREATE INDEX IF NOT EXISTS idx_analytics_events_bot ON analytics_events (is_bot, created_at);++CREATE INDEX IF NOT EXISTS idx_analytics_visitor_days_day ON analytics_visitor_days (service, day);@@@ -28,6 +28,7 @@"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.16.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.30.1",++ "pg": "^8.23.1","sanitize-html": "^2.17.7","sdp-transform": "^3.0.0","sharp": "^0.33.0",@@@ -2894,6 +2895,95 @@"integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==","license": "MIT"},++ "node_modules/pg": {++ "version": "8.23.1",++ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.1.tgz",++ "integrity": "sha512-aL96AHANtWjPLDOLqnhx+ngp9+UK7ETEU8VJrDCGvsSSi/mGLcWYsS6Herg7lmaBJe4uwrfqsa7gTEFaSizDoQ==",++ "license": "MIT",++ "dependencies": {++ "pg-connection-string": "^2.14.1",++ "pg-pool": "^3.14.0",++ "pg-protocol": "^1.16.1",++ "pg-types": "2.2.0",++ "pgpass": "1.0.5"++ },++ "engines": {++ "node": ">= 16.0.0"++ },++ "optionalDependencies": {++ "pg-cloudflare": "^1.4.1"++ },++ "peerDependencies": {++ "pg-native": ">=3.0.1"++ },++ "peerDependenciesMeta": {++ "pg-native": {++ "optional": true++ }++ }++ },++ "node_modules/pg-cloudflare": {++ "version": "1.4.1",++ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.1.tgz",++ "integrity": "sha512-6PQbsFWZcp9EmJEwy5cGQ2La+AMWpP46lgbb8X+U/XsHIUweYDNCpeuKck5RxL2MdVFi7krbbEi5nX4Zh7JhrQ==",++ "license": "MIT",++ "optional": true++ },++ "node_modules/pg-connection-string": {++ "version": "2.14.1",++ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.1.tgz",++ "integrity": "sha512-qR3kGNPBLpCNtz0evbKA0Y/MRFXwSSdT+pTJvYp/bXTcReZbvX1kzF0IyTc1QnxqF7AZbOeBhNL8R5mYQZV/MA==",++ "license": "MIT"++ },++ "node_modules/pg-int8": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",++ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",++ "license": "ISC",++ "engines": {++ "node": ">=4.0.0"++ }++ },++ "node_modules/pg-pool": {++ "version": "3.14.0",++ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",++ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",++ "license": "MIT",++ "peerDependencies": {++ "pg": ">=8.0"++ }++ },++ "node_modules/pg-protocol": {++ "version": "1.16.1",++ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.1.tgz",++ "integrity": "sha512-p9VOFMiHB/ZbJATetbg+99PxssTVSQRnyuPSQ67mN1+1KBOjZaZ83ZQzltnxPhJwSsC3nwVjJ10DVJlerbFzLg==",++ "license": "MIT"++ },++ "node_modules/pg-types": {++ "version": "2.2.0",++ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",++ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",++ "license": "MIT",++ "dependencies": {++ "pg-int8": "1.0.1",++ "postgres-array": "~2.0.0",++ "postgres-bytea": "~1.0.0",++ "postgres-date": "~1.0.4",++ "postgres-interval": "^1.1.0"++ },++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/pgpass": {++ "version": "1.0.5",++ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",++ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",++ "license": "MIT",++ "dependencies": {++ "split2": "^4.1.0"++ }++ },"node_modules/picocolors": {"version": "1.1.1","resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",@@@ -2928,6 +3018,45 @@"node": "^10 || ^12 || >=14"}},++ "node_modules/postgres-array": {++ "version": "2.0.0",++ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",++ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",++ "license": "MIT",++ "engines": {++ "node": ">=4"++ }++ },++ "node_modules/postgres-bytea": {++ "version": "1.0.1",++ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",++ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-date": {++ "version": "1.0.7",++ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",++ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",++ "license": "MIT",++ "engines": {++ "node": ">=0.10.0"++ }++ },++ "node_modules/postgres-interval": {++ "version": "1.2.0",++ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",++ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",++ "license": "MIT",++ "dependencies": {++ "xtend": "^4.0.0"++ },++ "engines": {++ "node": ">=0.10.0"++ }++ },"node_modules/prebuild-install": {"version": "7.1.3","resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",@@@ -3407,6 +3536,15 @@"node": ">=0.10.0"}},++ "node_modules/split2": {++ "version": "4.2.0",++ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",++ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",++ "license": "ISC",++ "engines": {++ "node": ">= 10.x"++ }++ },"node_modules/statuses": {"version": "2.0.2","resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",@@@ -7,11 +7,13 @@"start": "node server/index.js","dev": "NODE_ENV=development node server/index.js","init-db": "node server/db/init.js",++ "migrate:analytics-pg": "node scripts/analytics-pg-migrate.js","seed": "node server/db/seed.js","test": "node test/run.js","test:unit": "node test/run.js","test:powerchat": "for f in test/powerchat-*.test.js; do node \"$f\" || exit 1; done","test:browser": "node test/browser/smoke.js",++ "test:analytics-pg": "node test/analytics-pg.test.js","perf:page": "node scripts/perf/measure-page.js","perf:budget": "node scripts/perf/check-budgets.js","n-1:record": "node scripts/n-1-record.js"@@@ -47,6 +49,7 @@"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.16.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.30.1",++ "pg": "^8.23.1","sanitize-html": "^2.17.7","sdp-transform": "^3.0.0","sharp": "^0.33.0",nnew file mode 100644@@@ -0,0 +1,21 @@++#!/usr/bin/env node++'use strict';++++const path = require('path');++require('dotenv').config({ path: path.join(__dirname, '..', '.env') });++++async function main() {++ const url = process.env.DATABASE_DIRECT_URL;++ if (!url) throw new Error('DATABASE_DIRECT_URL is required for the analytics PostgreSQL migration');++ const { createDb } = require('openvibe-sdk/db');++ const db = createDb({ url, service: 'live-analytics-migrate' });++ try {++ const result = await db.migrate({ dir: path.join(__dirname, '..', 'migrations') });++ console.log(`[Analytics] migrations applied: ${result.applied.length}; held: ${result.held.length}`);++ } finally {++ await db.close();++ }++}++++if (require.main === module) main().catch((err) => { console.error(err); process.exitCode = 1; });++module.exports = { main };nnew file mode 100644@@@ -0,0 +1,50 @@++'use strict';++++const fs = require('fs');++const path = require('path');++const analyticsModule = require('openvibe-shared/analytics');++const paths = require('../paths');++++function createAnalyticsStore({ driver = process.env.ANALYTICS_DRIVER || 'sqlite',++ databaseUrl = process.env.DATABASE_URL, drill = false, timers = !drill } = {}) {++ if (driver !== 'sqlite' && driver !== 'postgres') {++ throw new Error(`ANALYTICS_DRIVER must be sqlite or postgres (got ${driver})`);++ }++++ // Restore drills always use their isolated SQLite analytics file and never contact PostgreSQL.++ if (driver === 'postgres' && !drill) {++ if (!databaseUrl) throw new Error('DATABASE_URL is required when ANALYTICS_DRIVER=postgres');++ const { createDb } = require('openvibe-sdk/db');++ const { AnalyticsTrackerPg, pruneRawEventsPg } = require('openvibe-shared/analytics/pg');++ const db = createDb({ url: databaseUrl, service: 'live-analytics' });++ const tracker = new AnalyticsTrackerPg(db, 'live', { retention: false, timers });++ let closing;++ return {++ tracker,++ async ready() {++ // The owner runs migrations before enabling this driver; runtime has no DDL rights.++ await db.value('SELECT 1 FROM analytics_events LIMIT 1');++ },++ prune: () => pruneRawEventsPg(db, { days: analyticsModule.retention.MAX_DAYS }),++ close() {++ if (!closing) closing = Promise.resolve(tracker.destroy()).finally(() => db.close());++ return closing;++ },++ };++ }++++ const file = paths.analyticsDbPath();++ fs.mkdirSync(path.dirname(file), { recursive: true });++ const BetterSqlite3 = require('better-sqlite3');++ const db = new BetterSqlite3(file);++ db.pragma('journal_mode = WAL');++ const tracker = new analyticsModule.AnalyticsTracker(db, 'live', { retention: false, timers });++ return {++ tracker,++ ready: async () => {},++ prune: () => analyticsModule.retention.pruneRawEvents(db, { days: analyticsModule.retention.MAX_DAYS }),++ close: async () => { tracker.destroy(); db.close(); },++ };++}++++module.exports = { createAnalyticsStore };@@@ -137,7 +137,7 @@ const vibeCodingRoutes = require('./vibe-coding/routes');// Restreamconst restreamRoutes = require('./streaming/restream-routes');const restreamManager = require('./streaming/restream-manager');--const analyticsModule = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, 30-day raw retention; Sec-GPC/DNT not recorded++// ADR-021 analytics: no IP/user id, route templates, 30-day raw retention; Sec-GPC/DNT not recorded.// WHIP (WebRTC-HTTP Ingestion Protocol)const whipHandler = require('./streaming/whip-handler');@@@ -395,14 +395,8 @@ const uploadLimiter = rateLimit({message: { error: 'Too many upload requests, please slow down' },});// ── Analytics Tracking ────────────────────────────────────────--const BetterSqlite3 = require('better-sqlite3');--// <data dir>/analytics.db (ANALYTICS_DB_PATH overrides it outside a drill; server/paths.js).--const analyticsDbPath = paths.analyticsDbPath();--fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });--const analyticsDb = new BetterSqlite3(analyticsDbPath);--analyticsDb.pragma('journal_mode = WAL');--// A drill records no page views and runs no flush/aggregate timers (its analytics.db is its own, empty).--const analytics = new analyticsModule.AnalyticsTracker(analyticsDb, 'live', { retention: false, timers: !drill.enabled }); // prune: job 8b2 below++const analyticsStore = require('./analytics/store').createAnalyticsStore({ drill: drill.enabled });++const analytics = analyticsStore.tracker;app.locals.analytics = analytics;if (!drill.enabled) app.use(analytics.middleware());@@@ -606,10 +600,10 @@ app.post('/internal/openre-events', require('./openre/mirror').webhookHandler);// Internal (server-to-server) routes — allow openvibe.network to call into this service// Summary numbers for the Network's navigation service (ordering sites by real use).// A service token with live.analytics.read (loopback only); returns totals, never rows.--app.get('/internal/analytics-summary', require('./net/service-guard').guard('live.analytics.read'), (req, res) => {++app.get('/internal/analytics-summary', require('./net/service-guard').guard('live.analytics.read'), async (req, res) => {try {const days = Math.min(parseInt(req.query.days, 10) || 7, 90);-- const st = analytics.getStats({ days }) || {};++ const st = await analytics.getStats({ days }) || {};res.json({ ok: true, summary: st.summary || {} });} catch (err) { res.status(500).json({ ok: false, error: err.message }); }});@@@ -736,19 +730,19 @@ app.use('/api/news', newsRoutes);// ── Internal Analytics API ───────────────────────────────────// Called by openvibe-tools admin panel to fetch this service's analytics--app.get('/api/admin/analytics', requireAuth, permissions.requireAdmin, (req, res) => {++app.get('/api/admin/analytics', requireAuth, permissions.requireAdmin, async (req, res) => {try {const days = Math.min(parseInt(req.query.days) || 30, 365);const hours = req.query.hours ? Math.min(parseInt(req.query.hours), 8760) : null;-- res.json({ ok: true, analytics: analytics.getStats({ days, hours }) });++ res.json({ ok: true, analytics: await analytics.getStats({ days, hours }) });} catch (err) {res.status(500).json({ ok: false, error: err.message });}});--app.get('/api/admin/analytics/bots', requireAuth, permissions.requireAdmin, (req, res) => {++app.get('/api/admin/analytics/bots', requireAuth, permissions.requireAdmin, async (req, res) => {try {const days = Math.min(parseInt(req.query.days) || 30, 365);-- res.json({ ok: true, bots: analytics.getBotAnalysis(days) });++ res.json({ ok: true, bots: await analytics.getBotAnalysis(days) });} catch (err) {res.status(500).json({ ok: false, error: err.message });}@@@ -1125,6 +1119,7 @@ async function start() {// 1. Initialize databasedb.initDb();++ await analyticsStore.ready();// Initialize cosmetics tablescosmeticsModule.ensureTables();// Chat tag tables (read-only tags, server/chat/tags.js)@@@ -1550,8 +1545,9 @@ async function start() {// 8b2. Raw analytics retention (ADR-021): events older than 30 days go, in bounded batches;// hourly/daily rollups stay. Nightly, first run a few minutes after boot.require('./utils/jobs').every('analytics-prune', 24 * 60 * 60 * 1000, async () => {-- const out = await analyticsModule.retention.pruneRawEvents(analyticsDb, { days: analyticsModule.retention.MAX_DAYS });-- if (out.deleted) console.log(`[Analytics] pruned ${out.deleted} raw events older than ${out.cutoff}`);++ const out = await analyticsStore.prune();++ const deleted = out.deleted || out.removed;++ if (deleted) console.log(`[Analytics] pruned ${deleted} raw events older than ${out.cutoff}`);}, { initialDelayMs: 5 * 60 * 1000, jitterMs: 60 * 1000 });// 8b3. Media requests whose OpenCoins charge never got an answer: charged again with the same@@@ -1614,7 +1610,7 @@ function shutdown() {// Nothing was started but the HTTP server and the two databases.console.log('[Drill] Shutting down');_bootComplete = false;-- try { analytics.destroy(); analyticsDb.close(); } catch { /* */ }++ analyticsStore.close().catch(() => {});server.close(() => { try { db.close(); } catch { /* */ } process.exit(0); });try { server.closeAllConnections(); } catch { /* */ }setTimeout(() => process.exit(0), 3000).unref();@@@ -1653,7 +1649,7 @@ function shutdown() {try { require('./streaming/live-events').closeAll(); } catch { /* */ }// Small delay to let the message reach clients before closing sockets-- setTimeout(() => {++ setTimeout(async () => {restreamManager.stopViewerCountPolling();restreamManager.stopAll();try { recorder.stopAll(); } catch {}@@@ -1664,8 +1660,7 @@ function shutdown() {jsmpegRelay.closeAll();webrtcSFU.closeAll();rtmpServer.stop();-- analytics.destroy();-- analyticsDb.close();++ try { await analyticsStore.close(); } catch (err) { console.error('[Analytics] Shutdown:', err); }db.close();server.close(() => {nnew file mode 100644@@@ -0,0 +1,86 @@++'use strict';++++const assert = require('assert');++const crypto = require('crypto');++const path = require('path');++++async function main() {++ const directUrl = process.env.OV_TEST_PG_DIRECT_URL;++ if (!directUrl) {++ console.log('analytics PostgreSQL: skipped (OV_TEST_PG_DIRECT_URL not set)');++ return;++ }++++ const { createDb } = require('openvibe-sdk/db');++ const admin = createDb({ url: directUrl, service: 'live-analytics-test-admin' });++ const healthy = await admin.ready();++ if (!healthy.ok) {++ await admin.close();++ console.log(`analytics PostgreSQL: skipped (test service unavailable: ${healthy.error})`);++ return;++ }++++ const name = `ov_live_analytics_${process.pid}_${crypto.randomBytes(4).toString('hex')}`;++ const url = new URL(directUrl);++ url.pathname = `/${name}`;++ let created = false;++ let owner;++ let store;++ try {++ await admin.query(`CREATE DATABASE ${name}`);++ created = true;++ owner = createDb({ url: url.toString(), service: 'live-analytics-test-owner' });++ const migrations = path.join(__dirname, '..', 'migrations');++ assert.strictEqual((await owner.migrate({ dir: migrations })).applied.length, 1);++ assert.strictEqual((await owner.migrate({ dir: migrations })).applied.length, 0);++++ const { createAnalyticsStore } = require('../server/analytics/store');++ store = createAnalyticsStore({ driver: 'postgres', databaseUrl: url.toString(), timers: false });++ await store.ready();++++ const express = require('express');++ const app = express();++ app.use(store.tracker.middleware());++ app.get('/watch/:id', (req, res) => res.send('ok'));++ const server = app.listen(0, '127.0.0.1');++ try {++ await new Promise((resolve) => server.once('listening', resolve));++ const response = await fetch(`http://127.0.0.1:${server.address().port}/watch/123?token=secret`, {++ headers: {++ 'user-agent': 'Mozilla/5.0 Chrome/126.0.0.0 Safari/537.36',++ referer: 'https://example.com/private/path?secret=1',++ 'x-forwarded-for': '203.0.113.77',++ },++ });++ assert.strictEqual(response.status, 200);++ await response.text();++ } finally {++ await new Promise((resolve) => server.close(resolve));++ }++ await store.tracker.flush();++ const rows = await owner.many('SELECT path, ip, user_id, city, user_agent, referer FROM analytics_events');++ assert.strictEqual(rows.length, 1);++ assert.strictEqual(rows[0].path, '/watch/:id');++ assert.strictEqual(rows[0].ip, null);++ assert.strictEqual(rows[0].user_id, null);++ assert.strictEqual(rows[0].city, null);++ assert.strictEqual(rows[0].referer, 'https://example.com');++ assert.ok(!JSON.stringify(rows[0]).includes('secret'));++ const stats = await store.tracker.getStats({ hours: 1 });++ assert.strictEqual(Number(stats.summary.total_pageviews), 1);++ assert.ok(Array.isArray((await store.tracker.getBotAnalysis(1)).botTrend));++++ await owner.query(`INSERT INTO analytics_events (service, path, created_at)++ VALUES ('live', '/old', '2020-01-01 00:00:00')`);++ assert.strictEqual((await store.prune()).removed, 1);++ assert.strictEqual(Number(await owner.value('SELECT COUNT(*) FROM analytics_events')), 1);++ console.log('analytics PostgreSQL: migration, record, privacy, reads, and prune passed');++ } finally {++ if (store) await store.close();++ if (owner) await owner.close();++ if (created) await admin.query(`DROP DATABASE ${name} WITH (FORCE)`);++ await admin.close();++ }++}++++main().catch((err) => { console.error(err); process.exitCode = 1; });nnew file mode 100644@@@ -0,0 +1,34 @@++'use strict';++++const assert = require('assert');++const fs = require('fs');++const os = require('os');++const path = require('path');++++async function main() {++ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-live-analytics-store-'));++ process.env.DATA_DIR = tmp;++ delete process.env.ANALYTICS_DB_PATH;++ const { createAnalyticsStore } = require('../server/analytics/store');++ const sqlite = createAnalyticsStore({ timers: false });++ try {++ await sqlite.ready();++ assert.ok(fs.existsSync(path.join(tmp, 'analytics.db')));++ assert.strictEqual(typeof sqlite.tracker.getStats({ days: 1 }), 'object');++ assert.ok('deleted' in await sqlite.prune());++ } finally {++ await sqlite.close();++ }++++ // Production's PostgreSQL setting must not open an outbound connection during a restore drill.++ const drill = createAnalyticsStore({ driver: 'postgres', drill: true, databaseUrl: 'postgres://unreachable', timers: false });++ try {++ await drill.ready();++ assert.ok(fs.existsSync(path.join(tmp, 'analytics.db')));++ } finally {++ await drill.close();++ }++ console.log('analytics store: SQLite default and isolated drill passed');++}++++main().catch((err) => { console.error(err); process.exitCode = 1; });
@@ -37,6 +37,10 @@ JWT_SECRET=change-me-to-a-random-secret-at-least-32-chars
@@ -37,6 +37,10 @@ JWT_SECRET=change-me-to-a-random-secret-at-least-32-chars
# Database
# Database
DB_PATH=./data/live.db
DB_PATH=./data/live.db
+# Page analytics only. Defaults to SQLite until its PostgreSQL schema is migrated.
+# ANALYTICS_DRIVER=postgres
+# DATABASE_URL=postgres://live_runtime:password@localhost:5432/ov_live
+# DATABASE_DIRECT_URL=postgres://live_owner:password@localhost:5432/ov_live
# Every file Live writes lives under one data directory (default ./data, relative to the working
# Every file Live writes lives under one data directory (default ./data, relative to the working
# directory). The *_PATH variables below still move single locations (server/paths.js).
# directory). The *_PATH variables below still move single locations (server/paths.js).
# DATA_DIR=./data
# DATA_DIR=./data
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+# Live PostgreSQL migration: slice 1
+
+Status: implementation slice, 2026-09-30. The default remains SQLite.
+
+## Decision
+
+Move page analytics first, behind `ANALYTICS_DRIVER=postgres`. It is Live's second SQLite database and already has a PostgreSQL implementation in the pinned `openvibe-shared` release. Its request-facing `record()` stays synchronous while the PostgreSQL tracker buffers writes; only the three analytics read routes and the retention job need to await database work. This makes the slice deployable independently of the 2,000-plus synchronous calls to Live's primary database.
+
+`ANALYTICS_DRIVER` defaults to `sqlite`. With `postgres`, `DATABASE_URL` connects the runtime tracker to PostgreSQL. Run `DATABASE_DIRECT_URL=... npm run migrate:analytics-pg` with the schema owner before enabling it. The migration is versioned and checked by the SDK. Startup checks that `analytics_events` exists; the runtime role never creates tables. `LIVE_DRILL` uses its isolated SQLite analytics file regardless of the driver setting and starts no analytics timers or outbound calls. Existing SQLite analytics data is not imported by this slice; preserve the file and import its history before a production analytics switch if historical dashboards must be continuous.
+
+## Target architecture
+
+The primary Live database will use `openvibe-sdk/db` with `DATABASE_URL`, versioned PostgreSQL migrations on `DATABASE_DIRECT_URL`, and async methods retaining the current exported names. Generate its schema from a booted SQLite database, omit the frozen tables, and move all boot-time DDL into migrations. PostgreSQL analytics then shares that database and migration ledger. Live's outbox and inbox use SDK PostgreSQL implementations; page analytics uses `openvibe-shared/analytics/pg`. The cutover imports and verifies data through the Host switch procedure. The SQLite file remains the rollback copy for the agreed window.
+
+## Next slices
+
+1. Land the remaining Chat table retirement and settle money ownership before generating the primary schema, so deleted tables are never imported.
+2. Add the primary database's async driver and generated schema in a separately tested slice; convert one bounded caller group at a time, including its transactions and route handlers. Keep SQLite as the default until every caller is async.
+3. Convert the outbox/inbox, analytics history, drill guard, readiness, remaining callers, and tests; rehearse the import with row counts and checksums before the Host cutover.
+
+The source survey is `openvibe/agents/jobs/brief-t4-live-pg.out.md`. The PostgreSQL analytics test runs against `OV_TEST_PG_DIRECT_URL`; it skips when the disposable service is unavailable.
new file mode 100644
new file mode 100644
@@ -0,0 +1,57 @@
@@ -0,0 +1,57 @@
+-- phase: expand
+-- Page analytics schema from openvibe-shared v1.30.1 analytics/pg.js.
+-- Run with DATABASE_DIRECT_URL before ANALYTICS_DRIVER=postgres is enabled.
+
+CREATE TABLE IF NOT EXISTS analytics_events (
+ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+ service text NOT NULL,
+ event_type text NOT NULL DEFAULT 'pageview',
+ path text,
+ method text DEFAULT 'GET',
+ status_code integer,
+ response_time_ms integer,
+ user_id bigint,
+ session_id text,
+ ip text,
+ country text,
+ city text,
+ user_agent text,
+ referer text,
+ is_bot integer DEFAULT 0,
+ bot_type text,
+ device_type text,
+ browser text,
+ os text,
+ authenticated integer NOT NULL DEFAULT 0,
+ created_at text NOT NULL
+);
+CREATE TABLE IF NOT EXISTS analytics_hourly (
+ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+ service text NOT NULL, hour text NOT NULL,
+ pageviews bigint DEFAULT 0, api_calls bigint DEFAULT 0, unique_visitors bigint DEFAULT 0, unique_users bigint DEFAULT 0,
+ bot_hits bigint DEFAULT 0, avg_response_ms integer DEFAULT 0, error_count bigint DEFAULT 0, bandwidth_bytes bigint DEFAULT 0,
+ top_paths text, top_referers text,
+ UNIQUE (service, hour)
+);
+CREATE TABLE IF NOT EXISTS analytics_daily (
+ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+ service text NOT NULL, date text NOT NULL,
+ pageviews bigint DEFAULT 0, api_calls bigint DEFAULT 0, unique_visitors bigint DEFAULT 0, unique_users bigint DEFAULT 0,
+ new_users bigint DEFAULT 0, bot_hits bigint DEFAULT 0, avg_response_ms integer DEFAULT 0, error_count bigint DEFAULT 0,
+ top_paths text, top_referers text, top_countries text, device_breakdown text, browser_breakdown text,
+ UNIQUE (service, date)
+);
+CREATE TABLE IF NOT EXISTS analytics_visitor_days (
+ service text NOT NULL, day text NOT NULL, hour text NOT NULL, vhash text NOT NULL,
+ authenticated integer NOT NULL DEFAULT 0,
+ PRIMARY KEY (service, hour, vhash)
+);
+CREATE TABLE IF NOT EXISTS analytics_day_salts (
+ service text NOT NULL, day text NOT NULL, salt text NOT NULL,
+ PRIMARY KEY (service, day)
+);
+CREATE INDEX IF NOT EXISTS idx_analytics_events_created ON analytics_events (created_at);
+CREATE INDEX IF NOT EXISTS idx_analytics_events_service ON analytics_events (service, created_at);
+CREATE INDEX IF NOT EXISTS idx_analytics_events_path ON analytics_events (service, path, created_at);
+CREATE INDEX IF NOT EXISTS idx_analytics_events_bot ON analytics_events (is_bot, created_at);
+CREATE INDEX IF NOT EXISTS idx_analytics_visitor_days_day ON analytics_visitor_days (service, day);
@@ -28,6 +28,7 @@
@@ -28,6 +28,7 @@
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.16.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.16.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.30.1",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.30.1",
+ "pg": "^8.23.1",
"sanitize-html": "^2.17.7",
"sanitize-html": "^2.17.7",
"sdp-transform": "^3.0.0",
"sdp-transform": "^3.0.0",
"sharp": "^0.33.0",
"sharp": "^0.33.0",
@@ -2894,6 +2895,95 @@
@@ -2894,6 +2895,95 @@
"integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==",
"integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==",
"license": "MIT"
"license": "MIT"
},
},
+ "node_modules/pg": {
+ "version": "8.23.1",
+ "resolved": "https://registry.npmjs.org/pg/-/pg-8.23.1.tgz",
+ "integrity": "sha512-aL96AHANtWjPLDOLqnhx+ngp9+UK7ETEU8VJrDCGvsSSi/mGLcWYsS6Herg7lmaBJe4uwrfqsa7gTEFaSizDoQ==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-connection-string": "^2.14.1",
+ "pg-pool": "^3.14.0",
+ "pg-protocol": "^1.16.1",
+ "pg-types": "2.2.0",
+ "pgpass": "1.0.5"
+ },
+ "engines": {
+ "node": ">= 16.0.0"
+ },
+ "optionalDependencies": {
+ "pg-cloudflare": "^1.4.1"
+ },
+ "peerDependencies": {
+ "pg-native": ">=3.0.1"
+ },
+ "peerDependenciesMeta": {
+ "pg-native": {
+ "optional": true
+ }
+ }
+ },
+ "node_modules/pg-cloudflare": {
+ "version": "1.4.1",
+ "resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.1.tgz",
+ "integrity": "sha512-6PQbsFWZcp9EmJEwy5cGQ2La+AMWpP46lgbb8X+U/XsHIUweYDNCpeuKck5RxL2MdVFi7krbbEi5nX4Zh7JhrQ==",
+ "license": "MIT",
+ "optional": true
+ },
+ "node_modules/pg-connection-string": {
+ "version": "2.14.1",
+ "resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.1.tgz",
+ "integrity": "sha512-qR3kGNPBLpCNtz0evbKA0Y/MRFXwSSdT+pTJvYp/bXTcReZbvX1kzF0IyTc1QnxqF7AZbOeBhNL8R5mYQZV/MA==",
+ "license": "MIT"
+ },
+ "node_modules/pg-int8": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
+ "integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
+ "license": "ISC",
+ "engines": {
+ "node": ">=4.0.0"
+ }
+ },
+ "node_modules/pg-pool": {
+ "version": "3.14.0",
+ "resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
+ "integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
+ "license": "MIT",
+ "peerDependencies": {
+ "pg": ">=8.0"
+ }
+ },
+ "node_modules/pg-protocol": {
+ "version": "1.16.1",
+ "resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.1.tgz",
+ "integrity": "sha512-p9VOFMiHB/ZbJATetbg+99PxssTVSQRnyuPSQ67mN1+1KBOjZaZ83ZQzltnxPhJwSsC3nwVjJ10DVJlerbFzLg==",
+ "license": "MIT"
+ },
+ "node_modules/pg-types": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
+ "integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
+ "license": "MIT",
+ "dependencies": {
+ "pg-int8": "1.0.1",
+ "postgres-array": "~2.0.0",
+ "postgres-bytea": "~1.0.0",
+ "postgres-date": "~1.0.4",
+ "postgres-interval": "^1.1.0"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/pgpass": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
+ "integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
+ "license": "MIT",
+ "dependencies": {
+ "split2": "^4.1.0"
+ }
+ },
"node_modules/picocolors": {
"node_modules/picocolors": {
"version": "1.1.1",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -2928,6 +3018,45 @@
@@ -2928,6 +3018,45 @@
"node": "^10 || ^12 || >=14"
"node": "^10 || ^12 || >=14"
}
}
},
},
+ "node_modules/postgres-array": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
+ "integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/postgres-bytea": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
+ "integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-date": {
+ "version": "1.0.7",
+ "resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
+ "integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/postgres-interval": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
+ "integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
+ "license": "MIT",
+ "dependencies": {
+ "xtend": "^4.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
"node_modules/prebuild-install": {
"node_modules/prebuild-install": {
"version": "7.1.3",
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
@@ -3407,6 +3536,15 @@
@@ -3407,6 +3536,15 @@
"node": ">=0.10.0"
"node": ">=0.10.0"
}
}
},
},
+ "node_modules/split2": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
+ "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
+ "license": "ISC",
+ "engines": {
+ "node": ">= 10.x"
+ }
+ },
"node_modules/statuses": {
"node_modules/statuses": {
"version": "2.0.2",
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@@ -7,11 +7,13 @@
@@ -7,11 +7,13 @@
"start": "node server/index.js",
"start": "node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"init-db": "node server/db/init.js",
"init-db": "node server/db/init.js",
+ "migrate:analytics-pg": "node scripts/analytics-pg-migrate.js",
"seed": "node server/db/seed.js",
"seed": "node server/db/seed.js",
"test": "node test/run.js",
"test": "node test/run.js",
"test:unit": "node test/run.js",
"test:unit": "node test/run.js",
"test:powerchat": "for f in test/powerchat-*.test.js; do node \"$f\" || exit 1; done",
"test:powerchat": "for f in test/powerchat-*.test.js; do node \"$f\" || exit 1; done",
"test:browser": "node test/browser/smoke.js",
"test:browser": "node test/browser/smoke.js",
+ "test:analytics-pg": "node test/analytics-pg.test.js",
"perf:page": "node scripts/perf/measure-page.js",
"perf:page": "node scripts/perf/measure-page.js",
"perf:budget": "node scripts/perf/check-budgets.js",
"perf:budget": "node scripts/perf/check-budgets.js",
"n-1:record": "node scripts/n-1-record.js"
"n-1:record": "node scripts/n-1-record.js"
@@ -47,6 +49,7 @@
@@ -47,6 +49,7 @@
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.71.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.16.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.16.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.30.1",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v1.30.1",
+ "pg": "^8.23.1",
"sanitize-html": "^2.17.7",
"sanitize-html": "^2.17.7",
"sdp-transform": "^3.0.0",
"sdp-transform": "^3.0.0",
"sharp": "^0.33.0",
"sharp": "^0.33.0",
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+#!/usr/bin/env node
+'use strict';
+
+const path = require('path');
+require('dotenv').config({ path: path.join(__dirname, '..', '.env') });
+
+async function main() {
+ const url = process.env.DATABASE_DIRECT_URL;
+ if (!url) throw new Error('DATABASE_DIRECT_URL is required for the analytics PostgreSQL migration');
+ const { createDb } = require('openvibe-sdk/db');
+ const db = createDb({ url, service: 'live-analytics-migrate' });
+ try {
+ const result = await db.migrate({ dir: path.join(__dirname, '..', 'migrations') });
+ console.log(`[Analytics] migrations applied: ${result.applied.length}; held: ${result.held.length}`);
+ } finally {
+ await db.close();
+ }
+}
+
+if (require.main === module) main().catch((err) => { console.error(err); process.exitCode = 1; });
+module.exports = { main };
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+'use strict';
+
+const fs = require('fs');
+const path = require('path');
+const analyticsModule = require('openvibe-shared/analytics');
+const paths = require('../paths');
+
+function createAnalyticsStore({ driver = process.env.ANALYTICS_DRIVER || 'sqlite',
+ databaseUrl = process.env.DATABASE_URL, drill = false, timers = !drill } = {}) {
+ if (driver !== 'sqlite' && driver !== 'postgres') {
+ throw new Error(`ANALYTICS_DRIVER must be sqlite or postgres (got ${driver})`);
+ }
+
+ // Restore drills always use their isolated SQLite analytics file and never contact PostgreSQL.
+ if (driver === 'postgres' && !drill) {
+ if (!databaseUrl) throw new Error('DATABASE_URL is required when ANALYTICS_DRIVER=postgres');
+ const { createDb } = require('openvibe-sdk/db');
+ const { AnalyticsTrackerPg, pruneRawEventsPg } = require('openvibe-shared/analytics/pg');
+ const db = createDb({ url: databaseUrl, service: 'live-analytics' });
+ const tracker = new AnalyticsTrackerPg(db, 'live', { retention: false, timers });
+ let closing;
+ return {
+ tracker,
+ async ready() {
+ // The owner runs migrations before enabling this driver; runtime has no DDL rights.
+ await db.value('SELECT 1 FROM analytics_events LIMIT 1');
+ },
+ prune: () => pruneRawEventsPg(db, { days: analyticsModule.retention.MAX_DAYS }),
+ close() {
+ if (!closing) closing = Promise.resolve(tracker.destroy()).finally(() => db.close());
+ return closing;
+ },
+ };
+ }
+
+ const file = paths.analyticsDbPath();
+ fs.mkdirSync(path.dirname(file), { recursive: true });
+ const BetterSqlite3 = require('better-sqlite3');
+ const db = new BetterSqlite3(file);
+ db.pragma('journal_mode = WAL');
+ const tracker = new analyticsModule.AnalyticsTracker(db, 'live', { retention: false, timers });
+ return {
+ tracker,
+ ready: async () => {},
+ prune: () => analyticsModule.retention.pruneRawEvents(db, { days: analyticsModule.retention.MAX_DAYS }),
+ close: async () => { tracker.destroy(); db.close(); },
+ };
+}
+
+module.exports = { createAnalyticsStore };
@@ -137,7 +137,7 @@ const vibeCodingRoutes = require('./vibe-coding/routes');
@@ -137,7 +137,7 @@ const vibeCodingRoutes = require('./vibe-coding/routes');
// Restream
// Restream
const restreamRoutes = require('./streaming/restream-routes');
const restreamRoutes = require('./streaming/restream-routes');
const restreamManager = require('./streaming/restream-manager');
const restreamManager = require('./streaming/restream-manager');
-const analyticsModule = require('openvibe-shared/analytics'); // ADR-021: no IP/user id, route templates, 30-day raw retention; Sec-GPC/DNT not recorded
+// ADR-021 analytics: no IP/user id, route templates, 30-day raw retention; Sec-GPC/DNT not recorded.
// WHIP (WebRTC-HTTP Ingestion Protocol)
// WHIP (WebRTC-HTTP Ingestion Protocol)
const whipHandler = require('./streaming/whip-handler');
const whipHandler = require('./streaming/whip-handler');
@@ -395,14 +395,8 @@ const uploadLimiter = rateLimit({
@@ -395,14 +395,8 @@ const uploadLimiter = rateLimit({
message: { error: 'Too many upload requests, please slow down' },
message: { error: 'Too many upload requests, please slow down' },
});
});
// ── Analytics Tracking ────────────────────────────────────────
// ── Analytics Tracking ────────────────────────────────────────
-const BetterSqlite3 = require('better-sqlite3');
-// <data dir>/analytics.db (ANALYTICS_DB_PATH overrides it outside a drill; server/paths.js).
-const analyticsDbPath = paths.analyticsDbPath();
-fs.mkdirSync(path.dirname(analyticsDbPath), { recursive: true });
-const analyticsDb = new BetterSqlite3(analyticsDbPath);
-analyticsDb.pragma('journal_mode = WAL');
-// A drill records no page views and runs no flush/aggregate timers (its analytics.db is its own, empty).
-const analytics = new analyticsModule.AnalyticsTracker(analyticsDb, 'live', { retention: false, timers: !drill.enabled }); // prune: job 8b2 below
+const analyticsStore = require('./analytics/store').createAnalyticsStore({ drill: drill.enabled });
+const analytics = analyticsStore.tracker;
app.locals.analytics = analytics;
app.locals.analytics = analytics;
if (!drill.enabled) app.use(analytics.middleware());
if (!drill.enabled) app.use(analytics.middleware());
@@ -606,10 +600,10 @@ app.post('/internal/openre-events', require('./openre/mirror').webhookHandler);
@@ -606,10 +600,10 @@ app.post('/internal/openre-events', require('./openre/mirror').webhookHandler);
// Internal (server-to-server) routes — allow openvibe.network to call into this service
// Internal (server-to-server) routes — allow openvibe.network to call into this service
// Summary numbers for the Network's navigation service (ordering sites by real use).
// Summary numbers for the Network's navigation service (ordering sites by real use).
// A service token with live.analytics.read (loopback only); returns totals, never rows.
// A service token with live.analytics.read (loopback only); returns totals, never rows.
-app.get('/internal/analytics-summary', require('./net/service-guard').guard('live.analytics.read'), (req, res) => {
+app.get('/internal/analytics-summary', require('./net/service-guard').guard('live.analytics.read'), async (req, res) => {
try {
try {
const days = Math.min(parseInt(req.query.days, 10) || 7, 90);
const days = Math.min(parseInt(req.query.days, 10) || 7, 90);
- const st = analytics.getStats({ days }) || {};
+ const st = await analytics.getStats({ days }) || {};
res.json({ ok: true, summary: st.summary || {} });
res.json({ ok: true, summary: st.summary || {} });
} catch (err) { res.status(500).json({ ok: false, error: err.message }); }
} catch (err) { res.status(500).json({ ok: false, error: err.message }); }
});
});
@@ -736,19 +730,19 @@ app.use('/api/news', newsRoutes);
@@ -736,19 +730,19 @@ app.use('/api/news', newsRoutes);
// ── Internal Analytics API ───────────────────────────────────
// ── Internal Analytics API ───────────────────────────────────
// Called by openvibe-tools admin panel to fetch this service's analytics
// Called by openvibe-tools admin panel to fetch this service's analytics
-app.get('/api/admin/analytics', requireAuth, permissions.requireAdmin, (req, res) => {
+app.get('/api/admin/analytics', requireAuth, permissions.requireAdmin, async (req, res) => {
try {
try {
const days = Math.min(parseInt(req.query.days) || 30, 365);
const days = Math.min(parseInt(req.query.days) || 30, 365);
const hours = req.query.hours ? Math.min(parseInt(req.query.hours), 8760) : null;
const hours = req.query.hours ? Math.min(parseInt(req.query.hours), 8760) : null;
- res.json({ ok: true, analytics: analytics.getStats({ days, hours }) });
+ res.json({ ok: true, analytics: await analytics.getStats({ days, hours }) });
} catch (err) {
} catch (err) {
res.status(500).json({ ok: false, error: err.message });
res.status(500).json({ ok: false, error: err.message });
}
}
});
});
-app.get('/api/admin/analytics/bots', requireAuth, permissions.requireAdmin, (req, res) => {
+app.get('/api/admin/analytics/bots', requireAuth, permissions.requireAdmin, async (req, res) => {
try {
try {
const days = Math.min(parseInt(req.query.days) || 30, 365);
const days = Math.min(parseInt(req.query.days) || 30, 365);
- res.json({ ok: true, bots: analytics.getBotAnalysis(days) });
+ res.json({ ok: true, bots: await analytics.getBotAnalysis(days) });
} catch (err) {
} catch (err) {
res.status(500).json({ ok: false, error: err.message });
res.status(500).json({ ok: false, error: err.message });
}
}
@@ -1125,6 +1119,7 @@ async function start() {
@@ -1125,6 +1119,7 @@ async function start() {
// 1. Initialize database
// 1. Initialize database
db.initDb();
db.initDb();
+ await analyticsStore.ready();
// Initialize cosmetics tables
// Initialize cosmetics tables
cosmeticsModule.ensureTables();
cosmeticsModule.ensureTables();
// Chat tag tables (read-only tags, server/chat/tags.js)
// Chat tag tables (read-only tags, server/chat/tags.js)
@@ -1550,8 +1545,9 @@ async function start() {
@@ -1550,8 +1545,9 @@ async function start() {
// 8b2. Raw analytics retention (ADR-021): events older than 30 days go, in bounded batches;
// 8b2. Raw analytics retention (ADR-021): events older than 30 days go, in bounded batches;
// hourly/daily rollups stay. Nightly, first run a few minutes after boot.
// hourly/daily rollups stay. Nightly, first run a few minutes after boot.
require('./utils/jobs').every('analytics-prune', 24 * 60 * 60 * 1000, async () => {
require('./utils/jobs').every('analytics-prune', 24 * 60 * 60 * 1000, async () => {
- const out = await analyticsModule.retention.pruneRawEvents(analyticsDb, { days: analyticsModule.retention.MAX_DAYS });
- if (out.deleted) console.log(`[Analytics] pruned ${out.deleted} raw events older than ${out.cutoff}`);
+ const out = await analyticsStore.prune();
+ const deleted = out.deleted || out.removed;
+ if (deleted) console.log(`[Analytics] pruned ${deleted} raw events older than ${out.cutoff}`);
}, { initialDelayMs: 5 * 60 * 1000, jitterMs: 60 * 1000 });
}, { initialDelayMs: 5 * 60 * 1000, jitterMs: 60 * 1000 });
// 8b3. Media requests whose OpenCoins charge never got an answer: charged again with the same
// 8b3. Media requests whose OpenCoins charge never got an answer: charged again with the same
@@ -1614,7 +1610,7 @@ function shutdown() {
@@ -1614,7 +1610,7 @@ function shutdown() {
// Nothing was started but the HTTP server and the two databases.
// Nothing was started but the HTTP server and the two databases.
console.log('[Drill] Shutting down');
console.log('[Drill] Shutting down');
_bootComplete = false;
_bootComplete = false;
- try { analytics.destroy(); analyticsDb.close(); } catch { /* */ }
+ analyticsStore.close().catch(() => {});
server.close(() => { try { db.close(); } catch { /* */ } process.exit(0); });
server.close(() => { try { db.close(); } catch { /* */ } process.exit(0); });
try { server.closeAllConnections(); } catch { /* */ }
try { server.closeAllConnections(); } catch { /* */ }
setTimeout(() => process.exit(0), 3000).unref();
setTimeout(() => process.exit(0), 3000).unref();
@@ -1653,7 +1649,7 @@ function shutdown() {
@@ -1653,7 +1649,7 @@ function shutdown() {
try { require('./streaming/live-events').closeAll(); } catch { /* */ }
try { require('./streaming/live-events').closeAll(); } catch { /* */ }
// Small delay to let the message reach clients before closing sockets
// Small delay to let the message reach clients before closing sockets
- setTimeout(() => {
+ setTimeout(async () => {
restreamManager.stopViewerCountPolling();
restreamManager.stopViewerCountPolling();
restreamManager.stopAll();
restreamManager.stopAll();
try { recorder.stopAll(); } catch {}
try { recorder.stopAll(); } catch {}
@@ -1664,8 +1660,7 @@ function shutdown() {
@@ -1664,8 +1660,7 @@ function shutdown() {
jsmpegRelay.closeAll();
jsmpegRelay.closeAll();
webrtcSFU.closeAll();
webrtcSFU.closeAll();
rtmpServer.stop();
rtmpServer.stop();
- analytics.destroy();
- analyticsDb.close();
+ try { await analyticsStore.close(); } catch (err) { console.error('[Analytics] Shutdown:', err); }
db.close();
db.close();
server.close(() => {
server.close(() => {
new file mode 100644
new file mode 100644
@@ -0,0 +1,86 @@
@@ -0,0 +1,86 @@
+'use strict';
+
+const assert = require('assert');
+const crypto = require('crypto');
+const path = require('path');
+
+async function main() {
+ const directUrl = process.env.OV_TEST_PG_DIRECT_URL;
+ if (!directUrl) {
+ console.log('analytics PostgreSQL: skipped (OV_TEST_PG_DIRECT_URL not set)');
+ return;
+ }
+
+ const { createDb } = require('openvibe-sdk/db');
+ const admin = createDb({ url: directUrl, service: 'live-analytics-test-admin' });
+ const healthy = await admin.ready();
+ if (!healthy.ok) {
+ await admin.close();
+ console.log(`analytics PostgreSQL: skipped (test service unavailable: ${healthy.error})`);
+ return;
+ }
+
+ const name = `ov_live_analytics_${process.pid}_${crypto.randomBytes(4).toString('hex')}`;
+ const url = new URL(directUrl);
+ url.pathname = `/${name}`;
+ let created = false;
+ let owner;
+ let store;
+ try {
+ await admin.query(`CREATE DATABASE ${name}`);
+ created = true;
+ owner = createDb({ url: url.toString(), service: 'live-analytics-test-owner' });
+ const migrations = path.join(__dirname, '..', 'migrations');
+ assert.strictEqual((await owner.migrate({ dir: migrations })).applied.length, 1);
+ assert.strictEqual((await owner.migrate({ dir: migrations })).applied.length, 0);
+
+ const { createAnalyticsStore } = require('../server/analytics/store');
+ store = createAnalyticsStore({ driver: 'postgres', databaseUrl: url.toString(), timers: false });
+ await store.ready();
+
+ const express = require('express');
+ const app = express();
+ app.use(store.tracker.middleware());
+ app.get('/watch/:id', (req, res) => res.send('ok'));
+ const server = app.listen(0, '127.0.0.1');
+ try {
+ await new Promise((resolve) => server.once('listening', resolve));
+ const response = await fetch(`http://127.0.0.1:${server.address().port}/watch/123?token=secret`, {
+ headers: {
+ 'user-agent': 'Mozilla/5.0 Chrome/126.0.0.0 Safari/537.36',
+ referer: 'https://example.com/private/path?secret=1',
+ 'x-forwarded-for': '203.0.113.77',
+ },
+ });
+ assert.strictEqual(response.status, 200);
+ await response.text();
+ } finally {
+ await new Promise((resolve) => server.close(resolve));
+ }
+ await store.tracker.flush();
+ const rows = await owner.many('SELECT path, ip, user_id, city, user_agent, referer FROM analytics_events');
+ assert.strictEqual(rows.length, 1);
+ assert.strictEqual(rows[0].path, '/watch/:id');
+ assert.strictEqual(rows[0].ip, null);
+ assert.strictEqual(rows[0].user_id, null);
+ assert.strictEqual(rows[0].city, null);
+ assert.strictEqual(rows[0].referer, 'https://example.com');
+ assert.ok(!JSON.stringify(rows[0]).includes('secret'));
+ const stats = await store.tracker.getStats({ hours: 1 });
+ assert.strictEqual(Number(stats.summary.total_pageviews), 1);
+ assert.ok(Array.isArray((await store.tracker.getBotAnalysis(1)).botTrend));
+
+ await owner.query(`INSERT INTO analytics_events (service, path, created_at)
+ VALUES ('live', '/old', '2020-01-01 00:00:00')`);
+ assert.strictEqual((await store.prune()).removed, 1);
+ assert.strictEqual(Number(await owner.value('SELECT COUNT(*) FROM analytics_events')), 1);
+ console.log('analytics PostgreSQL: migration, record, privacy, reads, and prune passed');
+ } finally {
+ if (store) await store.close();
+ if (owner) await owner.close();
+ if (created) await admin.query(`DROP DATABASE ${name} WITH (FORCE)`);
+ await admin.close();
+ }
+}
+
+main().catch((err) => { console.error(err); process.exitCode = 1; });
new file mode 100644
new file mode 100644
@@ -0,0 +1,34 @@
@@ -0,0 +1,34 @@
+'use strict';
+
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+
+async function main() {
+ const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-live-analytics-store-'));
+ process.env.DATA_DIR = tmp;
+ delete process.env.ANALYTICS_DB_PATH;
+ const { createAnalyticsStore } = require('../server/analytics/store');
+ const sqlite = createAnalyticsStore({ timers: false });
+ try {
+ await sqlite.ready();
+ assert.ok(fs.existsSync(path.join(tmp, 'analytics.db')));
+ assert.strictEqual(typeof sqlite.tracker.getStats({ days: 1 }), 'object');
+ assert.ok('deleted' in await sqlite.prune());
+ } finally {
+ await sqlite.close();
+ }
+
+ // Production's PostgreSQL setting must not open an outbound connection during a restore drill.
+ const drill = createAnalyticsStore({ driver: 'postgres', drill: true, databaseUrl: 'postgres://unreachable', timers: false });
+ try {
+ await drill.ready();
+ assert.ok(fs.existsSync(path.join(tmp, 'analytics.db')));
+ } finally {
+ await drill.close();
+ }
+ console.log('analytics store: SQLite default and isolated drill passed');
+}
+
+main().catch((err) => { console.error(err); process.exitCode = 1; });