openvibers/openvibe.shared Commits
4.113.0: count bot API calls at ingest, not as raw rows (#294)
Changes
@@@ -1,5 +1,19 @@# Changelog++## 4.113.0 — 2026-11-06++++- **Bot API calls can be counted at ingest, not stored row by row.** `AnalyticsTrackerPg` takes++ `countBotApiAtIngest` (off by default): a bot's API call increments `analytics_bot_api_hourly` and++ `analytics_bot_api_daily` by endpoint group, status class and bot family at flush instead of writing a raw++ `analytics_events` row. The hourly and daily rollups add those counters to raw rows, so a day's `api_calls`,++ `bot_hits` and `error_count` are what the raw rows gave before, and a day recomputed at close counts each call++ once. The admin's bot breakdown, status classes and sub-day summary include the counters; a raw page view by a++ bot still keeps its row, and the option off changes nothing. New tables are expand-only: `analyticsSchema()`++ creates them for a new install, and an existing service adds them in a migration.++++- **Phone minimums in the kit** (the fleet phone sweep of 11-05/06): `.ovk-badge` is 12 px (was 11.2 px), and a room's++ own links (`.ov-room > a`) are 44 px tall on touch screens (were 40.6 px).++## 4.112.1 — 2026-11-05- **The shared chrome meets the phone minimums** (the fleet phone sweep of 11-05, 322 pages at 390 px). Nothing@@@ -778,6 +792,17 @@ From a phone sweep of every site's home (11-02):## Unreleased++## 4.113.0 — 2026-11-06++++- **Bot API calls can be counted at ingest, not stored row by row.** `AnalyticsTrackerPg` takes++ `countBotApiAtIngest` (off by default): a bot's API call increments `analytics_bot_api_hourly` and++ `analytics_bot_api_daily` by endpoint group, status class and bot family at flush instead of writing a raw++ `analytics_events` row. The hourly and daily rollups add those counters to raw rows, so a day's `api_calls`,++ `bot_hits` and `error_count` are what the raw rows gave before, and a day recomputed at close counts each call++ once. The admin's bot breakdown, status classes and sub-day summary include the counters; a raw page view by a++ bot still keeps its row, and the option off changes nothing. New tables are expand-only: `analyticsSchema()`++ creates them for a new install, and an existing service adds them in a migration.++## 4.67.0 — 2026-10-31- **Share this look** (plan L45 step 4). The Look panel has a Share button beside "saved to your account". It copies@@@ -2,7 +2,7 @@> Versioned UI, the OpenVibe Frame (the navbar, footer and "shipped" views every site sits in), SEO, legal and release-client packages every OpenVibe site renders.--**Status:** alpha, v4.112.1 (every release is a tag; see [CHANGELOG.md](CHANGELOG.md)). Every deployed++**Status:** alpha, v4.113.0 (every release is a tag; see [CHANGELOG.md](CHANGELOG.md)). Every deployedconsumer installs a tagged release and none keeps a vendored copy([docs/migration-plan.md](docs/migration-plan.md) is done); each repository's `package.json` namesthe tag it pins.@@@ -497,6 +497,7 @@ const analytics = new AnalyticsTrackerPg(db, 'live', {retention: { days: 30 },paramPrefixes: ['avatar'],pathRules: [[/\/by-username\/[^/?#]+/gi, '/by-username/:username']],++ countBotApiAtIngest: true, // optional: bot API calls go into the rollups, not one raw row each});app.use(analytics.middleware());// A service may call pruneRawEventsPg(db, { days: 30 }) in its own job runner.@@@ -508,6 +509,16 @@ referer origin, a user-agent class, a rotating session id and a country. Raw eveall. The row shape is `analytics/event.v1` (`docs/schemas/analytics-event.v1.json`), and`require('openvibe-shared/analytics/event').checkRow(row)` checks a stored row against it. PostgreSQL retention uses `pruneRawEventsPg(db, { days })`; `retention.MAX_DAYS` is 30.++`countBotApiAtIngest` (off by default) stops writing a raw row for a bot's API call. At flush the++call increments `analytics_bot_api_hourly` and `analytics_bot_api_daily` by endpoint group (the++first three path segments of the route template), status class (`2xx`/`3xx`/`4xx`/`5xx`/`other`) and++bot family (the row's `bot_type`: `known_crawler`, `no_useragent`, `rate_limit`, `honeypot_hit`).++The hour's and day's rollups add those counters to raw rows when they are computed, so a day closes++to the same `api_calls`, `bot_hits` and `error_count` a raw-row rollup gave; the counters are totals,++so recomputing a day never counts a call twice. Page views by bots always keep their raw row, and a++call that cannot be counted (the bounded in-memory map is full) falls back to a raw row. The bot++breakdown and status classes in `getStats` and `getBotAnalysis` include the counters.++### Browser side: two options1. **From the Network** (what most pages do today):@@@ -4,7 +4,7 @@"code": true,"wave": "W2","package": "openvibe-shared",-- "version": "4.112.1",++ "version": "4.113.0","plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)","deployed": false,"deployment": "none of its own: a package released as git tags; OpenVibe.Network serves /shared/* from its pinned tag, and a site can serve its own pinned copy (openvibe-shared/serve)",@@@ -32,6 +32,9 @@ class AnalyticsTrackerCore {this.service = service;this._now = opts.now || Date.now;this._pathOpts = privacy.pathOptions({ paramPrefixes: opts.paramPrefixes, pathRules: opts.pathRules });++ // A site may count bot API calls into its rollups at ingest instead of writing a raw row for each (the++ // PostgreSQL tracker does the counting; see analytics/pg.js).++ this._countBotApiAtIngest = !!opts.countBotApiAtIngest;this.pruneJob = null;this._buffer = [];this._visitors = new Map();@@@ -110,7 +113,7 @@ class AnalyticsTrackerCore {const vhash = this._visitorHash(ip, ua, day);const authenticated = !!(req.user && (req.user.id != null || req.user.sub != null));const isApi = entryPath.startsWith('/api/') || entryPath.startsWith('/internal/') || entryPath.startsWith('/oauth/');-- this._push({++ const event = {service: this.service,event_type: isApi ? 'api_call' : 'pageview',path: privacy.routeTemplate(req, entryPath, this._pathOpts),@@@ -128,10 +131,18 @@ class AnalyticsTrackerCore {os: parsed.os,authenticated,created_at: sqlTime(nowMs),-- }, { day, hour: hourOf(nowMs), vhash, authenticated });++ };++ if (event.event_type === 'api_call' && event.is_bot && this._countBotApiAtIngest && this._ingestBotApi(event)) return true;++ this._push(event, { day, hour: hourOf(nowMs), vhash, authenticated });return true;}++ /**++ * Count one bot API call into the tracker's rollups instead of a raw row. The PostgreSQL tracker overrides++ * this when `countBotApiAtIngest` is on; the default writes the raw row as before. Returns whether it counted.++ */++ _ingestBotApi() { return false; }++/*** Express middleware: one row per finished request (health checks, static assets and opted-out* requests skipped).@@@ -25,8 +25,35 @@ const { AnalyticsTrackerCore, sqlTime, dayOf, hourOf } = require('./core');const privacy = require('./privacy');const MAX_BUFFER = 5000;++const MAX_BOT_API_KEYS = 1000; // 1000 rows x 6 values stays one INSERT; beyond it a call keeps its raw rowconst DAY = 86400000;++/** The status class of a code, the same grouping getStats's statusCodes uses. */++function statusClassOf(code) {++ const n = Number(code);++ if (n >= 200 && n < 300) return '2xx';++ if (n >= 300 && n < 400) return '3xx';++ if (n >= 400 && n < 500) return '4xx';++ if (n >= 500) return '5xx';++ return 'other';++}++++/** The first three path segments of a route template (/api/v1/streams for /api/v1/streams/:id), or two otherwise. */++function endpointGroupOf(path) {++ const parts = String(path || '').split('/').filter(Boolean);++ const n = parts[0] === 'api' || parts[0] === 'internal' || parts[0] === 'oauth' ? 3 : 2;++ return `/${parts.slice(0, n).join('/')}` || '/';++}++++/**++ * The bot family the admin's breakdown groups by: the tracker's bot_type (known_crawler, no_useragent,++ * rate_limit, honeypot_hit). The user-agent class stays on the raw row; it is not carried per counter.++ */++function botFamilyOf(botType) {++ const s = String(botType || 'unknown');++ return /^[a-z0-9_]{1,32}$/.test(s) ? s : 'unknown';++}++/** The DDL for a service's migrations (the runtime role creates no tables). */function analyticsSchema() {return `@@@ -69,6 +96,20 @@ CREATE TABLE IF NOT EXISTS analytics_daily (top_paths text, top_referers text, top_countries text, device_breakdown text, browser_breakdown text,UNIQUE (service, date));++CREATE TABLE IF NOT EXISTS analytics_bot_api_hourly (++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,++ service text NOT NULL, hour text NOT NULL,++ endpoint_group text NOT NULL, status_class text NOT NULL, bot_family text NOT NULL,++ hits bigint NOT NULL DEFAULT 0,++ UNIQUE (service, hour, endpoint_group, status_class, bot_family)++);++CREATE TABLE IF NOT EXISTS analytics_bot_api_daily (++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,++ service text NOT NULL, date text NOT NULL,++ endpoint_group text NOT NULL, status_class text NOT NULL, bot_family text NOT NULL,++ hits bigint NOT NULL DEFAULT 0,++ UNIQUE (service, date, endpoint_group, status_class, bot_family)++);CREATE TABLE IF NOT EXISTS analytics_visitor_days (service text NOT NULL, day text NOT NULL, hour text NOT NULL, vhash text NOT NULL,authenticated integer NOT NULL DEFAULT 0,@@@ -91,13 +132,16 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {* @param {object} db openvibe-sdk/db handle (the service's own database, analyticsSchema() migrated)* @param {string} service* @param {object} [opts] paramPrefixes, pathRules, retention ({ days = 30, botDays = days, intervalMs,-- * initialDelayMs } or false), timers, now++ * initialDelayMs } or false), timers, now; countBotApiAtIngest counts bot API calls++ * into analytics_bot_api_{hourly,daily} and the hour's/day's rollup instead of writing++ * a raw row (page views by bots always keep theirs)*/constructor(db, service, opts = {}) {super(db, service, opts);this._pending = []; // visitors waiting for the shared salt: { day, hour, ip, ua, authenticated }this._localKeys = new Map(); // day → process-local session keythis._salts = new Map(); // day → shared salt (cached after the first read)++ this._botApi = new Map(); // counted bot API calls waiting for the flush: hour|group|status|family → rowthis._flushing = null;if (opts.timers !== false) {const every = (fn, ms) => { const t = setInterval(fn, ms); if (t.unref) t.unref(); this._timers.push(t); };@@@ -138,6 +182,25 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {return true;}++ /** Count a bot API call into the hour's and day's counters (same transaction as the raw rows at flush). */++ _ingestBotApi(event) {++ const hour = `${event.created_at.slice(0, 13)}:00:00`;++ const endpoint_group = endpointGroupOf(event.path);++ const status_class = statusClassOf(event.status_code);++ const bot_family = botFamilyOf(event.bot_type);++ const key = `${hour}|${endpoint_group}|${status_class}|${bot_family}`;++ let row = this._botApi.get(key);++ if (!row) {++ // Bounded: if a site ever makes more distinct combinations than this, further calls keep their raw row++ // rather than lose their count.++ if (this._botApi.size >= MAX_BOT_API_KEYS) return false;++ row = { hour, endpoint_group, status_class, bot_family, hits: 0 };++ this._botApi.set(key, row);++ }++ row.hits++;++ return true;++ }++_push(event) {if (this._buffer.length >= MAX_BUFFER) this._buffer.shift();this._buffer.push(event);@@@ -163,9 +226,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {}async _flushNow() {-- if (!this._buffer.length && !this._pending.length) return;++ if (!this._buffer.length && !this._pending.length && !this._botApi.size) return;const events = this._buffer.splice(0);const pending = this._pending.splice(0);++ const botApi = [...this._botApi.entries()];++ this._botApi.clear();try {const visitors = new Map();for (const p of pending) {@@@ -195,11 +260,42 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {ON CONFLICT (service, hour, vhash) DO UPDATE SET authenticated = GREATEST(analytics_visitor_days.authenticated, excluded.authenticated)`,[this.service, v.day, v.hour, v.vhash, v.authenticated ? 1 : 0]);}++ if (botApi.length) {++ // One row per key: two hours of the same day share a daily key, and one INSERT may not touch it twice.++ const daily = new Map();++ const hourVals = []; const hourRows = [];++ for (const [, r] of botApi) {++ let b = hourVals.length;++ hourRows.push(`(${Array.from({ length: 6 }, (_, k) => `$${b + k + 1}`).join(', ')})`);++ hourVals.push(this.service, r.hour, r.endpoint_group, r.status_class, r.bot_family, r.hits);++ const dkey = `${r.hour.slice(0, 10)}|${r.endpoint_group}|${r.status_class}|${r.bot_family}`;++ const d = daily.get(dkey);++ if (d) d.hits += r.hits;++ else daily.set(dkey, { date: r.hour.slice(0, 10), endpoint_group: r.endpoint_group, status_class: r.status_class, bot_family: r.bot_family, hits: r.hits });++ }++ const dayVals = []; const dayRows = [];++ for (const r of daily.values()) {++ let b = dayVals.length;++ dayRows.push(`(${Array.from({ length: 6 }, (_, k) => `$${b + k + 1}`).join(', ')})`);++ dayVals.push(this.service, r.date, r.endpoint_group, r.status_class, r.bot_family, r.hits);++ }++ await this.db.query(`INSERT INTO analytics_bot_api_hourly (service, hour, endpoint_group, status_class, bot_family, hits)++ VALUES ${hourRows.join(', ')} ON CONFLICT (service, hour, endpoint_group, status_class, bot_family)++ DO UPDATE SET hits = analytics_bot_api_hourly.hits + excluded.hits`, hourVals);++ await this.db.query(`INSERT INTO analytics_bot_api_daily (service, date, endpoint_group, status_class, bot_family, hits)++ VALUES ${dayRows.join(', ')} ON CONFLICT (service, date, endpoint_group, status_class, bot_family)++ DO UPDATE SET hits = analytics_bot_api_daily.hits + excluded.hits`, dayVals);++ }});} catch (err) {console.error(`[Analytics:${this.service}] Flush error:`, err.message);this._buffer = events.concat(this._buffer).slice(-MAX_BUFFER);this._pending = pending.concat(this._pending).slice(-MAX_BUFFER);++ for (const [key, r] of botApi) {++ const prev = this._botApi.get(key);++ if (prev) prev.hits += r.hits;++ else if (this._botApi.size < MAX_BOT_API_KEYS) this._botApi.set(key, r);++ }}}@@@ -215,6 +311,10 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {for (const p of await this.db.many('SELECT DISTINCT day, hour FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today])) {hours.add(p.hour); days.add(p.day);}++ // Counted bot API calls have no raw row to discover their hour from: roll up the last day's counter hours.++ for (const p of await this.db.many('SELECT DISTINCT hour FROM analytics_bot_api_hourly WHERE service = $1 AND hour >= $2', [this.service, sqlTime(nowMs - DAY)])) {++ hours.add(p.hour);++ }for (const h of hours) await this._aggregateHour(h);for (const d of days) await this._aggregateDay(d);await this.db.query('DELETE FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today]);@@@ -233,7 +333,12 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,COUNT(*) FILTER (WHERE status_code >= 400) AS error_countFROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3`, range);-- if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0)) return;++ // Bot API calls counted at ingest are not raw rows: add them once here (raw + counted are disjoint).++ const botApi = await this.db.maybe(`SELECT COALESCE(SUM(hits), 0)::bigint AS hits,++ COALESCE(SUM(hits) FILTER (WHERE status_class IN ('4xx', '5xx')), 0)::bigint AS errors++ FROM analytics_bot_api_hourly WHERE service = $1 AND hour = $2`, [svc, hourStart]);++ const botHits = Number(botApi && botApi.hits || 0);++ if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0 && botHits === 0)) return;const v = await this.db.maybe('SELECT COUNT(*) AS visitors, COALESCE(SUM(authenticated), 0)::bigint AS users FROM analytics_visitor_days WHERE service = $1 AND hour = $2', [svc, hourStart]);const prev = (await this.db.maybe('SELECT unique_visitors, unique_users FROM analytics_hourly WHERE service = $1 AND hour = $2', [svc, hourStart])) || {};const topPaths = await this.db.many(`SELECT path, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3 AND is_bot = 0@@@ -246,9 +351,9 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {ON CONFLICT (service, hour) DO UPDATE SET pageviews = excluded.pageviews, api_calls = excluded.api_calls,unique_visitors = excluded.unique_visitors, unique_users = excluded.unique_users, bot_hits = excluded.bot_hits,avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count, top_paths = excluded.top_paths, top_referers = excluded.top_referers`,-- [svc, hourStart, stats.pageviews, stats.api_calls, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),-- Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), stats.bot_hits, stats.avg_response_ms || 0, stats.error_count,-- JSON.stringify(topPaths), JSON.stringify(topReferers)]);++ [svc, hourStart, stats.pageviews, Number(stats.api_calls) + botHits, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),++ Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), Number(stats.bot_hits) + botHits, stats.avg_response_ms || 0,++ Number(stats.error_count) + Number(botApi && botApi.errors || 0), JSON.stringify(topPaths), JSON.stringify(topReferers)]);}async _aggregateDay(date) {@@@ -259,7 +364,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,COUNT(*) FILTER (WHERE status_code >= 400) AS error_countFROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at <= $3`, range);-- if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0)) return;++ const botApi = await this.db.maybe(`SELECT COALESCE(SUM(hits), 0)::bigint AS hits,++ COALESCE(SUM(hits) FILTER (WHERE status_class IN ('4xx', '5xx')), 0)::bigint AS errors++ FROM analytics_bot_api_daily WHERE service = $1 AND date = $2`, [svc, date]);++ const botHits = Number(botApi && botApi.hits || 0);++ if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0 && botHits === 0)) return;const v = await this.db.maybe(`SELECT COUNT(DISTINCT vhash) AS visitors, COUNT(DISTINCT CASE WHEN authenticated = 1 THEN vhash END) AS usersFROM analytics_visitor_days WHERE service = $1 AND day = $2`, [svc, date]);const prev = (await this.db.maybe('SELECT unique_visitors, unique_users, new_users FROM analytics_daily WHERE service = $1 AND date = $2', [svc, date])) || {};@@@ -276,14 +385,28 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {bot_hits = excluded.bot_hits, avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count,top_paths = excluded.top_paths, top_referers = excluded.top_referers, top_countries = excluded.top_countries,device_breakdown = excluded.device_breakdown, browser_breakdown = excluded.browser_breakdown`,-- [svc, date, stats.pageviews, stats.api_calls, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),++ [svc, date, stats.pageviews, Number(stats.api_calls) + botHits, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), prev.new_users != null ? prev.new_users : null,-- stats.bot_hits, stats.avg_response_ms || 0, stats.error_count, JSON.stringify(topPaths), JSON.stringify(topReferers),-- JSON.stringify(topCountries), JSON.stringify(devices), JSON.stringify(browsers)]);++ Number(stats.bot_hits) + botHits, stats.avg_response_ms || 0, Number(stats.error_count) + Number(botApi && botApi.errors || 0),++ JSON.stringify(topPaths), JSON.stringify(topReferers), JSON.stringify(topCountries), JSON.stringify(devices), JSON.stringify(browsers)]);}// ── Reads (admin dashboards): dashboard shapes ──++ /**++ * Counted bot API calls for a dashboard window, as rows of endpoint group, status class, bot family and hits.++ * The hourly table answers an hours window; the daily table answers a days window.++ */++ async _botApiRange({ hours, days = 30, service = this.service } = {}) {++ if (hours) {++ const since = `${sqlTime(this._now() - hours * 3600000).slice(0, 13)}:00:00`;++ return this.db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits++ FROM analytics_bot_api_hourly WHERE service = $1 AND hour >= $2 GROUP BY 1, 2, 3`, [service, since]);++ }++ return this.db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits++ FROM analytics_bot_api_daily WHERE service = $1 AND date >= $2 GROUP BY 1, 2, 3`, [service, dayOf(this._now() - days * DAY)]);++ }++async getStats(options = {}) {const { days = 30, hours, service } = options;const svc = service || this.service;@@@ -339,8 +462,18 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {const statusRows = await all(`SELECT is_bot, bot_type, CASE WHEN status_code >= 200 AND status_code < 300 THEN '2xx' WHEN status_code >= 300 AND status_code < 400 THEN '3xx'WHEN status_code >= 400 AND status_code < 500 THEN '4xx' WHEN status_code >= 500 THEN '5xx' ELSE 'other' END AS group_code, COUNT(*) AS cntFROM analytics_events WHERE service = $1 AND created_at >= $2 GROUP BY 1, 2, 3`, human);-- const botBreakdown = tally(statusRows.filter((r) => r.is_bot === 1), 'bot_type', 'cnt', 'hits').slice(0, 15);-- const statusCodes = tally(statusRows, 'group_code', 'cnt');++ // Bot API calls counted at ingest have no raw row: their counters keep the admin's bot breakdown, status++ // classes and (for a sub-day window, which reads raw rows) the summary's bot hits and errors complete.++ const botApiRows = await this._botApiRange({ hours, days, service: svc });++ const botApiHits = botApiRows.reduce((n, r) => n + Number(r.hits), 0);++ const botApiErrors = botApiRows.filter((r) => r.status_class === '4xx' || r.status_class === '5xx').reduce((n, r) => n + Number(r.hits), 0);++ if (hours && hours < 48 && summary) {++ summary.total_bot_hits = Number(summary.total_bot_hits || 0) + botApiHits;++ summary.total_errors = Number(summary.total_errors || 0) + botApiErrors;++ }++ const statusRowsAll = statusRows.concat(botApiRows.map((r) => ({ is_bot: 1, bot_type: r.bot_family, group_code: r.status_class, cnt: Number(r.hits) })));++ const botBreakdown = tally(statusRowsAll.filter((r) => r.is_bot === 1), 'bot_type', 'cnt', 'hits').slice(0, 15);++ const statusCodes = tally(statusRowsAll, 'group_code', 'cnt');const dims = await all(`SELECT device_type, browser, os, country, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND is_bot = 0GROUP BY 1, 2, 3, 4`, human);const deviceBreakdown = tally(dims, 'device_type', 'cnt');@@@ -417,19 +550,33 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {}async getBotAnalysis(days = 30) {-- const since = sqlTime(this._now() - Math.min(Math.max(1, days), 365) * DAY);++ const windowDays = Math.min(Math.max(1, days), 365);++ const since = sqlTime(this._now() - windowDays * DAY);const topBotIPs = await this.db.many(`SELECT NULL AS ip, user_agent AS ua_class, bot_type, COUNT(*) AS hits, COUNT(DISTINCT session_id) AS sessions,MIN(created_at) AS first_seen, MAX(created_at) AS last_seenFROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY user_agent, bot_type ORDER BY hits DESC LIMIT 25`, [since]);const botTrend = await this.db.many(`SELECT date, (SUM(pageviews) + SUM(api_calls))::bigint AS human_hits, SUM(bot_hits)::bigint AS bot_hitsFROM analytics_daily WHERE date >= $1 GROUP BY date ORDER BY date ASC`, [since.slice(0, 10)]);++ const botApi = await this._botApiRange({ days: windowDays });const botTypes = await this.db.many(`SELECT bot_type, COUNT(*) AS hits, NULL AS unique_ips, COUNT(DISTINCT session_id) AS unique_sessionsFROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY bot_type ORDER BY hits DESC`, [since]);++ // Counted bot API calls have no raw row: add their families to the breakdown (and keep families whose raw++ // rows were pruned while their counters stayed).++ const byFamily = new Map();++ for (const r of botApi) byFamily.set(r.bot_family, (byFamily.get(r.bot_family) || 0) + Number(r.hits));++ for (const r of botTypes) {++ if (byFamily.has(r.bot_type)) {++ r.hits = Number(r.hits) + byFamily.get(r.bot_type);++ byFamily.delete(r.bot_type);++ }++ }++ for (const [bot_type, hits] of byFamily) botTypes.push({ bot_type, hits, unique_ips: null, unique_sessions: null });++ botTypes.sort((x, y) => Number(y.hits) - Number(x.hits));const suspiciousIPs = await this.db.many(`SELECT NULL AS ip, session_id, COUNT(*) AS total_hits, COUNT(*) FILTER (WHERE status_code >= 400) AS error_hits,COUNT(DISTINCT path) AS unique_paths, MIN(created_at) AS first_seen, MAX(created_at) AS last_seenFROM analytics_events WHERE created_at >= $1 AND is_bot = 0 AND session_id IS NOT NULLGROUP BY session_id HAVING COUNT(*) > 500 ORDER BY total_hits DESC LIMIT 20`, [sqlTime(this._now() - 7 * DAY)]);-- return { topBotIPs, botTrend, botTypes, suspiciousIPs };++ return { topBotIPs, botTrend, botTypes, suspiciousIPs, botApi };}async destroy() {@@@ -113,7 +113,7 @@ html[data-ov-motion=reduced] { --ov-dur-1: 0s; --ov-dur-2: 0s; --ov-dur-3: 0s; -.ovk-icon-btn--danger:hover { color: var(--ovk-bad-text); }.ovk-icon-btn .ovk-i { width: 20px; height: 20px; }@media (pointer: coarse) { .ovk-icon-btn { width: var(--ovk-tap); height: var(--ovk-tap); } }--.ovk-badge { display: inline-flex; align-items: center; flex: none; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--ovk-line); font-size: .7rem; font-weight: 800; letter-spacing: .03em; text-transform: uppercase; color: var(--ovk-muted); white-space: nowrap; }++.ovk-badge { display: inline-flex; align-items: center; flex: none; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--ovk-line); font-size: .75rem; font-weight: 800; letter-spacing: .03em; text-transform: uppercase; color: var(--ovk-muted); white-space: nowrap; }.ovk-badge--accent { color: var(--ovk-accent); border-color: color-mix(in srgb, var(--ovk-accent) 50%, transparent); }.ovk-badge--ok { color: var(--ovk-ok-text); border-color: color-mix(in srgb, var(--ovk-ok) 50%, transparent); }.ovk-badge--warn { color: var(--ovk-warn-text); border-color: color-mix(in srgb, var(--ovk-warn) 50%, transparent); }@@@ -1,12 +1,12 @@{"name": "openvibe-shared",-- "version": "4.112.1",++ "version": "4.113.0","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "openvibe-shared",-- "version": "4.112.1",++ "version": "4.113.0","license": "MIT","dependencies": {"jsonwebtoken": "^9.0.0",@@@ -1,6 +1,6 @@{"name": "openvibe-shared",-- "version": "4.112.1",++ "version": "4.113.0","description": "Shared navbar, footer, themes, icons, legal pages, SEO, auth and analytics for every OpenVibe site","main": "index.js","bin": {@@@ -86,6 +86,8 @@* quarter of the theme's text mixed in, so it stays at 4.5:1 on the room's tinted chips: the bare accent was 3.9-4.4:1);* --room-fill and --room-on: a filled button and its text (dark text on a dark theme's bright fill, white on light). ── */.ov-room { --room-accent: var(--ovk-accent); --room-strong: color-mix(in srgb, var(--ovk-accent) 75%, var(--ovk-text)); --room-fill: var(--ovk-fill); --room-on: var(--ovk-on-fill); }++/* 44 px to the touch (4.113.0, the phone sweep of 11-06): a room's own links (a brand, a place) were 40.6 px tall. */++@media (pointer: coarse) { .ov-room > a { min-height: 44px; display: inline-flex; align-items: center; } }.ov-room[style*="--room-dark"] { --room-accent: var(--room-dark); --room-strong: color-mix(in srgb, var(--room-dark) 75%, var(--ovk-text)); --room-fill: var(--room-dark); --room-on: #0b0d12; }:root[data-theme-mode="light"] .ov-room[style*="--room-light"] { --room-accent: var(--room-light); --room-strong: color-mix(in srgb, var(--room-light) 75%, var(--ovk-text)); --room-fill: var(--room-light); --room-on: #fff; }nnew file mode 100644@@@ -0,0 +1,109 @@++'use strict';++/**++ * countBotApiAtIngest: a bot API call is counted into analytics_bot_api_{hourly,daily} at flush and added to the++ * hour's and day's rollups instead of being written as a raw row. Page views by bots keep their raw row (the admin's++ * suspicious-session check reads raw rows); a tracked API call by a person is unchanged. The counters give the same++ * rollup numbers a raw-row rollup gives for the same input, and recomputing a closed day does not count them twice.++ * With the option off nothing changes.++ */++const assert = require('assert');++let createDb;++try { ({ createDb } = require('openvibe-sdk/db')); } catch { console.log('analytics-bot-ingest: skipped (openvibe-sdk not installed)'); process.exit(0); }++const { AnalyticsTrackerPg, analyticsSchema } = require('../analytics/pg');++++const DAY = 86400000;++const T0 = Date.parse('2026-11-06T10:05:00Z');++const BOT_UA = 'curl/8.5.0';++const HUMAN_UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36';++const req = (path, ua, ip = '203.0.113.10') => ({ path, method: 'GET', ip, headers: { 'user-agent': ua }, user: null });++const res = (status = 200) => ({ statusCode: status });++++(async () => {++ const db = createDb({ pglite: true, service: 'analytics-bot-ingest', log: { log() {}, warn() {}, error() {}, info() {} } });++ await db.query(analyticsSchema());++ let now = T0;++ const off = new AnalyticsTrackerPg(db, 'off', { timers: false, now: () => now });++ const on = new AnalyticsTrackerPg(db, 'on', { timers: false, now: () => now, countBotApiAtIngest: true });++++ const input = [++ ['/api/v1/streams/abc/clips', BOT_UA, 200],++ ['/api/v1/streams/abc/clips', BOT_UA, 404],++ ['/api/v1/vods/xyz', BOT_UA, 500],++ ['/about', BOT_UA, 200], // a bot page view keeps its raw row++ ['/api/v1/streams/abc', HUMAN_UA, 200],++ ];++ for (const [path, ua, status] of input) {++ assert.strictEqual(off.record(req(path, ua), res(status), path, 15), true);++ assert.strictEqual(on.record(req(path, ua), res(status), path, 15), true);++ }++ await off.flush(); await on.flush();++++ const raw = async (service) => db.many('SELECT event_type, is_bot, bot_type FROM analytics_events WHERE service = $1 ORDER BY id', [service]);++ const offRaw = await raw('off');++ const onRaw = await raw('on');++ assert.strictEqual(offRaw.length, 5, 'option off: every request is a raw row');++ assert.deepStrictEqual(onRaw.map((r) => [r.event_type, r.is_bot]), [['pageview', 1], ['api_call', 0]], 'option on: the bot page view and the person\'s API call only');++ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_events WHERE service = 'on' AND is_bot = 1 AND event_type = 'api_call'")), 0, 'no raw bot API row');++ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_events WHERE service = 'off' AND is_bot = 1 AND event_type = 'api_call'")), 3);++++ const counters = await db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits++ FROM analytics_bot_api_hourly WHERE service = 'on' GROUP BY 1, 2, 3 ORDER BY 1, 2`);++ assert.deepStrictEqual(counters.map((r) => [r.endpoint_group, r.status_class, r.bot_family, Number(r.hits)]), [++ ['/api/v1/streams', '2xx', 'known_crawler', 1],++ ['/api/v1/streams', '4xx', 'known_crawler', 1],++ ['/api/v1/vods', '5xx', 'known_crawler', 1],++ ]);++ const dayCounters = await db.many(`SELECT status_class, SUM(hits)::bigint AS hits FROM analytics_bot_api_daily WHERE service = 'on' GROUP BY 1 ORDER BY 1`);++ assert.deepStrictEqual(dayCounters.map((r) => [r.status_class, Number(r.hits)]), [['2xx', 1], ['4xx', 1], ['5xx', 1]]);++ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_bot_api_hourly WHERE service = 'off'")), 0, 'option off writes no counters');++++ await off.aggregate(); await on.aggregate();++ const rollup = async (service, table, bucket, at) => db.maybe(`SELECT pageviews, api_calls, bot_hits, error_count, unique_visitors++ FROM ${table} WHERE service = $1 AND ${bucket} = $2`, [service, at]);++ const hourAt = '2026-11-06 10:00:00';++ const [offHour, onHour] = [await rollup('off', 'analytics_hourly', 'hour', hourAt), await rollup('on', 'analytics_hourly', 'hour', hourAt)];++ assert.deepStrictEqual([onHour.pageviews, onHour.api_calls, onHour.bot_hits, onHour.error_count].map(Number), [1, 4, 4, 2], 'hourly: counters give the raw rollup\'s numbers');++ assert.deepStrictEqual([offHour.pageviews, offHour.api_calls, offHour.bot_hits, offHour.error_count].map(Number), [1, 4, 4, 2]);++ assert.strictEqual(Number(onHour.unique_visitors), Number(offHour.unique_visitors), 'the visitor count is unchanged');++ const [offDay, onDay] = [await rollup('off', 'analytics_daily', 'date', '2026-11-06'), await rollup('on', 'analytics_daily', 'date', '2026-11-06')];++ assert.deepStrictEqual([onDay.pageviews, onDay.api_calls, onDay.bot_hits, onDay.error_count].map(Number), [1, 4, 4, 2], 'daily: the same');++ assert.deepStrictEqual([offDay.pageviews, offDay.api_calls, offDay.bot_hits, offDay.error_count].map(Number), [1, 4, 4, 2]);++ assert.strictEqual(Number(onDay.unique_visitors), Number(offDay.unique_visitors), 'the daily visitor count is unchanged');++++ // The admin reads: the bot breakdown, status classes and a sub-day summary stay complete from the counters.++ const stats = await on.getStats({ hours: 6 });++ assert.deepStrictEqual(stats.botBreakdown.map((r) => [r.bot_type, Number(r.hits)]), [['known_crawler', 4]], '3 counted API calls + the raw bot page view');++ assert.deepStrictEqual(Object.fromEntries(stats.statusCodes.map((r) => [r.group_code, Number(r.cnt)])), { '2xx': 3, '4xx': 1, '5xx': 1 });++ const offStats = await off.getStats({ hours: 6 });++ for (const k of ['total_pageviews', 'total_api_calls', 'total_bot_hits', 'total_errors']) {++ assert.strictEqual(Number(stats.summary[k]), Number(offStats.summary[k]), `summary ${k} equals the raw-row summary`);++ }++ // getBotAnalysis is scoped to the database (one service per production database), not by service here.++ await db.query("DELETE FROM analytics_events WHERE service = 'off'");++ const bots = await on.getBotAnalysis(7);++ assert.deepStrictEqual(bots.botTypes.map((r) => [r.bot_type, Number(r.hits)]), [['known_crawler', 4]], 'raw page view + counted API calls');++++ // Recomputing does not double count (the rollups are read from raw rows plus the ingest counters).++ await on.aggregate();++ const again = await rollup('on', 'analytics_daily', 'date', '2026-11-06');++ assert.deepStrictEqual([again.pageviews, again.api_calls, again.bot_hits, again.error_count].map(Number), [1, 4, 4, 2]);++++ // A day with only counted bot API calls still gets an hourly and a daily rollup (there is no raw row to find it by).++ const only = new AnalyticsTrackerPg(db, 'only', { timers: false, now: () => now, countBotApiAtIngest: true });++ only.record(req('/api/v1/streams/abc', BOT_UA), res(200), '/api/v1/streams/abc', 10);++ await only.flush(); await only.aggregate();++ const onlyHour = await rollup('only', 'analytics_hourly', 'hour', hourAt);++ assert.deepStrictEqual([onlyHour.pageviews, onlyHour.api_calls, onlyHour.bot_hits].map(Number), [0, 1, 1]);++++ // Day close: the next day's aggregation recomputes yesterday from raw rows plus counters, once.++ now = T0 + DAY;++ await on.aggregate();++ await on.aggregate();++ const closed = await rollup('on', 'analytics_daily', 'date', '2026-11-06');++ assert.deepStrictEqual([closed.pageviews, closed.api_calls, closed.bot_hits, closed.error_count].map(Number), [1, 4, 4, 2], 'no double count at day close');++ const countersAfter = await db.many(`SELECT status_class, SUM(hits)::bigint AS hits FROM analytics_bot_api_daily WHERE service = 'on' GROUP BY 1 ORDER BY 1`);++ assert.deepStrictEqual(countersAfter.map((r) => [r.status_class, Number(r.hits)]), [['2xx', 1], ['4xx', 1], ['5xx', 1]], 'counters are totals, not accumulated again');++++ await off.destroy(); await on.destroy(); await only.destroy(); await db.close();++ console.log('analytics-bot-ingest: all checks passed');++})().catch((err) => { console.error(err); process.exit(1); });
@@ -1,5 +1,19 @@
@@ -1,5 +1,19 @@
# Changelog
# Changelog
+## 4.113.0 — 2026-11-06
+
+- **Bot API calls can be counted at ingest, not stored row by row.** `AnalyticsTrackerPg` takes
+ `countBotApiAtIngest` (off by default): a bot's API call increments `analytics_bot_api_hourly` and
+ `analytics_bot_api_daily` by endpoint group, status class and bot family at flush instead of writing a raw
+ `analytics_events` row. The hourly and daily rollups add those counters to raw rows, so a day's `api_calls`,
+ `bot_hits` and `error_count` are what the raw rows gave before, and a day recomputed at close counts each call
+ once. The admin's bot breakdown, status classes and sub-day summary include the counters; a raw page view by a
+ bot still keeps its row, and the option off changes nothing. New tables are expand-only: `analyticsSchema()`
+ creates them for a new install, and an existing service adds them in a migration.
+
+- **Phone minimums in the kit** (the fleet phone sweep of 11-05/06): `.ovk-badge` is 12 px (was 11.2 px), and a room's
+ own links (`.ov-room > a`) are 44 px tall on touch screens (were 40.6 px).
+
## 4.112.1 — 2026-11-05
## 4.112.1 — 2026-11-05
- **The shared chrome meets the phone minimums** (the fleet phone sweep of 11-05, 322 pages at 390 px). Nothing
- **The shared chrome meets the phone minimums** (the fleet phone sweep of 11-05, 322 pages at 390 px). Nothing
@@ -778,6 +792,17 @@ From a phone sweep of every site's home (11-02):
@@ -778,6 +792,17 @@ From a phone sweep of every site's home (11-02):
## Unreleased
## Unreleased
+## 4.113.0 — 2026-11-06
+
+- **Bot API calls can be counted at ingest, not stored row by row.** `AnalyticsTrackerPg` takes
+ `countBotApiAtIngest` (off by default): a bot's API call increments `analytics_bot_api_hourly` and
+ `analytics_bot_api_daily` by endpoint group, status class and bot family at flush instead of writing a raw
+ `analytics_events` row. The hourly and daily rollups add those counters to raw rows, so a day's `api_calls`,
+ `bot_hits` and `error_count` are what the raw rows gave before, and a day recomputed at close counts each call
+ once. The admin's bot breakdown, status classes and sub-day summary include the counters; a raw page view by a
+ bot still keeps its row, and the option off changes nothing. New tables are expand-only: `analyticsSchema()`
+ creates them for a new install, and an existing service adds them in a migration.
+
## 4.67.0 — 2026-10-31
## 4.67.0 — 2026-10-31
- **Share this look** (plan L45 step 4). The Look panel has a Share button beside "saved to your account". It copies
- **Share this look** (plan L45 step 4). The Look panel has a Share button beside "saved to your account". It copies
@@ -2,7 +2,7 @@
@@ -2,7 +2,7 @@
> Versioned UI, the OpenVibe Frame (the navbar, footer and "shipped" views every site sits in), SEO, legal and release-client packages every OpenVibe site renders.
> Versioned UI, the OpenVibe Frame (the navbar, footer and "shipped" views every site sits in), SEO, legal and release-client packages every OpenVibe site renders.
-**Status:** alpha, v4.112.1 (every release is a tag; see [CHANGELOG.md](CHANGELOG.md)). Every deployed
+**Status:** alpha, v4.113.0 (every release is a tag; see [CHANGELOG.md](CHANGELOG.md)). Every deployed
consumer installs a tagged release and none keeps a vendored copy
consumer installs a tagged release and none keeps a vendored copy
([docs/migration-plan.md](docs/migration-plan.md) is done); each repository's `package.json` names
([docs/migration-plan.md](docs/migration-plan.md) is done); each repository's `package.json` names
the tag it pins.
the tag it pins.
@@ -497,6 +497,7 @@ const analytics = new AnalyticsTrackerPg(db, 'live', {
@@ -497,6 +497,7 @@ const analytics = new AnalyticsTrackerPg(db, 'live', {
retention: { days: 30 },
retention: { days: 30 },
paramPrefixes: ['avatar'],
paramPrefixes: ['avatar'],
pathRules: [[/\/by-username\/[^/?#]+/gi, '/by-username/:username']],
pathRules: [[/\/by-username\/[^/?#]+/gi, '/by-username/:username']],
+ countBotApiAtIngest: true, // optional: bot API calls go into the rollups, not one raw row each
});
});
app.use(analytics.middleware());
app.use(analytics.middleware());
// A service may call pruneRawEventsPg(db, { days: 30 }) in its own job runner.
// A service may call pruneRawEventsPg(db, { days: 30 }) in its own job runner.
@@ -508,6 +509,16 @@ referer origin, a user-agent class, a rotating session id and a country. Raw eve
@@ -508,6 +509,16 @@ referer origin, a user-agent class, a rotating session id and a country. Raw eve
all. The row shape is `analytics/event.v1` (`docs/schemas/analytics-event.v1.json`), and
all. The row shape is `analytics/event.v1` (`docs/schemas/analytics-event.v1.json`), and
`require('openvibe-shared/analytics/event').checkRow(row)` checks a stored row against it. PostgreSQL retention uses `pruneRawEventsPg(db, { days })`; `retention.MAX_DAYS` is 30.
`require('openvibe-shared/analytics/event').checkRow(row)` checks a stored row against it. PostgreSQL retention uses `pruneRawEventsPg(db, { days })`; `retention.MAX_DAYS` is 30.
+`countBotApiAtIngest` (off by default) stops writing a raw row for a bot's API call. At flush the
+call increments `analytics_bot_api_hourly` and `analytics_bot_api_daily` by endpoint group (the
+first three path segments of the route template), status class (`2xx`/`3xx`/`4xx`/`5xx`/`other`) and
+bot family (the row's `bot_type`: `known_crawler`, `no_useragent`, `rate_limit`, `honeypot_hit`).
+The hour's and day's rollups add those counters to raw rows when they are computed, so a day closes
+to the same `api_calls`, `bot_hits` and `error_count` a raw-row rollup gave; the counters are totals,
+so recomputing a day never counts a call twice. Page views by bots always keep their raw row, and a
+call that cannot be counted (the bounded in-memory map is full) falls back to a raw row. The bot
+breakdown and status classes in `getStats` and `getBotAnalysis` include the counters.
+
### Browser side: two options
### Browser side: two options
1. **From the Network** (what most pages do today):
1. **From the Network** (what most pages do today):
@@ -4,7 +4,7 @@
@@ -4,7 +4,7 @@
"code": true,
"code": true,
"wave": "W2",
"wave": "W2",
"package": "openvibe-shared",
"package": "openvibe-shared",
- "version": "4.112.1",
+ "version": "4.113.0",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"deployed": false,
"deployed": false,
"deployment": "none of its own: a package released as git tags; OpenVibe.Network serves /shared/* from its pinned tag, and a site can serve its own pinned copy (openvibe-shared/serve)",
"deployment": "none of its own: a package released as git tags; OpenVibe.Network serves /shared/* from its pinned tag, and a site can serve its own pinned copy (openvibe-shared/serve)",
@@ -32,6 +32,9 @@ class AnalyticsTrackerCore {
@@ -32,6 +32,9 @@ class AnalyticsTrackerCore {
this.service = service;
this.service = service;
this._now = opts.now || Date.now;
this._now = opts.now || Date.now;
this._pathOpts = privacy.pathOptions({ paramPrefixes: opts.paramPrefixes, pathRules: opts.pathRules });
this._pathOpts = privacy.pathOptions({ paramPrefixes: opts.paramPrefixes, pathRules: opts.pathRules });
+ // A site may count bot API calls into its rollups at ingest instead of writing a raw row for each (the
+ // PostgreSQL tracker does the counting; see analytics/pg.js).
+ this._countBotApiAtIngest = !!opts.countBotApiAtIngest;
this.pruneJob = null;
this.pruneJob = null;
this._buffer = [];
this._buffer = [];
this._visitors = new Map();
this._visitors = new Map();
@@ -110,7 +113,7 @@ class AnalyticsTrackerCore {
@@ -110,7 +113,7 @@ class AnalyticsTrackerCore {
const vhash = this._visitorHash(ip, ua, day);
const vhash = this._visitorHash(ip, ua, day);
const authenticated = !!(req.user && (req.user.id != null || req.user.sub != null));
const authenticated = !!(req.user && (req.user.id != null || req.user.sub != null));
const isApi = entryPath.startsWith('/api/') || entryPath.startsWith('/internal/') || entryPath.startsWith('/oauth/');
const isApi = entryPath.startsWith('/api/') || entryPath.startsWith('/internal/') || entryPath.startsWith('/oauth/');
- this._push({
+ const event = {
service: this.service,
service: this.service,
event_type: isApi ? 'api_call' : 'pageview',
event_type: isApi ? 'api_call' : 'pageview',
path: privacy.routeTemplate(req, entryPath, this._pathOpts),
path: privacy.routeTemplate(req, entryPath, this._pathOpts),
@@ -128,10 +131,18 @@ class AnalyticsTrackerCore {
@@ -128,10 +131,18 @@ class AnalyticsTrackerCore {
os: parsed.os,
os: parsed.os,
authenticated,
authenticated,
created_at: sqlTime(nowMs),
created_at: sqlTime(nowMs),
- }, { day, hour: hourOf(nowMs), vhash, authenticated });
+ };
+ if (event.event_type === 'api_call' && event.is_bot && this._countBotApiAtIngest && this._ingestBotApi(event)) return true;
+ this._push(event, { day, hour: hourOf(nowMs), vhash, authenticated });
return true;
return true;
}
}
+ /**
+ * Count one bot API call into the tracker's rollups instead of a raw row. The PostgreSQL tracker overrides
+ * this when `countBotApiAtIngest` is on; the default writes the raw row as before. Returns whether it counted.
+ */
+ _ingestBotApi() { return false; }
+
/**
/**
* Express middleware: one row per finished request (health checks, static assets and opted-out
* Express middleware: one row per finished request (health checks, static assets and opted-out
* requests skipped).
* requests skipped).
@@ -25,8 +25,35 @@ const { AnalyticsTrackerCore, sqlTime, dayOf, hourOf } = require('./core');
@@ -25,8 +25,35 @@ const { AnalyticsTrackerCore, sqlTime, dayOf, hourOf } = require('./core');
const privacy = require('./privacy');
const privacy = require('./privacy');
const MAX_BUFFER = 5000;
const MAX_BUFFER = 5000;
+const MAX_BOT_API_KEYS = 1000; // 1000 rows x 6 values stays one INSERT; beyond it a call keeps its raw row
const DAY = 86400000;
const DAY = 86400000;
+/** The status class of a code, the same grouping getStats's statusCodes uses. */
+function statusClassOf(code) {
+ const n = Number(code);
+ if (n >= 200 && n < 300) return '2xx';
+ if (n >= 300 && n < 400) return '3xx';
+ if (n >= 400 && n < 500) return '4xx';
+ if (n >= 500) return '5xx';
+ return 'other';
+}
+
+/** The first three path segments of a route template (/api/v1/streams for /api/v1/streams/:id), or two otherwise. */
+function endpointGroupOf(path) {
+ const parts = String(path || '').split('/').filter(Boolean);
+ const n = parts[0] === 'api' || parts[0] === 'internal' || parts[0] === 'oauth' ? 3 : 2;
+ return `/${parts.slice(0, n).join('/')}` || '/';
+}
+
+/**
+ * The bot family the admin's breakdown groups by: the tracker's bot_type (known_crawler, no_useragent,
+ * rate_limit, honeypot_hit). The user-agent class stays on the raw row; it is not carried per counter.
+ */
+function botFamilyOf(botType) {
+ const s = String(botType || 'unknown');
+ return /^[a-z0-9_]{1,32}$/.test(s) ? s : 'unknown';
+}
+
/** The DDL for a service's migrations (the runtime role creates no tables). */
/** The DDL for a service's migrations (the runtime role creates no tables). */
function analyticsSchema() {
function analyticsSchema() {
return `
return `
@@ -69,6 +96,20 @@ CREATE TABLE IF NOT EXISTS analytics_daily (
@@ -69,6 +96,20 @@ CREATE TABLE IF NOT EXISTS analytics_daily (
top_paths text, top_referers text, top_countries text, device_breakdown text, browser_breakdown text,
top_paths text, top_referers text, top_countries text, device_breakdown text, browser_breakdown text,
UNIQUE (service, date)
UNIQUE (service, date)
);
);
+CREATE TABLE IF NOT EXISTS analytics_bot_api_hourly (
+ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+ service text NOT NULL, hour text NOT NULL,
+ endpoint_group text NOT NULL, status_class text NOT NULL, bot_family text NOT NULL,
+ hits bigint NOT NULL DEFAULT 0,
+ UNIQUE (service, hour, endpoint_group, status_class, bot_family)
+);
+CREATE TABLE IF NOT EXISTS analytics_bot_api_daily (
+ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+ service text NOT NULL, date text NOT NULL,
+ endpoint_group text NOT NULL, status_class text NOT NULL, bot_family text NOT NULL,
+ hits bigint NOT NULL DEFAULT 0,
+ UNIQUE (service, date, endpoint_group, status_class, bot_family)
+);
CREATE TABLE IF NOT EXISTS analytics_visitor_days (
CREATE TABLE IF NOT EXISTS analytics_visitor_days (
service text NOT NULL, day text NOT NULL, hour text NOT NULL, vhash text NOT NULL,
service text NOT NULL, day text NOT NULL, hour text NOT NULL, vhash text NOT NULL,
authenticated integer NOT NULL DEFAULT 0,
authenticated integer NOT NULL DEFAULT 0,
@@ -91,13 +132,16 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -91,13 +132,16 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
* @param {object} db openvibe-sdk/db handle (the service's own database, analyticsSchema() migrated)
* @param {object} db openvibe-sdk/db handle (the service's own database, analyticsSchema() migrated)
* @param {string} service
* @param {string} service
* @param {object} [opts] paramPrefixes, pathRules, retention ({ days = 30, botDays = days, intervalMs,
* @param {object} [opts] paramPrefixes, pathRules, retention ({ days = 30, botDays = days, intervalMs,
- * initialDelayMs } or false), timers, now
+ * initialDelayMs } or false), timers, now; countBotApiAtIngest counts bot API calls
+ * into analytics_bot_api_{hourly,daily} and the hour's/day's rollup instead of writing
+ * a raw row (page views by bots always keep theirs)
*/
*/
constructor(db, service, opts = {}) {
constructor(db, service, opts = {}) {
super(db, service, opts);
super(db, service, opts);
this._pending = []; // visitors waiting for the shared salt: { day, hour, ip, ua, authenticated }
this._pending = []; // visitors waiting for the shared salt: { day, hour, ip, ua, authenticated }
this._localKeys = new Map(); // day → process-local session key
this._localKeys = new Map(); // day → process-local session key
this._salts = new Map(); // day → shared salt (cached after the first read)
this._salts = new Map(); // day → shared salt (cached after the first read)
+ this._botApi = new Map(); // counted bot API calls waiting for the flush: hour|group|status|family → row
this._flushing = null;
this._flushing = null;
if (opts.timers !== false) {
if (opts.timers !== false) {
const every = (fn, ms) => { const t = setInterval(fn, ms); if (t.unref) t.unref(); this._timers.push(t); };
const every = (fn, ms) => { const t = setInterval(fn, ms); if (t.unref) t.unref(); this._timers.push(t); };
@@ -138,6 +182,25 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -138,6 +182,25 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
return true;
return true;
}
}
+ /** Count a bot API call into the hour's and day's counters (same transaction as the raw rows at flush). */
+ _ingestBotApi(event) {
+ const hour = `${event.created_at.slice(0, 13)}:00:00`;
+ const endpoint_group = endpointGroupOf(event.path);
+ const status_class = statusClassOf(event.status_code);
+ const bot_family = botFamilyOf(event.bot_type);
+ const key = `${hour}|${endpoint_group}|${status_class}|${bot_family}`;
+ let row = this._botApi.get(key);
+ if (!row) {
+ // Bounded: if a site ever makes more distinct combinations than this, further calls keep their raw row
+ // rather than lose their count.
+ if (this._botApi.size >= MAX_BOT_API_KEYS) return false;
+ row = { hour, endpoint_group, status_class, bot_family, hits: 0 };
+ this._botApi.set(key, row);
+ }
+ row.hits++;
+ return true;
+ }
+
_push(event) {
_push(event) {
if (this._buffer.length >= MAX_BUFFER) this._buffer.shift();
if (this._buffer.length >= MAX_BUFFER) this._buffer.shift();
this._buffer.push(event);
this._buffer.push(event);
@@ -163,9 +226,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -163,9 +226,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
}
}
async _flushNow() {
async _flushNow() {
- if (!this._buffer.length && !this._pending.length) return;
+ if (!this._buffer.length && !this._pending.length && !this._botApi.size) return;
const events = this._buffer.splice(0);
const events = this._buffer.splice(0);
const pending = this._pending.splice(0);
const pending = this._pending.splice(0);
+ const botApi = [...this._botApi.entries()];
+ this._botApi.clear();
try {
try {
const visitors = new Map();
const visitors = new Map();
for (const p of pending) {
for (const p of pending) {
@@ -195,11 +260,42 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -195,11 +260,42 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
ON CONFLICT (service, hour, vhash) DO UPDATE SET authenticated = GREATEST(analytics_visitor_days.authenticated, excluded.authenticated)`,
ON CONFLICT (service, hour, vhash) DO UPDATE SET authenticated = GREATEST(analytics_visitor_days.authenticated, excluded.authenticated)`,
[this.service, v.day, v.hour, v.vhash, v.authenticated ? 1 : 0]);
[this.service, v.day, v.hour, v.vhash, v.authenticated ? 1 : 0]);
}
}
+ if (botApi.length) {
+ // One row per key: two hours of the same day share a daily key, and one INSERT may not touch it twice.
+ const daily = new Map();
+ const hourVals = []; const hourRows = [];
+ for (const [, r] of botApi) {
+ let b = hourVals.length;
+ hourRows.push(`(${Array.from({ length: 6 }, (_, k) => `$${b + k + 1}`).join(', ')})`);
+ hourVals.push(this.service, r.hour, r.endpoint_group, r.status_class, r.bot_family, r.hits);
+ const dkey = `${r.hour.slice(0, 10)}|${r.endpoint_group}|${r.status_class}|${r.bot_family}`;
+ const d = daily.get(dkey);
+ if (d) d.hits += r.hits;
+ else daily.set(dkey, { date: r.hour.slice(0, 10), endpoint_group: r.endpoint_group, status_class: r.status_class, bot_family: r.bot_family, hits: r.hits });
+ }
+ const dayVals = []; const dayRows = [];
+ for (const r of daily.values()) {
+ let b = dayVals.length;
+ dayRows.push(`(${Array.from({ length: 6 }, (_, k) => `$${b + k + 1}`).join(', ')})`);
+ dayVals.push(this.service, r.date, r.endpoint_group, r.status_class, r.bot_family, r.hits);
+ }
+ await this.db.query(`INSERT INTO analytics_bot_api_hourly (service, hour, endpoint_group, status_class, bot_family, hits)
+ VALUES ${hourRows.join(', ')} ON CONFLICT (service, hour, endpoint_group, status_class, bot_family)
+ DO UPDATE SET hits = analytics_bot_api_hourly.hits + excluded.hits`, hourVals);
+ await this.db.query(`INSERT INTO analytics_bot_api_daily (service, date, endpoint_group, status_class, bot_family, hits)
+ VALUES ${dayRows.join(', ')} ON CONFLICT (service, date, endpoint_group, status_class, bot_family)
+ DO UPDATE SET hits = analytics_bot_api_daily.hits + excluded.hits`, dayVals);
+ }
});
});
} catch (err) {
} catch (err) {
console.error(`[Analytics:${this.service}] Flush error:`, err.message);
console.error(`[Analytics:${this.service}] Flush error:`, err.message);
this._buffer = events.concat(this._buffer).slice(-MAX_BUFFER);
this._buffer = events.concat(this._buffer).slice(-MAX_BUFFER);
this._pending = pending.concat(this._pending).slice(-MAX_BUFFER);
this._pending = pending.concat(this._pending).slice(-MAX_BUFFER);
+ for (const [key, r] of botApi) {
+ const prev = this._botApi.get(key);
+ if (prev) prev.hits += r.hits;
+ else if (this._botApi.size < MAX_BOT_API_KEYS) this._botApi.set(key, r);
+ }
}
}
}
}
@@ -215,6 +311,10 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -215,6 +311,10 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
for (const p of await this.db.many('SELECT DISTINCT day, hour FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today])) {
for (const p of await this.db.many('SELECT DISTINCT day, hour FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today])) {
hours.add(p.hour); days.add(p.day);
hours.add(p.hour); days.add(p.day);
}
}
+ // Counted bot API calls have no raw row to discover their hour from: roll up the last day's counter hours.
+ for (const p of await this.db.many('SELECT DISTINCT hour FROM analytics_bot_api_hourly WHERE service = $1 AND hour >= $2', [this.service, sqlTime(nowMs - DAY)])) {
+ hours.add(p.hour);
+ }
for (const h of hours) await this._aggregateHour(h);
for (const h of hours) await this._aggregateHour(h);
for (const d of days) await this._aggregateDay(d);
for (const d of days) await this._aggregateDay(d);
await this.db.query('DELETE FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today]);
await this.db.query('DELETE FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today]);
@@ -233,7 +333,12 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -233,7 +333,12 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,
COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,
COUNT(*) FILTER (WHERE status_code >= 400) AS error_count
COUNT(*) FILTER (WHERE status_code >= 400) AS error_count
FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3`, range);
FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3`, range);
- if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0)) return;
+ // Bot API calls counted at ingest are not raw rows: add them once here (raw + counted are disjoint).
+ const botApi = await this.db.maybe(`SELECT COALESCE(SUM(hits), 0)::bigint AS hits,
+ COALESCE(SUM(hits) FILTER (WHERE status_class IN ('4xx', '5xx')), 0)::bigint AS errors
+ FROM analytics_bot_api_hourly WHERE service = $1 AND hour = $2`, [svc, hourStart]);
+ const botHits = Number(botApi && botApi.hits || 0);
+ if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0 && botHits === 0)) return;
const v = await this.db.maybe('SELECT COUNT(*) AS visitors, COALESCE(SUM(authenticated), 0)::bigint AS users FROM analytics_visitor_days WHERE service = $1 AND hour = $2', [svc, hourStart]);
const v = await this.db.maybe('SELECT COUNT(*) AS visitors, COALESCE(SUM(authenticated), 0)::bigint AS users FROM analytics_visitor_days WHERE service = $1 AND hour = $2', [svc, hourStart]);
const prev = (await this.db.maybe('SELECT unique_visitors, unique_users FROM analytics_hourly WHERE service = $1 AND hour = $2', [svc, hourStart])) || {};
const prev = (await this.db.maybe('SELECT unique_visitors, unique_users FROM analytics_hourly WHERE service = $1 AND hour = $2', [svc, hourStart])) || {};
const topPaths = await this.db.many(`SELECT path, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3 AND is_bot = 0
const topPaths = await this.db.many(`SELECT path, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3 AND is_bot = 0
@@ -246,9 +351,9 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -246,9 +351,9 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
ON CONFLICT (service, hour) DO UPDATE SET pageviews = excluded.pageviews, api_calls = excluded.api_calls,
ON CONFLICT (service, hour) DO UPDATE SET pageviews = excluded.pageviews, api_calls = excluded.api_calls,
unique_visitors = excluded.unique_visitors, unique_users = excluded.unique_users, bot_hits = excluded.bot_hits,
unique_visitors = excluded.unique_visitors, unique_users = excluded.unique_users, bot_hits = excluded.bot_hits,
avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count, top_paths = excluded.top_paths, top_referers = excluded.top_referers`,
avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count, top_paths = excluded.top_paths, top_referers = excluded.top_referers`,
- [svc, hourStart, stats.pageviews, stats.api_calls, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
- Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), stats.bot_hits, stats.avg_response_ms || 0, stats.error_count,
- JSON.stringify(topPaths), JSON.stringify(topReferers)]);
+ [svc, hourStart, stats.pageviews, Number(stats.api_calls) + botHits, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
+ Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), Number(stats.bot_hits) + botHits, stats.avg_response_ms || 0,
+ Number(stats.error_count) + Number(botApi && botApi.errors || 0), JSON.stringify(topPaths), JSON.stringify(topReferers)]);
}
}
async _aggregateDay(date) {
async _aggregateDay(date) {
@@ -259,7 +364,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -259,7 +364,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,
COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,
COUNT(*) FILTER (WHERE status_code >= 400) AS error_count
COUNT(*) FILTER (WHERE status_code >= 400) AS error_count
FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at <= $3`, range);
FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at <= $3`, range);
- if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0)) return;
+ const botApi = await this.db.maybe(`SELECT COALESCE(SUM(hits), 0)::bigint AS hits,
+ COALESCE(SUM(hits) FILTER (WHERE status_class IN ('4xx', '5xx')), 0)::bigint AS errors
+ FROM analytics_bot_api_daily WHERE service = $1 AND date = $2`, [svc, date]);
+ const botHits = Number(botApi && botApi.hits || 0);
+ if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0 && botHits === 0)) return;
const v = await this.db.maybe(`SELECT COUNT(DISTINCT vhash) AS visitors, COUNT(DISTINCT CASE WHEN authenticated = 1 THEN vhash END) AS users
const v = await this.db.maybe(`SELECT COUNT(DISTINCT vhash) AS visitors, COUNT(DISTINCT CASE WHEN authenticated = 1 THEN vhash END) AS users
FROM analytics_visitor_days WHERE service = $1 AND day = $2`, [svc, date]);
FROM analytics_visitor_days WHERE service = $1 AND day = $2`, [svc, date]);
const prev = (await this.db.maybe('SELECT unique_visitors, unique_users, new_users FROM analytics_daily WHERE service = $1 AND date = $2', [svc, date])) || {};
const prev = (await this.db.maybe('SELECT unique_visitors, unique_users, new_users FROM analytics_daily WHERE service = $1 AND date = $2', [svc, date])) || {};
@@ -276,14 +385,28 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -276,14 +385,28 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
bot_hits = excluded.bot_hits, avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count,
bot_hits = excluded.bot_hits, avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count,
top_paths = excluded.top_paths, top_referers = excluded.top_referers, top_countries = excluded.top_countries,
top_paths = excluded.top_paths, top_referers = excluded.top_referers, top_countries = excluded.top_countries,
device_breakdown = excluded.device_breakdown, browser_breakdown = excluded.browser_breakdown`,
device_breakdown = excluded.device_breakdown, browser_breakdown = excluded.browser_breakdown`,
- [svc, date, stats.pageviews, stats.api_calls, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
+ [svc, date, stats.pageviews, Number(stats.api_calls) + botHits, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), prev.new_users != null ? prev.new_users : null,
Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), prev.new_users != null ? prev.new_users : null,
- stats.bot_hits, stats.avg_response_ms || 0, stats.error_count, JSON.stringify(topPaths), JSON.stringify(topReferers),
- JSON.stringify(topCountries), JSON.stringify(devices), JSON.stringify(browsers)]);
+ Number(stats.bot_hits) + botHits, stats.avg_response_ms || 0, Number(stats.error_count) + Number(botApi && botApi.errors || 0),
+ JSON.stringify(topPaths), JSON.stringify(topReferers), JSON.stringify(topCountries), JSON.stringify(devices), JSON.stringify(browsers)]);
}
}
// ── Reads (admin dashboards): dashboard shapes ──
// ── Reads (admin dashboards): dashboard shapes ──
+ /**
+ * Counted bot API calls for a dashboard window, as rows of endpoint group, status class, bot family and hits.
+ * The hourly table answers an hours window; the daily table answers a days window.
+ */
+ async _botApiRange({ hours, days = 30, service = this.service } = {}) {
+ if (hours) {
+ const since = `${sqlTime(this._now() - hours * 3600000).slice(0, 13)}:00:00`;
+ return this.db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits
+ FROM analytics_bot_api_hourly WHERE service = $1 AND hour >= $2 GROUP BY 1, 2, 3`, [service, since]);
+ }
+ return this.db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits
+ FROM analytics_bot_api_daily WHERE service = $1 AND date >= $2 GROUP BY 1, 2, 3`, [service, dayOf(this._now() - days * DAY)]);
+ }
+
async getStats(options = {}) {
async getStats(options = {}) {
const { days = 30, hours, service } = options;
const { days = 30, hours, service } = options;
const svc = service || this.service;
const svc = service || this.service;
@@ -339,8 +462,18 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -339,8 +462,18 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
const statusRows = await all(`SELECT is_bot, bot_type, CASE WHEN status_code >= 200 AND status_code < 300 THEN '2xx' WHEN status_code >= 300 AND status_code < 400 THEN '3xx'
const statusRows = await all(`SELECT is_bot, bot_type, CASE WHEN status_code >= 200 AND status_code < 300 THEN '2xx' WHEN status_code >= 300 AND status_code < 400 THEN '3xx'
WHEN status_code >= 400 AND status_code < 500 THEN '4xx' WHEN status_code >= 500 THEN '5xx' ELSE 'other' END AS group_code, COUNT(*) AS cnt
WHEN status_code >= 400 AND status_code < 500 THEN '4xx' WHEN status_code >= 500 THEN '5xx' ELSE 'other' END AS group_code, COUNT(*) AS cnt
FROM analytics_events WHERE service = $1 AND created_at >= $2 GROUP BY 1, 2, 3`, human);
FROM analytics_events WHERE service = $1 AND created_at >= $2 GROUP BY 1, 2, 3`, human);
- const botBreakdown = tally(statusRows.filter((r) => r.is_bot === 1), 'bot_type', 'cnt', 'hits').slice(0, 15);
- const statusCodes = tally(statusRows, 'group_code', 'cnt');
+ // Bot API calls counted at ingest have no raw row: their counters keep the admin's bot breakdown, status
+ // classes and (for a sub-day window, which reads raw rows) the summary's bot hits and errors complete.
+ const botApiRows = await this._botApiRange({ hours, days, service: svc });
+ const botApiHits = botApiRows.reduce((n, r) => n + Number(r.hits), 0);
+ const botApiErrors = botApiRows.filter((r) => r.status_class === '4xx' || r.status_class === '5xx').reduce((n, r) => n + Number(r.hits), 0);
+ if (hours && hours < 48 && summary) {
+ summary.total_bot_hits = Number(summary.total_bot_hits || 0) + botApiHits;
+ summary.total_errors = Number(summary.total_errors || 0) + botApiErrors;
+ }
+ const statusRowsAll = statusRows.concat(botApiRows.map((r) => ({ is_bot: 1, bot_type: r.bot_family, group_code: r.status_class, cnt: Number(r.hits) })));
+ const botBreakdown = tally(statusRowsAll.filter((r) => r.is_bot === 1), 'bot_type', 'cnt', 'hits').slice(0, 15);
+ const statusCodes = tally(statusRowsAll, 'group_code', 'cnt');
const dims = await all(`SELECT device_type, browser, os, country, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND is_bot = 0
const dims = await all(`SELECT device_type, browser, os, country, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND is_bot = 0
GROUP BY 1, 2, 3, 4`, human);
GROUP BY 1, 2, 3, 4`, human);
const deviceBreakdown = tally(dims, 'device_type', 'cnt');
const deviceBreakdown = tally(dims, 'device_type', 'cnt');
@@ -417,19 +550,33 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
@@ -417,19 +550,33 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
}
}
async getBotAnalysis(days = 30) {
async getBotAnalysis(days = 30) {
- const since = sqlTime(this._now() - Math.min(Math.max(1, days), 365) * DAY);
+ const windowDays = Math.min(Math.max(1, days), 365);
+ const since = sqlTime(this._now() - windowDays * DAY);
const topBotIPs = await this.db.many(`SELECT NULL AS ip, user_agent AS ua_class, bot_type, COUNT(*) AS hits, COUNT(DISTINCT session_id) AS sessions,
const topBotIPs = await this.db.many(`SELECT NULL AS ip, user_agent AS ua_class, bot_type, COUNT(*) AS hits, COUNT(DISTINCT session_id) AS sessions,
MIN(created_at) AS first_seen, MAX(created_at) AS last_seen
MIN(created_at) AS first_seen, MAX(created_at) AS last_seen
FROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY user_agent, bot_type ORDER BY hits DESC LIMIT 25`, [since]);
FROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY user_agent, bot_type ORDER BY hits DESC LIMIT 25`, [since]);
const botTrend = await this.db.many(`SELECT date, (SUM(pageviews) + SUM(api_calls))::bigint AS human_hits, SUM(bot_hits)::bigint AS bot_hits
const botTrend = await this.db.many(`SELECT date, (SUM(pageviews) + SUM(api_calls))::bigint AS human_hits, SUM(bot_hits)::bigint AS bot_hits
FROM analytics_daily WHERE date >= $1 GROUP BY date ORDER BY date ASC`, [since.slice(0, 10)]);
FROM analytics_daily WHERE date >= $1 GROUP BY date ORDER BY date ASC`, [since.slice(0, 10)]);
+ const botApi = await this._botApiRange({ days: windowDays });
const botTypes = await this.db.many(`SELECT bot_type, COUNT(*) AS hits, NULL AS unique_ips, COUNT(DISTINCT session_id) AS unique_sessions
const botTypes = await this.db.many(`SELECT bot_type, COUNT(*) AS hits, NULL AS unique_ips, COUNT(DISTINCT session_id) AS unique_sessions
FROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY bot_type ORDER BY hits DESC`, [since]);
FROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY bot_type ORDER BY hits DESC`, [since]);
+ // Counted bot API calls have no raw row: add their families to the breakdown (and keep families whose raw
+ // rows were pruned while their counters stayed).
+ const byFamily = new Map();
+ for (const r of botApi) byFamily.set(r.bot_family, (byFamily.get(r.bot_family) || 0) + Number(r.hits));
+ for (const r of botTypes) {
+ if (byFamily.has(r.bot_type)) {
+ r.hits = Number(r.hits) + byFamily.get(r.bot_type);
+ byFamily.delete(r.bot_type);
+ }
+ }
+ for (const [bot_type, hits] of byFamily) botTypes.push({ bot_type, hits, unique_ips: null, unique_sessions: null });
+ botTypes.sort((x, y) => Number(y.hits) - Number(x.hits));
const suspiciousIPs = await this.db.many(`SELECT NULL AS ip, session_id, COUNT(*) AS total_hits, COUNT(*) FILTER (WHERE status_code >= 400) AS error_hits,
const suspiciousIPs = await this.db.many(`SELECT NULL AS ip, session_id, COUNT(*) AS total_hits, COUNT(*) FILTER (WHERE status_code >= 400) AS error_hits,
COUNT(DISTINCT path) AS unique_paths, MIN(created_at) AS first_seen, MAX(created_at) AS last_seen
COUNT(DISTINCT path) AS unique_paths, MIN(created_at) AS first_seen, MAX(created_at) AS last_seen
FROM analytics_events WHERE created_at >= $1 AND is_bot = 0 AND session_id IS NOT NULL
FROM analytics_events WHERE created_at >= $1 AND is_bot = 0 AND session_id IS NOT NULL
GROUP BY session_id HAVING COUNT(*) > 500 ORDER BY total_hits DESC LIMIT 20`, [sqlTime(this._now() - 7 * DAY)]);
GROUP BY session_id HAVING COUNT(*) > 500 ORDER BY total_hits DESC LIMIT 20`, [sqlTime(this._now() - 7 * DAY)]);
- return { topBotIPs, botTrend, botTypes, suspiciousIPs };
+ return { topBotIPs, botTrend, botTypes, suspiciousIPs, botApi };
}
}
async destroy() {
async destroy() {
@@ -113,7 +113,7 @@ html[data-ov-motion=reduced] { --ov-dur-1: 0s; --ov-dur-2: 0s; --ov-dur-3: 0s; -
@@ -113,7 +113,7 @@ html[data-ov-motion=reduced] { --ov-dur-1: 0s; --ov-dur-2: 0s; --ov-dur-3: 0s; -
.ovk-icon-btn--danger:hover { color: var(--ovk-bad-text); }
.ovk-icon-btn--danger:hover { color: var(--ovk-bad-text); }
.ovk-icon-btn .ovk-i { width: 20px; height: 20px; }
.ovk-icon-btn .ovk-i { width: 20px; height: 20px; }
@media (pointer: coarse) { .ovk-icon-btn { width: var(--ovk-tap); height: var(--ovk-tap); } }
@media (pointer: coarse) { .ovk-icon-btn { width: var(--ovk-tap); height: var(--ovk-tap); } }
-.ovk-badge { display: inline-flex; align-items: center; flex: none; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--ovk-line); font-size: .7rem; font-weight: 800; letter-spacing: .03em; text-transform: uppercase; color: var(--ovk-muted); white-space: nowrap; }
+.ovk-badge { display: inline-flex; align-items: center; flex: none; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--ovk-line); font-size: .75rem; font-weight: 800; letter-spacing: .03em; text-transform: uppercase; color: var(--ovk-muted); white-space: nowrap; }
.ovk-badge--accent { color: var(--ovk-accent); border-color: color-mix(in srgb, var(--ovk-accent) 50%, transparent); }
.ovk-badge--accent { color: var(--ovk-accent); border-color: color-mix(in srgb, var(--ovk-accent) 50%, transparent); }
.ovk-badge--ok { color: var(--ovk-ok-text); border-color: color-mix(in srgb, var(--ovk-ok) 50%, transparent); }
.ovk-badge--ok { color: var(--ovk-ok-text); border-color: color-mix(in srgb, var(--ovk-ok) 50%, transparent); }
.ovk-badge--warn { color: var(--ovk-warn-text); border-color: color-mix(in srgb, var(--ovk-warn) 50%, transparent); }
.ovk-badge--warn { color: var(--ovk-warn-text); border-color: color-mix(in srgb, var(--ovk-warn) 50%, transparent); }
@@ -1,12 +1,12 @@
@@ -1,12 +1,12 @@
{
{
"name": "openvibe-shared",
"name": "openvibe-shared",
- "version": "4.112.1",
+ "version": "4.113.0",
"lockfileVersion": 3,
"lockfileVersion": 3,
"requires": true,
"requires": true,
"packages": {
"packages": {
"": {
"": {
"name": "openvibe-shared",
"name": "openvibe-shared",
- "version": "4.112.1",
+ "version": "4.113.0",
"license": "MIT",
"license": "MIT",
"dependencies": {
"dependencies": {
"jsonwebtoken": "^9.0.0",
"jsonwebtoken": "^9.0.0",
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
{
{
"name": "openvibe-shared",
"name": "openvibe-shared",
- "version": "4.112.1",
+ "version": "4.113.0",
"description": "Shared navbar, footer, themes, icons, legal pages, SEO, auth and analytics for every OpenVibe site",
"description": "Shared navbar, footer, themes, icons, legal pages, SEO, auth and analytics for every OpenVibe site",
"main": "index.js",
"main": "index.js",
"bin": {
"bin": {
@@ -86,6 +86,8 @@
@@ -86,6 +86,8 @@
* quarter of the theme's text mixed in, so it stays at 4.5:1 on the room's tinted chips: the bare accent was 3.9-4.4:1);
* quarter of the theme's text mixed in, so it stays at 4.5:1 on the room's tinted chips: the bare accent was 3.9-4.4:1);
* --room-fill and --room-on: a filled button and its text (dark text on a dark theme's bright fill, white on light). ── */
* --room-fill and --room-on: a filled button and its text (dark text on a dark theme's bright fill, white on light). ── */
.ov-room { --room-accent: var(--ovk-accent); --room-strong: color-mix(in srgb, var(--ovk-accent) 75%, var(--ovk-text)); --room-fill: var(--ovk-fill); --room-on: var(--ovk-on-fill); }
.ov-room { --room-accent: var(--ovk-accent); --room-strong: color-mix(in srgb, var(--ovk-accent) 75%, var(--ovk-text)); --room-fill: var(--ovk-fill); --room-on: var(--ovk-on-fill); }
+/* 44 px to the touch (4.113.0, the phone sweep of 11-06): a room's own links (a brand, a place) were 40.6 px tall. */
+@media (pointer: coarse) { .ov-room > a { min-height: 44px; display: inline-flex; align-items: center; } }
.ov-room[style*="--room-dark"] { --room-accent: var(--room-dark); --room-strong: color-mix(in srgb, var(--room-dark) 75%, var(--ovk-text)); --room-fill: var(--room-dark); --room-on: #0b0d12; }
.ov-room[style*="--room-dark"] { --room-accent: var(--room-dark); --room-strong: color-mix(in srgb, var(--room-dark) 75%, var(--ovk-text)); --room-fill: var(--room-dark); --room-on: #0b0d12; }
:root[data-theme-mode="light"] .ov-room[style*="--room-light"] { --room-accent: var(--room-light); --room-strong: color-mix(in srgb, var(--room-light) 75%, var(--ovk-text)); --room-fill: var(--room-light); --room-on: #fff; }
:root[data-theme-mode="light"] .ov-room[style*="--room-light"] { --room-accent: var(--room-light); --room-strong: color-mix(in srgb, var(--room-light) 75%, var(--ovk-text)); --room-fill: var(--room-light); --room-on: #fff; }
new file mode 100644
new file mode 100644
@@ -0,0 +1,109 @@
@@ -0,0 +1,109 @@
+'use strict';
+/**
+ * countBotApiAtIngest: a bot API call is counted into analytics_bot_api_{hourly,daily} at flush and added to the
+ * hour's and day's rollups instead of being written as a raw row. Page views by bots keep their raw row (the admin's
+ * suspicious-session check reads raw rows); a tracked API call by a person is unchanged. The counters give the same
+ * rollup numbers a raw-row rollup gives for the same input, and recomputing a closed day does not count them twice.
+ * With the option off nothing changes.
+ */
+const assert = require('assert');
+let createDb;
+try { ({ createDb } = require('openvibe-sdk/db')); } catch { console.log('analytics-bot-ingest: skipped (openvibe-sdk not installed)'); process.exit(0); }
+const { AnalyticsTrackerPg, analyticsSchema } = require('../analytics/pg');
+
+const DAY = 86400000;
+const T0 = Date.parse('2026-11-06T10:05:00Z');
+const BOT_UA = 'curl/8.5.0';
+const HUMAN_UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36';
+const req = (path, ua, ip = '203.0.113.10') => ({ path, method: 'GET', ip, headers: { 'user-agent': ua }, user: null });
+const res = (status = 200) => ({ statusCode: status });
+
+(async () => {
+ const db = createDb({ pglite: true, service: 'analytics-bot-ingest', log: { log() {}, warn() {}, error() {}, info() {} } });
+ await db.query(analyticsSchema());
+ let now = T0;
+ const off = new AnalyticsTrackerPg(db, 'off', { timers: false, now: () => now });
+ const on = new AnalyticsTrackerPg(db, 'on', { timers: false, now: () => now, countBotApiAtIngest: true });
+
+ const input = [
+ ['/api/v1/streams/abc/clips', BOT_UA, 200],
+ ['/api/v1/streams/abc/clips', BOT_UA, 404],
+ ['/api/v1/vods/xyz', BOT_UA, 500],
+ ['/about', BOT_UA, 200], // a bot page view keeps its raw row
+ ['/api/v1/streams/abc', HUMAN_UA, 200],
+ ];
+ for (const [path, ua, status] of input) {
+ assert.strictEqual(off.record(req(path, ua), res(status), path, 15), true);
+ assert.strictEqual(on.record(req(path, ua), res(status), path, 15), true);
+ }
+ await off.flush(); await on.flush();
+
+ const raw = async (service) => db.many('SELECT event_type, is_bot, bot_type FROM analytics_events WHERE service = $1 ORDER BY id', [service]);
+ const offRaw = await raw('off');
+ const onRaw = await raw('on');
+ assert.strictEqual(offRaw.length, 5, 'option off: every request is a raw row');
+ assert.deepStrictEqual(onRaw.map((r) => [r.event_type, r.is_bot]), [['pageview', 1], ['api_call', 0]], 'option on: the bot page view and the person\'s API call only');
+ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_events WHERE service = 'on' AND is_bot = 1 AND event_type = 'api_call'")), 0, 'no raw bot API row');
+ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_events WHERE service = 'off' AND is_bot = 1 AND event_type = 'api_call'")), 3);
+
+ const counters = await db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits
+ FROM analytics_bot_api_hourly WHERE service = 'on' GROUP BY 1, 2, 3 ORDER BY 1, 2`);
+ assert.deepStrictEqual(counters.map((r) => [r.endpoint_group, r.status_class, r.bot_family, Number(r.hits)]), [
+ ['/api/v1/streams', '2xx', 'known_crawler', 1],
+ ['/api/v1/streams', '4xx', 'known_crawler', 1],
+ ['/api/v1/vods', '5xx', 'known_crawler', 1],
+ ]);
+ const dayCounters = await db.many(`SELECT status_class, SUM(hits)::bigint AS hits FROM analytics_bot_api_daily WHERE service = 'on' GROUP BY 1 ORDER BY 1`);
+ assert.deepStrictEqual(dayCounters.map((r) => [r.status_class, Number(r.hits)]), [['2xx', 1], ['4xx', 1], ['5xx', 1]]);
+ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_bot_api_hourly WHERE service = 'off'")), 0, 'option off writes no counters');
+
+ await off.aggregate(); await on.aggregate();
+ const rollup = async (service, table, bucket, at) => db.maybe(`SELECT pageviews, api_calls, bot_hits, error_count, unique_visitors
+ FROM ${table} WHERE service = $1 AND ${bucket} = $2`, [service, at]);
+ const hourAt = '2026-11-06 10:00:00';
+ const [offHour, onHour] = [await rollup('off', 'analytics_hourly', 'hour', hourAt), await rollup('on', 'analytics_hourly', 'hour', hourAt)];
+ assert.deepStrictEqual([onHour.pageviews, onHour.api_calls, onHour.bot_hits, onHour.error_count].map(Number), [1, 4, 4, 2], 'hourly: counters give the raw rollup\'s numbers');
+ assert.deepStrictEqual([offHour.pageviews, offHour.api_calls, offHour.bot_hits, offHour.error_count].map(Number), [1, 4, 4, 2]);
+ assert.strictEqual(Number(onHour.unique_visitors), Number(offHour.unique_visitors), 'the visitor count is unchanged');
+ const [offDay, onDay] = [await rollup('off', 'analytics_daily', 'date', '2026-11-06'), await rollup('on', 'analytics_daily', 'date', '2026-11-06')];
+ assert.deepStrictEqual([onDay.pageviews, onDay.api_calls, onDay.bot_hits, onDay.error_count].map(Number), [1, 4, 4, 2], 'daily: the same');
+ assert.deepStrictEqual([offDay.pageviews, offDay.api_calls, offDay.bot_hits, offDay.error_count].map(Number), [1, 4, 4, 2]);
+ assert.strictEqual(Number(onDay.unique_visitors), Number(offDay.unique_visitors), 'the daily visitor count is unchanged');
+
+ // The admin reads: the bot breakdown, status classes and a sub-day summary stay complete from the counters.
+ const stats = await on.getStats({ hours: 6 });
+ assert.deepStrictEqual(stats.botBreakdown.map((r) => [r.bot_type, Number(r.hits)]), [['known_crawler', 4]], '3 counted API calls + the raw bot page view');
+ assert.deepStrictEqual(Object.fromEntries(stats.statusCodes.map((r) => [r.group_code, Number(r.cnt)])), { '2xx': 3, '4xx': 1, '5xx': 1 });
+ const offStats = await off.getStats({ hours: 6 });
+ for (const k of ['total_pageviews', 'total_api_calls', 'total_bot_hits', 'total_errors']) {
+ assert.strictEqual(Number(stats.summary[k]), Number(offStats.summary[k]), `summary ${k} equals the raw-row summary`);
+ }
+ // getBotAnalysis is scoped to the database (one service per production database), not by service here.
+ await db.query("DELETE FROM analytics_events WHERE service = 'off'");
+ const bots = await on.getBotAnalysis(7);
+ assert.deepStrictEqual(bots.botTypes.map((r) => [r.bot_type, Number(r.hits)]), [['known_crawler', 4]], 'raw page view + counted API calls');
+
+ // Recomputing does not double count (the rollups are read from raw rows plus the ingest counters).
+ await on.aggregate();
+ const again = await rollup('on', 'analytics_daily', 'date', '2026-11-06');
+ assert.deepStrictEqual([again.pageviews, again.api_calls, again.bot_hits, again.error_count].map(Number), [1, 4, 4, 2]);
+
+ // A day with only counted bot API calls still gets an hourly and a daily rollup (there is no raw row to find it by).
+ const only = new AnalyticsTrackerPg(db, 'only', { timers: false, now: () => now, countBotApiAtIngest: true });
+ only.record(req('/api/v1/streams/abc', BOT_UA), res(200), '/api/v1/streams/abc', 10);
+ await only.flush(); await only.aggregate();
+ const onlyHour = await rollup('only', 'analytics_hourly', 'hour', hourAt);
+ assert.deepStrictEqual([onlyHour.pageviews, onlyHour.api_calls, onlyHour.bot_hits].map(Number), [0, 1, 1]);
+
+ // Day close: the next day's aggregation recomputes yesterday from raw rows plus counters, once.
+ now = T0 + DAY;
+ await on.aggregate();
+ await on.aggregate();
+ const closed = await rollup('on', 'analytics_daily', 'date', '2026-11-06');
+ assert.deepStrictEqual([closed.pageviews, closed.api_calls, closed.bot_hits, closed.error_count].map(Number), [1, 4, 4, 2], 'no double count at day close');
+ const countersAfter = await db.many(`SELECT status_class, SUM(hits)::bigint AS hits FROM analytics_bot_api_daily WHERE service = 'on' GROUP BY 1 ORDER BY 1`);
+ assert.deepStrictEqual(countersAfter.map((r) => [r.status_class, Number(r.hits)]), [['2xx', 1], ['4xx', 1], ['5xx', 1]], 'counters are totals, not accumulated again');
+
+ await off.destroy(); await on.destroy(); await only.destroy(); await db.close();
+ console.log('analytics-bot-ingest: all checks passed');
+})().catch((err) => { console.error(err); process.exit(1); });