openvibers/openvibe.shared Commits

4.113.0: count bot API calls at ingest, not as raw rows (#294)

OP OpenVibers committed 7 days ago · 3e58dcaaa0 · 10 files, +327 −22

* 4.113.0: count bot API calls at ingest

* 4.113.0 also: the kit badge at 12 px and a room's links 44 px tall on touch

From the fleet phone sweep: .ovk-badge was 0.7rem (11.2 px, Food's /boxes and wherever the kit badges), and a room's
own links (.ov-room > a) were 40.6 px tall. Carried in this release so the fleet takes one pin round.

Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex 788cf37..cfbe4bb 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -1,5 +1,19 @@
# Changelog
++## 4.113.0 — 2026-11-06
++
++- **Bot API calls can be counted at ingest, not stored row by row.** `AnalyticsTrackerPg` takes
++ `countBotApiAtIngest` (off by default): a bot's API call increments `analytics_bot_api_hourly` and
++ `analytics_bot_api_daily` by endpoint group, status class and bot family at flush instead of writing a raw
++ `analytics_events` row. The hourly and daily rollups add those counters to raw rows, so a day's `api_calls`,
++ `bot_hits` and `error_count` are what the raw rows gave before, and a day recomputed at close counts each call
++ once. The admin's bot breakdown, status classes and sub-day summary include the counters; a raw page view by a
++ bot still keeps its row, and the option off changes nothing. New tables are expand-only: `analyticsSchema()`
++ creates them for a new install, and an existing service adds them in a migration.
++
++- **Phone minimums in the kit** (the fleet phone sweep of 11-05/06): `.ovk-badge` is 12 px (was 11.2 px), and a room's
++ own links (`.ov-room > a`) are 44 px tall on touch screens (were 40.6 px).
++
## 4.112.1 — 2026-11-05
- **The shared chrome meets the phone minimums** (the fleet phone sweep of 11-05, 322 pages at 390 px). Nothing
@@@ -778,6 +792,17 @@ From a phone sweep of every site's home (11-02):
## Unreleased
++## 4.113.0 — 2026-11-06
++
++- **Bot API calls can be counted at ingest, not stored row by row.** `AnalyticsTrackerPg` takes
++ `countBotApiAtIngest` (off by default): a bot's API call increments `analytics_bot_api_hourly` and
++ `analytics_bot_api_daily` by endpoint group, status class and bot family at flush instead of writing a raw
++ `analytics_events` row. The hourly and daily rollups add those counters to raw rows, so a day's `api_calls`,
++ `bot_hits` and `error_count` are what the raw rows gave before, and a day recomputed at close counts each call
++ once. The admin's bot breakdown, status classes and sub-day summary include the counters; a raw page view by a
++ bot still keeps its row, and the option off changes nothing. New tables are expand-only: `analyticsSchema()`
++ creates them for a new install, and an existing service adds them in a migration.
++
## 4.67.0 — 2026-10-31
- **Share this look** (plan L45 step 4). The Look panel has a Share button beside "saved to your account". It copies
ddiff --git a/README.md b/README.md
iindex 1d3e56e..96d003a 100644
---- a/README.md
++++ b/README.md
@@@ -2,7 +2,7 @@
> Versioned UI, the OpenVibe Frame (the navbar, footer and "shipped" views every site sits in), SEO, legal and release-client packages every OpenVibe site renders.
--**Status:** alpha, v4.112.1 (every release is a tag; see [CHANGELOG.md](CHANGELOG.md)). Every deployed
++**Status:** alpha, v4.113.0 (every release is a tag; see [CHANGELOG.md](CHANGELOG.md)). Every deployed
consumer installs a tagged release and none keeps a vendored copy
([docs/migration-plan.md](docs/migration-plan.md) is done); each repository's `package.json` names
the tag it pins.
@@@ -497,6 +497,7 @@ const analytics = new AnalyticsTrackerPg(db, 'live', {
retention: { days: 30 },
paramPrefixes: ['avatar'],
pathRules: [[/\/by-username\/[^/?#]+/gi, '/by-username/:username']],
++ countBotApiAtIngest: true, // optional: bot API calls go into the rollups, not one raw row each
});
app.use(analytics.middleware());
// A service may call pruneRawEventsPg(db, { days: 30 }) in its own job runner.
@@@ -508,6 +509,16 @@ referer origin, a user-agent class, a rotating session id and a country. Raw eve
all. The row shape is `analytics/event.v1` (`docs/schemas/analytics-event.v1.json`), and
`require('openvibe-shared/analytics/event').checkRow(row)` checks a stored row against it. PostgreSQL retention uses `pruneRawEventsPg(db, { days })`; `retention.MAX_DAYS` is 30.
++`countBotApiAtIngest` (off by default) stops writing a raw row for a bot's API call. At flush the
++call increments `analytics_bot_api_hourly` and `analytics_bot_api_daily` by endpoint group (the
++first three path segments of the route template), status class (`2xx`/`3xx`/`4xx`/`5xx`/`other`) and
++bot family (the row's `bot_type`: `known_crawler`, `no_useragent`, `rate_limit`, `honeypot_hit`).
++The hour's and day's rollups add those counters to raw rows when they are computed, so a day closes
++to the same `api_calls`, `bot_hits` and `error_count` a raw-row rollup gave; the counters are totals,
++so recomputing a day never counts a call twice. Page views by bots always keep their raw row, and a
++call that cannot be counted (the bounded in-memory map is full) falls back to a raw row. The bot
++breakdown and status classes in `getStats` and `getBotAnalysis` include the counters.
++
### Browser side: two options
1. **From the Network** (what most pages do today):
ddiff --git a/STATUS.json b/STATUS.json
iindex 53551ce..06f1c3f 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -4,7 +4,7 @@
"code": true,
"wave": "W2",
"package": "openvibe-shared",
-- "version": "4.112.1",
++ "version": "4.113.0",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026)",
"deployed": false,
"deployment": "none of its own: a package released as git tags; OpenVibe.Network serves /shared/* from its pinned tag, and a site can serve its own pinned copy (openvibe-shared/serve)",
ddiff --git a/analytics/core.js b/analytics/core.js
iindex 0f33075..b3e77ab 100644
---- a/analytics/core.js
++++ b/analytics/core.js
@@@ -32,6 +32,9 @@ class AnalyticsTrackerCore {
this.service = service;
this._now = opts.now || Date.now;
this._pathOpts = privacy.pathOptions({ paramPrefixes: opts.paramPrefixes, pathRules: opts.pathRules });
++ // A site may count bot API calls into its rollups at ingest instead of writing a raw row for each (the
++ // PostgreSQL tracker does the counting; see analytics/pg.js).
++ this._countBotApiAtIngest = !!opts.countBotApiAtIngest;
this.pruneJob = null;
this._buffer = [];
this._visitors = new Map();
@@@ -110,7 +113,7 @@ class AnalyticsTrackerCore {
const vhash = this._visitorHash(ip, ua, day);
const authenticated = !!(req.user && (req.user.id != null || req.user.sub != null));
const isApi = entryPath.startsWith('/api/') || entryPath.startsWith('/internal/') || entryPath.startsWith('/oauth/');
-- this._push({
++ const event = {
service: this.service,
event_type: isApi ? 'api_call' : 'pageview',
path: privacy.routeTemplate(req, entryPath, this._pathOpts),
@@@ -128,10 +131,18 @@ class AnalyticsTrackerCore {
os: parsed.os,
authenticated,
created_at: sqlTime(nowMs),
-- }, { day, hour: hourOf(nowMs), vhash, authenticated });
++ };
++ if (event.event_type === 'api_call' && event.is_bot && this._countBotApiAtIngest && this._ingestBotApi(event)) return true;
++ this._push(event, { day, hour: hourOf(nowMs), vhash, authenticated });
return true;
}
++ /**
++ * Count one bot API call into the tracker's rollups instead of a raw row. The PostgreSQL tracker overrides
++ * this when `countBotApiAtIngest` is on; the default writes the raw row as before. Returns whether it counted.
++ */
++ _ingestBotApi() { return false; }
++
/**
* Express middleware: one row per finished request (health checks, static assets and opted-out
* requests skipped).
ddiff --git a/analytics/pg.js b/analytics/pg.js
iindex 28c2d6a..5463cfd 100644
---- a/analytics/pg.js
++++ b/analytics/pg.js
@@@ -25,8 +25,35 @@ const { AnalyticsTrackerCore, sqlTime, dayOf, hourOf } = require('./core');
const privacy = require('./privacy');
const MAX_BUFFER = 5000;
++const MAX_BOT_API_KEYS = 1000; // 1000 rows x 6 values stays one INSERT; beyond it a call keeps its raw row
const DAY = 86400000;
++/** The status class of a code, the same grouping getStats's statusCodes uses. */
++function statusClassOf(code) {
++ const n = Number(code);
++ if (n >= 200 && n < 300) return '2xx';
++ if (n >= 300 && n < 400) return '3xx';
++ if (n >= 400 && n < 500) return '4xx';
++ if (n >= 500) return '5xx';
++ return 'other';
++}
++
++/** The first three path segments of a route template (/api/v1/streams for /api/v1/streams/:id), or two otherwise. */
++function endpointGroupOf(path) {
++ const parts = String(path || '').split('/').filter(Boolean);
++ const n = parts[0] === 'api' || parts[0] === 'internal' || parts[0] === 'oauth' ? 3 : 2;
++ return `/${parts.slice(0, n).join('/')}` || '/';
++}
++
++/**
++ * The bot family the admin's breakdown groups by: the tracker's bot_type (known_crawler, no_useragent,
++ * rate_limit, honeypot_hit). The user-agent class stays on the raw row; it is not carried per counter.
++ */
++function botFamilyOf(botType) {
++ const s = String(botType || 'unknown');
++ return /^[a-z0-9_]{1,32}$/.test(s) ? s : 'unknown';
++}
++
/** The DDL for a service's migrations (the runtime role creates no tables). */
function analyticsSchema() {
return `
@@@ -69,6 +96,20 @@ CREATE TABLE IF NOT EXISTS analytics_daily (
top_paths text, top_referers text, top_countries text, device_breakdown text, browser_breakdown text,
UNIQUE (service, date)
);
++CREATE TABLE IF NOT EXISTS analytics_bot_api_hourly (
++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
++ service text NOT NULL, hour text NOT NULL,
++ endpoint_group text NOT NULL, status_class text NOT NULL, bot_family text NOT NULL,
++ hits bigint NOT NULL DEFAULT 0,
++ UNIQUE (service, hour, endpoint_group, status_class, bot_family)
++);
++CREATE TABLE IF NOT EXISTS analytics_bot_api_daily (
++ id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
++ service text NOT NULL, date text NOT NULL,
++ endpoint_group text NOT NULL, status_class text NOT NULL, bot_family text NOT NULL,
++ hits bigint NOT NULL DEFAULT 0,
++ UNIQUE (service, date, endpoint_group, status_class, bot_family)
++);
CREATE TABLE IF NOT EXISTS analytics_visitor_days (
service text NOT NULL, day text NOT NULL, hour text NOT NULL, vhash text NOT NULL,
authenticated integer NOT NULL DEFAULT 0,
@@@ -91,13 +132,16 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
* @param {object} db openvibe-sdk/db handle (the service's own database, analyticsSchema() migrated)
* @param {string} service
* @param {object} [opts] paramPrefixes, pathRules, retention ({ days = 30, botDays = days, intervalMs,
-- * initialDelayMs } or false), timers, now
++ * initialDelayMs } or false), timers, now; countBotApiAtIngest counts bot API calls
++ * into analytics_bot_api_{hourly,daily} and the hour's/day's rollup instead of writing
++ * a raw row (page views by bots always keep theirs)
*/
constructor(db, service, opts = {}) {
super(db, service, opts);
this._pending = []; // visitors waiting for the shared salt: { day, hour, ip, ua, authenticated }
this._localKeys = new Map(); // day → process-local session key
this._salts = new Map(); // day → shared salt (cached after the first read)
++ this._botApi = new Map(); // counted bot API calls waiting for the flush: hour|group|status|family → row
this._flushing = null;
if (opts.timers !== false) {
const every = (fn, ms) => { const t = setInterval(fn, ms); if (t.unref) t.unref(); this._timers.push(t); };
@@@ -138,6 +182,25 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
return true;
}
++ /** Count a bot API call into the hour's and day's counters (same transaction as the raw rows at flush). */
++ _ingestBotApi(event) {
++ const hour = `${event.created_at.slice(0, 13)}:00:00`;
++ const endpoint_group = endpointGroupOf(event.path);
++ const status_class = statusClassOf(event.status_code);
++ const bot_family = botFamilyOf(event.bot_type);
++ const key = `${hour}|${endpoint_group}|${status_class}|${bot_family}`;
++ let row = this._botApi.get(key);
++ if (!row) {
++ // Bounded: if a site ever makes more distinct combinations than this, further calls keep their raw row
++ // rather than lose their count.
++ if (this._botApi.size >= MAX_BOT_API_KEYS) return false;
++ row = { hour, endpoint_group, status_class, bot_family, hits: 0 };
++ this._botApi.set(key, row);
++ }
++ row.hits++;
++ return true;
++ }
++
_push(event) {
if (this._buffer.length >= MAX_BUFFER) this._buffer.shift();
this._buffer.push(event);
@@@ -163,9 +226,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
}
async _flushNow() {
-- if (!this._buffer.length && !this._pending.length) return;
++ if (!this._buffer.length && !this._pending.length && !this._botApi.size) return;
const events = this._buffer.splice(0);
const pending = this._pending.splice(0);
++ const botApi = [...this._botApi.entries()];
++ this._botApi.clear();
try {
const visitors = new Map();
for (const p of pending) {
@@@ -195,11 +260,42 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
ON CONFLICT (service, hour, vhash) DO UPDATE SET authenticated = GREATEST(analytics_visitor_days.authenticated, excluded.authenticated)`,
[this.service, v.day, v.hour, v.vhash, v.authenticated ? 1 : 0]);
}
++ if (botApi.length) {
++ // One row per key: two hours of the same day share a daily key, and one INSERT may not touch it twice.
++ const daily = new Map();
++ const hourVals = []; const hourRows = [];
++ for (const [, r] of botApi) {
++ let b = hourVals.length;
++ hourRows.push(`(${Array.from({ length: 6 }, (_, k) => `$${b + k + 1}`).join(', ')})`);
++ hourVals.push(this.service, r.hour, r.endpoint_group, r.status_class, r.bot_family, r.hits);
++ const dkey = `${r.hour.slice(0, 10)}|${r.endpoint_group}|${r.status_class}|${r.bot_family}`;
++ const d = daily.get(dkey);
++ if (d) d.hits += r.hits;
++ else daily.set(dkey, { date: r.hour.slice(0, 10), endpoint_group: r.endpoint_group, status_class: r.status_class, bot_family: r.bot_family, hits: r.hits });
++ }
++ const dayVals = []; const dayRows = [];
++ for (const r of daily.values()) {
++ let b = dayVals.length;
++ dayRows.push(`(${Array.from({ length: 6 }, (_, k) => `$${b + k + 1}`).join(', ')})`);
++ dayVals.push(this.service, r.date, r.endpoint_group, r.status_class, r.bot_family, r.hits);
++ }
++ await this.db.query(`INSERT INTO analytics_bot_api_hourly (service, hour, endpoint_group, status_class, bot_family, hits)
++ VALUES ${hourRows.join(', ')} ON CONFLICT (service, hour, endpoint_group, status_class, bot_family)
++ DO UPDATE SET hits = analytics_bot_api_hourly.hits + excluded.hits`, hourVals);
++ await this.db.query(`INSERT INTO analytics_bot_api_daily (service, date, endpoint_group, status_class, bot_family, hits)
++ VALUES ${dayRows.join(', ')} ON CONFLICT (service, date, endpoint_group, status_class, bot_family)
++ DO UPDATE SET hits = analytics_bot_api_daily.hits + excluded.hits`, dayVals);
++ }
});
} catch (err) {
console.error(`[Analytics:${this.service}] Flush error:`, err.message);
this._buffer = events.concat(this._buffer).slice(-MAX_BUFFER);
this._pending = pending.concat(this._pending).slice(-MAX_BUFFER);
++ for (const [key, r] of botApi) {
++ const prev = this._botApi.get(key);
++ if (prev) prev.hits += r.hits;
++ else if (this._botApi.size < MAX_BOT_API_KEYS) this._botApi.set(key, r);
++ }
}
}
@@@ -215,6 +311,10 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
for (const p of await this.db.many('SELECT DISTINCT day, hour FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today])) {
hours.add(p.hour); days.add(p.day);
}
++ // Counted bot API calls have no raw row to discover their hour from: roll up the last day's counter hours.
++ for (const p of await this.db.many('SELECT DISTINCT hour FROM analytics_bot_api_hourly WHERE service = $1 AND hour >= $2', [this.service, sqlTime(nowMs - DAY)])) {
++ hours.add(p.hour);
++ }
for (const h of hours) await this._aggregateHour(h);
for (const d of days) await this._aggregateDay(d);
await this.db.query('DELETE FROM analytics_visitor_days WHERE service = $1 AND day < $2', [this.service, today]);
@@@ -233,7 +333,12 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,
COUNT(*) FILTER (WHERE status_code >= 400) AS error_count
FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3`, range);
-- if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0)) return;
++ // Bot API calls counted at ingest are not raw rows: add them once here (raw + counted are disjoint).
++ const botApi = await this.db.maybe(`SELECT COALESCE(SUM(hits), 0)::bigint AS hits,
++ COALESCE(SUM(hits) FILTER (WHERE status_class IN ('4xx', '5xx')), 0)::bigint AS errors
++ FROM analytics_bot_api_hourly WHERE service = $1 AND hour = $2`, [svc, hourStart]);
++ const botHits = Number(botApi && botApi.hits || 0);
++ if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0 && botHits === 0)) return;
const v = await this.db.maybe('SELECT COUNT(*) AS visitors, COALESCE(SUM(authenticated), 0)::bigint AS users FROM analytics_visitor_days WHERE service = $1 AND hour = $2', [svc, hourStart]);
const prev = (await this.db.maybe('SELECT unique_visitors, unique_users FROM analytics_hourly WHERE service = $1 AND hour = $2', [svc, hourStart])) || {};
const topPaths = await this.db.many(`SELECT path, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at < $3 AND is_bot = 0
@@@ -246,9 +351,9 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
ON CONFLICT (service, hour) DO UPDATE SET pageviews = excluded.pageviews, api_calls = excluded.api_calls,
unique_visitors = excluded.unique_visitors, unique_users = excluded.unique_users, bot_hits = excluded.bot_hits,
avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count, top_paths = excluded.top_paths, top_referers = excluded.top_referers`,
-- [svc, hourStart, stats.pageviews, stats.api_calls, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
-- Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), stats.bot_hits, stats.avg_response_ms || 0, stats.error_count,
-- JSON.stringify(topPaths), JSON.stringify(topReferers)]);
++ [svc, hourStart, stats.pageviews, Number(stats.api_calls) + botHits, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
++ Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), Number(stats.bot_hits) + botHits, stats.avg_response_ms || 0,
++ Number(stats.error_count) + Number(botApi && botApi.errors || 0), JSON.stringify(topPaths), JSON.stringify(topReferers)]);
}
async _aggregateDay(date) {
@@@ -259,7 +364,11 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
COUNT(*) FILTER (WHERE is_bot = 1) AS bot_hits, CAST(AVG(response_time_ms) AS integer) AS avg_response_ms,
COUNT(*) FILTER (WHERE status_code >= 400) AS error_count
FROM analytics_events WHERE service = $1 AND created_at >= $2 AND created_at <= $3`, range);
-- if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0)) return;
++ const botApi = await this.db.maybe(`SELECT COALESCE(SUM(hits), 0)::bigint AS hits,
++ COALESCE(SUM(hits) FILTER (WHERE status_class IN ('4xx', '5xx')), 0)::bigint AS errors
++ FROM analytics_bot_api_daily WHERE service = $1 AND date = $2`, [svc, date]);
++ const botHits = Number(botApi && botApi.hits || 0);
++ if (!stats || (Number(stats.pageviews) === 0 && Number(stats.api_calls) === 0 && botHits === 0)) return;
const v = await this.db.maybe(`SELECT COUNT(DISTINCT vhash) AS visitors, COUNT(DISTINCT CASE WHEN authenticated = 1 THEN vhash END) AS users
FROM analytics_visitor_days WHERE service = $1 AND day = $2`, [svc, date]);
const prev = (await this.db.maybe('SELECT unique_visitors, unique_users, new_users FROM analytics_daily WHERE service = $1 AND date = $2', [svc, date])) || {};
@@@ -276,14 +385,28 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
bot_hits = excluded.bot_hits, avg_response_ms = excluded.avg_response_ms, error_count = excluded.error_count,
top_paths = excluded.top_paths, top_referers = excluded.top_referers, top_countries = excluded.top_countries,
device_breakdown = excluded.device_breakdown, browser_breakdown = excluded.browser_breakdown`,
-- [svc, date, stats.pageviews, stats.api_calls, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
++ [svc, date, stats.pageviews, Number(stats.api_calls) + botHits, Math.max(Number(prev.unique_visitors || 0), Number(v.visitors || 0)),
Math.max(Number(prev.unique_users || 0), Number(v.users || 0)), prev.new_users != null ? prev.new_users : null,
-- stats.bot_hits, stats.avg_response_ms || 0, stats.error_count, JSON.stringify(topPaths), JSON.stringify(topReferers),
-- JSON.stringify(topCountries), JSON.stringify(devices), JSON.stringify(browsers)]);
++ Number(stats.bot_hits) + botHits, stats.avg_response_ms || 0, Number(stats.error_count) + Number(botApi && botApi.errors || 0),
++ JSON.stringify(topPaths), JSON.stringify(topReferers), JSON.stringify(topCountries), JSON.stringify(devices), JSON.stringify(browsers)]);
}
// ── Reads (admin dashboards): dashboard shapes ──
++ /**
++ * Counted bot API calls for a dashboard window, as rows of endpoint group, status class, bot family and hits.
++ * The hourly table answers an hours window; the daily table answers a days window.
++ */
++ async _botApiRange({ hours, days = 30, service = this.service } = {}) {
++ if (hours) {
++ const since = `${sqlTime(this._now() - hours * 3600000).slice(0, 13)}:00:00`;
++ return this.db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits
++ FROM analytics_bot_api_hourly WHERE service = $1 AND hour >= $2 GROUP BY 1, 2, 3`, [service, since]);
++ }
++ return this.db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits
++ FROM analytics_bot_api_daily WHERE service = $1 AND date >= $2 GROUP BY 1, 2, 3`, [service, dayOf(this._now() - days * DAY)]);
++ }
++
async getStats(options = {}) {
const { days = 30, hours, service } = options;
const svc = service || this.service;
@@@ -339,8 +462,18 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
const statusRows = await all(`SELECT is_bot, bot_type, CASE WHEN status_code >= 200 AND status_code < 300 THEN '2xx' WHEN status_code >= 300 AND status_code < 400 THEN '3xx'
WHEN status_code >= 400 AND status_code < 500 THEN '4xx' WHEN status_code >= 500 THEN '5xx' ELSE 'other' END AS group_code, COUNT(*) AS cnt
FROM analytics_events WHERE service = $1 AND created_at >= $2 GROUP BY 1, 2, 3`, human);
-- const botBreakdown = tally(statusRows.filter((r) => r.is_bot === 1), 'bot_type', 'cnt', 'hits').slice(0, 15);
-- const statusCodes = tally(statusRows, 'group_code', 'cnt');
++ // Bot API calls counted at ingest have no raw row: their counters keep the admin's bot breakdown, status
++ // classes and (for a sub-day window, which reads raw rows) the summary's bot hits and errors complete.
++ const botApiRows = await this._botApiRange({ hours, days, service: svc });
++ const botApiHits = botApiRows.reduce((n, r) => n + Number(r.hits), 0);
++ const botApiErrors = botApiRows.filter((r) => r.status_class === '4xx' || r.status_class === '5xx').reduce((n, r) => n + Number(r.hits), 0);
++ if (hours && hours < 48 && summary) {
++ summary.total_bot_hits = Number(summary.total_bot_hits || 0) + botApiHits;
++ summary.total_errors = Number(summary.total_errors || 0) + botApiErrors;
++ }
++ const statusRowsAll = statusRows.concat(botApiRows.map((r) => ({ is_bot: 1, bot_type: r.bot_family, group_code: r.status_class, cnt: Number(r.hits) })));
++ const botBreakdown = tally(statusRowsAll.filter((r) => r.is_bot === 1), 'bot_type', 'cnt', 'hits').slice(0, 15);
++ const statusCodes = tally(statusRowsAll, 'group_code', 'cnt');
const dims = await all(`SELECT device_type, browser, os, country, COUNT(*) AS cnt FROM analytics_events WHERE service = $1 AND created_at >= $2 AND is_bot = 0
GROUP BY 1, 2, 3, 4`, human);
const deviceBreakdown = tally(dims, 'device_type', 'cnt');
@@@ -417,19 +550,33 @@ class AnalyticsTrackerPg extends AnalyticsTrackerCore {
}
async getBotAnalysis(days = 30) {
-- const since = sqlTime(this._now() - Math.min(Math.max(1, days), 365) * DAY);
++ const windowDays = Math.min(Math.max(1, days), 365);
++ const since = sqlTime(this._now() - windowDays * DAY);
const topBotIPs = await this.db.many(`SELECT NULL AS ip, user_agent AS ua_class, bot_type, COUNT(*) AS hits, COUNT(DISTINCT session_id) AS sessions,
MIN(created_at) AS first_seen, MAX(created_at) AS last_seen
FROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY user_agent, bot_type ORDER BY hits DESC LIMIT 25`, [since]);
const botTrend = await this.db.many(`SELECT date, (SUM(pageviews) + SUM(api_calls))::bigint AS human_hits, SUM(bot_hits)::bigint AS bot_hits
FROM analytics_daily WHERE date >= $1 GROUP BY date ORDER BY date ASC`, [since.slice(0, 10)]);
++ const botApi = await this._botApiRange({ days: windowDays });
const botTypes = await this.db.many(`SELECT bot_type, COUNT(*) AS hits, NULL AS unique_ips, COUNT(DISTINCT session_id) AS unique_sessions
FROM analytics_events WHERE is_bot = 1 AND created_at >= $1 GROUP BY bot_type ORDER BY hits DESC`, [since]);
++ // Counted bot API calls have no raw row: add their families to the breakdown (and keep families whose raw
++ // rows were pruned while their counters stayed).
++ const byFamily = new Map();
++ for (const r of botApi) byFamily.set(r.bot_family, (byFamily.get(r.bot_family) || 0) + Number(r.hits));
++ for (const r of botTypes) {
++ if (byFamily.has(r.bot_type)) {
++ r.hits = Number(r.hits) + byFamily.get(r.bot_type);
++ byFamily.delete(r.bot_type);
++ }
++ }
++ for (const [bot_type, hits] of byFamily) botTypes.push({ bot_type, hits, unique_ips: null, unique_sessions: null });
++ botTypes.sort((x, y) => Number(y.hits) - Number(x.hits));
const suspiciousIPs = await this.db.many(`SELECT NULL AS ip, session_id, COUNT(*) AS total_hits, COUNT(*) FILTER (WHERE status_code >= 400) AS error_hits,
COUNT(DISTINCT path) AS unique_paths, MIN(created_at) AS first_seen, MAX(created_at) AS last_seen
FROM analytics_events WHERE created_at >= $1 AND is_bot = 0 AND session_id IS NOT NULL
GROUP BY session_id HAVING COUNT(*) > 500 ORDER BY total_hits DESC LIMIT 20`, [sqlTime(this._now() - 7 * DAY)]);
-- return { topBotIPs, botTrend, botTypes, suspiciousIPs };
++ return { topBotIPs, botTrend, botTypes, suspiciousIPs, botApi };
}
async destroy() {
ddiff --git a/kit.css b/kit.css
iindex 1e4ba70..bcb4ba6 100644
---- a/kit.css
++++ b/kit.css
@@@ -113,7 +113,7 @@ html[data-ov-motion=reduced] { --ov-dur-1: 0s; --ov-dur-2: 0s; --ov-dur-3: 0s; -
.ovk-icon-btn--danger:hover { color: var(--ovk-bad-text); }
.ovk-icon-btn .ovk-i { width: 20px; height: 20px; }
@media (pointer: coarse) { .ovk-icon-btn { width: var(--ovk-tap); height: var(--ovk-tap); } }
--.ovk-badge { display: inline-flex; align-items: center; flex: none; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--ovk-line); font-size: .7rem; font-weight: 800; letter-spacing: .03em; text-transform: uppercase; color: var(--ovk-muted); white-space: nowrap; }
++.ovk-badge { display: inline-flex; align-items: center; flex: none; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--ovk-line); font-size: .75rem; font-weight: 800; letter-spacing: .03em; text-transform: uppercase; color: var(--ovk-muted); white-space: nowrap; }
.ovk-badge--accent { color: var(--ovk-accent); border-color: color-mix(in srgb, var(--ovk-accent) 50%, transparent); }
.ovk-badge--ok { color: var(--ovk-ok-text); border-color: color-mix(in srgb, var(--ovk-ok) 50%, transparent); }
.ovk-badge--warn { color: var(--ovk-warn-text); border-color: color-mix(in srgb, var(--ovk-warn) 50%, transparent); }
ddiff --git a/package-lock.json b/package-lock.json
iindex be1e018..c7de92c 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -1,12 +1,12 @@
{
"name": "openvibe-shared",
-- "version": "4.112.1",
++ "version": "4.113.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "openvibe-shared",
-- "version": "4.112.1",
++ "version": "4.113.0",
"license": "MIT",
"dependencies": {
"jsonwebtoken": "^9.0.0",
ddiff --git a/package.json b/package.json
iindex 2eec489..cd95c31 100644
---- a/package.json
++++ b/package.json
@@@ -1,6 +1,6 @@
{
"name": "openvibe-shared",
-- "version": "4.112.1",
++ "version": "4.113.0",
"description": "Shared navbar, footer, themes, icons, legal pages, SEO, auth and analytics for every OpenVibe site",
"main": "index.js",
"bin": {
ddiff --git a/room.css b/room.css
iindex 1a1210a..a9a5614 100644
---- a/room.css
++++ b/room.css
@@@ -86,6 +86,8 @@
* quarter of the theme's text mixed in, so it stays at 4.5:1 on the room's tinted chips: the bare accent was 3.9-4.4:1);
* --room-fill and --room-on: a filled button and its text (dark text on a dark theme's bright fill, white on light). ── */
.ov-room { --room-accent: var(--ovk-accent); --room-strong: color-mix(in srgb, var(--ovk-accent) 75%, var(--ovk-text)); --room-fill: var(--ovk-fill); --room-on: var(--ovk-on-fill); }
++/* 44 px to the touch (4.113.0, the phone sweep of 11-06): a room's own links (a brand, a place) were 40.6 px tall. */
++@media (pointer: coarse) { .ov-room > a { min-height: 44px; display: inline-flex; align-items: center; } }
.ov-room[style*="--room-dark"] { --room-accent: var(--room-dark); --room-strong: color-mix(in srgb, var(--room-dark) 75%, var(--ovk-text)); --room-fill: var(--room-dark); --room-on: #0b0d12; }
:root[data-theme-mode="light"] .ov-room[style*="--room-light"] { --room-accent: var(--room-light); --room-strong: color-mix(in srgb, var(--room-light) 75%, var(--ovk-text)); --room-fill: var(--room-light); --room-on: #fff; }
ddiff --git a/test/analytics-bot-ingest.test.js b/test/analytics-bot-ingest.test.js
nnew file mode 100644
iindex 0000000..f377f53
---- /dev/null
++++ b/test/analytics-bot-ingest.test.js
@@@ -0,0 +1,109 @@
++'use strict';
++/**
++ * countBotApiAtIngest: a bot API call is counted into analytics_bot_api_{hourly,daily} at flush and added to the
++ * hour's and day's rollups instead of being written as a raw row. Page views by bots keep their raw row (the admin's
++ * suspicious-session check reads raw rows); a tracked API call by a person is unchanged. The counters give the same
++ * rollup numbers a raw-row rollup gives for the same input, and recomputing a closed day does not count them twice.
++ * With the option off nothing changes.
++ */
++const assert = require('assert');
++let createDb;
++try { ({ createDb } = require('openvibe-sdk/db')); } catch { console.log('analytics-bot-ingest: skipped (openvibe-sdk not installed)'); process.exit(0); }
++const { AnalyticsTrackerPg, analyticsSchema } = require('../analytics/pg');
++
++const DAY = 86400000;
++const T0 = Date.parse('2026-11-06T10:05:00Z');
++const BOT_UA = 'curl/8.5.0';
++const HUMAN_UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36';
++const req = (path, ua, ip = '203.0.113.10') => ({ path, method: 'GET', ip, headers: { 'user-agent': ua }, user: null });
++const res = (status = 200) => ({ statusCode: status });
++
++(async () => {
++ const db = createDb({ pglite: true, service: 'analytics-bot-ingest', log: { log() {}, warn() {}, error() {}, info() {} } });
++ await db.query(analyticsSchema());
++ let now = T0;
++ const off = new AnalyticsTrackerPg(db, 'off', { timers: false, now: () => now });
++ const on = new AnalyticsTrackerPg(db, 'on', { timers: false, now: () => now, countBotApiAtIngest: true });
++
++ const input = [
++ ['/api/v1/streams/abc/clips', BOT_UA, 200],
++ ['/api/v1/streams/abc/clips', BOT_UA, 404],
++ ['/api/v1/vods/xyz', BOT_UA, 500],
++ ['/about', BOT_UA, 200], // a bot page view keeps its raw row
++ ['/api/v1/streams/abc', HUMAN_UA, 200],
++ ];
++ for (const [path, ua, status] of input) {
++ assert.strictEqual(off.record(req(path, ua), res(status), path, 15), true);
++ assert.strictEqual(on.record(req(path, ua), res(status), path, 15), true);
++ }
++ await off.flush(); await on.flush();
++
++ const raw = async (service) => db.many('SELECT event_type, is_bot, bot_type FROM analytics_events WHERE service = $1 ORDER BY id', [service]);
++ const offRaw = await raw('off');
++ const onRaw = await raw('on');
++ assert.strictEqual(offRaw.length, 5, 'option off: every request is a raw row');
++ assert.deepStrictEqual(onRaw.map((r) => [r.event_type, r.is_bot]), [['pageview', 1], ['api_call', 0]], 'option on: the bot page view and the person\'s API call only');
++ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_events WHERE service = 'on' AND is_bot = 1 AND event_type = 'api_call'")), 0, 'no raw bot API row');
++ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_events WHERE service = 'off' AND is_bot = 1 AND event_type = 'api_call'")), 3);
++
++ const counters = await db.many(`SELECT endpoint_group, status_class, bot_family, SUM(hits)::bigint AS hits
++ FROM analytics_bot_api_hourly WHERE service = 'on' GROUP BY 1, 2, 3 ORDER BY 1, 2`);
++ assert.deepStrictEqual(counters.map((r) => [r.endpoint_group, r.status_class, r.bot_family, Number(r.hits)]), [
++ ['/api/v1/streams', '2xx', 'known_crawler', 1],
++ ['/api/v1/streams', '4xx', 'known_crawler', 1],
++ ['/api/v1/vods', '5xx', 'known_crawler', 1],
++ ]);
++ const dayCounters = await db.many(`SELECT status_class, SUM(hits)::bigint AS hits FROM analytics_bot_api_daily WHERE service = 'on' GROUP BY 1 ORDER BY 1`);
++ assert.deepStrictEqual(dayCounters.map((r) => [r.status_class, Number(r.hits)]), [['2xx', 1], ['4xx', 1], ['5xx', 1]]);
++ assert.strictEqual(Number(await db.value("SELECT COUNT(*) FROM analytics_bot_api_hourly WHERE service = 'off'")), 0, 'option off writes no counters');
++
++ await off.aggregate(); await on.aggregate();
++ const rollup = async (service, table, bucket, at) => db.maybe(`SELECT pageviews, api_calls, bot_hits, error_count, unique_visitors
++ FROM ${table} WHERE service = $1 AND ${bucket} = $2`, [service, at]);
++ const hourAt = '2026-11-06 10:00:00';
++ const [offHour, onHour] = [await rollup('off', 'analytics_hourly', 'hour', hourAt), await rollup('on', 'analytics_hourly', 'hour', hourAt)];
++ assert.deepStrictEqual([onHour.pageviews, onHour.api_calls, onHour.bot_hits, onHour.error_count].map(Number), [1, 4, 4, 2], 'hourly: counters give the raw rollup\'s numbers');
++ assert.deepStrictEqual([offHour.pageviews, offHour.api_calls, offHour.bot_hits, offHour.error_count].map(Number), [1, 4, 4, 2]);
++ assert.strictEqual(Number(onHour.unique_visitors), Number(offHour.unique_visitors), 'the visitor count is unchanged');
++ const [offDay, onDay] = [await rollup('off', 'analytics_daily', 'date', '2026-11-06'), await rollup('on', 'analytics_daily', 'date', '2026-11-06')];
++ assert.deepStrictEqual([onDay.pageviews, onDay.api_calls, onDay.bot_hits, onDay.error_count].map(Number), [1, 4, 4, 2], 'daily: the same');
++ assert.deepStrictEqual([offDay.pageviews, offDay.api_calls, offDay.bot_hits, offDay.error_count].map(Number), [1, 4, 4, 2]);
++ assert.strictEqual(Number(onDay.unique_visitors), Number(offDay.unique_visitors), 'the daily visitor count is unchanged');
++
++ // The admin reads: the bot breakdown, status classes and a sub-day summary stay complete from the counters.
++ const stats = await on.getStats({ hours: 6 });
++ assert.deepStrictEqual(stats.botBreakdown.map((r) => [r.bot_type, Number(r.hits)]), [['known_crawler', 4]], '3 counted API calls + the raw bot page view');
++ assert.deepStrictEqual(Object.fromEntries(stats.statusCodes.map((r) => [r.group_code, Number(r.cnt)])), { '2xx': 3, '4xx': 1, '5xx': 1 });
++ const offStats = await off.getStats({ hours: 6 });
++ for (const k of ['total_pageviews', 'total_api_calls', 'total_bot_hits', 'total_errors']) {
++ assert.strictEqual(Number(stats.summary[k]), Number(offStats.summary[k]), `summary ${k} equals the raw-row summary`);
++ }
++ // getBotAnalysis is scoped to the database (one service per production database), not by service here.
++ await db.query("DELETE FROM analytics_events WHERE service = 'off'");
++ const bots = await on.getBotAnalysis(7);
++ assert.deepStrictEqual(bots.botTypes.map((r) => [r.bot_type, Number(r.hits)]), [['known_crawler', 4]], 'raw page view + counted API calls');
++
++ // Recomputing does not double count (the rollups are read from raw rows plus the ingest counters).
++ await on.aggregate();
++ const again = await rollup('on', 'analytics_daily', 'date', '2026-11-06');
++ assert.deepStrictEqual([again.pageviews, again.api_calls, again.bot_hits, again.error_count].map(Number), [1, 4, 4, 2]);
++
++ // A day with only counted bot API calls still gets an hourly and a daily rollup (there is no raw row to find it by).
++ const only = new AnalyticsTrackerPg(db, 'only', { timers: false, now: () => now, countBotApiAtIngest: true });
++ only.record(req('/api/v1/streams/abc', BOT_UA), res(200), '/api/v1/streams/abc', 10);
++ await only.flush(); await only.aggregate();
++ const onlyHour = await rollup('only', 'analytics_hourly', 'hour', hourAt);
++ assert.deepStrictEqual([onlyHour.pageviews, onlyHour.api_calls, onlyHour.bot_hits].map(Number), [0, 1, 1]);
++
++ // Day close: the next day's aggregation recomputes yesterday from raw rows plus counters, once.
++ now = T0 + DAY;
++ await on.aggregate();
++ await on.aggregate();
++ const closed = await rollup('on', 'analytics_daily', 'date', '2026-11-06');
++ assert.deepStrictEqual([closed.pageviews, closed.api_calls, closed.bot_hits, closed.error_count].map(Number), [1, 4, 4, 2], 'no double count at day close');
++ const countersAfter = await db.many(`SELECT status_class, SUM(hits)::bigint AS hits FROM analytics_bot_api_daily WHERE service = 'on' GROUP BY 1 ORDER BY 1`);
++ assert.deepStrictEqual(countersAfter.map((r) => [r.status_class, Number(r.hits)]), [['2xx', 1], ['4xx', 1], ['5xx', 1]], 'counters are totals, not accumulated again');
++
++ await off.destroy(); await on.destroy(); await only.destroy(); await db.close();
++ console.log('analytics-bot-ingest: all checks passed');
++})().catch((err) => { console.error(err); process.exit(1); });