openvibers/openvibe.search Commits

Notify saved searches of new matches through Network behind a flag

OP OpenVibers committed 4 Oct 2026 · de3bc9f19e · 14 files, +605 −22

Asked: T9 step 1 — J9 — OpenVibe.Search: saved-search notifications through Network (dossier /home/workstation/openvibe/agents/ds/prep/T9.md section "T9 step 1", attached by the plan tag). ONE pull request from current origin/main (git fetch first; read with `git show origin/main:<path>`, the local checko…
Harness job search-t9-step-1-j9-195816 (plan T9).

Changes

ddiff --git a/.env.example b/.env.example
iindex 488a902..6b70120 100644
---- a/.env.example
++++ b/.env.example
@@@ -22,7 +22,8 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
#SEARCH_USER_AUDIENCES=openvibe.search,openvibe.network
# This service's own principal (svc:search) — publishes search.document.* to Events
--# (events.event.publish) and creates its subscription (events.subscription.manage).
++# (events.event.publish) and creates its subscription (events.subscription.manage); with
++# SEARCH_SAVED_NOTIFY=1 also resolves people and pushes notifications on Network.
OV_OAUTH_CLIENT_ID=search
OV_OAUTH_CLIENT_SECRET=
@@@ -55,6 +56,12 @@ VALKEY_PREFIX=
# Saved searches per signed-in person.
#SEARCH_SAVED_MAX_PER_SUBJECT=50
++# Notifications of new matches through OpenVibe.Network (README "Saved searches"): off unless 1, and
++# then only with OV_OAUTH_CLIENT_SECRET above. Network must grant the principal `search`
++# network.notifications.push (own service `search`) and identity.subject.resolve.
++#SEARCH_SAVED_NOTIFY=1
++#SEARCH_SAVED_NOTIFY_INTERVAL_MS=900000
++#SEARCH_SAVED_NOTIFY_BATCH=100
# Per-actor limits (server/actor-limits.js): requests one caller (a service principal, a signed-in
# person, else an address) may make to a read route per minute and per hour; past them 429
ddiff --git a/README.md b/README.md
iindex 6e78feb..36e93f2 100644
---- a/README.md
++++ b/README.md
@@@ -111,9 +111,23 @@ query again returns the existing one (renamed if a new name is given). Only the
every run is a fresh query as that person at that moment, so what they lost access to (or what was
deleted) is gone and what they gained appears. Someone else's saved search is the same 404 as a
missing one; guests cannot save; a cookie-authenticated POST/DELETE must carry this origin's
--`Origin`. Notifications of new matches wait for OpenVibe.Network's notifications (`last_run_at` is
--kept for that job), deferred to plan tracks T9/T11; the notifying job will need the
--`network.notifications.push` capability. Signing in on `search.openvibe.network` itself needs a
++`Origin`.
++
++**Notifications of new matches** ([server/saved-notifier.js](server/saved-notifier.js)) are off
++unless `SEARCH_SAVED_NOTIFY=1`; they also need `OV_OAUTH_CLIENT_SECRET` (the Network OAuth client
++`search`, client credentials) and stay off without it, which `/api/ready` reports as
++`saved_notify: "off (…)"`, never as a failure. Every `SEARCH_SAVED_NOTIFY_INTERVAL_MS` (15 min) the
++notifier takes up to `SEARCH_SAVED_NOTIFY_BATCH` (100) saved searches, longest waiting first, and runs
++each as its owner for documents indexed (a new revision) after its watermark, `last_run_at` (else
++`created_at`). A search with new hits gets one notification through OpenVibe.Network (type
++`SEARCH_SAVED_MATCH`, service `search`, category `service`: how many, the top title, a link to the
++search page with that query); the person is resolved to Network's user id first
++(`GET /internal/identity/resolve`, then `POST /internal/notifications/push`). The watermark moves to
++the tick's start only once the push went out (or there was nothing new, or Network no longer knows
++the person): a failed push is retried on the next tick and a delivered batch is never notified
++twice. Opening the results moves the watermark too, so what the person has seen is not announced.
++Network must grant the principal `search` `network.notifications.push` (for its own service) and
++`identity.subject.resolve`. Signing in on `search.openvibe.network` itself needs a
Network OAuth client (`search`), which does not exist yet: today saved searches are used through
Bearer tokens or a product acting for its visitor.
@@@ -210,6 +224,8 @@ Called elsewhere, as the service principal `search` (client credentials from Ope
|---|---|---|
| OpenVibe.Events | `events.event.publish` (audience `openvibe.events`) | the outbox relays `search.document.indexed` / `search.document.removed` |
| OpenVibe.Events | `events.subscription.manage` (`npm run subscribe`, once) | the HMAC-signed subscription to `*.index_document.*` that delivers index events |
++| OpenVibe.Network | `identity.subject.resolve` (audience `openvibe.network`) | saved-search notifications: a `usr_` subject to Network's user id (only with `SEARCH_SAVED_NOTIFY=1`) |
++| OpenVibe.Network | `network.notifications.push` (audience `openvibe.network`, own service `search`) | saved-search notifications: one notification per saved search with new hits |
Released in `openvibe-contracts` v0.76.0 with the service manifest (this repo pins v0.49.0);
[server/auth.js](server/auth.js) decides them with the contracts grant rule, and CI runs
@@@ -251,6 +267,9 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea
- freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`)
- saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits
(`test/saved-searches.test.js`)
++- saved-search notifications: one push per search with new hits, as its owner, the watermark only
++ after a push, a failed push retried, never twice, off without the flag and the secret
++ (`test/saved-notify.test.js`, `test/network-push.test.js`)
- the search page: public only, escaped, no-store, noindex results, paging (`test/page.test.js`)
- per-actor limits: 429 `rate_limited` with Retry-After per principal, person or address, another
caller unaffected, deliveries and health never limited (`test/actor-limits.test.js`)
@@@ -279,7 +298,7 @@ Reporting: [SECURITY.md](SECURITY.md). The rules the code keeps:
results never carry the ACL, and every response is `no-store`. Saved searches store the query only.
- **Network exposure.** The owner API, `/internal/events` and document writes are loopback-only in the
vhost; `/metrics` answers direct loopback callers only; client addresses come from `$remote_addr`.
--- **Egress.** Search calls only OpenVibe.Network (keys, tokens), OpenVibe.Events (`EVENTS_URL`) and,
++- **Egress.** Search calls only OpenVibe.Network (keys, tokens, and with `SEARCH_SAVED_NOTIFY=1` identity resolve and notification push), OpenVibe.Events (`EVENTS_URL`) and,
when `CLOUDFLARE_PURGE_TOKEN` is set, the Cloudflare purge API. It never fetches a URL a user chose.
- **Secrets.** `OV_OAUTH_CLIENT_SECRET`, `SEARCH_EVENTS_SECRET` and `CLOUDFLARE_PURGE_TOKEN` come from
the environment (`/etc/openvibe/search.env`, 0600) and are never stored or logged. Event deliveries
ddiff --git a/server/api/query.js b/server/api/query.js
iindex 0443fd6..3350aec 100644
---- a/server/api/query.js
++++ b/server/api/query.js
@@@ -115,14 +115,16 @@ function canView(viewer, row, doc) {
/**
* One search as a viewer: the query API, saved-search runs and the HTML page all go through here.
-- * run({ text, query, viewer, limit, cursor, facetKeys, now }) → { results, next_cursor, facets? }
-- * `query` is the query-string-shaped filter source (owner, type, lang, facet.<key>). Throws
-- * QueryError for a malformed request.
++ * run({ text, query, viewer, limit, cursor, facetKeys, now, since }) → { results, next_cursor, facets? }
++ * `query` is the query-string-shaped filter source (owner, type, lang, facet.<key>). `since` (epoch ms,
++ * internal callers only: the saved-search notifier) keeps documents indexed after it; the public
++ * API never passes it. Throws QueryError for a malformed request.
*/
function createSearcher({ config, store, engine, now: clock = () => Date.now() }) {
-- async function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock() }) {
++ async function run({ text = '', query = {}, filters = null, viewer, limit, cursor, facetKeys = [], now = clock(), since = null }) {
if (text.length > 500) throw new QueryError('search.bad_query', 'q is longer than 500 characters');
filters = filters || parseFilters(query, config);
++ if (since != null) filters = { ...filters, since: Number(since) };
limit = Math.min(Math.max(parseInt(limit, 10) || config.query.defaultLimit, 1), config.query.maxLimit);
if (facetKeys.length > 5 || facetKeys.some(k => !FACET_KEY_RE.test(k))) throw new QueryError('search.bad_query', 'facets: at most 5 well-formed keys');
ddiff --git a/server/app.js b/server/app.js
iindex 32aa268..7923d40 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -14,7 +14,7 @@ const { webhookRouter } = require('./api/webhook');
const { createLimits } = require('./actor-limits');
const pkg = require('../package.json');
--function createApp({ config, db, store, engine, auth, outbox, relay, purges, purger, saved, log = console, now }) {
++function createApp({ config, db, store, engine, auth, outbox, relay, purges, purger, saved, searcher: givenSearcher = null, notifier = null, log = console, now }) {
const app = express();
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
@@@ -46,12 +46,12 @@ function createApp({ config, db, store, engine, auth, outbox, relay, purges, pur
// Readiness (openvibe-shared/ready): 503 only when the database (documents and full-text tables)
// fails; the Network key and index consistency are optional and degrade it (see observability.js).
-- const readiness = createSearchReadiness({ db, config, engine, store, outbox, relay, purges, purger, saved, release: release.release });
++ const readiness = createSearchReadiness({ db, config, engine, store, outbox, relay, purges, purger, saved, notifier, release: release.release });
app.get('/api/ready', readiness.handler);
// GET /release.json (ADR-016) and POST /release-metrics (open tabs' update reports into /metrics).
release.mount(app, { registry: metrics.registry });
-- const searcher = createSearcher({ config, store, engine, now });
++ const searcher = givenSearcher || createSearcher({ config, store, engine, now });
// One per-actor limiter for the API routes below (server/actor-limits.js); the Events webhook,
// health, ready, release.json and metrics above are never limited.
const limits = createLimits({ config, now: now || (() => Date.now()), registry: metrics.registry, log });
ddiff --git a/server/config.js b/server/config.js
iindex a45d7ab..5f9d38d 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -98,6 +98,11 @@ function load(env = process.env) {
savedSearches: {
maxPerSubject: Math.max(1, int(env.SEARCH_SAVED_MAX_PER_SUBJECT, 50)),
++ // Notifications of new matches through OpenVibe.Network (server/saved-notifier.js): off unless
++ // SEARCH_SAVED_NOTIFY=1, and then only with OV_OAUTH_CLIENT_SECRET (the Network OAuth client `search`).
++ notify: env.SEARCH_SAVED_NOTIFY === '1',
++ notifyIntervalMs: Math.max(1000, int(env.SEARCH_SAVED_NOTIFY_INTERVAL_MS, 15 * 60 * 1000)),
++ notifyBatch: Math.max(1, int(env.SEARCH_SAVED_NOTIFY_BATCH, 100)),
},
// Per-actor limits at the API routes (server/actor-limits.js, roadmap WS-R task 4): the requests
ddiff --git a/server/engine/pg.js b/server/engine/pg.js
iindex 1c310a7..6f81744 100644
---- a/server/engine/pg.js
++++ b/server/engine/pg.js
@@@ -115,6 +115,8 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
if (filters.owner) { parts.push('d.owner = @f_owner'); params.f_owner = filters.owner; }
if (filters.type) { parts.push('d.type = @f_type'); params.f_type = filters.type; }
if (filters.language) { parts.push('d.language = @f_lang'); params.f_lang = filters.language; }
++ // Internal only (saved-search notifications): documents indexed (a new revision) after `since`, epoch ms.
++ if (filters.since != null) { parts.push('d.indexed_at > @f_since::bigint'); params.f_since = Number(filters.since); }
(filters.facets || []).forEach(([key, values], i) => {
params[`f_fk${i}`] = key;
params[`f_fv${i}`] = values.map(String);
ddiff --git a/server/index.js b/server/index.js
iindex 4cec4a3..ab06726 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -17,9 +17,12 @@ const { gracefulStop } = require('openvibe-sdk/service');
const { createAuth } = require('./auth');
const { createPurgeQueue, createPurger } = require('./purge');
const { createSavedSearches } = require('./saved');
++const { createSearcher } = require('./api/query');
++const { createNetworkPush } = require('./network-push');
++const { createSavedNotifier } = require('./saved-notifier');
const { createApp } = require('./app');
--async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, log = console, listen = true } = {}) {
++async function start({ config, db: givenDb = null, now = () => Date.now(), fetchImpl = globalThis.fetch, tokenClient, networkPush = null, log = console, listen = true } = {}) {
config = config || load();
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
const db = givenDb || await openDb(config, { log });
@@@ -42,7 +45,13 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
const jwksUrl = `${config.networkInternalUrl}/api/.well-known/jwks`;
const jwks = jwksClient(jwksUrl, { log, fetch: fetchImpl });
const auth = createAuth({ config, jwks, log });
-- const app = createApp({ config, db, store, engine, auth, outbox, relay, purges, purger, saved, log, now });
++ const searcher = createSearcher({ config, store, engine, now });
++ // Saved-search notifications through Network: off unless SEARCH_SAVED_NOTIFY=1 with the `search` client's secret.
++ const notifier = createSavedNotifier({
++ config, saved, searcher, now, log,
++ network: networkPush || createNetworkPush({ config, fetchImpl }),
++ });
++ const app = createApp({ config, db, store, engine, auth, outbox, relay, purges, purger, saved, searcher, notifier, log, now });
// The background refresher is not started under the test runtime (the tests stub the fetch and load
// the keys once below); a real boot refreshes on an unref'd timer. keyLoaded settles the first load.
@@@ -50,6 +59,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
const keyLoaded = jwks.keys().catch(() => null);
relay.start();
purger.start();
++ notifier.start();
const pruneTimer = setInterval(async () => { try { await outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
pruneTimer.unref?.();
@@@ -67,6 +77,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
jwks.stop();
await relay.stop();
await purger.stop();
++ await notifier.stop();
if (server) {
server.closeAllConnections?.();
await new Promise(resolve => server.close(() => resolve()));
@@@ -74,15 +85,15 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
if (!givenDb) await db.close();
}
-- return { config, db, engine, store, outbox, relay, purges, purger, saved, jwks, keyLoaded, auth, app, server, close };
++ return { config, db, engine, store, outbox, relay, purges, purger, saved, searcher, notifier, jwks, keyLoaded, auth, app, server, close };
}
if (require.main === module) {
require('dotenv').config();
start().then((handles) => {
// SIGTERM/SIGINT (openvibe-sdk/service, docs/service.md's handles family): requests in flight get 8 s,
-- // then handles.close() (the prune timer and JWKS refresher stopped, the relay and purger stopped, the
-- // server closed, the database closed; a rejection exits 1); past 10 s the process exits 1.
++ // then handles.close() (the prune timer and JWKS refresher stopped, the relay, purger and saved-search
++ // notifier stopped, the server closed, the database closed; a rejection exits 1); past 10 s the process exits 1.
gracefulStop({ name: 'search', server: handles.server, handles, drainMs: 8000, deadlineMs: 10000 });
}).catch((err) => {
console.error(`[search] failed to start: ${err.stack || err}`);
ddiff --git a/server/network-push.js b/server/network-push.js
nnew file mode 100644
iindex 0000000..58463eb
---- /dev/null
++++ b/server/network-push.js
@@@ -0,0 +1,76 @@
++'use strict';
++/**
++ * OpenVibe.Network client for saved-search notifications (server/saved-notifier.js):
++ *
++ * resolve(subject) GET /internal/identity/resolve?subject_id=usr_… identity.subject.resolve
++ * → Network's numeric user id, or null (no such person, or a deleted account)
++ * push(subject, notification) resolve, then POST /internal/notifications/push network.notifications.push
++ * (network.notification-push-request@1) → { sent: true, skipped } | { sent: false, reason }
++ *
++ * Client-credentials tokens for audience openvibe.network from the Network OAuth client `search`
++ * (config.oauth), one per capability; a 401 drops the cached token so the next call fetches a new one.
++ * Any other refusal throws a NetworkError carrying the status only: tokens, the client secret and
++ * Network's answer bodies are never logged.
++ */
++const { serviceAuth } = require('openvibe-contracts');
++
++class NetworkError extends Error {
++ constructor(message, status = null) {
++ super(message);
++ this.name = 'NetworkError';
++ this.status = status;
++ }
++}
++
++/**
++ * createNetworkPush({ config, fetchImpl?, tokenClient?, timeoutMs? }). `tokenClient` (tests) replaces
++ * both token clients.
++ */
++function createNetworkPush({ config, fetchImpl = globalThis.fetch, tokenClient = null, timeoutMs = 5000 }) {
++ const base = config.networkInternalUrl;
++ const tokens = (scope) => tokenClient || serviceAuth.createTokenClient({
++ tokenUrl: `${base}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,
++ audience: 'openvibe.network', scope, fetchImpl, timeoutMs,
++ });
++ const resolveTokens = tokens('identity.subject.resolve');
++ const pushTokens = tokens('network.notifications.push');
++
++ async function call(what, tokenSource, path, body) {
++ let res;
++ try {
++ res = await fetchImpl(`${base}${path}`, {
++ method: body ? 'POST' : 'GET',
++ headers: { Accept: 'application/json', ...(body ? { 'Content-Type': 'application/json' } : {}), ...(await tokenSource.authHeaders()) },
++ body: body ? JSON.stringify(body) : undefined,
++ signal: AbortSignal.timeout(timeoutMs),
++ });
++ } catch (err) {
++ throw new NetworkError(`${what}: ${err.name === 'TimeoutError' ? `no answer within ${timeoutMs} ms` : err.message}`, err.status || null);
++ }
++ if (res.status === 401) tokenSource.invalidate?.();
++ return res;
++ }
++
++ async function resolve(subject) {
++ const res = await call('identity resolve', resolveTokens, `/internal/identity/resolve?subject_id=${encodeURIComponent(subject)}`);
++ if (res.status === 404) return null;
++ if (!res.ok) throw new NetworkError(`identity resolve: Network answered ${res.status}`, res.status);
++ const body = await res.json().catch(() => null);
++ if (!body || body.deleted || body.network_user_id == null) return null;
++ return body.network_user_id;
++ }
++
++ async function push(subject, notification) {
++ const userId = await resolve(subject);
++ if (userId == null) return { sent: false, reason: 'unknown_subject' };
++ const res = await call('notification push', pushTokens, '/internal/notifications/push', { ...notification, user_id: userId });
++ if (!res.ok) throw new NetworkError(`notification push: Network answered ${res.status}`, res.status);
++ const body = await res.json().catch(() => ({}));
++ // skipped: the person switched the category off; Network made its decision, so it counts as delivered.
++ return { sent: true, skipped: Boolean(body && body.skipped) };
++ }
++
++ return { resolve, push };
++}
++
++module.exports = { createNetworkPush, NetworkError };
ddiff --git a/server/observability.js b/server/observability.js
iindex d75b813..1a67b0f 100644
---- a/server/observability.js
++++ b/server/observability.js
@@@ -35,7 +35,7 @@ function readers(db, engine) {
};
}
--function createSearchReadiness({ db, config, engine, store, outbox, relay, purges = null, purger = null, saved = null, release = null }) {
++function createSearchReadiness({ db, config, engine, store, outbox, relay, purges = null, purger = null, saved = null, notifier = null, release = null }) {
const read = readers(db, engine);
return createReadiness({
service: 'search',
@@@ -91,6 +91,8 @@ function createSearchReadiness({ db, config, engine, store, outbox, relay, purge
...await purges.cdnCounts(),
} : null,
saved_searches: dbOk && saved ? await saved.total() : null,
++ // Off by default (SEARCH_SAVED_NOTIFY); off is a setting, never a failure.
++ saved_notify: notifier ? notifier.state() : null,
freshness: config.freshness,
};
},
ddiff --git a/server/saved-notifier.js b/server/saved-notifier.js
nnew file mode 100644
iindex 0000000..d785a43
---- /dev/null
++++ b/server/saved-notifier.js
@@@ -0,0 +1,121 @@
++'use strict';
++/**
++ * Saved-search notifications (README "Saved searches"), off unless SEARCH_SAVED_NOTIFY=1 and the
++ * Network OAuth client `search` is configured. Every notifyIntervalMs it takes up to notifyBatch saved
++ * searches, longest without a run first (server/saved.js due()), and runs each as its owner
++ * (viewer = the usr_ subject, so the ACL applies at that moment) for documents indexed after its
++ * watermark (searcher `since`). A search with new hits gets ONE notification through OpenVibe.Network
++ * (server/network-push.js): how many, the top title and a link to the search page with that query.
++ *
++ * The watermark is last_run_at, else created_at: a person opening the results moves it too, so what
++ * they have seen is not announced again. It advances to the tick's start only when the push went
++ * out, there was nothing new, or the person no longer exists on Network. A failed push (or run)
++ * leaves it, so the same hits are retried on the next tick; a delivered batch is never notified twice.
++ *
++ * createSavedNotifier({ config, saved, searcher, network, now?, log? }) → { start, stop, tick, running, state }
++ */
++
++const MAX_TITLE = 120;
++const clip = (s, n) => (s.length > n ? `${s.slice(0, n - 1)}…` : s);
++
++/** Why the notifier stays off for this config, or null when it may run. */
++function disabledReason(config) {
++ if (!config.savedSearches.notify) return 'off (SEARCH_SAVED_NOTIFY unset)';
++ if (!config.networkInternalUrl) return 'off (OV_NETWORK_INTERNAL_URL unset)';
++ if (!config.oauth.clientSecret) return 'off (OV_OAUTH_CLIENT_SECRET unset)';
++ return null;
++}
++
++/**
++ * The search page with the saved query and filters, in the query API's parameter names (the page,
++ * server/web/page.js, applies q, owner and type today).
++ */
++function pageUrl(baseUrl, s) {
++ const p = new URLSearchParams();
++ if (s.q) p.set('q', s.q);
++ if (s.filters.owner) p.set('owner', s.filters.owner);
++ if (s.filters.type) p.set('type', s.filters.type);
++ if (s.filters.language) p.set('lang', s.filters.language);
++ for (const [k, vs] of s.filters.facets || []) for (const v of vs) p.append(`facet.${k}`, v);
++ const qs = p.toString();
++ return `${baseUrl}/${qs ? `?${qs}` : ''}`;
++}
++
++function notificationFor(config, s, body) {
++ const count = `${body.results.length}${body.next_cursor ? '+' : ''}`;
++ const top = body.results[0].title || 'Untitled';
++ return {
++ type: 'SEARCH_SAVED_MATCH',
++ service: 'search',
++ category: 'service',
++ priority: 'normal',
++ icon: '🔍',
++ title: clip(`New results for “${s.name}”`, MAX_TITLE),
++ message: body.results.length === 1 && !body.next_cursor ? clip(top, 300) : clip(`${count} new results, including “${top}”`, 300),
++ url: pageUrl(config.baseUrl, s),
++ rich_content: { saved_search_id: s.id, count: body.results.length, more: Boolean(body.next_cursor) },
++ };
++}
++
++function createSavedNotifier({ config, saved, searcher, network, now = () => Date.now(), log = console }) {
++ const off = disabledReason(config);
++ let timer = null;
++ let ticking = null;
++
++ async function doTick() {
++ const at = now();
++ const out = { checked: 0, notified: 0, unchanged: 0, unknown: 0, failed: 0 };
++ for (const s of await saved.due(config.savedSearches.notifyBatch)) {
++ out.checked++;
++ try {
++ const body = await searcher.run({
++ text: s.q,
++ filters: { ...s.filters, facets: s.filters.facets || [] },
++ viewer: { subject: s.subject },
++ limit: config.query.maxLimit,
++ since: s.watermark,
++ now: at,
++ });
++ if (body.results.length) {
++ const sent = await network.push(s.subject, notificationFor(config, s, body));
++ if (sent.sent) out.notified++; else out.unknown++;
++ } else {
++ out.unchanged++;
++ }
++ await saved.advance(s.id, at);
++ } catch (err) {
++ out.failed++;
++ if (out.failed === 1) log.warn(`[saved-notify] ${s.id}: ${err.message}`);
++ }
++ }
++ if (out.failed) log.warn(`[saved-notify] ${out.failed} saved search(es) not notified yet; will retry`);
++ return out;
++ }
++
++ /** One pass (overlapping calls share it). */
++ function tick() {
++ if (!ticking) ticking = doTick().finally(() => { ticking = null; });
++ return ticking;
++ }
++
++ function start() {
++ if (timer || off) return;
++ timer = setInterval(() => { tick().catch(err => log.warn(`[saved-notify] ${err.message}`)); }, config.savedSearches.notifyIntervalMs);
++ timer.unref?.();
++ }
++
++ function stop() {
++ if (timer) clearInterval(timer);
++ timer = null;
++ return ticking ? ticking.catch(() => {}) : Promise.resolve();
++ }
++
++ return {
++ start, stop, tick,
++ running: () => Boolean(timer),
++ /** For readiness: 'running', 'stopped', or why it is off. */
++ state: () => off || (timer ? 'running' : 'stopped'),
++ };
++}
++
++module.exports = { createSavedNotifier, disabledReason, pageUrl };
ddiff --git a/server/saved.js b/server/saved.js
iindex 0a249a5..ddb660a 100644
---- a/server/saved.js
++++ b/server/saved.js
@@@ -5,8 +5,8 @@
* that person at that moment, so a document that became private or was deleted since the save
* is simply not there, and one they gained access to is.
*
-- * Notifications for new matches wait for OpenVibe.Network's notifications (README "Saved
-- * searches"); last_run_at is kept so that job can tell what is new.
++ * last_run_at is also the watermark of the saved-search notifier (server/saved-notifier.js): what was
++ * indexed after it is new.
*/
const crypto = require('crypto');
const { ids } = require('openvibe-contracts');
@@@ -41,6 +41,9 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }
rename: db.prepare('UPDATE saved_searches SET name = ?, updated_at = ? WHERE id = ?'),
del: db.prepare('DELETE FROM saved_searches WHERE id = ? AND subject = ?'),
ran: db.prepare('UPDATE saved_searches SET last_run_at = ? WHERE id = ?'),
++ // Never moves back: a person opening the results during a notifier tick keeps the later time.
++ advance: db.prepare('UPDATE saved_searches SET last_run_at = GREATEST(COALESCE(last_run_at, 0), ?) WHERE id = ?'),
++ due: db.prepare('SELECT * FROM saved_searches ORDER BY COALESCE(last_run_at, created_at), id LIMIT ?'),
total: db.prepare('SELECT COUNT(*) AS n FROM saved_searches'),
};
@@@ -88,6 +91,13 @@ function createSavedSearches(db, { maxPerSubject = 50, now = () => Date.now() }
save,
remove: async (subject, id) => ID_RE.test(String(id)) && (await st.del.run(id, subject)).changes > 0,
markRun: async (id) => await st.ran.run(now(), id),
++ /**
++ * The notifier's queue: up to `limit` saved searches, longest without a run first, each with its
++ * subject and watermark (last_run_at, else created_at; epoch ms).
++ */
++ due: async (limit) => (await st.due.all(limit)).map(r => ({ ...view(r), subject: r.subject, watermark: Number(r.last_run_at ?? r.created_at) })),
++ /** Move the watermark to `at` (epoch ms) after a notification went out or there was nothing new. */
++ advance: async (id, at) => await st.advance.run(at, id),
total: async () => (await st.total.get()).n,
maxPerSubject,
};
ddiff --git a/test/helpers.js b/test/helpers.js
iindex ae863cb..1659d7f 100644
---- a/test/helpers.js
++++ b/test/helpers.js
@@@ -43,7 +43,7 @@ function userToken({ subjectId = ids.newId('user'), role = 'user', aud = ['openv
}, key);
}
--async function boot({ env = {}, fetchImpl, jwksFetch, tokenClient, now } = {}) {
++async function boot({ env = {}, fetchImpl, jwksFetch, tokenClient, networkPush, now } = {}) {
const config = load({
NODE_ENV: 'test',
PORT: '0',
@@@ -55,7 +55,7 @@ async function boot({ env = {}, fetchImpl, jwksFetch, tokenClient, now } = {}) {
const inner = fetchImpl || globalThis.fetch;
const jwks = jwksFetch || stubJwks;
const fetchImplBoth = async (url, opts) => String(url).includes('/api/.well-known/jwks') ? jwks(url, opts) : inner(url, opts);
-- const h = await start({ config, db: testdb.db, log: silent, fetchImpl: fetchImplBoth, tokenClient, now });
++ const h = await start({ config, db: testdb.db, log: silent, fetchImpl: fetchImplBoth, tokenClient, networkPush, now });
const base = `http://127.0.0.1:${h.server.address().port}`;
return { ...h, base, async stop() { await h.close(); await testdb.close(); } };
}
ddiff --git a/test/network-push.test.js b/test/network-push.test.js
nnew file mode 100644
iindex 0000000..fe1cfa4
---- /dev/null
++++ b/test/network-push.test.js
@@@ -0,0 +1,118 @@
++'use strict';
++/**
++ * The Network client of the saved-search notifier: resolve a usr_ subject, then push one notification
++ * (network.notification-push-request@1) with client-credentials tokens for audience openvibe.network;
++ * an unknown or deleted person is not pushed to; a 401 drops the cached token; refusals and 5xx throw
++ * with the status only, never a token.
++ */
++const assert = require('assert');
++const { load } = require('../server/config');
++const { createNetworkPush, NetworkError } = require('../server/network-push');
++const { suite } = require('./helpers');
++
++const t = suite('network-push');
++const config = load({ NODE_ENV: 'test', OV_NETWORK_INTERNAL_URL: 'http://network.test', OV_OAUTH_CLIENT_SECRET: 'client-secret-xyz' });
++const SUBJECT = 'usr_01j9zzzzzzzzzzzzzzzzzzzzzz';
++const NOTE = { type: 'SEARCH_SAVED_MATCH', service: 'search', category: 'service', priority: 'normal', title: 'New results', message: 'One', url: 'https://search.test/?q=x' };
++
++const json = (status, body) => ({ ok: status >= 200 && status < 300, status, json: async () => body, text: async () => JSON.stringify(body) });
++
++function fakeTokens() {
++ const tk = { invalidated: 0, async authHeaders() { return { Authorization: 'Bearer tok-SECRET' }; }, invalidate() { tk.invalidated++; } };
++ return tk;
++}
++
++function fakeFetch(answers) {
++ const calls = [];
++ const fn = async (url, opts = {}) => {
++ calls.push({ url: String(url), opts });
++ const a = answers.shift();
++ if (!a) throw new Error(`unexpected call ${url}`);
++ return typeof a === 'function' ? a(url, opts) : a;
++ };
++ fn.calls = calls;
++ return fn;
++}
++
++t('resolves the subject, then pushes one notification to the numeric user id', async () => {
++ const tokens = fakeTokens();
++ const fetchImpl = fakeFetch([json(200, { subject: { type: 'user', id: SUBJECT }, network_user_id: 42 }), json(200, { ok: true, notification: { id: 7 } })]);
++ const out = await createNetworkPush({ config, fetchImpl, tokenClient: tokens }).push(SUBJECT, NOTE);
++ assert.deepStrictEqual(out, { sent: true, skipped: false });
++ const [resolve, push] = fetchImpl.calls;
++ assert.strictEqual(resolve.url, `http://network.test/internal/identity/resolve?subject_id=${SUBJECT}`);
++ assert.strictEqual(resolve.opts.method, 'GET');
++ assert.strictEqual(resolve.opts.headers.Authorization, 'Bearer tok-SECRET');
++ assert.ok(resolve.opts.signal, 'a request timeout');
++ assert.strictEqual(push.url, 'http://network.test/internal/notifications/push');
++ assert.strictEqual(push.opts.method, 'POST');
++ assert.strictEqual(push.opts.headers['Content-Type'], 'application/json');
++ assert.ok(push.opts.signal);
++ assert.deepStrictEqual(JSON.parse(push.opts.body), { ...NOTE, user_id: 42 });
++});
++
++t('a category the person switched off (skipped) still counts as delivered', async () => {
++ const fetchImpl = fakeFetch([json(200, { network_user_id: 42 }), json(200, { ok: true, skipped: true })]);
++ assert.deepStrictEqual(await createNetworkPush({ config, fetchImpl, tokenClient: fakeTokens() }).push(SUBJECT, NOTE), { sent: true, skipped: true });
++});
++
++t('an unknown subject (404) or a deleted account is not pushed to', async () => {
++ let fetchImpl = fakeFetch([json(404, { type: 'identity.subject_not_found' })]);
++ assert.deepStrictEqual(await createNetworkPush({ config, fetchImpl, tokenClient: fakeTokens() }).push(SUBJECT, NOTE), { sent: false, reason: 'unknown_subject' });
++ assert.strictEqual(fetchImpl.calls.length, 1);
++ fetchImpl = fakeFetch([json(200, { network_user_id: 42, deleted: true })]);
++ assert.deepStrictEqual(await createNetworkPush({ config, fetchImpl, tokenClient: fakeTokens() }).push(SUBJECT, NOTE), { sent: false, reason: 'unknown_subject' });
++ assert.strictEqual(fetchImpl.calls.length, 1);
++});
++
++t('a 401 throws with the status, drops the cached token and never shows the token', async () => {
++ const tokens = fakeTokens();
++ const fetchImpl = fakeFetch([json(200, { network_user_id: 42 }), json(401, { error: 'invalid token tok-SECRET' })]);
++ const err = await createNetworkPush({ config, fetchImpl, tokenClient: tokens }).push(SUBJECT, NOTE).then(() => null, e => e);
++ assert.ok(err instanceof NetworkError);
++ assert.strictEqual(err.status, 401);
++ assert.strictEqual(tokens.invalidated, 1);
++ assert.ok(!/tok-SECRET|client-secret/.test(err.message), err.message);
++});
++
++t('a 5xx from resolve throws before any push; a 5xx from push throws', async () => {
++ let fetchImpl = fakeFetch([json(503, {})]);
++ let err = await createNetworkPush({ config, fetchImpl, tokenClient: fakeTokens() }).push(SUBJECT, NOTE).then(() => null, e => e);
++ assert.strictEqual(err.status, 503);
++ assert.strictEqual(fetchImpl.calls.length, 1);
++ fetchImpl = fakeFetch([json(200, { network_user_id: 42 }), json(500, { error: 'boom' })]);
++ err = await createNetworkPush({ config, fetchImpl, tokenClient: fakeTokens() }).push(SUBJECT, NOTE).then(() => null, e => e);
++ assert.strictEqual(err.status, 500);
++ assert.match(err.message, /notification push: Network answered 500/);
++});
++
++t('a timeout or a dropped connection throws a NetworkError', async () => {
++ const fetchImpl = fakeFetch([async () => { const e = new Error('aborted'); e.name = 'TimeoutError'; throw e; }]);
++ const err = await createNetworkPush({ config, fetchImpl, tokenClient: fakeTokens(), timeoutMs: 50 }).push(SUBJECT, NOTE).then(() => null, e => e);
++ assert.ok(err instanceof NetworkError);
++ assert.match(err.message, /no answer within 50 ms/);
++});
++
++t('without an injected client: client-credentials tokens for openvibe.network, one per capability', async () => {
++ const forms = [];
++ const fetchImpl = async (url, opts = {}) => {
++ if (String(url).endsWith('/oauth/token')) {
++ const form = new URLSearchParams(String(opts.body));
++ forms.push(Object.fromEntries(form));
++ return json(200, { access_token: `tok-${form.get('scope')}`, expires_in: 300 });
++ }
++ if (String(url).includes('/identity/resolve')) {
++ assert.strictEqual(opts.headers.Authorization, 'Bearer tok-identity.subject.resolve');
++ return json(200, { network_user_id: 42 });
++ }
++ assert.strictEqual(opts.headers.Authorization, 'Bearer tok-network.notifications.push');
++ return json(200, { ok: true });
++ };
++ assert.deepStrictEqual(await createNetworkPush({ config, fetchImpl }).push(SUBJECT, NOTE), { sent: true, skipped: false });
++ assert.deepStrictEqual(forms.map(f => [f.client_id, f.audience, f.scope]), [
++ ['search', 'openvibe.network', 'identity.subject.resolve'],
++ ['search', 'openvibe.network', 'network.notifications.push'],
++ ]);
++});
++
++t.run();
ddiff --git a/test/saved-notify.test.js b/test/saved-notify.test.js
nnew file mode 100644
iindex 0000000..3b58eb6
---- /dev/null
++++ b/test/saved-notify.test.js
@@@ -0,0 +1,210 @@
++'use strict';
++/**
++ * Saved-search notifications: the engine's internal `since` filter (ranked and listed search) keeps
++ * documents indexed after a watermark; the notifier pushes ONE notification per saved search with new
++ * hits (as its owner: the ACL applies), advances the watermark only after a push went out, retries a
++ * failed push on the next tick, never notifies a delivered batch twice, and stays off (reported in
++ * readiness, not failing) without SEARCH_SAVED_NOTIFY=1 and the `search` client's secret.
++ */
++const assert = require('assert');
++const { ids } = require('openvibe-contracts');
++const { load } = require('../server/config');
++const { createSavedNotifier } = require('../server/saved-notifier');
++const { boot, request, doc, suite } = require('./helpers');
++
++const t = suite('saved-notify');
++const ORIGIN = 'https://search.openvibe.network';
++const alice = ids.newId('user');
++const bob = ids.newId('user');
++
++let clock = Date.parse('2026-10-01T00:00:00Z');
++const tickClock = (ms = 1000) => { clock += ms; return clock; };
++
++// The Network client stub: records pushes; `fail` makes the next push throw, `unknown` answers "no such person".
++const pushes = [];
++const network = {
++ fail: false,
++ unknown: false,
++ async push(subject, notification) {
++ if (network.fail) { const e = new Error('notification push: Network answered 503'); e.status = 503; throw e; }
++ pushes.push({ subject, notification });
++ return network.unknown ? { sent: false, reason: 'unknown_subject' } : { sent: true, skipped: false };
++ },
++};
++
++let svc;
++let zebras;
++let giraffes;
++let t0;
++const watermark = async (id) => (await svc.saved.due(100)).find(s => s.id === id).watermark;
++
++t('boot (notifier off: the test drives tick())', async () => {
++ svc = await boot({ env: { BASE_URL: ORIGIN }, now: () => clock, networkPush: network });
++ assert.strictEqual(svc.notifier.running(), false);
++ t0 = clock;
++ await svc.store.apply(doc({ id: 'old', title: 'Zebra old news', canonical_url: 'https://openvibe.wiki/p/old' }));
++ tickClock();
++});
++
++t('since keeps only documents indexed after it, in ranked and listed search', async () => {
++ const mark = clock;
++ tickClock();
++ await svc.store.apply(doc({ id: 'fresh', title: 'Zebra fresh arrival', canonical_url: 'https://openvibe.wiki/p/fresh' }));
++ const viewer = { subject: null };
++ const ranked = await svc.searcher.run({ text: 'zebra', viewer, since: mark });
++ assert.deepStrictEqual(ranked.results.map(r => r.id), ['fresh']);
++ const listed = await svc.searcher.run({ text: '', viewer, since: mark });
++ assert.deepStrictEqual(listed.results.map(r => r.id), ['fresh']);
++ assert.deepStrictEqual((await svc.searcher.run({ text: 'zebra', viewer, since: t0 - 1 })).results.map(r => r.id).sort(), ['fresh', 'old']);
++ assert.deepStrictEqual((await svc.searcher.run({ text: 'zebra', viewer })).results.length, 2, 'no since: everything');
++ assert.deepStrictEqual((await svc.searcher.run({ text: 'zebra', viewer, since: clock })).results, []);
++ // A new revision is indexed again: it is new to a watermark before it.
++ tickClock();
++ const before = clock;
++ tickClock();
++ await svc.store.apply(doc({ id: 'old', revision: 2, title: 'Zebra old news, revised', canonical_url: 'https://openvibe.wiki/p/old' }));
++ assert.deepStrictEqual((await svc.searcher.run({ text: 'zebra', viewer, since: before })).results.map(r => r.id), ['old']);
++ // The public query API has no `since`: an unknown parameter is ignored.
++ const api = await request(svc.base, 'GET', `/api/v1/search?q=zebra&since=${clock}`);
++ assert.strictEqual(api.status, 200, api.text);
++ assert.strictEqual(api.body.results.length, 2);
++ tickClock();
++});
++
++t('a saved search with nothing new since it was saved: no push, watermark advanced', async () => {
++ zebras = (await svc.saved.save(alice, { name: 'Zebras', q: 'zebra', filters: { owner: 'wiki' } })).saved;
++ giraffes = (await svc.saved.save(bob, { name: 'Giraffes', q: 'giraffe', filters: {} })).saved;
++ assert.strictEqual(await watermark(zebras.id), clock, 'never run: the watermark is created_at');
++ const at = tickClock();
++ const out = await svc.notifier.tick();
++ assert.deepStrictEqual(out, { checked: 2, notified: 0, unchanged: 2, unknown: 0, failed: 0 });
++ assert.strictEqual(pushes.length, 0);
++ assert.strictEqual(await watermark(zebras.id), at);
++ assert.strictEqual(await watermark(giraffes.id), at);
++});
++
++t('new hits: ONE push to the owner, the restricted document of someone else is not announced', async () => {
++ tickClock();
++ await svc.store.apply(doc({ id: 'z1', title: 'Zebra herd spotted', canonical_url: 'https://openvibe.wiki/p/z1' }));
++ await svc.store.apply(doc({ id: 'z2', title: 'Zebra stripes explained', canonical_url: 'https://openvibe.wiki/p/z2' }));
++ await svc.store.apply(doc({ id: 'zp', visibility: 'private', acl: { subjects: [bob] }, title: 'Zebra secret of bob' }));
++ await svc.store.apply(doc({ owner: 'blog', type: 'post', id: 'zb', title: 'Zebra blog post', canonical_url: 'https://openvibe.blog/p/zb' }));
++ const at = tickClock();
++ const out = await svc.notifier.tick();
++ assert.strictEqual(out.notified, 1);
++ assert.strictEqual(pushes.length, 1);
++ const [{ subject, notification: n }] = pushes;
++ assert.strictEqual(subject, alice);
++ assert.strictEqual(n.service, 'search');
++ assert.strictEqual(n.type, 'SEARCH_SAVED_MATCH');
++ assert.strictEqual(n.category, 'service');
++ assert.strictEqual(n.priority, 'normal');
++ assert.strictEqual(n.title, 'New results for “Zebras”');
++ assert.match(n.message, /^2 new results, including “Zebra (herd spotted|stripes explained)”$/);
++ assert.strictEqual(n.url, `${ORIGIN}/?q=zebra&owner=wiki`);
++ assert.deepStrictEqual(n.rich_content, { saved_search_id: zebras.id, count: 2, more: false });
++ assert.strictEqual(await watermark(zebras.id), at);
++});
++
++t('a second tick with nothing newer pushes nothing (a delivered batch is never notified twice)', async () => {
++ tickClock();
++ const out = await svc.notifier.tick();
++ assert.strictEqual(out.notified, 0);
++ assert.strictEqual(pushes.length, 1);
++});
++
++t('a failed push leaves the watermark; the next tick retries the same hits once', async () => {
++ tickClock();
++ await svc.store.apply(doc({ id: 'z3', title: 'Zebra crossing', canonical_url: 'https://openvibe.wiki/p/z3' }));
++ const before = await watermark(zebras.id);
++ tickClock();
++ network.fail = true;
++ const failed = await svc.notifier.tick();
++ assert.strictEqual(failed.failed, 1);
++ assert.strictEqual(pushes.length, 1);
++ assert.strictEqual(await watermark(zebras.id), before, 'not advanced');
++ network.fail = false;
++ const at = tickClock();
++ const retried = await svc.notifier.tick();
++ assert.strictEqual(retried.notified, 1);
++ assert.strictEqual(pushes.length, 2);
++ assert.strictEqual(pushes[1].notification.message, 'Zebra crossing');
++ assert.strictEqual(await watermark(zebras.id), at);
++ tickClock();
++ await svc.notifier.tick();
++ assert.strictEqual(pushes.length, 2);
++});
++
++t('a person Network no longer knows: no notification, watermark advanced', async () => {
++ tickClock();
++ await svc.store.apply(doc({ id: 'g1', title: 'Giraffe neck facts', canonical_url: 'https://openvibe.wiki/p/g1' }));
++ network.unknown = true;
++ const at = tickClock();
++ const out = await svc.notifier.tick();
++ network.unknown = false;
++ assert.strictEqual(out.unknown, 1);
++ assert.strictEqual(await watermark(giraffes.id), at);
++});
++
++t('opening the results moves the watermark; a notifier tick never moves it back', async () => {
++ const at = tickClock();
++ const r = await svc.saved.advance(zebras.id, at);
++ assert.ok(r);
++ tickClock(5000);
++ await svc.saved.markRun(zebras.id);
++ const opened = clock;
++ await svc.saved.advance(zebras.id, at);
++ assert.strictEqual(await watermark(zebras.id), opened);
++});
++
++t('the batch bound: one tick runs at most notifyBatch saved searches, longest waiting first', async () => {
++ const config = { ...svc.config, savedSearches: { ...svc.config.savedSearches, notifyBatch: 1 } };
++ const runs = [];
++ const searcher = { run: async (q) => { runs.push(q); return { results: [], next_cursor: null }; } };
++ const n = createSavedNotifier({ config, saved: svc.saved, searcher, network, now: () => clock });
++ tickClock();
++ await n.tick();
++ assert.strictEqual(runs.length, 1);
++ assert.strictEqual(runs[0].text, 'giraffe', 'the oldest watermark first');
++ assert.deepStrictEqual(runs[0].viewer, { subject: bob });
++});
++
++t('readiness reports the notifier as off, not failing', async () => {
++ const ready = await request(svc.base, 'GET', '/api/ready');
++ assert.strictEqual(ready.body.saved_notify, 'off (SEARCH_SAVED_NOTIFY unset)');
++});
++
++t('disabled config: not started without the flag, or with the flag but no client secret', async () => {
++ const make = (env) => createSavedNotifier({ config: load({ NODE_ENV: 'test', ...env }), saved: svc.saved, searcher: svc.searcher, network });
++ for (const [env, state] of [
++ [{}, 'off (SEARCH_SAVED_NOTIFY unset)'],
++ [{ SEARCH_SAVED_NOTIFY: '1' }, 'off (OV_OAUTH_CLIENT_SECRET unset)'],
++ [{ SEARCH_SAVED_NOTIFY: 'yes', OV_OAUTH_CLIENT_SECRET: 's' }, 'off (SEARCH_SAVED_NOTIFY unset)'],
++ ]) {
++ const n = make(env);
++ n.start();
++ assert.strictEqual(n.running(), false);
++ assert.strictEqual(n.state(), state);
++ await n.stop();
++ }
++ const on = make({ SEARCH_SAVED_NOTIFY: '1', OV_OAUTH_CLIENT_SECRET: 's' });
++ on.start();
++ assert.strictEqual(on.state(), 'running');
++ await on.stop();
++ assert.strictEqual(on.state(), 'stopped');
++});
++
++t('with the flag and the secret, start() runs it and close() stops it', async () => {
++ const other = await boot({ env: { SEARCH_SAVED_NOTIFY: '1', OV_OAUTH_CLIENT_SECRET: 'test-secret' }, networkPush: network });
++ try {
++ assert.strictEqual(other.notifier.running(), true);
++ assert.strictEqual((await request(other.base, 'GET', '/api/ready')).body.saved_notify, 'running');
++ } finally {
++ await other.stop();
++ }
++ assert.strictEqual(other.notifier.running(), false);
++});
++
++t('shutdown', async () => { await svc.stop(); });
++
++t.run();