openvibers/openvibe.search Commits
The Search Fabric v1 (Contracts 0.225.0 search.fabric.query): POST/GET /api/v1/query over network, mine, web and the vertical scopes; the web ladder open index → DeepSeek (signed-in, daily cap) → Wikipedia through AI's research, cached by freshness; route.explain and cost; GET /api/v1/query/providers (#119)
Changes
@@@ -54,6 +54,16 @@ VALKEY_PREFIX=#SEARCH_FRESHNESS_WEIGHT=1#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30++# Search Fabric (README "Search Fabric"): the web ladder through OpenVibe.AI.++# AI's internal origin; unset uses the AI manifest's internalOrigin. Needs OV_OAUTH_CLIENT_SECRET above.++#OV_AI_INTERNAL_URL=http://127.0.0.1:4700++# off leaves the paid/free web providers off; the open index and cached answers still answer.++#SEARCH_WEB=on++# Paid DeepSeek calls per day across all callers; past it the provider stands down.++#SEARCH_WEB_DAILY=500++# Owners whose documents the ladder's own open index reads first.++#SEARCH_OPEN_INDEX_OWNERS=sources++# Saved searches per signed-in person.#SEARCH_SAVED_MAX_PER_SUBJECT=50# Notifications of new matches through OpenVibe.Network (README "Saved searches"): off unless 1, and@@@ -155,6 +155,67 @@ Search so it reads `OV_OAUTH_CLIENT_ID` and `OV_OAUTH_CLIENT_SECRET`), andsaved-search notifications are switched on here with `SEARCH_SAVED_NOTIFY=1`; until someone signsin that way, saved searches are used through Bearer tokens or a product acting for its visitor.++## Search Fabric++++One search over several scopes, routed the cheapest way that answers (`search.fabric.query`):++++| Scope | What it reads |++|---|---|++| `network` | Search's own index, with the same visibility rules as `GET /api/v1/search` |++| `mine` | the same search kept to the caller's non-public documents (needs a signed-in person) |++| `web` | OpenVibe's open index, then the paid/free web providers in a ladder |++| `news`, `video`, `products`, `places`, `code` | facets of the same index, narrowed to the owner and type Search holds |++| `images`, `people` | nothing yet: `skipped: "not indexed yet"` in `route.explain` |++++The web ladder is `open_index` first (free, the configured `SEARCH_OPEN_INDEX_OWNERS`), then++`deepseek` (paid), then `wikipedia` (free). Every step says what happened in `route.explain`: `ok`,++`count`, `ms`, `cached`, `cost_usd`, or `skipped` with the reason. Depth `quick` reads the index and++the cache only; `standard` stops climbing once it has `min(limit, 3)` results; `deep` runs every++step and merges them, de-duplicating by URL.++++Only a signed-in person, or a first-party service, may trigger a paid provider.++Anonymous callers get the open index and cached answers; the cache key is the normalized query,++scope, step, freshness and site, so the same question typed two ways is one entry. DeepSeek calls are++capped per day (`SEARCH_WEB_DAILY`, default 500); past the cap the step is skipped with++`daily cap reached`. Cache TTLs: `day` freshness 1 h, `week` 6 h, otherwise 24 h.++++```http++POST /api/v1/query { "q": "solar eclipse", "scopes": ["network", "web"], "depth": "standard" }++GET /api/v1/query?q=solar+eclipse&scopes=network,web++GET /api/v1/query/providers++```++++The GET query string takes `q`, `scopes` (comma-separated), `freshness`, `depth`, `limit`, `cursor`,++`owner`, `type`, `lang` and `site`. GET and `/providers` answer `Access-Control-Allow-Origin: *`++(anonymous rules apply); POST does not. Results are grouped in the requested scope order, at most++`limit` per scope and 100 in all; `citations` number the web results and `next_cursor` pages the++network or mine scope when it was the only one asked. The answer never contains the query text.++++```json++{++ "results": [++ { "scope": "network", "kind": "document", "title": "Eclipse timings", "url": "https://openvibe.news/s/1", "site": "openvibe.news", "owner": "news", "type": "story", "id": "story_1" },++ { "scope": "web", "kind": "page", "title": "Solar eclipse", "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "site": "en.wikipedia.org", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" }++ ],++ "groups": { "network": 1, "web": 1 },++ "answer": null,++ "citations": [{ "n": 1, "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "title": "Solar eclipse", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" }],++ "route": {++ "explain": [++ { "scope": "network", "step": "index", "ok": true, "count": 1, "ms": 4 },++ { "scope": "web", "step": "open_index", "ok": true, "count": 0, "ms": 3 },++ { "scope": "web", "step": "deepseek", "ok": false, "count": 0, "skipped": "paid providers need a signed-in person" },++ { "scope": "web", "step": "wikipedia", "ok": true, "count": 1, "ms": 190, "cost_usd": 0 }++ ],++ "cost_usd": 0++ },++ "next_cursor": null++}++```++++Asking `"answer": true` answers `answer: null` and adds an `answer` explain row: answers are not on++yet.++## IndexingOwners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding@@@ -269,6 +330,8 @@ signed-out visitors is not counted (it speaks for all of them). Past a limit: `4| Route | Per caller ||---|---|| Queries, direct gets, owner reconciliation reads, saved-search reads and runs | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` (120 a minute, 3000 an hour) |++| `POST/GET /api/v1/query` (`search.fabric`, a paid-provider ladder) | 30 / 600 |++| `GET /api/v1/query/providers` | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` || `GET /api/v1/suggest` | 300 / 6000 (a search box as the person types) || `PUT/DELETE /api/v1/documents/…` | 120 / 1200 (a catalog sync is a burst; bulk content comes through Events) || Saved-search create, delete | 30 / 300 |@@@ -293,6 +356,8 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea- freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`)- saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits(`test/saved-searches.test.js`)++- the Search Fabric: network visibility, mine for a person, the web ladder, cache hits, the daily++ cap, provider offers and GET/POST parity (`test/fabric.test.js`)- saved-search notifications: one push per search with new hits, as its owner, the watermark onlyafter a push, a failed push retried, never twice, off without the flag and the secret(`test/saved-notify.test.js`, `test/network-push.test.js`)nnew file mode 100644@@@ -0,0 +1,21 @@++-- phase: expand++-- Search Fabric: cached provider answers and per-provider daily call stats (server/fabric).++CREATE TABLE IF NOT EXISTS query_cache (++ key text PRIMARY KEY,++ scope text NOT NULL,++ step text NOT NULL,++ results jsonb NOT NULL,++ cost_usd numeric NOT NULL DEFAULT 0,++ created_at timestamptz NOT NULL,++ expires_at timestamptz NOT NULL++);++CREATE INDEX IF NOT EXISTS query_cache_expires_at_idx ON query_cache (expires_at);++++CREATE TABLE IF NOT EXISTS query_provider_stats (++ day date NOT NULL,++ provider text NOT NULL,++ calls integer NOT NULL DEFAULT 0,++ answered integer NOT NULL DEFAULT 0,++ ms_total bigint NOT NULL DEFAULT 0,++ PRIMARY KEY (day, provider)++);@@@ -13,7 +13,7 @@"ajv-formats": "^3.0.1","dotenv": "^16.4.7","express": "^4.21.2",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0","pg": "^8.23.0"@@@ -750,9 +750,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.222.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",-- "integrity": "sha512-wdUQCilq9glRMwfxEswbIj3l0QQqKDXZAi2+CR5Mlh34rZjI1Q+7GlAoR0UtNwZAgtLwXhPxFyJ/aPk8lcDG1Q==",++ "version": "0.225.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",++ "integrity": "sha512-msLaRYa2Byyabnf/+L3aRPwkx8MI3zWbofFhwupq+RyBCdgbA4gLlkPsJu9nK/1UTCWdNdSTfWXW5pZY1/1zbQ==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -25,7 +25,7 @@"ajv-formats": "^3.0.1","dotenv": "^16.4.7","express": "^4.21.2",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0","pg": "^8.23.0"@@@ -10,8 +10,9 @@* is not counted: it speaks for all of them and limits them itself, so one budget would refuse its* whole site. Past a limit the route answers 429 problem+json `rate_limited` with Retry-After before* it does any work; the refusal is logged once and counted in search_rate_limited_total{limit,window}.-- * Read routes get SEARCH_LIMITS_MINUTE / SEARCH_LIMITS_HOUR (120 and 3000). Counters live in this-- * process: a restart forgets them.++ * Read routes get SEARCH_LIMITS_MINUTE / SEARCH_LIMITS_HOUR (120 and 3000). The Search Fabric++ * query (POST/GET /api/v1/query) allows 30 a minute and 600 an hour because a step may reach a paid++ * provider; its providers list takes the defaults. Counters live in this process: a restart forgets them.** Never limited: /api/health, /api/ready, /release.json, /metrics, and the signed Events deliveries* (POST /internal/events: Events pushes at its own pace, and a 429 would only make it retry and fallnnew file mode 100644@@@ -0,0 +1,60 @@++'use strict';++/**++ * Search Fabric routes (capability search.fabric.query):++ *++ * POST /api/v1/query search.fabric-request@1 body → search.fabric-result@1++ * GET /api/v1/query the same request in the query string (scopes comma-separated)++ * GET /api/v1/query/providers the web ladder's offers (search.provider-offer@1)++ *++ * GET routes answer any origin (Access-Control-Allow-Origin *): a cross-site browser is anonymous++ * anyway, and the answer it reads is exactly what anonymous gets. POST does not: a browser sends it++ * with its own credentials.++ */++const express = require('express');++const { withViewer, one } = require('./query');++++const SEARCH_LIMITS = { minute: 30, hour: 600 };++++/** The query string as a search.fabric-request@1 body; identity never comes from here. */++function requestFromQuery(query) {++ const body = { q: one(query.q) === undefined ? '' : String(one(query.q)) };++ const scopes = one(query.scopes);++ if (scopes !== undefined) {++ body.scopes = String(scopes).split(',').map((s) => s.trim()).filter(Boolean);++ }++ for (const key of ['freshness', 'depth', 'cursor']) {++ const value = one(query[key]);++ if (value !== undefined) body[key] = String(value);++ }++ if (one(query.limit) !== undefined) body.limit = parseInt(one(query.limit), 10);++ const filters = {};++ for (const [param, key] of [['owner', 'owner'], ['type', 'type'], ['lang', 'lang'], ['site', 'site']]) {++ const value = one(query[param]);++ if (value !== undefined) filters[key] = String(value);++ }++ if (Object.keys(filters).length) body.filters = filters;++ return body;++}++++function fabricRouter({ fabric, auth, limits }) {++ const router = express.Router();++ const guarded = (limit, h) => withViewer(auth, h, limit);++ const search = limits('search.fabric', SEARCH_LIMITS);++ const cors = (_req, res, next) => { res.set('Access-Control-Allow-Origin', '*'); next(); };++++ router.post('/api/v1/query', guarded(search, async (req, res) => {++ res.json(await fabric.query(req.body, req.viewer));++ }));++++ router.get('/api/v1/query', cors, guarded(search, async (req, res) => {++ res.json(await fabric.query(requestFromQuery(req.query), req.viewer));++ }));++++ router.get('/api/v1/query/providers', cors, guarded(limits('search.fabric.providers'), async (_req, res) => {++ res.json({ providers: await fabric.providers() });++ }));++++ return router;++}++++module.exports = { fabricRouter, requestFromQuery, SEARCH_LIMITS };@@@ -8,6 +8,10 @@ const { createRelease } = require('openvibe-shared/release');const { createSearchReadiness, registerSearchGauges } = require('./observability');const { documentsRouter } = require('./api/documents');const { queryRouter, createSearcher } = require('./api/query');++const { fabricRouter } = require('./api/fabric');++const { createQueryCache } = require('./fabric/cache');++const { createWebProviders } = require('./fabric/web');++const { createFabric } = require('./fabric');const { createFrontData } = require('./web/front-data');const { savedRouter } = require('./api/saved');const { pageRouter } = require('./web/page');@@@ -16,7 +20,7 @@ const { createLimits } = require('./actor-limits');const { createIndexNow } = require('openvibe-shared/indexnow');const pkg = require('../package.json');--function createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher: givenSearcher = null, notifier = null, log = console, now }) {++function createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher: givenSearcher = null, notifier = null, fetchImpl = globalThis.fetch, log = console, now }) {const app = express();app.disable('x-powered-by');app.set('trust proxy', 'loopback');@@@ -70,8 +74,17 @@ function createApp({ config, db, store, engine, auth, outbox, purges, purger, sa// One per-actor limiter for the API routes below (server/actor-limits.js); the Events webhook,// health, ready, release.json and metrics above are never limited.const limits = createLimits({ config, now: now || (() => Date.now()), registry: metrics.registry, log });++ // The Search Fabric: one cache and one web-provider ladder for the process. The hourly prune timer++ // lives in server/index.js, which clears it on close.++ const clock = now || (() => Date.now());++ const queryCache = createQueryCache({ db, now: clock });++ const web = createWebProviders({ config, fetchImpl, now: clock, log });++ const fabric = createFabric({ config, searcher, cache: queryCache, web, db, now: clock, log });++ app.locals.queryCache = queryCache;++ app.locals.fabric = fabric;app.use(documentsRouter({ store, auth, db, outbox, purges, limits }));app.use(queryRouter({ config, store, engine, auth, searcher, limits }));++ app.use(fabricRouter({ fabric, auth, limits }));app.use(savedRouter({ config, saved, searcher, auth, limits }));// GET / (HTML search page for browsers, the text route index otherwise), /robots.txt, /llms.txt and// /sitemap.xml (whose lastmods come from the index and the deployed release, never from the clock).@@@ -17,6 +17,9 @@ const num = (v, d) => {return Number.isFinite(n) ? n : d;};++// The AI service's loopback address, from Contracts, so a development boot reaches it without an env var.++const AI_SERVICE = require('openvibe-contracts/manifests/services/ai.json');++const ZONE_ID_RE = /^[0-9a-f]{32}$/;const HOST_RE = /^(?=.{1,253}$)([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;@@@ -95,6 +98,16 @@ function load(env = process.env) {},maxBodyBytes: int(env.SEARCH_MAX_BODY_BYTES, 256 * 1024),++ // The Search Fabric's web ladder (server/fabric): OpenVibe.AI's research endpoint for the web++ // providers, its own open index first. SEARCH_WEB=off leaves the ladder's providers off (the++ // open index and cached answers still work); SEARCH_WEB_DAILY bounds paid DeepSeek calls per day.++ fabric: {++ aiUrl: strip(env.OV_AI_INTERNAL_URL || AI_SERVICE.internalOrigin || ''),++ web: env.SEARCH_WEB !== 'off',++ dailyCap: int(env.SEARCH_WEB_DAILY, 500),++ openIndexOwners: list(env.SEARCH_OPEN_INDEX_OWNERS, ['sources']),++ },++// Freshness weighting of full-text relevance (README "Ranking"):// score = bm25 × (1 + weight × 2^(−age_days / halfLifeDays)); weight 0 turns it off.freshness: {nnew file mode 100644@@@ -0,0 +1,62 @@++'use strict';++/**++ * The Search Fabric's answer cache (query_cache, migrations/0005_fabric.sql): one provider step's++ * results for one normalized question, kept until it expires. A hit costs nothing and never reaches++ * a provider; a miss is the caller's to fetch and fill. `key` is shared by every caller — the query++ * text is only ever used to compute it and is never stored or returned.++ */++const crypto = require('crypto');++++/** Lowers, collapses whitespace and trims: the same question typed two ways is one cache entry. */++function normalizeQuery(q) {++ return String(q == null ? '' : q).toLowerCase().replace(/\s+/g, ' ').trim();++}++++/** sha256(base64url) of [q, scope, step, freshness, site], the q normalized as above. */++function cacheKey(parts) {++ const [q, scope, step, freshness, site] = Array.isArray(parts)++ ? parts++ : [parts.q, parts.scope, parts.step, parts.freshness, parts.site];++ return crypto.createHash('sha256')++ .update(JSON.stringify([normalizeQuery(q), scope, step, freshness, site]))++ .digest('base64url');++}++++function createQueryCache({ db, now = () => Date.now() }) {++ const st = {++ // An expired row is as good as absent; prune() removes it later.++ get: db.prepare('SELECT key, scope, step, results, cost_usd FROM query_cache WHERE key = ? AND expires_at > ?'),++ put: db.prepare(`INSERT INTO query_cache (key, scope, step, results, cost_usd, created_at, expires_at)++ VALUES (@key, @scope, @step, @results::jsonb, @cost_usd, @created_at, @expires_at)++ ON CONFLICT (key) DO UPDATE SET scope = EXCLUDED.scope, step = EXCLUDED.step,++ results = EXCLUDED.results, cost_usd = EXCLUDED.cost_usd,++ created_at = EXCLUDED.created_at, expires_at = EXCLUDED.expires_at`),++ prune: db.prepare('DELETE FROM query_cache WHERE expires_at <= ?'),++ };++++ async function get(key) {++ const row = await st.get.get(key, new Date(now()).toISOString());++ if (!row) return null;++ return { scope: row.scope, step: row.step, results: row.results || [], cost_usd: Number(row.cost_usd) || 0 };++ }++++ async function put(key, { scope, step, results, cost_usd = 0 }, ttlMs) {++ const at = now();++ await st.put.run({++ key, scope, step,++ results: JSON.stringify(results || []),++ cost_usd,++ created_at: new Date(at).toISOString(),++ expires_at: new Date(at + ttlMs).toISOString(),++ });++ }++++ /** Drop every expired row; called at boot's hourly timer (server/index.js). */++ async function prune() {++ return (await st.prune.run(new Date(now()).toISOString())).changes;++ }++++ return { get, put, prune };++}++++module.exports = { createQueryCache, cacheKey, normalizeQuery };nnew file mode 100644@@@ -0,0 +1,451 @@++'use strict';++/**++ * The Search Fabric (capability search.fabric.query): one search over several scopes, routed the++ * cheapest way that answers.++ *++ * network Search's own index, permission-aware exactly as the query API (searcher.run)++ * mine the same search as the caller, keeping only what is not public++ * web the open index first, then paid/free AI providers in a ladder, cached per step++ * news, images, video, products, people, places, code++ * facets of the same index, narrowed to the owner/type Search actually holds++ *++ * Every scope runs in parallel and comes back grouped in the requested order; route.explain has one++ * row per step (what answered, from cache, or why it did not run) and route.cost_usd what the paid++ * steps cost. The query text is never part of the answer: a page renders results without writing it++ * into its HTML.++ */++const contracts = require('openvibe-contracts');++const { QueryError } = require('../api/query');++const { cacheKey } = require('./cache');++const { hostOf, plainText, PRICE_USD } = require('./web');++++const DEFAULT_SCOPES = ['network'];++const MAX_RESULTS = 100;++const MAX_EXPLAIN = 20;++const MAX_CITATIONS = 20;++const FRESHNESS_MS = { day: 86400000, week: 7 * 86400000, month: 30 * 86400000, year: 365 * 86400000 };++++/**++ * The vertical scopes and the document kinds Search actually indexes for them. A scope with no entry++ * (images and people today) has nothing to narrow yet and says so instead of guessing. Several++ * matchers are allowed where one product has two kinds (a recorded stream and an uploaded video).++ */++const VERTICALS = {++ news: [{ owner: 'news', type: 'story' }],++ video: [{ owner: 'live', type: 'vod' }, { owner: 'media-hub', type: 'video' }],++ products: [{ owner: 'reviews', type: 'entity' }],++ places: [{ owner: 'rent', type: 'city' }, { owner: 'rent', type: 'listing' }, { owner: 'reviews', type: 'entity' }],++ code: [{ owner: 'codes' }],++};++++function clip(value, max) {++ return String(value == null ? '' : value).slice(0, max);++}++++function msSince(start, now) {++ return Math.max(0, Math.round(now() - start));++}++++/** URL identity for de-duplication: no #fragment and no trailing slash. */++function urlKey(url) {++ try {++ const u = new URL(String(url));++ u.hash = '';++ const s = u.toString();++ return s.endsWith('/') ? s.slice(0, -1) : s;++ } catch {++ const s = String(url).split('#')[0];++ return s.endsWith('/') ? s.slice(0, -1) : s;++ }++}++++function hostMatches(host, site) {++ const h = String(host || '').toLowerCase();++ const s = String(site || '').toLowerCase();++ return Boolean(h && s) && (h === s || h.endsWith(`.${s}`));++}++++function isFresh(publishedAt, freshness, at) {++ const window = FRESHNESS_MS[freshness];++ if (!window || !publishedAt) return true;++ const t = Date.parse(publishedAt);++ return !Number.isFinite(t) || t >= at - window;++}++++/** One document from Search's index as a fabric result. `scope` decides the group it belongs to. */++function documentResult(doc, scope) {++ if (!doc || !doc.canonical_url) return null;++ const url = clip(doc.canonical_url, 2048);++ const snippet = doc.snippet_html ? plainText(doc.snippet_html, 600)++ : doc.summary ? plainText(doc.summary, 600) : null;++ const site = hostOf(url);++ return {++ scope,++ kind: 'document',++ title: clip(doc.title || url, 300),++ url,++ snippet: snippet || null,++ site: site ? clip(site, 253) : null,++ image: null,++ published_at: doc.published_at ? clip(doc.published_at, 40) : null,++ retrieved_at: null,++ provider: null,++ owner: doc.owner ? clip(doc.owner, 32) : null,++ type: doc.type ? clip(doc.type, 32) : null,++ id: doc.id ? clip(doc.id, 200) : null,++ score: null,++ };++}++++/** An open-index document as a web page: it is in Search's index, but it cites like any web source. */++function openIndexResult(doc, at) {++ const r = documentResult(doc, 'web');++ if (!r) return null;++ return {++ ...r,++ kind: 'page',++ retrieved_at: new Date(at).toISOString(),++ provider: 'open_index',++ owner: null, type: null, id: null,++ };++}++++/** Cache TTLs by freshness: a day's news is stale in an hour, the week's in six, the rest in a day. */++function cacheTtl(freshness) {++ if (freshness === 'day') return 3600000;++ if (freshness === 'week') return 6 * 3600000;++ return 24 * 3600000;++}++++function createFabric({ config, searcher, cache, web, db, now = () => Date.now(), log = console }) {++ const fabric = config.fabric || {};++ const st = {++ stat: db.prepare('SELECT calls, answered, ms_total FROM query_provider_stats WHERE day = ? AND provider = ?'),++ bump: db.prepare(`INSERT INTO query_provider_stats (day, provider, calls, answered, ms_total)++ VALUES (@day, @provider, 1, @answered, @ms_total)++ ON CONFLICT (day, provider) DO UPDATE SET++ calls = query_provider_stats.calls + 1,++ answered = query_provider_stats.answered + EXCLUDED.answered,++ ms_total = query_provider_stats.ms_total + EXCLUDED.ms_total`),++ };++++ const day = () => new Date(now()).toISOString().slice(0, 10);++ const dailyCap = () => {++ const cap = Number(fabric.dailyCap);++ return Number.isFinite(cap) ? cap : 500;++ };++++ async function providerStats(provider) {++ const row = await st.stat.get(day(), provider);++ return {++ calls: row ? Number(row.calls) : 0,++ answered: row ? Number(row.answered) : 0,++ ms_total: row ? Number(row.ms_total) : 0,++ };++ }++++ /** One provider call: counted whether or not it answered; `answered` only when it returned results. */++ async function recordProvider(provider, r) {++ await st.bump.run({++ day: day(), provider,++ answered: r.ok && r.results.length ? 1 : 0,++ ms_total: Math.max(0, Math.round(Number(r.ms) || 0)),++ });++ }++++ function networkFilters(requested) {++ const filters = { facets: [] };++ if (requested.owner) filters.owner = requested.owner;++ if (requested.type) filters.type = requested.type;++ if (requested.lang) filters.language = requested.lang;++ return filters;++ }++++ function runNetwork({ q, viewer, limit, cursor, requested }) {++ return (async () => {++ const started = now();++ const page = await searcher.run({ text: q, filters: networkFilters(requested), viewer, limit, cursor });++ const results = [];++ for (const doc of page.results || []) {++ const r = documentResult(doc, 'network');++ if (r) results.push(r);++ if (results.length >= limit) break;++ }++ return {++ results,++ next_cursor: page.next_cursor || null,++ explain: [{ scope: 'network', step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],++ };++ })();++ }++++ function runMine({ q, viewer, limit, cursor, requested }) {++ return (async () => {++ if (!viewer.subject) {++ return {++ results: [], next_cursor: null,++ explain: [{ scope: 'mine', step: 'index', ok: false, count: 0, skipped: 'needs a signed-in person' }],++ };++ }++ const started = now();++ const page = await searcher.run({ text: q, filters: networkFilters(requested), viewer, limit, cursor });++ const results = [];++ for (const doc of page.results || []) {++ // The scope is the caller's own private material; public hits are the network scope's.++ if (doc.visibility === 'public') continue;++ const r = documentResult(doc, 'mine');++ if (r) results.push(r);++ if (results.length >= limit) break;++ }++ return {++ results,++ next_cursor: page.next_cursor || null,++ explain: [{ scope: 'mine', step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],++ };++ })();++ }++++ function runVertical({ scope, q, viewer, limit }) {++ const matchers = VERTICALS[scope];++ if (!matchers || !matchers.length) {++ return Promise.resolve({++ results: [], next_cursor: null,++ explain: [{ scope, step: 'index', ok: false, count: 0, skipped: 'not indexed yet' }],++ });++ }++ return (async () => {++ const started = now();++ const pages = await Promise.all(matchers.map((m) => searcher.run({++ text: q, filters: { facets: [], ...m }, viewer, limit,++ })));++ const results = [];++ const seen = new Set();++ for (const page of pages) {++ for (const doc of page.results || []) {++ const key = `${doc.owner}/${doc.type}/${doc.id}`;++ if (seen.has(key)) continue;++ seen.add(key);++ const r = documentResult(doc, scope);++ if (r) results.push(r);++ if (results.length >= limit) break;++ }++ if (results.length >= limit) break;++ }++ return {++ results, next_cursor: null,++ explain: [{ scope, step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],++ };++ })();++ }++++ /** The web ladder and its merged, de-duplicated results (one explain row per step). */++ async function runWeb({ q, viewer, limit, freshness, site, depth }) {++ const explain = [];++ const merged = [];++ const seen = new Set();++ const threshold = Math.min(limit, 3);++ // standard stops climbing once earlier steps found enough; deep runs every step.++ const enough = () => depth !== 'deep' && merged.length >= threshold;++++ function add(results) {++ const added = [];++ for (const r of results) {++ if (!r || !r.url) continue;++ const key = urlKey(r.url);++ if (!key || seen.has(key)) continue;++ seen.add(key);++ added.push(r);++ merged.push(r);++ }++ return added;++ }++++ function row(step, fields) {++ explain.push({ scope: 'web', step, ...fields });++ }++++ // 1. OpenVibe's own open index: free and always first, for anonymous and signed-in callers alike.++ {++ const started = now();++ const owners = (Array.isArray(fabric.openIndexOwners) ? fabric.openIndexOwners : ['sources'])++ .filter((o) => typeof o === 'string' && o);++ const pages = await Promise.all(owners.map((owner) => searcher.run({++ text: q, filters: { facets: [], owner }, viewer, limit,++ })));++ const stepResults = [];++ const at = now();++ for (const page of pages) {++ for (const doc of page.results || []) {++ const r = openIndexResult(doc, at);++ if (r) stepResults.push(r);++ if (stepResults.length >= limit) break;++ }++ if (stepResults.length >= limit) break;++ }++ const added = add(stepResults);++ row('open_index', { ok: true, count: added.length, ms: msSince(started, now) });++ }++++ async function ladder(step, providers, { paid }) {++ const started = now();++ const skip = (why) => row(step, { ok: false, count: 0, skipped: why });++ if (enough()) return skip('earlier steps answered');++++ // A cached answer costs nothing and is served to anyone the earlier steps did not satisfy.++ const key = cacheKey([q, 'web', step, freshness, site]);++ let cached = null;++ try { cached = await cache.get(key); } catch (err) { log.warn(`[fabric] cache read: ${err.message}`); }++ if (cached) {++ const results = (Array.isArray(cached.results) ? cached.results : []).map((r) => ({ ...r, scope: 'web' }));++ const added = add(results);++ return row(step, { ok: true, count: added.length, ms: msSince(started, now), cached: true, cost_usd: 0 });++ }++++ if (depth === 'quick') return skip('quick search reads indexes only');++ if (paid && !(viewer.subject || viewer.kind === 'service')) return skip('paid providers need a signed-in person');++ if (fabric.web === false) return skip('web search is off');++ const availability = web.available();++ if (!availability.configured) return skip(availability.reason || 'not configured');++ if (paid) {++ const stats = await providerStats('deepseek');++ if (stats.calls >= dailyCap()) return skip('daily cap reached');++ }++++ const r = await web.research(q, { providers, limit });++ await recordProvider(step, r);++ if (!r.ok) return row(step, { ok: false, count: 0, ms: r.ms });++ const results = r.results.slice(0, limit);++ try {++ await cache.put(key, { scope: 'web', step, results, cost_usd: r.cost_usd }, cacheTtl(freshness));++ } catch (err) {++ log.warn(`[fabric] cache write: ${err.message}`);++ }++ add(results);++ return row(step, { ok: true, count: results.length, ms: r.ms, cost_usd: r.cost_usd });++ }++++ await ladder('deepseek', ['deepseek'], { paid: true });++ await ladder('wikipedia', ['wikipedia'], { paid: false });++ return { results: merged.slice(0, limit), next_cursor: null, explain };++ }++++ /**++ * Answer one request. Validation, defaults and QueryError are the API's contract; this function++ * never throws for a scope the caller may not use (that is a route.explain row).++ */++ async function query(body, viewer) {++ viewer = viewer || { kind: 'anonymous', subject: null };++ const checked = contracts.validate('search.fabric-request@1', body);++ if (!checked.valid) {++ const first = checked.errors[0];++ const detail = first ? `${first.path === '/' ? '' : `${first.path}: `}${first.message}` : 'body does not match search.fabric-request@1';++ throw new QueryError('search.bad_query', detail);++ }++++ const q = body.q;++ const scopes = body.scopes && body.scopes.length ? body.scopes : DEFAULT_SCOPES;++ const freshness = body.freshness || 'any';++ const depth = body.depth || 'standard';++ const limit = body.limit || 20;++ const cursor = body.cursor;++ const requested = body.filters || {};++ const site = requested.site || null;++ const askedForAnswer = body.answer === true;++++ const runs = await Promise.all(scopes.map((scope) => {++ if (scope === 'network') return runNetwork({ q, viewer, limit, cursor, requested });++ if (scope === 'mine') return runMine({ q, viewer, limit, cursor, requested });++ if (scope === 'web') return runWeb({ q, viewer, limit, freshness, site, depth });++ return runVertical({ scope, q, viewer, limit });++ }));++++ let results = [];++ let explain = [];++ const cursors = {};++ for (let i = 0; i < scopes.length; i++) {++ const run = runs[i];++ results.push(...run.results);++ explain.push(...run.explain);++ if (run.next_cursor) cursors[scopes[i]] = run.next_cursor;++ }++++ // Freshness is a property of the result, whichever scope found it; a site filter only narrows web pages.++ const at = now();++ if (freshness !== 'any') results = results.filter((r) => isFresh(r.published_at, freshness, at));++ if (site) results = results.filter((r) => r.scope !== 'web' || hostMatches(r.site, site));++ results = results.slice(0, MAX_RESULTS);++++ const groups = {};++ for (const scope of scopes) groups[scope] = results.filter((r) => r.scope === scope).length;++++ if (askedForAnswer) {++ explain.push({ scope: scopes[0], step: 'answer', ok: false, skipped: 'answers are not on yet' });++ }++++ const citations = results.filter((r) => r.scope === 'web').slice(0, MAX_CITATIONS).map((r, i) => ({++ n: i + 1,++ url: r.url,++ title: r.title,++ provider: r.provider,++ retrieved_at: r.retrieved_at,++ }));++++ const cost_usd = explain.reduce((sum, r) => sum + (Number(r.cost_usd) || 0), 0);++ const answer = {++ results,++ groups,++ answer: null,++ citations,++ route: { explain: explain.slice(0, MAX_EXPLAIN), cost_usd },++ next_cursor: scopes.length === 1 && (scopes[0] === 'network' || scopes[0] === 'mine')++ ? (cursors[scopes[0]] || null)++ : null,++ };++++ // Development and tests catch a schema drift before it ships; production keeps answering.++ if (!config.isProduction) {++ const valid = contracts.validate('search.fabric-result@1', answer);++ if (!valid.valid) {++ const first = valid.errors[0];++ log.error(`[fabric] answer violates search.fabric-result@1: ${first ? `${first.path} ${first.message}` : 'unknown'}`);++ }++ }++ return answer;++ }++++ /** GET /api/v1/query/providers: the ladder's offers, with today's paid-provider usage. */++ async function providers() {++ const checkedAt = new Date(now()).toISOString();++ const stats = await providerStats('deepseek');++ const cap = dailyCap();++ const availability = web.available();++ const reason = !availability.configured ? availability.reason++ : fabric.web === false ? 'web search is off'++ : stats.calls >= cap ? 'daily cap reached' : null;++ return [++ {++ provider: 'open_index', scopes: ['web'], price_usd_per_call: 0,++ independent_index: true, available: true, checked_at: checkedAt,++ },++ {++ provider: 'deepseek', scopes: ['web', 'news'], price_usd_per_call: PRICE_USD,++ independent_index: false, available: !reason, reason: reason || null,++ daily_cap: cap, used_today: stats.calls,++ latency_ms_p50: stats.calls ? Math.round(stats.ms_total / stats.calls) : null,++ answered_rate: stats.calls ? stats.answered / stats.calls : null,++ terms_url: 'https://api-docs.deepseek.com/guides/search',++ checked_at: checkedAt,++ },++ {++ provider: 'wikipedia', scopes: ['web'], price_usd_per_call: 0,++ independent_index: true, available: true,++ terms_url: 'https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use',++ checked_at: checkedAt,++ },++ ];++ }++++ return { query, providers };++}++++module.exports = { createFabric, VERTICALS, urlKey, cacheTtl };nnew file mode 100644@@@ -0,0 +1,145 @@++'use strict';++/**++ * The web ladder's paid/free providers, through OpenVibe.AI's research workflow:++ *++ * POST <config.fabric.aiUrl>/api/v1/research?wait=15000++ * { query, providers, images: false, max_sources, content_chars }++ *++ * AI answers { run }; run.output is the evidence pack (sources, providers, usage). Each source becomes++ * one Search Fabric page result with its provider, URL and dates. research() never throws: a network++ * failure, a non-2xx answer or a run that did not succeed comes back as { ok: false, error } so a web++ * step can be reported in route.explain without failing the whole query.++ *++ * The token is the SDK's service-token client for audience openvibe.ai; it is fetched lazily and only++ * when the provider is configured, so development and the tests boot without AI.++ */++const { createServiceTokenClient } = require('openvibe-sdk/auth');++++const PRICE_USD = 0.00075;++const TIMEOUT_MS = 20000;++++const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', '#39': '\'', '#x27': '\'', apos: '\'' };++++/** Plain text for a snippet: tags out, the five common entities decoded, whitespace collapsed. */++function plainText(value, max = 600) {++ return String(value == null ? '' : value)++ .replace(/<[^>]*>/g, ' ')++ .replace(/&(amp|lt|gt|quot|#39|#x27|apos);/g, (_m, e) => ENTITIES[e])++ .replace(/\s+/g, ' ')++ .trim()++ .slice(0, max);++}++++/** A URL's host, lowercased, or null when it does not parse. */++function hostOf(url) {++ try { return new URL(String(url)).hostname.toLowerCase() || null; } catch { return null; }++}++++/** Short, safe error text for route.explain/status; never a token or a full response body. */++function shortError(err) {++ const text = String((err && (err.message || err.name)) || 'request failed').replace(/\s+/g, ' ').trim();++ return text.slice(0, 160) || 'request failed';++}++++function createWebProviders({ config, fetchImpl = globalThis.fetch, now = () => Date.now(), log = console }) {++ const fabric = config.fabric || {};++ const oauth = config.oauth || {};++ const aiUrl = String(fabric.aiUrl || '').replace(/\/$/, '');++ let tokens = null;++++ function available() {++ const missing = [];++ if (!aiUrl) missing.push('OV_AI_INTERNAL_URL');++ if (!oauth.clientSecret) missing.push('OV_OAUTH_CLIENT_SECRET');++ return { configured: missing.length === 0, reason: missing.length ? `${missing.join(' and ')} unset` : null };++ }++++ function tokenClient() {++ if (!tokens) {++ tokens = createServiceTokenClient({++ network: config.networkInternalUrl,++ clientId: oauth.clientId,++ clientSecret: oauth.clientSecret,++ audience: 'openvibe.ai',++ fetch: fetchImpl,++ now,++ });++ }++ return tokens;++ }++++ /** One AI research call: { results, ok, ms, cost_usd, error }. */++ async function research(q, { providers = [], limit = 20, signal = null } = {}) {++ const started = now();++ const ms = () => Math.max(0, Math.round(now() - started));++ const fail = (error) => ({ results: [], ok: false, ms: ms(), cost_usd: 0, error });++++ const availability = available();++ if (!availability.configured) return fail(availability.reason);++ if (!Array.isArray(providers) || !providers.length) return fail('no providers asked for');++++ const body = {++ query: String(q).slice(0, 200),++ providers,++ images: false,++ max_sources: Math.min(Math.max(1, Number(limit) || 20), 20),++ content_chars: 500,++ };++++ try {++ const token = await tokenClient().getToken({ audience: 'openvibe.ai' });++ const timeout = AbortSignal.timeout(TIMEOUT_MS);++ const res = await fetchImpl(`${aiUrl}/api/v1/research?wait=15000`, {++ method: 'POST',++ headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json', Accept: 'application/json' },++ body: JSON.stringify(body),++ signal: signal ? AbortSignal.any([signal, timeout]) : timeout,++ });++ if (!res.ok) return fail(`AI answered ${res.status}`);++ let json;++ try { json = await res.json(); } catch { return fail('AI answered no JSON'); }++ const run = json && json.run;++ if (!run) return fail('AI answered no run');++ if (run.status !== 'succeeded' && run.status !== 'cached') return fail(`AI run ${run.status || 'did not finish'}`);++ const output = run.output;++ if (!output || !Array.isArray(output.sources)) return fail('AI run has no sources');++++ const mapped = [];++ for (const source of output.sources) {++ if (!source || !source.url) continue;++ const url = String(source.url).slice(0, 2048);++ const site = hostOf(url);++ if (!site) continue;++ const provider = source.provider || (source.provenance && source.provenance.provider) || providers[0] || null;++ const published = source.published_at || source.publishedAt || source.date || null;++ mapped.push({++ scope: 'web',++ kind: 'page',++ title: plainText(source.title || site, 300) || site,++ url,++ snippet: plainText(source.snippet || source.summary || source.excerpt || source.content, 600) || null,++ site: site.slice(0, 253),++ image: null,++ published_at: published ? String(published).slice(0, 40) : null,++ retrieved_at: String(source.retrieved_at || output.retrieved_at || new Date(now()).toISOString()).slice(0, 40),++ provider: provider ? String(provider).slice(0, 40) : null,++ owner: null, type: null, id: null,++ score: null,++ });++ }++ // 1 for the first source, falling evenly to 1/n for the last: order is the provider's rank.++ mapped.forEach((r, i) => { r.score = 1 - i / mapped.length; });++++ const deepseek = output.providers && output.providers.deepseek;++ const billed = providers.includes('deepseek') && deepseek && deepseek.ok === true && Number(deepseek.count) > 0;++ return { results: mapped, ok: true, ms: ms(), cost_usd: billed ? PRICE_USD : 0, error: null };++ } catch (err) {++ log.warn(`[fabric] AI research: ${shortError(err)}`);++ return fail(shortError(err));++ }++ }++++ return { research, available };++}++++module.exports = { createWebProviders, plainText, hostOf, PRICE_USD };@@@ -54,7 +54,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetchconfig, saved, searcher, now, log,network: networkPush || createNetworkPush({ config, fetchImpl }),});-- const app = createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher, notifier, log, now });++ const app = createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher, notifier, fetchImpl, log, now });// The background refresher is not started under the test runtime (the tests stub the fetch and load// the keys once below); a real boot refreshes on an unref'd timer. keyLoaded settles the first load.@@@ -65,6 +65,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetchnotifier.start();const pruneTimer = setInterval(async () => { try { await outbox.outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);pruneTimer.unref?.();++ // Expired Search Fabric answers are cheap to keep but pointless to serve; drop them hourly.++ const queryCache = app.locals.queryCache;++ const cachePruneTimer = setInterval(async () => { try { await queryCache.prune(); } catch (err) { log.error(`[fabric] cache prune: ${err.message}`); } }, 3600 * 1000);++ cachePruneTimer.unref?.();let server = null;if (listen) {@@@ -77,6 +81,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetchasync function close() {clearInterval(pruneTimer);++ clearInterval(cachePruneTimer);jwks.stop();await outbox.stop();await purger.stop();@@@ -88,7 +93,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetchif (!givenDb) await db.close();}-- return { config, db, engine, store, outbox, purges, purger, saved, searcher, notifier, jwks, keyLoaded, auth, app, server, close };++ return {++ config, db, engine, store, outbox, purges, purger, saved, searcher, notifier, jwks, keyLoaded, auth, app, server, close,++ queryCache, fabric: app.locals.fabric,++ };}if (require.main === module) {nnew file mode 100644@@@ -0,0 +1,242 @@++'use strict';++/**++ * Search Fabric (server/fabric, POST/GET /api/v1/query and /api/v1/query/providers):++ * network visibility, mine only for a person, the web ladder (open index, paid DeepSeek, free++ * Wikipedia), cache hits, quick depth, provider failures that do not sink the network scope, the++ * daily cap, a bad request, GET/POST parity and CORS, provider offers, and freshness/site filters.++ */++const assert = require('assert');++const contracts = require('openvibe-contracts');++const { ids } = require('openvibe-contracts');++const { boot, request, userToken, doc, suite } = require('./helpers');++++const t = suite('fabric');++const alice = ids.newId('user');++const asAlice = { token: userToken({ subjectId: alice, aud: ['openvibe.search'] }) };++const AI_ENV = { OV_AI_INTERNAL_URL: 'http://ai.test', OV_OAUTH_CLIENT_SECRET: 'test-ai-secret', SEARCH_WEB_DAILY: '500' };++const RETRIEVED = '2026-10-28T12:00:00.000Z';++++let svc;++const call = (method, p, opts = {}) => request(svc.base, method, p, opts);++++let aiCalls = [];++let research = null;++++function response(status, json) {++ return { ok: status >= 200 && status < 300, status, json: async () => json, text: async () => JSON.stringify(json) };++}++++function runWith(sources, providers = {}, retrievedAt = RETRIEVED) {++ return { run: { status: 'succeeded', output: { sources, providers, usage: {}, retrieved_at: retrievedAt } } };++}++++function wikiSource(overrides = {}) {++ return {++ source_type: 'wikipedia', url: 'https://en.wikipedia.org/wiki/Test', title: 'Test — Wikipedia',++ snippet: '<p>An <b>article</b> & its summary.</p>', retrieved_at: RETRIEVED,++ provenance: { provider: 'wikipedia' }, ...overrides,++ };++}++++function webSource(overrides = {}) {++ return {++ source_type: 'web', url: 'https://deep.example/quasar', title: 'Quasar result',++ snippet: 'A web snippet', retrieved_at: RETRIEVED, published_at: '2026-10-27T00:00:00Z',++ provenance: { provider: 'deepseek' }, ...overrides,++ };++}++++function defaultResearch(body) {++ if (body.providers.includes('deepseek')) {++ return runWith([webSource()], { deepseek: { ok: true, count: 1, ms: 10 } });++ }++ return runWith([wikiSource()], { wikipedia: { ok: true, count: 1, ms: 10 } });++}++++async function fetchImpl(url, opts = {}) {++ const u = String(url);++ if (u.includes('/oauth/token')) return response(200, { access_token: 'test-ai-token', token_type: 'Bearer', expires_in: 300 });++ if (u.includes('/api/v1/research')) {++ const body = opts.body ? JSON.parse(opts.body) : {};++ aiCalls.push(body);++ if (research) return research(body, opts);++ return response(200, defaultResearch(body));++ }++ throw new Error(`unexpected fetch ${u}`);++}++++t('boot with the web ladder configured and the fixtures indexed', async () => {++ svc = await boot({ env: AI_ENV, fetchImpl });++ await svc.store.apply(doc({ id: 'solstice-public', title: 'Solstice migration public', body: 'solstice public body', canonical_url: 'https://openvibe.wiki/p/solstice' }));++ await svc.store.apply(doc({++ id: 'solstice-private', visibility: 'private', acl: { subjects: [alice] },++ title: 'Solstice migration private', body: 'solstice private body', canonical_url: 'https://openvibe.wiki/p/solstice-private',++ }));++ await svc.store.apply(doc({ owner: 'sources', type: 'item', id: 'src_pulsar', title: 'Pulsar source page', body: 'pulsar', canonical_url: 'https://sources.example/pulsar' }));++ await svc.store.apply(doc({ id: 'failure-public', title: 'Failure report public', body: 'failure', canonical_url: 'https://openvibe.wiki/p/failure' }));++});++++t('anonymous network: public published documents only, as document results, and the answer validates', async () => {++ const r = await call('POST', '/api/v1/query', { body: { q: 'solstice', scopes: ['network'] } });++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(r.body.results.map((x) => x.id), ['solstice-public']);++ assert.ok(r.body.results.every((x) => x.scope === 'network' && x.kind === 'document' && x.owner === 'wiki' && x.type === 'page'));++ assert.strictEqual(r.body.groups.network, 1);++ assert.ok(!('q' in r.body), 'the answer never echoes the query');++ const v = contracts.validate('search.fabric-result@1', r.body);++ assert.ok(v.valid, JSON.stringify(v.errors));++});++++t('mine: anonymous is skipped, a signed-in person gets their private document', async () => {++ const anon = await call('POST', '/api/v1/query', { body: { q: 'solstice', scopes: ['mine'] } });++ assert.strictEqual(anon.status, 200);++ assert.deepStrictEqual(anon.body.results, []);++ assert.strictEqual(anon.body.route.explain.find((e) => e.scope === 'mine').skipped, 'needs a signed-in person');++++ const mine = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'solstice', scopes: ['mine'] } });++ assert.strictEqual(mine.status, 200, mine.text);++ assert.deepStrictEqual(mine.body.results.map((x) => x.id), ['solstice-private']);++ assert.ok(mine.body.results.every((x) => x.scope === 'mine' && x.kind === 'document'));++});++++t('web anonymous: the open index runs, DeepSeek is skipped, Wikipedia answers and citations are numbered', async () => {++ aiCalls = [];++ const r = await call('POST', '/api/v1/query', { body: { q: 'pulsar', scopes: ['web'] } });++ assert.strictEqual(r.status, 200, r.text);++ const explain = r.body.route.explain;++ assert.strictEqual(explain.find((e) => e.step === 'open_index').ok, true);++ assert.strictEqual(explain.find((e) => e.step === 'deepseek').skipped, 'paid providers need a signed-in person');++ assert.strictEqual(explain.find((e) => e.step === 'wikipedia').ok, true);++ assert.deepStrictEqual(aiCalls.map((b) => b.providers), [['wikipedia']], 'only the free provider is called');++ assert.ok(r.body.results.some((x) => x.provider === 'wikipedia'), r.text);++ assert.strictEqual(r.body.citations[0].n, 1);++ assert.deepStrictEqual(r.body.citations.map((c) => c.n), r.body.citations.map((_c, i) => i + 1));++ assert.strictEqual(r.body.citations[0].url, r.body.results.find((x) => x.scope === 'web').url);++});++++t('web signed in: DeepSeek answers and costs $0.00075; the same query again is a free cache hit', async () => {++ aiCalls = [];++ const first = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quasar', scopes: ['web'] } });++ assert.strictEqual(first.status, 200, first.text);++ assert.deepStrictEqual(aiCalls[0].providers, ['deepseek']);++ assert.strictEqual(first.body.route.cost_usd, 0.00075);++ assert.strictEqual(first.body.route.explain.find((e) => e.step === 'deepseek').cost_usd, 0.00075);++ assert.ok(aiCalls.length >= 1);++++ aiCalls = [];++ const again = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quasar', scopes: ['web'] } });++ assert.strictEqual(again.status, 200, again.text);++ const deep = again.body.route.explain.find((e) => e.step === 'deepseek');++ assert.deepStrictEqual([deep.cached, deep.cost_usd, deep.ok], [true, 0, true]);++ assert.strictEqual(again.body.route.cost_usd, 0);++ assert.strictEqual(aiCalls.length, 0, 'a cache hit never reaches AI');++});++++t('depth quick reads indexes and caches only: AI is never called', async () => {++ aiCalls = [];++ const r = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quick only', scopes: ['web'], depth: 'quick' } });++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(aiCalls.length, 0);++ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'deepseek').skipped, 'quick search reads indexes only');++ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'wikipedia').skipped, 'quick search reads indexes only');++});++++t('AI answers 500: that step is not ok, while the network scope in the same request still answers', async () => {++ research = () => response(500, { error: 'boom' });++ try {++ const r = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'failure', scopes: ['network', 'web'] } });++ assert.strictEqual(r.status, 200, r.text);++ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'deepseek').ok, false);++ assert.deepStrictEqual(r.body.results.filter((x) => x.scope === 'network').map((x) => x.id), ['failure-public']);++ } finally {++ research = null;++ }++});++++t('daily cap: a second distinct signed-in query skips DeepSeek with daily cap reached', async () => {++ const calls = [];++ const capFetch = async (url, opts = {}) => {++ const u = String(url);++ if (u.includes('/oauth/token')) return response(200, { access_token: 'cap-token', token_type: 'Bearer', expires_in: 300 });++ if (u.includes('/api/v1/research')) {++ const body = opts.body ? JSON.parse(opts.body) : {};++ calls.push(body);++ return response(200, defaultResearch(body));++ }++ throw new Error(`unexpected fetch ${u}`);++ };++ const capSvc = await boot({ env: { ...AI_ENV, SEARCH_WEB_DAILY: '1' }, fetchImpl: capFetch });++ try {++ const one = await request(capSvc.base, 'POST', '/api/v1/query', { ...asAlice, body: { q: 'cap one', scopes: ['web'] } });++ assert.strictEqual(one.status, 200, one.text);++ assert.strictEqual(one.body.route.explain.find((e) => e.step === 'deepseek').ok, true);++ const two = await request(capSvc.base, 'POST', '/api/v1/query', { ...asAlice, body: { q: 'cap two', scopes: ['web'] } });++ assert.strictEqual(two.status, 200, two.text);++ assert.strictEqual(two.body.route.explain.find((e) => e.step === 'deepseek').skipped, 'daily cap reached');++ } finally {++ await capSvc.stop();++ }++});++++t('a bad request is 400 search.bad_query', async () => {++ const r = await call('POST', '/api/v1/query', { body: { q: 'x', scopes: ['everything'] } });++ assert.strictEqual(r.status, 400, r.text);++ assert.strictEqual(r.body.code, 'search.bad_query');++});++++t('GET /api/v1/query equals the POST answer in shape and results; CORS is on GET only', async () => {++ const posted = await call('POST', '/api/v1/query', { body: { q: 'both', scopes: ['network', 'web'] } });++ assert.strictEqual(posted.status, 200, posted.text);++ assert.strictEqual(posted.headers.get('access-control-allow-origin'), null, 'POST is not cross-site readable');++++ const got = await call('GET', '/api/v1/query?q=both&scopes=network,web');++ assert.strictEqual(got.status, 200, got.text);++ assert.strictEqual(got.headers.get('access-control-allow-origin'), '*');++ assert.deepStrictEqual(Object.keys(got.body).sort(), Object.keys(posted.body).sort());++ assert.deepStrictEqual(got.body.groups, posted.body.groups);++ assert.deepStrictEqual(got.body.results, posted.body.results, 'the second answer is the same cached page');++});++++t('provider offers validate against search.provider-offer@1', async () => {++ const r = await call('GET', '/api/v1/query/providers');++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(r.body.providers.map((p) => p.provider), ['open_index', 'deepseek', 'wikipedia']);++ for (const offer of r.body.providers) {++ const v = contracts.validate('search.provider-offer@1', offer);++ assert.ok(v.valid, `${offer.provider}: ${JSON.stringify(v.errors)}`);++ }++ const deep = r.body.providers.find((p) => p.provider === 'deepseek');++ assert.strictEqual(deep.daily_cap, 500);++ assert.ok(Number.isInteger(deep.used_today) && deep.used_today >= 1);++ assert.strictEqual(deep.terms_url, 'https://api-docs.deepseek.com/guides/search');++});++++t('freshness day drops an old dated result; filters.site keeps only that host', async () => {++ research = (body) => {++ if (body.query === 'fresh check') {++ return response(200, runWith([++ wikiSource({ url: 'https://example.com/old', title: 'Old', published_at: '2020-01-01T00:00:00Z' }),++ wikiSource({ url: 'https://example.com/new', title: 'New', published_at: new Date().toISOString() }),++ ], { wikipedia: { ok: true, count: 2, ms: 5 } }));++ }++ return response(200, runWith([++ wikiSource({ url: 'https://example.com/kept', title: 'Kept' }),++ wikiSource({ url: 'https://other.test/dropped', title: 'Dropped' }),++ ], { wikipedia: { ok: true, count: 2, ms: 5 } }));++ };++ try {++ const fresh = await call('POST', '/api/v1/query', { body: { q: 'fresh check', scopes: ['web'], freshness: 'day' } });++ assert.strictEqual(fresh.status, 200, fresh.text);++ assert.deepStrictEqual(fresh.body.results.map((x) => x.url), ['https://example.com/new']);++++ const site = await call('POST', '/api/v1/query', { body: { q: 'site check', scopes: ['web'], filters: { site: 'example.com' } } });++ assert.strictEqual(site.status, 200, site.text);++ assert.deepStrictEqual(site.body.results.map((x) => x.url), ['https://example.com/kept']);++ } finally {++ research = null;++ }++});++++t('stop', async () => { await svc.stop(); });++++t.run();
@@ -54,6 +54,16 @@ VALKEY_PREFIX=
@@ -54,6 +54,16 @@ VALKEY_PREFIX=
#SEARCH_FRESHNESS_WEIGHT=1
#SEARCH_FRESHNESS_WEIGHT=1
#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30
#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30
+# Search Fabric (README "Search Fabric"): the web ladder through OpenVibe.AI.
+# AI's internal origin; unset uses the AI manifest's internalOrigin. Needs OV_OAUTH_CLIENT_SECRET above.
+#OV_AI_INTERNAL_URL=http://127.0.0.1:4700
+# off leaves the paid/free web providers off; the open index and cached answers still answer.
+#SEARCH_WEB=on
+# Paid DeepSeek calls per day across all callers; past it the provider stands down.
+#SEARCH_WEB_DAILY=500
+# Owners whose documents the ladder's own open index reads first.
+#SEARCH_OPEN_INDEX_OWNERS=sources
+
# Saved searches per signed-in person.
# Saved searches per signed-in person.
#SEARCH_SAVED_MAX_PER_SUBJECT=50
#SEARCH_SAVED_MAX_PER_SUBJECT=50
# Notifications of new matches through OpenVibe.Network (README "Saved searches"): off unless 1, and
# Notifications of new matches through OpenVibe.Network (README "Saved searches"): off unless 1, and
@@ -155,6 +155,67 @@ Search so it reads `OV_OAUTH_CLIENT_ID` and `OV_OAUTH_CLIENT_SECRET`), and
@@ -155,6 +155,67 @@ Search so it reads `OV_OAUTH_CLIENT_ID` and `OV_OAUTH_CLIENT_SECRET`), and
saved-search notifications are switched on here with `SEARCH_SAVED_NOTIFY=1`; until someone signs
saved-search notifications are switched on here with `SEARCH_SAVED_NOTIFY=1`; until someone signs
in that way, saved searches are used through Bearer tokens or a product acting for its visitor.
in that way, saved searches are used through Bearer tokens or a product acting for its visitor.
+## Search Fabric
+
+One search over several scopes, routed the cheapest way that answers (`search.fabric.query`):
+
+| Scope | What it reads |
+|---|---|
+| `network` | Search's own index, with the same visibility rules as `GET /api/v1/search` |
+| `mine` | the same search kept to the caller's non-public documents (needs a signed-in person) |
+| `web` | OpenVibe's open index, then the paid/free web providers in a ladder |
+| `news`, `video`, `products`, `places`, `code` | facets of the same index, narrowed to the owner and type Search holds |
+| `images`, `people` | nothing yet: `skipped: "not indexed yet"` in `route.explain` |
+
+The web ladder is `open_index` first (free, the configured `SEARCH_OPEN_INDEX_OWNERS`), then
+`deepseek` (paid), then `wikipedia` (free). Every step says what happened in `route.explain`: `ok`,
+`count`, `ms`, `cached`, `cost_usd`, or `skipped` with the reason. Depth `quick` reads the index and
+the cache only; `standard` stops climbing once it has `min(limit, 3)` results; `deep` runs every
+step and merges them, de-duplicating by URL.
+
+Only a signed-in person, or a first-party service, may trigger a paid provider.
+Anonymous callers get the open index and cached answers; the cache key is the normalized query,
+scope, step, freshness and site, so the same question typed two ways is one entry. DeepSeek calls are
+capped per day (`SEARCH_WEB_DAILY`, default 500); past the cap the step is skipped with
+`daily cap reached`. Cache TTLs: `day` freshness 1 h, `week` 6 h, otherwise 24 h.
+
+```http
+POST /api/v1/query { "q": "solar eclipse", "scopes": ["network", "web"], "depth": "standard" }
+GET /api/v1/query?q=solar+eclipse&scopes=network,web
+GET /api/v1/query/providers
+```
+
+The GET query string takes `q`, `scopes` (comma-separated), `freshness`, `depth`, `limit`, `cursor`,
+`owner`, `type`, `lang` and `site`. GET and `/providers` answer `Access-Control-Allow-Origin: *`
+(anonymous rules apply); POST does not. Results are grouped in the requested scope order, at most
+`limit` per scope and 100 in all; `citations` number the web results and `next_cursor` pages the
+network or mine scope when it was the only one asked. The answer never contains the query text.
+
+```json
+{
+ "results": [
+ { "scope": "network", "kind": "document", "title": "Eclipse timings", "url": "https://openvibe.news/s/1", "site": "openvibe.news", "owner": "news", "type": "story", "id": "story_1" },
+ { "scope": "web", "kind": "page", "title": "Solar eclipse", "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "site": "en.wikipedia.org", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" }
+ ],
+ "groups": { "network": 1, "web": 1 },
+ "answer": null,
+ "citations": [{ "n": 1, "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "title": "Solar eclipse", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" }],
+ "route": {
+ "explain": [
+ { "scope": "network", "step": "index", "ok": true, "count": 1, "ms": 4 },
+ { "scope": "web", "step": "open_index", "ok": true, "count": 0, "ms": 3 },
+ { "scope": "web", "step": "deepseek", "ok": false, "count": 0, "skipped": "paid providers need a signed-in person" },
+ { "scope": "web", "step": "wikipedia", "ok": true, "count": 1, "ms": 190, "cost_usd": 0 }
+ ],
+ "cost_usd": 0
+ },
+ "next_cursor": null
+}
+```
+
+Asking `"answer": true` answers `answer: null` and adds an `answer` explain row: answers are not on
+yet.
+
## Indexing
## Indexing
Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding
Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding
@@ -269,6 +330,8 @@ signed-out visitors is not counted (it speaks for all of them). Past a limit: `4
@@ -269,6 +330,8 @@ signed-out visitors is not counted (it speaks for all of them). Past a limit: `4
| Route | Per caller |
| Route | Per caller |
|---|---|
|---|---|
| Queries, direct gets, owner reconciliation reads, saved-search reads and runs | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` (120 a minute, 3000 an hour) |
| Queries, direct gets, owner reconciliation reads, saved-search reads and runs | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` (120 a minute, 3000 an hour) |
+| `POST/GET /api/v1/query` (`search.fabric`, a paid-provider ladder) | 30 / 600 |
+| `GET /api/v1/query/providers` | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` |
| `GET /api/v1/suggest` | 300 / 6000 (a search box as the person types) |
| `GET /api/v1/suggest` | 300 / 6000 (a search box as the person types) |
| `PUT/DELETE /api/v1/documents/…` | 120 / 1200 (a catalog sync is a burst; bulk content comes through Events) |
| `PUT/DELETE /api/v1/documents/…` | 120 / 1200 (a catalog sync is a burst; bulk content comes through Events) |
| Saved-search create, delete | 30 / 300 |
| Saved-search create, delete | 30 / 300 |
@@ -293,6 +356,8 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea
@@ -293,6 +356,8 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea
- freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`)
- freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`)
- saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits
- saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits
(`test/saved-searches.test.js`)
(`test/saved-searches.test.js`)
+- the Search Fabric: network visibility, mine for a person, the web ladder, cache hits, the daily
+ cap, provider offers and GET/POST parity (`test/fabric.test.js`)
- saved-search notifications: one push per search with new hits, as its owner, the watermark only
- saved-search notifications: one push per search with new hits, as its owner, the watermark only
after a push, a failed push retried, never twice, off without the flag and the secret
after a push, a failed push retried, never twice, off without the flag and the secret
(`test/saved-notify.test.js`, `test/network-push.test.js`)
(`test/saved-notify.test.js`, `test/network-push.test.js`)
new file mode 100644
new file mode 100644
@@ -0,0 +1,21 @@
@@ -0,0 +1,21 @@
+-- phase: expand
+-- Search Fabric: cached provider answers and per-provider daily call stats (server/fabric).
+CREATE TABLE IF NOT EXISTS query_cache (
+ key text PRIMARY KEY,
+ scope text NOT NULL,
+ step text NOT NULL,
+ results jsonb NOT NULL,
+ cost_usd numeric NOT NULL DEFAULT 0,
+ created_at timestamptz NOT NULL,
+ expires_at timestamptz NOT NULL
+);
+CREATE INDEX IF NOT EXISTS query_cache_expires_at_idx ON query_cache (expires_at);
+
+CREATE TABLE IF NOT EXISTS query_provider_stats (
+ day date NOT NULL,
+ provider text NOT NULL,
+ calls integer NOT NULL DEFAULT 0,
+ answered integer NOT NULL DEFAULT 0,
+ ms_total bigint NOT NULL DEFAULT 0,
+ PRIMARY KEY (day, provider)
+);
@@ -13,7 +13,7 @@
@@ -13,7 +13,7 @@
"ajv-formats": "^3.0.1",
"ajv-formats": "^3.0.1",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -750,9 +750,9 @@
@@ -750,9 +750,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.222.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",
- "integrity": "sha512-wdUQCilq9glRMwfxEswbIj3l0QQqKDXZAi2+CR5Mlh34rZjI1Q+7GlAoR0UtNwZAgtLwXhPxFyJ/aPk8lcDG1Q==",
+ "version": "0.225.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
+ "integrity": "sha512-msLaRYa2Byyabnf/+L3aRPwkx8MI3zWbofFhwupq+RyBCdgbA4gLlkPsJu9nK/1UTCWdNdSTfWXW5pZY1/1zbQ==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -25,7 +25,7 @@
@@ -25,7 +25,7 @@
"ajv-formats": "^3.0.1",
"ajv-formats": "^3.0.1",
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -10,8 +10,9 @@
@@ -10,8 +10,9 @@
* is not counted: it speaks for all of them and limits them itself, so one budget would refuse its
* is not counted: it speaks for all of them and limits them itself, so one budget would refuse its
* whole site. Past a limit the route answers 429 problem+json `rate_limited` with Retry-After before
* whole site. Past a limit the route answers 429 problem+json `rate_limited` with Retry-After before
* it does any work; the refusal is logged once and counted in search_rate_limited_total{limit,window}.
* it does any work; the refusal is logged once and counted in search_rate_limited_total{limit,window}.
- * Read routes get SEARCH_LIMITS_MINUTE / SEARCH_LIMITS_HOUR (120 and 3000). Counters live in this
- * process: a restart forgets them.
+ * Read routes get SEARCH_LIMITS_MINUTE / SEARCH_LIMITS_HOUR (120 and 3000). The Search Fabric
+ * query (POST/GET /api/v1/query) allows 30 a minute and 600 an hour because a step may reach a paid
+ * provider; its providers list takes the defaults. Counters live in this process: a restart forgets them.
*
*
* Never limited: /api/health, /api/ready, /release.json, /metrics, and the signed Events deliveries
* Never limited: /api/health, /api/ready, /release.json, /metrics, and the signed Events deliveries
* (POST /internal/events: Events pushes at its own pace, and a 429 would only make it retry and fall
* (POST /internal/events: Events pushes at its own pace, and a 429 would only make it retry and fall
new file mode 100644
new file mode 100644
@@ -0,0 +1,60 @@
@@ -0,0 +1,60 @@
+'use strict';
+/**
+ * Search Fabric routes (capability search.fabric.query):
+ *
+ * POST /api/v1/query search.fabric-request@1 body → search.fabric-result@1
+ * GET /api/v1/query the same request in the query string (scopes comma-separated)
+ * GET /api/v1/query/providers the web ladder's offers (search.provider-offer@1)
+ *
+ * GET routes answer any origin (Access-Control-Allow-Origin *): a cross-site browser is anonymous
+ * anyway, and the answer it reads is exactly what anonymous gets. POST does not: a browser sends it
+ * with its own credentials.
+ */
+const express = require('express');
+const { withViewer, one } = require('./query');
+
+const SEARCH_LIMITS = { minute: 30, hour: 600 };
+
+/** The query string as a search.fabric-request@1 body; identity never comes from here. */
+function requestFromQuery(query) {
+ const body = { q: one(query.q) === undefined ? '' : String(one(query.q)) };
+ const scopes = one(query.scopes);
+ if (scopes !== undefined) {
+ body.scopes = String(scopes).split(',').map((s) => s.trim()).filter(Boolean);
+ }
+ for (const key of ['freshness', 'depth', 'cursor']) {
+ const value = one(query[key]);
+ if (value !== undefined) body[key] = String(value);
+ }
+ if (one(query.limit) !== undefined) body.limit = parseInt(one(query.limit), 10);
+ const filters = {};
+ for (const [param, key] of [['owner', 'owner'], ['type', 'type'], ['lang', 'lang'], ['site', 'site']]) {
+ const value = one(query[param]);
+ if (value !== undefined) filters[key] = String(value);
+ }
+ if (Object.keys(filters).length) body.filters = filters;
+ return body;
+}
+
+function fabricRouter({ fabric, auth, limits }) {
+ const router = express.Router();
+ const guarded = (limit, h) => withViewer(auth, h, limit);
+ const search = limits('search.fabric', SEARCH_LIMITS);
+ const cors = (_req, res, next) => { res.set('Access-Control-Allow-Origin', '*'); next(); };
+
+ router.post('/api/v1/query', guarded(search, async (req, res) => {
+ res.json(await fabric.query(req.body, req.viewer));
+ }));
+
+ router.get('/api/v1/query', cors, guarded(search, async (req, res) => {
+ res.json(await fabric.query(requestFromQuery(req.query), req.viewer));
+ }));
+
+ router.get('/api/v1/query/providers', cors, guarded(limits('search.fabric.providers'), async (_req, res) => {
+ res.json({ providers: await fabric.providers() });
+ }));
+
+ return router;
+}
+
+module.exports = { fabricRouter, requestFromQuery, SEARCH_LIMITS };
@@ -8,6 +8,10 @@ const { createRelease } = require('openvibe-shared/release');
@@ -8,6 +8,10 @@ const { createRelease } = require('openvibe-shared/release');
const { createSearchReadiness, registerSearchGauges } = require('./observability');
const { createSearchReadiness, registerSearchGauges } = require('./observability');
const { documentsRouter } = require('./api/documents');
const { documentsRouter } = require('./api/documents');
const { queryRouter, createSearcher } = require('./api/query');
const { queryRouter, createSearcher } = require('./api/query');
+const { fabricRouter } = require('./api/fabric');
+const { createQueryCache } = require('./fabric/cache');
+const { createWebProviders } = require('./fabric/web');
+const { createFabric } = require('./fabric');
const { createFrontData } = require('./web/front-data');
const { createFrontData } = require('./web/front-data');
const { savedRouter } = require('./api/saved');
const { savedRouter } = require('./api/saved');
const { pageRouter } = require('./web/page');
const { pageRouter } = require('./web/page');
@@ -16,7 +20,7 @@ const { createLimits } = require('./actor-limits');
@@ -16,7 +20,7 @@ const { createLimits } = require('./actor-limits');
const { createIndexNow } = require('openvibe-shared/indexnow');
const { createIndexNow } = require('openvibe-shared/indexnow');
const pkg = require('../package.json');
const pkg = require('../package.json');
-function createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher: givenSearcher = null, notifier = null, log = console, now }) {
+function createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher: givenSearcher = null, notifier = null, fetchImpl = globalThis.fetch, log = console, now }) {
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
app.set('trust proxy', 'loopback');
@@ -70,8 +74,17 @@ function createApp({ config, db, store, engine, auth, outbox, purges, purger, sa
@@ -70,8 +74,17 @@ function createApp({ config, db, store, engine, auth, outbox, purges, purger, sa
// One per-actor limiter for the API routes below (server/actor-limits.js); the Events webhook,
// One per-actor limiter for the API routes below (server/actor-limits.js); the Events webhook,
// health, ready, release.json and metrics above are never limited.
// health, ready, release.json and metrics above are never limited.
const limits = createLimits({ config, now: now || (() => Date.now()), registry: metrics.registry, log });
const limits = createLimits({ config, now: now || (() => Date.now()), registry: metrics.registry, log });
+ // The Search Fabric: one cache and one web-provider ladder for the process. The hourly prune timer
+ // lives in server/index.js, which clears it on close.
+ const clock = now || (() => Date.now());
+ const queryCache = createQueryCache({ db, now: clock });
+ const web = createWebProviders({ config, fetchImpl, now: clock, log });
+ const fabric = createFabric({ config, searcher, cache: queryCache, web, db, now: clock, log });
+ app.locals.queryCache = queryCache;
+ app.locals.fabric = fabric;
app.use(documentsRouter({ store, auth, db, outbox, purges, limits }));
app.use(documentsRouter({ store, auth, db, outbox, purges, limits }));
app.use(queryRouter({ config, store, engine, auth, searcher, limits }));
app.use(queryRouter({ config, store, engine, auth, searcher, limits }));
+ app.use(fabricRouter({ fabric, auth, limits }));
app.use(savedRouter({ config, saved, searcher, auth, limits }));
app.use(savedRouter({ config, saved, searcher, auth, limits }));
// GET / (HTML search page for browsers, the text route index otherwise), /robots.txt, /llms.txt and
// GET / (HTML search page for browsers, the text route index otherwise), /robots.txt, /llms.txt and
// /sitemap.xml (whose lastmods come from the index and the deployed release, never from the clock).
// /sitemap.xml (whose lastmods come from the index and the deployed release, never from the clock).
@@ -17,6 +17,9 @@ const num = (v, d) => {
@@ -17,6 +17,9 @@ const num = (v, d) => {
return Number.isFinite(n) ? n : d;
return Number.isFinite(n) ? n : d;
};
};
+// The AI service's loopback address, from Contracts, so a development boot reaches it without an env var.
+const AI_SERVICE = require('openvibe-contracts/manifests/services/ai.json');
+
const ZONE_ID_RE = /^[0-9a-f]{32}$/;
const ZONE_ID_RE = /^[0-9a-f]{32}$/;
const HOST_RE = /^(?=.{1,253}$)([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;
const HOST_RE = /^(?=.{1,253}$)([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;
@@ -95,6 +98,16 @@ function load(env = process.env) {
@@ -95,6 +98,16 @@ function load(env = process.env) {
},
},
maxBodyBytes: int(env.SEARCH_MAX_BODY_BYTES, 256 * 1024),
maxBodyBytes: int(env.SEARCH_MAX_BODY_BYTES, 256 * 1024),
+ // The Search Fabric's web ladder (server/fabric): OpenVibe.AI's research endpoint for the web
+ // providers, its own open index first. SEARCH_WEB=off leaves the ladder's providers off (the
+ // open index and cached answers still work); SEARCH_WEB_DAILY bounds paid DeepSeek calls per day.
+ fabric: {
+ aiUrl: strip(env.OV_AI_INTERNAL_URL || AI_SERVICE.internalOrigin || ''),
+ web: env.SEARCH_WEB !== 'off',
+ dailyCap: int(env.SEARCH_WEB_DAILY, 500),
+ openIndexOwners: list(env.SEARCH_OPEN_INDEX_OWNERS, ['sources']),
+ },
+
// Freshness weighting of full-text relevance (README "Ranking"):
// Freshness weighting of full-text relevance (README "Ranking"):
// score = bm25 × (1 + weight × 2^(−age_days / halfLifeDays)); weight 0 turns it off.
// score = bm25 × (1 + weight × 2^(−age_days / halfLifeDays)); weight 0 turns it off.
freshness: {
freshness: {
new file mode 100644
new file mode 100644
@@ -0,0 +1,62 @@
@@ -0,0 +1,62 @@
+'use strict';
+/**
+ * The Search Fabric's answer cache (query_cache, migrations/0005_fabric.sql): one provider step's
+ * results for one normalized question, kept until it expires. A hit costs nothing and never reaches
+ * a provider; a miss is the caller's to fetch and fill. `key` is shared by every caller — the query
+ * text is only ever used to compute it and is never stored or returned.
+ */
+const crypto = require('crypto');
+
+/** Lowers, collapses whitespace and trims: the same question typed two ways is one cache entry. */
+function normalizeQuery(q) {
+ return String(q == null ? '' : q).toLowerCase().replace(/\s+/g, ' ').trim();
+}
+
+/** sha256(base64url) of [q, scope, step, freshness, site], the q normalized as above. */
+function cacheKey(parts) {
+ const [q, scope, step, freshness, site] = Array.isArray(parts)
+ ? parts
+ : [parts.q, parts.scope, parts.step, parts.freshness, parts.site];
+ return crypto.createHash('sha256')
+ .update(JSON.stringify([normalizeQuery(q), scope, step, freshness, site]))
+ .digest('base64url');
+}
+
+function createQueryCache({ db, now = () => Date.now() }) {
+ const st = {
+ // An expired row is as good as absent; prune() removes it later.
+ get: db.prepare('SELECT key, scope, step, results, cost_usd FROM query_cache WHERE key = ? AND expires_at > ?'),
+ put: db.prepare(`INSERT INTO query_cache (key, scope, step, results, cost_usd, created_at, expires_at)
+ VALUES (@key, @scope, @step, @results::jsonb, @cost_usd, @created_at, @expires_at)
+ ON CONFLICT (key) DO UPDATE SET scope = EXCLUDED.scope, step = EXCLUDED.step,
+ results = EXCLUDED.results, cost_usd = EXCLUDED.cost_usd,
+ created_at = EXCLUDED.created_at, expires_at = EXCLUDED.expires_at`),
+ prune: db.prepare('DELETE FROM query_cache WHERE expires_at <= ?'),
+ };
+
+ async function get(key) {
+ const row = await st.get.get(key, new Date(now()).toISOString());
+ if (!row) return null;
+ return { scope: row.scope, step: row.step, results: row.results || [], cost_usd: Number(row.cost_usd) || 0 };
+ }
+
+ async function put(key, { scope, step, results, cost_usd = 0 }, ttlMs) {
+ const at = now();
+ await st.put.run({
+ key, scope, step,
+ results: JSON.stringify(results || []),
+ cost_usd,
+ created_at: new Date(at).toISOString(),
+ expires_at: new Date(at + ttlMs).toISOString(),
+ });
+ }
+
+ /** Drop every expired row; called at boot's hourly timer (server/index.js). */
+ async function prune() {
+ return (await st.prune.run(new Date(now()).toISOString())).changes;
+ }
+
+ return { get, put, prune };
+}
+
+module.exports = { createQueryCache, cacheKey, normalizeQuery };
new file mode 100644
new file mode 100644
@@ -0,0 +1,451 @@
@@ -0,0 +1,451 @@
+'use strict';
+/**
+ * The Search Fabric (capability search.fabric.query): one search over several scopes, routed the
+ * cheapest way that answers.
+ *
+ * network Search's own index, permission-aware exactly as the query API (searcher.run)
+ * mine the same search as the caller, keeping only what is not public
+ * web the open index first, then paid/free AI providers in a ladder, cached per step
+ * news, images, video, products, people, places, code
+ * facets of the same index, narrowed to the owner/type Search actually holds
+ *
+ * Every scope runs in parallel and comes back grouped in the requested order; route.explain has one
+ * row per step (what answered, from cache, or why it did not run) and route.cost_usd what the paid
+ * steps cost. The query text is never part of the answer: a page renders results without writing it
+ * into its HTML.
+ */
+const contracts = require('openvibe-contracts');
+const { QueryError } = require('../api/query');
+const { cacheKey } = require('./cache');
+const { hostOf, plainText, PRICE_USD } = require('./web');
+
+const DEFAULT_SCOPES = ['network'];
+const MAX_RESULTS = 100;
+const MAX_EXPLAIN = 20;
+const MAX_CITATIONS = 20;
+const FRESHNESS_MS = { day: 86400000, week: 7 * 86400000, month: 30 * 86400000, year: 365 * 86400000 };
+
+/**
+ * The vertical scopes and the document kinds Search actually indexes for them. A scope with no entry
+ * (images and people today) has nothing to narrow yet and says so instead of guessing. Several
+ * matchers are allowed where one product has two kinds (a recorded stream and an uploaded video).
+ */
+const VERTICALS = {
+ news: [{ owner: 'news', type: 'story' }],
+ video: [{ owner: 'live', type: 'vod' }, { owner: 'media-hub', type: 'video' }],
+ products: [{ owner: 'reviews', type: 'entity' }],
+ places: [{ owner: 'rent', type: 'city' }, { owner: 'rent', type: 'listing' }, { owner: 'reviews', type: 'entity' }],
+ code: [{ owner: 'codes' }],
+};
+
+function clip(value, max) {
+ return String(value == null ? '' : value).slice(0, max);
+}
+
+function msSince(start, now) {
+ return Math.max(0, Math.round(now() - start));
+}
+
+/** URL identity for de-duplication: no #fragment and no trailing slash. */
+function urlKey(url) {
+ try {
+ const u = new URL(String(url));
+ u.hash = '';
+ const s = u.toString();
+ return s.endsWith('/') ? s.slice(0, -1) : s;
+ } catch {
+ const s = String(url).split('#')[0];
+ return s.endsWith('/') ? s.slice(0, -1) : s;
+ }
+}
+
+function hostMatches(host, site) {
+ const h = String(host || '').toLowerCase();
+ const s = String(site || '').toLowerCase();
+ return Boolean(h && s) && (h === s || h.endsWith(`.${s}`));
+}
+
+function isFresh(publishedAt, freshness, at) {
+ const window = FRESHNESS_MS[freshness];
+ if (!window || !publishedAt) return true;
+ const t = Date.parse(publishedAt);
+ return !Number.isFinite(t) || t >= at - window;
+}
+
+/** One document from Search's index as a fabric result. `scope` decides the group it belongs to. */
+function documentResult(doc, scope) {
+ if (!doc || !doc.canonical_url) return null;
+ const url = clip(doc.canonical_url, 2048);
+ const snippet = doc.snippet_html ? plainText(doc.snippet_html, 600)
+ : doc.summary ? plainText(doc.summary, 600) : null;
+ const site = hostOf(url);
+ return {
+ scope,
+ kind: 'document',
+ title: clip(doc.title || url, 300),
+ url,
+ snippet: snippet || null,
+ site: site ? clip(site, 253) : null,
+ image: null,
+ published_at: doc.published_at ? clip(doc.published_at, 40) : null,
+ retrieved_at: null,
+ provider: null,
+ owner: doc.owner ? clip(doc.owner, 32) : null,
+ type: doc.type ? clip(doc.type, 32) : null,
+ id: doc.id ? clip(doc.id, 200) : null,
+ score: null,
+ };
+}
+
+/** An open-index document as a web page: it is in Search's index, but it cites like any web source. */
+function openIndexResult(doc, at) {
+ const r = documentResult(doc, 'web');
+ if (!r) return null;
+ return {
+ ...r,
+ kind: 'page',
+ retrieved_at: new Date(at).toISOString(),
+ provider: 'open_index',
+ owner: null, type: null, id: null,
+ };
+}
+
+/** Cache TTLs by freshness: a day's news is stale in an hour, the week's in six, the rest in a day. */
+function cacheTtl(freshness) {
+ if (freshness === 'day') return 3600000;
+ if (freshness === 'week') return 6 * 3600000;
+ return 24 * 3600000;
+}
+
+function createFabric({ config, searcher, cache, web, db, now = () => Date.now(), log = console }) {
+ const fabric = config.fabric || {};
+ const st = {
+ stat: db.prepare('SELECT calls, answered, ms_total FROM query_provider_stats WHERE day = ? AND provider = ?'),
+ bump: db.prepare(`INSERT INTO query_provider_stats (day, provider, calls, answered, ms_total)
+ VALUES (@day, @provider, 1, @answered, @ms_total)
+ ON CONFLICT (day, provider) DO UPDATE SET
+ calls = query_provider_stats.calls + 1,
+ answered = query_provider_stats.answered + EXCLUDED.answered,
+ ms_total = query_provider_stats.ms_total + EXCLUDED.ms_total`),
+ };
+
+ const day = () => new Date(now()).toISOString().slice(0, 10);
+ const dailyCap = () => {
+ const cap = Number(fabric.dailyCap);
+ return Number.isFinite(cap) ? cap : 500;
+ };
+
+ async function providerStats(provider) {
+ const row = await st.stat.get(day(), provider);
+ return {
+ calls: row ? Number(row.calls) : 0,
+ answered: row ? Number(row.answered) : 0,
+ ms_total: row ? Number(row.ms_total) : 0,
+ };
+ }
+
+ /** One provider call: counted whether or not it answered; `answered` only when it returned results. */
+ async function recordProvider(provider, r) {
+ await st.bump.run({
+ day: day(), provider,
+ answered: r.ok && r.results.length ? 1 : 0,
+ ms_total: Math.max(0, Math.round(Number(r.ms) || 0)),
+ });
+ }
+
+ function networkFilters(requested) {
+ const filters = { facets: [] };
+ if (requested.owner) filters.owner = requested.owner;
+ if (requested.type) filters.type = requested.type;
+ if (requested.lang) filters.language = requested.lang;
+ return filters;
+ }
+
+ function runNetwork({ q, viewer, limit, cursor, requested }) {
+ return (async () => {
+ const started = now();
+ const page = await searcher.run({ text: q, filters: networkFilters(requested), viewer, limit, cursor });
+ const results = [];
+ for (const doc of page.results || []) {
+ const r = documentResult(doc, 'network');
+ if (r) results.push(r);
+ if (results.length >= limit) break;
+ }
+ return {
+ results,
+ next_cursor: page.next_cursor || null,
+ explain: [{ scope: 'network', step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],
+ };
+ })();
+ }
+
+ function runMine({ q, viewer, limit, cursor, requested }) {
+ return (async () => {
+ if (!viewer.subject) {
+ return {
+ results: [], next_cursor: null,
+ explain: [{ scope: 'mine', step: 'index', ok: false, count: 0, skipped: 'needs a signed-in person' }],
+ };
+ }
+ const started = now();
+ const page = await searcher.run({ text: q, filters: networkFilters(requested), viewer, limit, cursor });
+ const results = [];
+ for (const doc of page.results || []) {
+ // The scope is the caller's own private material; public hits are the network scope's.
+ if (doc.visibility === 'public') continue;
+ const r = documentResult(doc, 'mine');
+ if (r) results.push(r);
+ if (results.length >= limit) break;
+ }
+ return {
+ results,
+ next_cursor: page.next_cursor || null,
+ explain: [{ scope: 'mine', step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],
+ };
+ })();
+ }
+
+ function runVertical({ scope, q, viewer, limit }) {
+ const matchers = VERTICALS[scope];
+ if (!matchers || !matchers.length) {
+ return Promise.resolve({
+ results: [], next_cursor: null,
+ explain: [{ scope, step: 'index', ok: false, count: 0, skipped: 'not indexed yet' }],
+ });
+ }
+ return (async () => {
+ const started = now();
+ const pages = await Promise.all(matchers.map((m) => searcher.run({
+ text: q, filters: { facets: [], ...m }, viewer, limit,
+ })));
+ const results = [];
+ const seen = new Set();
+ for (const page of pages) {
+ for (const doc of page.results || []) {
+ const key = `${doc.owner}/${doc.type}/${doc.id}`;
+ if (seen.has(key)) continue;
+ seen.add(key);
+ const r = documentResult(doc, scope);
+ if (r) results.push(r);
+ if (results.length >= limit) break;
+ }
+ if (results.length >= limit) break;
+ }
+ return {
+ results, next_cursor: null,
+ explain: [{ scope, step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],
+ };
+ })();
+ }
+
+ /** The web ladder and its merged, de-duplicated results (one explain row per step). */
+ async function runWeb({ q, viewer, limit, freshness, site, depth }) {
+ const explain = [];
+ const merged = [];
+ const seen = new Set();
+ const threshold = Math.min(limit, 3);
+ // standard stops climbing once earlier steps found enough; deep runs every step.
+ const enough = () => depth !== 'deep' && merged.length >= threshold;
+
+ function add(results) {
+ const added = [];
+ for (const r of results) {
+ if (!r || !r.url) continue;
+ const key = urlKey(r.url);
+ if (!key || seen.has(key)) continue;
+ seen.add(key);
+ added.push(r);
+ merged.push(r);
+ }
+ return added;
+ }
+
+ function row(step, fields) {
+ explain.push({ scope: 'web', step, ...fields });
+ }
+
+ // 1. OpenVibe's own open index: free and always first, for anonymous and signed-in callers alike.
+ {
+ const started = now();
+ const owners = (Array.isArray(fabric.openIndexOwners) ? fabric.openIndexOwners : ['sources'])
+ .filter((o) => typeof o === 'string' && o);
+ const pages = await Promise.all(owners.map((owner) => searcher.run({
+ text: q, filters: { facets: [], owner }, viewer, limit,
+ })));
+ const stepResults = [];
+ const at = now();
+ for (const page of pages) {
+ for (const doc of page.results || []) {
+ const r = openIndexResult(doc, at);
+ if (r) stepResults.push(r);
+ if (stepResults.length >= limit) break;
+ }
+ if (stepResults.length >= limit) break;
+ }
+ const added = add(stepResults);
+ row('open_index', { ok: true, count: added.length, ms: msSince(started, now) });
+ }
+
+ async function ladder(step, providers, { paid }) {
+ const started = now();
+ const skip = (why) => row(step, { ok: false, count: 0, skipped: why });
+ if (enough()) return skip('earlier steps answered');
+
+ // A cached answer costs nothing and is served to anyone the earlier steps did not satisfy.
+ const key = cacheKey([q, 'web', step, freshness, site]);
+ let cached = null;
+ try { cached = await cache.get(key); } catch (err) { log.warn(`[fabric] cache read: ${err.message}`); }
+ if (cached) {
+ const results = (Array.isArray(cached.results) ? cached.results : []).map((r) => ({ ...r, scope: 'web' }));
+ const added = add(results);
+ return row(step, { ok: true, count: added.length, ms: msSince(started, now), cached: true, cost_usd: 0 });
+ }
+
+ if (depth === 'quick') return skip('quick search reads indexes only');
+ if (paid && !(viewer.subject || viewer.kind === 'service')) return skip('paid providers need a signed-in person');
+ if (fabric.web === false) return skip('web search is off');
+ const availability = web.available();
+ if (!availability.configured) return skip(availability.reason || 'not configured');
+ if (paid) {
+ const stats = await providerStats('deepseek');
+ if (stats.calls >= dailyCap()) return skip('daily cap reached');
+ }
+
+ const r = await web.research(q, { providers, limit });
+ await recordProvider(step, r);
+ if (!r.ok) return row(step, { ok: false, count: 0, ms: r.ms });
+ const results = r.results.slice(0, limit);
+ try {
+ await cache.put(key, { scope: 'web', step, results, cost_usd: r.cost_usd }, cacheTtl(freshness));
+ } catch (err) {
+ log.warn(`[fabric] cache write: ${err.message}`);
+ }
+ add(results);
+ return row(step, { ok: true, count: results.length, ms: r.ms, cost_usd: r.cost_usd });
+ }
+
+ await ladder('deepseek', ['deepseek'], { paid: true });
+ await ladder('wikipedia', ['wikipedia'], { paid: false });
+ return { results: merged.slice(0, limit), next_cursor: null, explain };
+ }
+
+ /**
+ * Answer one request. Validation, defaults and QueryError are the API's contract; this function
+ * never throws for a scope the caller may not use (that is a route.explain row).
+ */
+ async function query(body, viewer) {
+ viewer = viewer || { kind: 'anonymous', subject: null };
+ const checked = contracts.validate('search.fabric-request@1', body);
+ if (!checked.valid) {
+ const first = checked.errors[0];
+ const detail = first ? `${first.path === '/' ? '' : `${first.path}: `}${first.message}` : 'body does not match search.fabric-request@1';
+ throw new QueryError('search.bad_query', detail);
+ }
+
+ const q = body.q;
+ const scopes = body.scopes && body.scopes.length ? body.scopes : DEFAULT_SCOPES;
+ const freshness = body.freshness || 'any';
+ const depth = body.depth || 'standard';
+ const limit = body.limit || 20;
+ const cursor = body.cursor;
+ const requested = body.filters || {};
+ const site = requested.site || null;
+ const askedForAnswer = body.answer === true;
+
+ const runs = await Promise.all(scopes.map((scope) => {
+ if (scope === 'network') return runNetwork({ q, viewer, limit, cursor, requested });
+ if (scope === 'mine') return runMine({ q, viewer, limit, cursor, requested });
+ if (scope === 'web') return runWeb({ q, viewer, limit, freshness, site, depth });
+ return runVertical({ scope, q, viewer, limit });
+ }));
+
+ let results = [];
+ let explain = [];
+ const cursors = {};
+ for (let i = 0; i < scopes.length; i++) {
+ const run = runs[i];
+ results.push(...run.results);
+ explain.push(...run.explain);
+ if (run.next_cursor) cursors[scopes[i]] = run.next_cursor;
+ }
+
+ // Freshness is a property of the result, whichever scope found it; a site filter only narrows web pages.
+ const at = now();
+ if (freshness !== 'any') results = results.filter((r) => isFresh(r.published_at, freshness, at));
+ if (site) results = results.filter((r) => r.scope !== 'web' || hostMatches(r.site, site));
+ results = results.slice(0, MAX_RESULTS);
+
+ const groups = {};
+ for (const scope of scopes) groups[scope] = results.filter((r) => r.scope === scope).length;
+
+ if (askedForAnswer) {
+ explain.push({ scope: scopes[0], step: 'answer', ok: false, skipped: 'answers are not on yet' });
+ }
+
+ const citations = results.filter((r) => r.scope === 'web').slice(0, MAX_CITATIONS).map((r, i) => ({
+ n: i + 1,
+ url: r.url,
+ title: r.title,
+ provider: r.provider,
+ retrieved_at: r.retrieved_at,
+ }));
+
+ const cost_usd = explain.reduce((sum, r) => sum + (Number(r.cost_usd) || 0), 0);
+ const answer = {
+ results,
+ groups,
+ answer: null,
+ citations,
+ route: { explain: explain.slice(0, MAX_EXPLAIN), cost_usd },
+ next_cursor: scopes.length === 1 && (scopes[0] === 'network' || scopes[0] === 'mine')
+ ? (cursors[scopes[0]] || null)
+ : null,
+ };
+
+ // Development and tests catch a schema drift before it ships; production keeps answering.
+ if (!config.isProduction) {
+ const valid = contracts.validate('search.fabric-result@1', answer);
+ if (!valid.valid) {
+ const first = valid.errors[0];
+ log.error(`[fabric] answer violates search.fabric-result@1: ${first ? `${first.path} ${first.message}` : 'unknown'}`);
+ }
+ }
+ return answer;
+ }
+
+ /** GET /api/v1/query/providers: the ladder's offers, with today's paid-provider usage. */
+ async function providers() {
+ const checkedAt = new Date(now()).toISOString();
+ const stats = await providerStats('deepseek');
+ const cap = dailyCap();
+ const availability = web.available();
+ const reason = !availability.configured ? availability.reason
+ : fabric.web === false ? 'web search is off'
+ : stats.calls >= cap ? 'daily cap reached' : null;
+ return [
+ {
+ provider: 'open_index', scopes: ['web'], price_usd_per_call: 0,
+ independent_index: true, available: true, checked_at: checkedAt,
+ },
+ {
+ provider: 'deepseek', scopes: ['web', 'news'], price_usd_per_call: PRICE_USD,
+ independent_index: false, available: !reason, reason: reason || null,
+ daily_cap: cap, used_today: stats.calls,
+ latency_ms_p50: stats.calls ? Math.round(stats.ms_total / stats.calls) : null,
+ answered_rate: stats.calls ? stats.answered / stats.calls : null,
+ terms_url: 'https://api-docs.deepseek.com/guides/search',
+ checked_at: checkedAt,
+ },
+ {
+ provider: 'wikipedia', scopes: ['web'], price_usd_per_call: 0,
+ independent_index: true, available: true,
+ terms_url: 'https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use',
+ checked_at: checkedAt,
+ },
+ ];
+ }
+
+ return { query, providers };
+}
+
+module.exports = { createFabric, VERTICALS, urlKey, cacheTtl };
new file mode 100644
new file mode 100644
@@ -0,0 +1,145 @@
@@ -0,0 +1,145 @@
+'use strict';
+/**
+ * The web ladder's paid/free providers, through OpenVibe.AI's research workflow:
+ *
+ * POST <config.fabric.aiUrl>/api/v1/research?wait=15000
+ * { query, providers, images: false, max_sources, content_chars }
+ *
+ * AI answers { run }; run.output is the evidence pack (sources, providers, usage). Each source becomes
+ * one Search Fabric page result with its provider, URL and dates. research() never throws: a network
+ * failure, a non-2xx answer or a run that did not succeed comes back as { ok: false, error } so a web
+ * step can be reported in route.explain without failing the whole query.
+ *
+ * The token is the SDK's service-token client for audience openvibe.ai; it is fetched lazily and only
+ * when the provider is configured, so development and the tests boot without AI.
+ */
+const { createServiceTokenClient } = require('openvibe-sdk/auth');
+
+const PRICE_USD = 0.00075;
+const TIMEOUT_MS = 20000;
+
+const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', '#39': '\'', '#x27': '\'', apos: '\'' };
+
+/** Plain text for a snippet: tags out, the five common entities decoded, whitespace collapsed. */
+function plainText(value, max = 600) {
+ return String(value == null ? '' : value)
+ .replace(/<[^>]*>/g, ' ')
+ .replace(/&(amp|lt|gt|quot|#39|#x27|apos);/g, (_m, e) => ENTITIES[e])
+ .replace(/\s+/g, ' ')
+ .trim()
+ .slice(0, max);
+}
+
+/** A URL's host, lowercased, or null when it does not parse. */
+function hostOf(url) {
+ try { return new URL(String(url)).hostname.toLowerCase() || null; } catch { return null; }
+}
+
+/** Short, safe error text for route.explain/status; never a token or a full response body. */
+function shortError(err) {
+ const text = String((err && (err.message || err.name)) || 'request failed').replace(/\s+/g, ' ').trim();
+ return text.slice(0, 160) || 'request failed';
+}
+
+function createWebProviders({ config, fetchImpl = globalThis.fetch, now = () => Date.now(), log = console }) {
+ const fabric = config.fabric || {};
+ const oauth = config.oauth || {};
+ const aiUrl = String(fabric.aiUrl || '').replace(/\/$/, '');
+ let tokens = null;
+
+ function available() {
+ const missing = [];
+ if (!aiUrl) missing.push('OV_AI_INTERNAL_URL');
+ if (!oauth.clientSecret) missing.push('OV_OAUTH_CLIENT_SECRET');
+ return { configured: missing.length === 0, reason: missing.length ? `${missing.join(' and ')} unset` : null };
+ }
+
+ function tokenClient() {
+ if (!tokens) {
+ tokens = createServiceTokenClient({
+ network: config.networkInternalUrl,
+ clientId: oauth.clientId,
+ clientSecret: oauth.clientSecret,
+ audience: 'openvibe.ai',
+ fetch: fetchImpl,
+ now,
+ });
+ }
+ return tokens;
+ }
+
+ /** One AI research call: { results, ok, ms, cost_usd, error }. */
+ async function research(q, { providers = [], limit = 20, signal = null } = {}) {
+ const started = now();
+ const ms = () => Math.max(0, Math.round(now() - started));
+ const fail = (error) => ({ results: [], ok: false, ms: ms(), cost_usd: 0, error });
+
+ const availability = available();
+ if (!availability.configured) return fail(availability.reason);
+ if (!Array.isArray(providers) || !providers.length) return fail('no providers asked for');
+
+ const body = {
+ query: String(q).slice(0, 200),
+ providers,
+ images: false,
+ max_sources: Math.min(Math.max(1, Number(limit) || 20), 20),
+ content_chars: 500,
+ };
+
+ try {
+ const token = await tokenClient().getToken({ audience: 'openvibe.ai' });
+ const timeout = AbortSignal.timeout(TIMEOUT_MS);
+ const res = await fetchImpl(`${aiUrl}/api/v1/research?wait=15000`, {
+ method: 'POST',
+ headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json', Accept: 'application/json' },
+ body: JSON.stringify(body),
+ signal: signal ? AbortSignal.any([signal, timeout]) : timeout,
+ });
+ if (!res.ok) return fail(`AI answered ${res.status}`);
+ let json;
+ try { json = await res.json(); } catch { return fail('AI answered no JSON'); }
+ const run = json && json.run;
+ if (!run) return fail('AI answered no run');
+ if (run.status !== 'succeeded' && run.status !== 'cached') return fail(`AI run ${run.status || 'did not finish'}`);
+ const output = run.output;
+ if (!output || !Array.isArray(output.sources)) return fail('AI run has no sources');
+
+ const mapped = [];
+ for (const source of output.sources) {
+ if (!source || !source.url) continue;
+ const url = String(source.url).slice(0, 2048);
+ const site = hostOf(url);
+ if (!site) continue;
+ const provider = source.provider || (source.provenance && source.provenance.provider) || providers[0] || null;
+ const published = source.published_at || source.publishedAt || source.date || null;
+ mapped.push({
+ scope: 'web',
+ kind: 'page',
+ title: plainText(source.title || site, 300) || site,
+ url,
+ snippet: plainText(source.snippet || source.summary || source.excerpt || source.content, 600) || null,
+ site: site.slice(0, 253),
+ image: null,
+ published_at: published ? String(published).slice(0, 40) : null,
+ retrieved_at: String(source.retrieved_at || output.retrieved_at || new Date(now()).toISOString()).slice(0, 40),
+ provider: provider ? String(provider).slice(0, 40) : null,
+ owner: null, type: null, id: null,
+ score: null,
+ });
+ }
+ // 1 for the first source, falling evenly to 1/n for the last: order is the provider's rank.
+ mapped.forEach((r, i) => { r.score = 1 - i / mapped.length; });
+
+ const deepseek = output.providers && output.providers.deepseek;
+ const billed = providers.includes('deepseek') && deepseek && deepseek.ok === true && Number(deepseek.count) > 0;
+ return { results: mapped, ok: true, ms: ms(), cost_usd: billed ? PRICE_USD : 0, error: null };
+ } catch (err) {
+ log.warn(`[fabric] AI research: ${shortError(err)}`);
+ return fail(shortError(err));
+ }
+ }
+
+ return { research, available };
+}
+
+module.exports = { createWebProviders, plainText, hostOf, PRICE_USD };
@@ -54,7 +54,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -54,7 +54,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
config, saved, searcher, now, log,
config, saved, searcher, now, log,
network: networkPush || createNetworkPush({ config, fetchImpl }),
network: networkPush || createNetworkPush({ config, fetchImpl }),
});
});
- const app = createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher, notifier, log, now });
+ const app = createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher, notifier, fetchImpl, log, now });
// The background refresher is not started under the test runtime (the tests stub the fetch and load
// The background refresher is not started under the test runtime (the tests stub the fetch and load
// the keys once below); a real boot refreshes on an unref'd timer. keyLoaded settles the first load.
// the keys once below); a real boot refreshes on an unref'd timer. keyLoaded settles the first load.
@@ -65,6 +65,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -65,6 +65,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
notifier.start();
notifier.start();
const pruneTimer = setInterval(async () => { try { await outbox.outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
const pruneTimer = setInterval(async () => { try { await outbox.outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
pruneTimer.unref?.();
pruneTimer.unref?.();
+ // Expired Search Fabric answers are cheap to keep but pointless to serve; drop them hourly.
+ const queryCache = app.locals.queryCache;
+ const cachePruneTimer = setInterval(async () => { try { await queryCache.prune(); } catch (err) { log.error(`[fabric] cache prune: ${err.message}`); } }, 3600 * 1000);
+ cachePruneTimer.unref?.();
let server = null;
let server = null;
if (listen) {
if (listen) {
@@ -77,6 +81,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -77,6 +81,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
async function close() {
async function close() {
clearInterval(pruneTimer);
clearInterval(pruneTimer);
+ clearInterval(cachePruneTimer);
jwks.stop();
jwks.stop();
await outbox.stop();
await outbox.stop();
await purger.stop();
await purger.stop();
@@ -88,7 +93,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
@@ -88,7 +93,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
if (!givenDb) await db.close();
if (!givenDb) await db.close();
}
}
- return { config, db, engine, store, outbox, purges, purger, saved, searcher, notifier, jwks, keyLoaded, auth, app, server, close };
+ return {
+ config, db, engine, store, outbox, purges, purger, saved, searcher, notifier, jwks, keyLoaded, auth, app, server, close,
+ queryCache, fabric: app.locals.fabric,
+ };
}
}
if (require.main === module) {
if (require.main === module) {
new file mode 100644
new file mode 100644
@@ -0,0 +1,242 @@
@@ -0,0 +1,242 @@
+'use strict';
+/**
+ * Search Fabric (server/fabric, POST/GET /api/v1/query and /api/v1/query/providers):
+ * network visibility, mine only for a person, the web ladder (open index, paid DeepSeek, free
+ * Wikipedia), cache hits, quick depth, provider failures that do not sink the network scope, the
+ * daily cap, a bad request, GET/POST parity and CORS, provider offers, and freshness/site filters.
+ */
+const assert = require('assert');
+const contracts = require('openvibe-contracts');
+const { ids } = require('openvibe-contracts');
+const { boot, request, userToken, doc, suite } = require('./helpers');
+
+const t = suite('fabric');
+const alice = ids.newId('user');
+const asAlice = { token: userToken({ subjectId: alice, aud: ['openvibe.search'] }) };
+const AI_ENV = { OV_AI_INTERNAL_URL: 'http://ai.test', OV_OAUTH_CLIENT_SECRET: 'test-ai-secret', SEARCH_WEB_DAILY: '500' };
+const RETRIEVED = '2026-10-28T12:00:00.000Z';
+
+let svc;
+const call = (method, p, opts = {}) => request(svc.base, method, p, opts);
+
+let aiCalls = [];
+let research = null;
+
+function response(status, json) {
+ return { ok: status >= 200 && status < 300, status, json: async () => json, text: async () => JSON.stringify(json) };
+}
+
+function runWith(sources, providers = {}, retrievedAt = RETRIEVED) {
+ return { run: { status: 'succeeded', output: { sources, providers, usage: {}, retrieved_at: retrievedAt } } };
+}
+
+function wikiSource(overrides = {}) {
+ return {
+ source_type: 'wikipedia', url: 'https://en.wikipedia.org/wiki/Test', title: 'Test — Wikipedia',
+ snippet: '<p>An <b>article</b> & its summary.</p>', retrieved_at: RETRIEVED,
+ provenance: { provider: 'wikipedia' }, ...overrides,
+ };
+}
+
+function webSource(overrides = {}) {
+ return {
+ source_type: 'web', url: 'https://deep.example/quasar', title: 'Quasar result',
+ snippet: 'A web snippet', retrieved_at: RETRIEVED, published_at: '2026-10-27T00:00:00Z',
+ provenance: { provider: 'deepseek' }, ...overrides,
+ };
+}
+
+function defaultResearch(body) {
+ if (body.providers.includes('deepseek')) {
+ return runWith([webSource()], { deepseek: { ok: true, count: 1, ms: 10 } });
+ }
+ return runWith([wikiSource()], { wikipedia: { ok: true, count: 1, ms: 10 } });
+}
+
+async function fetchImpl(url, opts = {}) {
+ const u = String(url);
+ if (u.includes('/oauth/token')) return response(200, { access_token: 'test-ai-token', token_type: 'Bearer', expires_in: 300 });
+ if (u.includes('/api/v1/research')) {
+ const body = opts.body ? JSON.parse(opts.body) : {};
+ aiCalls.push(body);
+ if (research) return research(body, opts);
+ return response(200, defaultResearch(body));
+ }
+ throw new Error(`unexpected fetch ${u}`);
+}
+
+t('boot with the web ladder configured and the fixtures indexed', async () => {
+ svc = await boot({ env: AI_ENV, fetchImpl });
+ await svc.store.apply(doc({ id: 'solstice-public', title: 'Solstice migration public', body: 'solstice public body', canonical_url: 'https://openvibe.wiki/p/solstice' }));
+ await svc.store.apply(doc({
+ id: 'solstice-private', visibility: 'private', acl: { subjects: [alice] },
+ title: 'Solstice migration private', body: 'solstice private body', canonical_url: 'https://openvibe.wiki/p/solstice-private',
+ }));
+ await svc.store.apply(doc({ owner: 'sources', type: 'item', id: 'src_pulsar', title: 'Pulsar source page', body: 'pulsar', canonical_url: 'https://sources.example/pulsar' }));
+ await svc.store.apply(doc({ id: 'failure-public', title: 'Failure report public', body: 'failure', canonical_url: 'https://openvibe.wiki/p/failure' }));
+});
+
+t('anonymous network: public published documents only, as document results, and the answer validates', async () => {
+ const r = await call('POST', '/api/v1/query', { body: { q: 'solstice', scopes: ['network'] } });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(r.body.results.map((x) => x.id), ['solstice-public']);
+ assert.ok(r.body.results.every((x) => x.scope === 'network' && x.kind === 'document' && x.owner === 'wiki' && x.type === 'page'));
+ assert.strictEqual(r.body.groups.network, 1);
+ assert.ok(!('q' in r.body), 'the answer never echoes the query');
+ const v = contracts.validate('search.fabric-result@1', r.body);
+ assert.ok(v.valid, JSON.stringify(v.errors));
+});
+
+t('mine: anonymous is skipped, a signed-in person gets their private document', async () => {
+ const anon = await call('POST', '/api/v1/query', { body: { q: 'solstice', scopes: ['mine'] } });
+ assert.strictEqual(anon.status, 200);
+ assert.deepStrictEqual(anon.body.results, []);
+ assert.strictEqual(anon.body.route.explain.find((e) => e.scope === 'mine').skipped, 'needs a signed-in person');
+
+ const mine = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'solstice', scopes: ['mine'] } });
+ assert.strictEqual(mine.status, 200, mine.text);
+ assert.deepStrictEqual(mine.body.results.map((x) => x.id), ['solstice-private']);
+ assert.ok(mine.body.results.every((x) => x.scope === 'mine' && x.kind === 'document'));
+});
+
+t('web anonymous: the open index runs, DeepSeek is skipped, Wikipedia answers and citations are numbered', async () => {
+ aiCalls = [];
+ const r = await call('POST', '/api/v1/query', { body: { q: 'pulsar', scopes: ['web'] } });
+ assert.strictEqual(r.status, 200, r.text);
+ const explain = r.body.route.explain;
+ assert.strictEqual(explain.find((e) => e.step === 'open_index').ok, true);
+ assert.strictEqual(explain.find((e) => e.step === 'deepseek').skipped, 'paid providers need a signed-in person');
+ assert.strictEqual(explain.find((e) => e.step === 'wikipedia').ok, true);
+ assert.deepStrictEqual(aiCalls.map((b) => b.providers), [['wikipedia']], 'only the free provider is called');
+ assert.ok(r.body.results.some((x) => x.provider === 'wikipedia'), r.text);
+ assert.strictEqual(r.body.citations[0].n, 1);
+ assert.deepStrictEqual(r.body.citations.map((c) => c.n), r.body.citations.map((_c, i) => i + 1));
+ assert.strictEqual(r.body.citations[0].url, r.body.results.find((x) => x.scope === 'web').url);
+});
+
+t('web signed in: DeepSeek answers and costs $0.00075; the same query again is a free cache hit', async () => {
+ aiCalls = [];
+ const first = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quasar', scopes: ['web'] } });
+ assert.strictEqual(first.status, 200, first.text);
+ assert.deepStrictEqual(aiCalls[0].providers, ['deepseek']);
+ assert.strictEqual(first.body.route.cost_usd, 0.00075);
+ assert.strictEqual(first.body.route.explain.find((e) => e.step === 'deepseek').cost_usd, 0.00075);
+ assert.ok(aiCalls.length >= 1);
+
+ aiCalls = [];
+ const again = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quasar', scopes: ['web'] } });
+ assert.strictEqual(again.status, 200, again.text);
+ const deep = again.body.route.explain.find((e) => e.step === 'deepseek');
+ assert.deepStrictEqual([deep.cached, deep.cost_usd, deep.ok], [true, 0, true]);
+ assert.strictEqual(again.body.route.cost_usd, 0);
+ assert.strictEqual(aiCalls.length, 0, 'a cache hit never reaches AI');
+});
+
+t('depth quick reads indexes and caches only: AI is never called', async () => {
+ aiCalls = [];
+ const r = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quick only', scopes: ['web'], depth: 'quick' } });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(aiCalls.length, 0);
+ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'deepseek').skipped, 'quick search reads indexes only');
+ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'wikipedia').skipped, 'quick search reads indexes only');
+});
+
+t('AI answers 500: that step is not ok, while the network scope in the same request still answers', async () => {
+ research = () => response(500, { error: 'boom' });
+ try {
+ const r = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'failure', scopes: ['network', 'web'] } });
+ assert.strictEqual(r.status, 200, r.text);
+ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'deepseek').ok, false);
+ assert.deepStrictEqual(r.body.results.filter((x) => x.scope === 'network').map((x) => x.id), ['failure-public']);
+ } finally {
+ research = null;
+ }
+});
+
+t('daily cap: a second distinct signed-in query skips DeepSeek with daily cap reached', async () => {
+ const calls = [];
+ const capFetch = async (url, opts = {}) => {
+ const u = String(url);
+ if (u.includes('/oauth/token')) return response(200, { access_token: 'cap-token', token_type: 'Bearer', expires_in: 300 });
+ if (u.includes('/api/v1/research')) {
+ const body = opts.body ? JSON.parse(opts.body) : {};
+ calls.push(body);
+ return response(200, defaultResearch(body));
+ }
+ throw new Error(`unexpected fetch ${u}`);
+ };
+ const capSvc = await boot({ env: { ...AI_ENV, SEARCH_WEB_DAILY: '1' }, fetchImpl: capFetch });
+ try {
+ const one = await request(capSvc.base, 'POST', '/api/v1/query', { ...asAlice, body: { q: 'cap one', scopes: ['web'] } });
+ assert.strictEqual(one.status, 200, one.text);
+ assert.strictEqual(one.body.route.explain.find((e) => e.step === 'deepseek').ok, true);
+ const two = await request(capSvc.base, 'POST', '/api/v1/query', { ...asAlice, body: { q: 'cap two', scopes: ['web'] } });
+ assert.strictEqual(two.status, 200, two.text);
+ assert.strictEqual(two.body.route.explain.find((e) => e.step === 'deepseek').skipped, 'daily cap reached');
+ } finally {
+ await capSvc.stop();
+ }
+});
+
+t('a bad request is 400 search.bad_query', async () => {
+ const r = await call('POST', '/api/v1/query', { body: { q: 'x', scopes: ['everything'] } });
+ assert.strictEqual(r.status, 400, r.text);
+ assert.strictEqual(r.body.code, 'search.bad_query');
+});
+
+t('GET /api/v1/query equals the POST answer in shape and results; CORS is on GET only', async () => {
+ const posted = await call('POST', '/api/v1/query', { body: { q: 'both', scopes: ['network', 'web'] } });
+ assert.strictEqual(posted.status, 200, posted.text);
+ assert.strictEqual(posted.headers.get('access-control-allow-origin'), null, 'POST is not cross-site readable');
+
+ const got = await call('GET', '/api/v1/query?q=both&scopes=network,web');
+ assert.strictEqual(got.status, 200, got.text);
+ assert.strictEqual(got.headers.get('access-control-allow-origin'), '*');
+ assert.deepStrictEqual(Object.keys(got.body).sort(), Object.keys(posted.body).sort());
+ assert.deepStrictEqual(got.body.groups, posted.body.groups);
+ assert.deepStrictEqual(got.body.results, posted.body.results, 'the second answer is the same cached page');
+});
+
+t('provider offers validate against search.provider-offer@1', async () => {
+ const r = await call('GET', '/api/v1/query/providers');
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(r.body.providers.map((p) => p.provider), ['open_index', 'deepseek', 'wikipedia']);
+ for (const offer of r.body.providers) {
+ const v = contracts.validate('search.provider-offer@1', offer);
+ assert.ok(v.valid, `${offer.provider}: ${JSON.stringify(v.errors)}`);
+ }
+ const deep = r.body.providers.find((p) => p.provider === 'deepseek');
+ assert.strictEqual(deep.daily_cap, 500);
+ assert.ok(Number.isInteger(deep.used_today) && deep.used_today >= 1);
+ assert.strictEqual(deep.terms_url, 'https://api-docs.deepseek.com/guides/search');
+});
+
+t('freshness day drops an old dated result; filters.site keeps only that host', async () => {
+ research = (body) => {
+ if (body.query === 'fresh check') {
+ return response(200, runWith([
+ wikiSource({ url: 'https://example.com/old', title: 'Old', published_at: '2020-01-01T00:00:00Z' }),
+ wikiSource({ url: 'https://example.com/new', title: 'New', published_at: new Date().toISOString() }),
+ ], { wikipedia: { ok: true, count: 2, ms: 5 } }));
+ }
+ return response(200, runWith([
+ wikiSource({ url: 'https://example.com/kept', title: 'Kept' }),
+ wikiSource({ url: 'https://other.test/dropped', title: 'Dropped' }),
+ ], { wikipedia: { ok: true, count: 2, ms: 5 } }));
+ };
+ try {
+ const fresh = await call('POST', '/api/v1/query', { body: { q: 'fresh check', scopes: ['web'], freshness: 'day' } });
+ assert.strictEqual(fresh.status, 200, fresh.text);
+ assert.deepStrictEqual(fresh.body.results.map((x) => x.url), ['https://example.com/new']);
+
+ const site = await call('POST', '/api/v1/query', { body: { q: 'site check', scopes: ['web'], filters: { site: 'example.com' } } });
+ assert.strictEqual(site.status, 200, site.text);
+ assert.deepStrictEqual(site.body.results.map((x) => x.url), ['https://example.com/kept']);
+ } finally {
+ research = null;
+ }
+});
+
+t('stop', async () => { await svc.stop(); });
+
+t.run();