openvibers/openvibe.search Commits

The Search Fabric v1 (Contracts 0.225.0 search.fabric.query): POST/GET /api/v1/query over network, mine, web and the vertical scopes; the web ladder open index → DeepSeek (signed-in, daily cap) → Wikipedia through AI's research, cached by freshness; route.explain and cost; GET /api/v1/query/providers (#119)

OP OpenVibers committed 16 days ago · 7c2b1b6e87 · 14 files, +1101 −10

Changes

ddiff --git a/.env.example b/.env.example
iindex 64b4a69..db808b0 100644
---- a/.env.example
++++ b/.env.example
@@@ -54,6 +54,16 @@ VALKEY_PREFIX=
#SEARCH_FRESHNESS_WEIGHT=1
#SEARCH_FRESHNESS_HALF_LIFE_DAYS=30
++# Search Fabric (README "Search Fabric"): the web ladder through OpenVibe.AI.
++# AI's internal origin; unset uses the AI manifest's internalOrigin. Needs OV_OAUTH_CLIENT_SECRET above.
++#OV_AI_INTERNAL_URL=http://127.0.0.1:4700
++# off leaves the paid/free web providers off; the open index and cached answers still answer.
++#SEARCH_WEB=on
++# Paid DeepSeek calls per day across all callers; past it the provider stands down.
++#SEARCH_WEB_DAILY=500
++# Owners whose documents the ladder's own open index reads first.
++#SEARCH_OPEN_INDEX_OWNERS=sources
++
# Saved searches per signed-in person.
#SEARCH_SAVED_MAX_PER_SUBJECT=50
# Notifications of new matches through OpenVibe.Network (README "Saved searches"): off unless 1, and
ddiff --git a/README.md b/README.md
iindex 2b231da..d055cdd 100644
---- a/README.md
++++ b/README.md
@@@ -155,6 +155,67 @@ Search so it reads `OV_OAUTH_CLIENT_ID` and `OV_OAUTH_CLIENT_SECRET`), and
saved-search notifications are switched on here with `SEARCH_SAVED_NOTIFY=1`; until someone signs
in that way, saved searches are used through Bearer tokens or a product acting for its visitor.
++## Search Fabric
++
++One search over several scopes, routed the cheapest way that answers (`search.fabric.query`):
++
++| Scope | What it reads |
++|---|---|
++| `network` | Search's own index, with the same visibility rules as `GET /api/v1/search` |
++| `mine` | the same search kept to the caller's non-public documents (needs a signed-in person) |
++| `web` | OpenVibe's open index, then the paid/free web providers in a ladder |
++| `news`, `video`, `products`, `places`, `code` | facets of the same index, narrowed to the owner and type Search holds |
++| `images`, `people` | nothing yet: `skipped: "not indexed yet"` in `route.explain` |
++
++The web ladder is `open_index` first (free, the configured `SEARCH_OPEN_INDEX_OWNERS`), then
++`deepseek` (paid), then `wikipedia` (free). Every step says what happened in `route.explain`: `ok`,
++`count`, `ms`, `cached`, `cost_usd`, or `skipped` with the reason. Depth `quick` reads the index and
++the cache only; `standard` stops climbing once it has `min(limit, 3)` results; `deep` runs every
++step and merges them, de-duplicating by URL.
++
++Only a signed-in person, or a first-party service, may trigger a paid provider.
++Anonymous callers get the open index and cached answers; the cache key is the normalized query,
++scope, step, freshness and site, so the same question typed two ways is one entry. DeepSeek calls are
++capped per day (`SEARCH_WEB_DAILY`, default 500); past the cap the step is skipped with
++`daily cap reached`. Cache TTLs: `day` freshness 1 h, `week` 6 h, otherwise 24 h.
++
++```http
++POST /api/v1/query { "q": "solar eclipse", "scopes": ["network", "web"], "depth": "standard" }
++GET /api/v1/query?q=solar+eclipse&scopes=network,web
++GET /api/v1/query/providers
++```
++
++The GET query string takes `q`, `scopes` (comma-separated), `freshness`, `depth`, `limit`, `cursor`,
++`owner`, `type`, `lang` and `site`. GET and `/providers` answer `Access-Control-Allow-Origin: *`
++(anonymous rules apply); POST does not. Results are grouped in the requested scope order, at most
++`limit` per scope and 100 in all; `citations` number the web results and `next_cursor` pages the
++network or mine scope when it was the only one asked. The answer never contains the query text.
++
++```json
++{
++ "results": [
++ { "scope": "network", "kind": "document", "title": "Eclipse timings", "url": "https://openvibe.news/s/1", "site": "openvibe.news", "owner": "news", "type": "story", "id": "story_1" },
++ { "scope": "web", "kind": "page", "title": "Solar eclipse", "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "site": "en.wikipedia.org", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" }
++ ],
++ "groups": { "network": 1, "web": 1 },
++ "answer": null,
++ "citations": [{ "n": 1, "url": "https://en.wikipedia.org/wiki/Solar_eclipse", "title": "Solar eclipse", "provider": "wikipedia", "retrieved_at": "2026-10-28T12:00:00.000Z" }],
++ "route": {
++ "explain": [
++ { "scope": "network", "step": "index", "ok": true, "count": 1, "ms": 4 },
++ { "scope": "web", "step": "open_index", "ok": true, "count": 0, "ms": 3 },
++ { "scope": "web", "step": "deepseek", "ok": false, "count": 0, "skipped": "paid providers need a signed-in person" },
++ { "scope": "web", "step": "wikipedia", "ok": true, "count": 1, "ms": 190, "cost_usd": 0 }
++ ],
++ "cost_usd": 0
++ },
++ "next_cursor": null
++}
++```
++
++Asking `"answer": true` answers `answer: null` and adds an `answer` explain row: answers are not on
++yet.
++
## Indexing
Owners call with an OpenVibe.Network client-credentials token (`audience=openvibe.search`) holding
@@@ -269,6 +330,8 @@ signed-out visitors is not counted (it speaks for all of them). Past a limit: `4
| Route | Per caller |
|---|---|
| Queries, direct gets, owner reconciliation reads, saved-search reads and runs | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` (120 a minute, 3000 an hour) |
++| `POST/GET /api/v1/query` (`search.fabric`, a paid-provider ladder) | 30 / 600 |
++| `GET /api/v1/query/providers` | `SEARCH_LIMITS_MINUTE` / `SEARCH_LIMITS_HOUR` |
| `GET /api/v1/suggest` | 300 / 6000 (a search box as the person types) |
| `PUT/DELETE /api/v1/documents/…` | 120 / 1200 (a catalog sync is a burst; bulk content comes through Events) |
| Saved-search create, delete | 30 / 300 |
@@@ -293,6 +356,8 @@ Never limited: the signed Events deliveries (`POST /internal/events`), `/api/hea
- freshness decay, newer-first between equals, relevance over age, stable cursors (`test/freshness.test.js`)
- saved searches: signed-in only, own only, run-time ACL, Origin check for cookies, limits
(`test/saved-searches.test.js`)
++- the Search Fabric: network visibility, mine for a person, the web ladder, cache hits, the daily
++ cap, provider offers and GET/POST parity (`test/fabric.test.js`)
- saved-search notifications: one push per search with new hits, as its owner, the watermark only
after a push, a failed push retried, never twice, off without the flag and the secret
(`test/saved-notify.test.js`, `test/network-push.test.js`)
ddiff --git a/migrations/0005_fabric.sql b/migrations/0005_fabric.sql
nnew file mode 100644
iindex 0000000..a2b924d
---- /dev/null
++++ b/migrations/0005_fabric.sql
@@@ -0,0 +1,21 @@
++-- phase: expand
++-- Search Fabric: cached provider answers and per-provider daily call stats (server/fabric).
++CREATE TABLE IF NOT EXISTS query_cache (
++ key text PRIMARY KEY,
++ scope text NOT NULL,
++ step text NOT NULL,
++ results jsonb NOT NULL,
++ cost_usd numeric NOT NULL DEFAULT 0,
++ created_at timestamptz NOT NULL,
++ expires_at timestamptz NOT NULL
++);
++CREATE INDEX IF NOT EXISTS query_cache_expires_at_idx ON query_cache (expires_at);
++
++CREATE TABLE IF NOT EXISTS query_provider_stats (
++ day date NOT NULL,
++ provider text NOT NULL,
++ calls integer NOT NULL DEFAULT 0,
++ answered integer NOT NULL DEFAULT 0,
++ ms_total bigint NOT NULL DEFAULT 0,
++ PRIMARY KEY (day, provider)
++);
ddiff --git a/package-lock.json b/package-lock.json
iindex 7bfa16c..f4721a8 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -13,7 +13,7 @@
"ajv-formats": "^3.0.1",
"dotenv": "^16.4.7",
"express": "^4.21.2",
-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",
++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0",
"pg": "^8.23.0"
@@@ -750,9 +750,9 @@
}
},
"node_modules/openvibe-contracts": {
-- "version": "0.222.0",
-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",
-- "integrity": "sha512-wdUQCilq9glRMwfxEswbIj3l0QQqKDXZAi2+CR5Mlh34rZjI1Q+7GlAoR0UtNwZAgtLwXhPxFyJ/aPk8lcDG1Q==",
++ "version": "0.225.0",
++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
++ "integrity": "sha512-msLaRYa2Byyabnf/+L3aRPwkx8MI3zWbofFhwupq+RyBCdgbA4gLlkPsJu9nK/1UTCWdNdSTfWXW5pZY1/1zbQ==",
"license": "AGPL-3.0-only",
"dependencies": {
"ajv": "^8.17.1",
ddiff --git a/package.json b/package.json
iindex 3b22425..d1e7e40 100644
---- a/package.json
++++ b/package.json
@@@ -25,7 +25,7 @@
"ajv-formats": "^3.0.1",
"dotenv": "^16.4.7",
"express": "^4.21.2",
-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.222.0",
++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.225.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.67.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.27.0",
"pg": "^8.23.0"
ddiff --git a/server/actor-limits.js b/server/actor-limits.js
iindex 31041ed..1828608 100644
---- a/server/actor-limits.js
++++ b/server/actor-limits.js
@@@ -10,8 +10,9 @@
* is not counted: it speaks for all of them and limits them itself, so one budget would refuse its
* whole site. Past a limit the route answers 429 problem+json `rate_limited` with Retry-After before
* it does any work; the refusal is logged once and counted in search_rate_limited_total{limit,window}.
-- * Read routes get SEARCH_LIMITS_MINUTE / SEARCH_LIMITS_HOUR (120 and 3000). Counters live in this
-- * process: a restart forgets them.
++ * Read routes get SEARCH_LIMITS_MINUTE / SEARCH_LIMITS_HOUR (120 and 3000). The Search Fabric
++ * query (POST/GET /api/v1/query) allows 30 a minute and 600 an hour because a step may reach a paid
++ * provider; its providers list takes the defaults. Counters live in this process: a restart forgets them.
*
* Never limited: /api/health, /api/ready, /release.json, /metrics, and the signed Events deliveries
* (POST /internal/events: Events pushes at its own pace, and a 429 would only make it retry and fall
ddiff --git a/server/api/fabric.js b/server/api/fabric.js
nnew file mode 100644
iindex 0000000..a9eb80d
---- /dev/null
++++ b/server/api/fabric.js
@@@ -0,0 +1,60 @@
++'use strict';
++/**
++ * Search Fabric routes (capability search.fabric.query):
++ *
++ * POST /api/v1/query search.fabric-request@1 body → search.fabric-result@1
++ * GET /api/v1/query the same request in the query string (scopes comma-separated)
++ * GET /api/v1/query/providers the web ladder's offers (search.provider-offer@1)
++ *
++ * GET routes answer any origin (Access-Control-Allow-Origin *): a cross-site browser is anonymous
++ * anyway, and the answer it reads is exactly what anonymous gets. POST does not: a browser sends it
++ * with its own credentials.
++ */
++const express = require('express');
++const { withViewer, one } = require('./query');
++
++const SEARCH_LIMITS = { minute: 30, hour: 600 };
++
++/** The query string as a search.fabric-request@1 body; identity never comes from here. */
++function requestFromQuery(query) {
++ const body = { q: one(query.q) === undefined ? '' : String(one(query.q)) };
++ const scopes = one(query.scopes);
++ if (scopes !== undefined) {
++ body.scopes = String(scopes).split(',').map((s) => s.trim()).filter(Boolean);
++ }
++ for (const key of ['freshness', 'depth', 'cursor']) {
++ const value = one(query[key]);
++ if (value !== undefined) body[key] = String(value);
++ }
++ if (one(query.limit) !== undefined) body.limit = parseInt(one(query.limit), 10);
++ const filters = {};
++ for (const [param, key] of [['owner', 'owner'], ['type', 'type'], ['lang', 'lang'], ['site', 'site']]) {
++ const value = one(query[param]);
++ if (value !== undefined) filters[key] = String(value);
++ }
++ if (Object.keys(filters).length) body.filters = filters;
++ return body;
++}
++
++function fabricRouter({ fabric, auth, limits }) {
++ const router = express.Router();
++ const guarded = (limit, h) => withViewer(auth, h, limit);
++ const search = limits('search.fabric', SEARCH_LIMITS);
++ const cors = (_req, res, next) => { res.set('Access-Control-Allow-Origin', '*'); next(); };
++
++ router.post('/api/v1/query', guarded(search, async (req, res) => {
++ res.json(await fabric.query(req.body, req.viewer));
++ }));
++
++ router.get('/api/v1/query', cors, guarded(search, async (req, res) => {
++ res.json(await fabric.query(requestFromQuery(req.query), req.viewer));
++ }));
++
++ router.get('/api/v1/query/providers', cors, guarded(limits('search.fabric.providers'), async (_req, res) => {
++ res.json({ providers: await fabric.providers() });
++ }));
++
++ return router;
++}
++
++module.exports = { fabricRouter, requestFromQuery, SEARCH_LIMITS };
ddiff --git a/server/app.js b/server/app.js
iindex 0b4eac9..5b92cfb 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -8,6 +8,10 @@ const { createRelease } = require('openvibe-shared/release');
const { createSearchReadiness, registerSearchGauges } = require('./observability');
const { documentsRouter } = require('./api/documents');
const { queryRouter, createSearcher } = require('./api/query');
++const { fabricRouter } = require('./api/fabric');
++const { createQueryCache } = require('./fabric/cache');
++const { createWebProviders } = require('./fabric/web');
++const { createFabric } = require('./fabric');
const { createFrontData } = require('./web/front-data');
const { savedRouter } = require('./api/saved');
const { pageRouter } = require('./web/page');
@@@ -16,7 +20,7 @@ const { createLimits } = require('./actor-limits');
const { createIndexNow } = require('openvibe-shared/indexnow');
const pkg = require('../package.json');
--function createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher: givenSearcher = null, notifier = null, log = console, now }) {
++function createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher: givenSearcher = null, notifier = null, fetchImpl = globalThis.fetch, log = console, now }) {
const app = express();
app.disable('x-powered-by');
app.set('trust proxy', 'loopback');
@@@ -70,8 +74,17 @@ function createApp({ config, db, store, engine, auth, outbox, purges, purger, sa
// One per-actor limiter for the API routes below (server/actor-limits.js); the Events webhook,
// health, ready, release.json and metrics above are never limited.
const limits = createLimits({ config, now: now || (() => Date.now()), registry: metrics.registry, log });
++ // The Search Fabric: one cache and one web-provider ladder for the process. The hourly prune timer
++ // lives in server/index.js, which clears it on close.
++ const clock = now || (() => Date.now());
++ const queryCache = createQueryCache({ db, now: clock });
++ const web = createWebProviders({ config, fetchImpl, now: clock, log });
++ const fabric = createFabric({ config, searcher, cache: queryCache, web, db, now: clock, log });
++ app.locals.queryCache = queryCache;
++ app.locals.fabric = fabric;
app.use(documentsRouter({ store, auth, db, outbox, purges, limits }));
app.use(queryRouter({ config, store, engine, auth, searcher, limits }));
++ app.use(fabricRouter({ fabric, auth, limits }));
app.use(savedRouter({ config, saved, searcher, auth, limits }));
// GET / (HTML search page for browsers, the text route index otherwise), /robots.txt, /llms.txt and
// /sitemap.xml (whose lastmods come from the index and the deployed release, never from the clock).
ddiff --git a/server/config.js b/server/config.js
iindex 907acd1..6dfac9c 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -17,6 +17,9 @@ const num = (v, d) => {
return Number.isFinite(n) ? n : d;
};
++// The AI service's loopback address, from Contracts, so a development boot reaches it without an env var.
++const AI_SERVICE = require('openvibe-contracts/manifests/services/ai.json');
++
const ZONE_ID_RE = /^[0-9a-f]{32}$/;
const HOST_RE = /^(?=.{1,253}$)([a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;
@@@ -95,6 +98,16 @@ function load(env = process.env) {
},
maxBodyBytes: int(env.SEARCH_MAX_BODY_BYTES, 256 * 1024),
++ // The Search Fabric's web ladder (server/fabric): OpenVibe.AI's research endpoint for the web
++ // providers, its own open index first. SEARCH_WEB=off leaves the ladder's providers off (the
++ // open index and cached answers still work); SEARCH_WEB_DAILY bounds paid DeepSeek calls per day.
++ fabric: {
++ aiUrl: strip(env.OV_AI_INTERNAL_URL || AI_SERVICE.internalOrigin || ''),
++ web: env.SEARCH_WEB !== 'off',
++ dailyCap: int(env.SEARCH_WEB_DAILY, 500),
++ openIndexOwners: list(env.SEARCH_OPEN_INDEX_OWNERS, ['sources']),
++ },
++
// Freshness weighting of full-text relevance (README "Ranking"):
// score = bm25 × (1 + weight × 2^(−age_days / halfLifeDays)); weight 0 turns it off.
freshness: {
ddiff --git a/server/fabric/cache.js b/server/fabric/cache.js
nnew file mode 100644
iindex 0000000..aa21023
---- /dev/null
++++ b/server/fabric/cache.js
@@@ -0,0 +1,62 @@
++'use strict';
++/**
++ * The Search Fabric's answer cache (query_cache, migrations/0005_fabric.sql): one provider step's
++ * results for one normalized question, kept until it expires. A hit costs nothing and never reaches
++ * a provider; a miss is the caller's to fetch and fill. `key` is shared by every caller — the query
++ * text is only ever used to compute it and is never stored or returned.
++ */
++const crypto = require('crypto');
++
++/** Lowers, collapses whitespace and trims: the same question typed two ways is one cache entry. */
++function normalizeQuery(q) {
++ return String(q == null ? '' : q).toLowerCase().replace(/\s+/g, ' ').trim();
++}
++
++/** sha256(base64url) of [q, scope, step, freshness, site], the q normalized as above. */
++function cacheKey(parts) {
++ const [q, scope, step, freshness, site] = Array.isArray(parts)
++ ? parts
++ : [parts.q, parts.scope, parts.step, parts.freshness, parts.site];
++ return crypto.createHash('sha256')
++ .update(JSON.stringify([normalizeQuery(q), scope, step, freshness, site]))
++ .digest('base64url');
++}
++
++function createQueryCache({ db, now = () => Date.now() }) {
++ const st = {
++ // An expired row is as good as absent; prune() removes it later.
++ get: db.prepare('SELECT key, scope, step, results, cost_usd FROM query_cache WHERE key = ? AND expires_at > ?'),
++ put: db.prepare(`INSERT INTO query_cache (key, scope, step, results, cost_usd, created_at, expires_at)
++ VALUES (@key, @scope, @step, @results::jsonb, @cost_usd, @created_at, @expires_at)
++ ON CONFLICT (key) DO UPDATE SET scope = EXCLUDED.scope, step = EXCLUDED.step,
++ results = EXCLUDED.results, cost_usd = EXCLUDED.cost_usd,
++ created_at = EXCLUDED.created_at, expires_at = EXCLUDED.expires_at`),
++ prune: db.prepare('DELETE FROM query_cache WHERE expires_at <= ?'),
++ };
++
++ async function get(key) {
++ const row = await st.get.get(key, new Date(now()).toISOString());
++ if (!row) return null;
++ return { scope: row.scope, step: row.step, results: row.results || [], cost_usd: Number(row.cost_usd) || 0 };
++ }
++
++ async function put(key, { scope, step, results, cost_usd = 0 }, ttlMs) {
++ const at = now();
++ await st.put.run({
++ key, scope, step,
++ results: JSON.stringify(results || []),
++ cost_usd,
++ created_at: new Date(at).toISOString(),
++ expires_at: new Date(at + ttlMs).toISOString(),
++ });
++ }
++
++ /** Drop every expired row; called at boot's hourly timer (server/index.js). */
++ async function prune() {
++ return (await st.prune.run(new Date(now()).toISOString())).changes;
++ }
++
++ return { get, put, prune };
++}
++
++module.exports = { createQueryCache, cacheKey, normalizeQuery };
ddiff --git a/server/fabric/index.js b/server/fabric/index.js
nnew file mode 100644
iindex 0000000..53132cd
---- /dev/null
++++ b/server/fabric/index.js
@@@ -0,0 +1,451 @@
++'use strict';
++/**
++ * The Search Fabric (capability search.fabric.query): one search over several scopes, routed the
++ * cheapest way that answers.
++ *
++ * network Search's own index, permission-aware exactly as the query API (searcher.run)
++ * mine the same search as the caller, keeping only what is not public
++ * web the open index first, then paid/free AI providers in a ladder, cached per step
++ * news, images, video, products, people, places, code
++ * facets of the same index, narrowed to the owner/type Search actually holds
++ *
++ * Every scope runs in parallel and comes back grouped in the requested order; route.explain has one
++ * row per step (what answered, from cache, or why it did not run) and route.cost_usd what the paid
++ * steps cost. The query text is never part of the answer: a page renders results without writing it
++ * into its HTML.
++ */
++const contracts = require('openvibe-contracts');
++const { QueryError } = require('../api/query');
++const { cacheKey } = require('./cache');
++const { hostOf, plainText, PRICE_USD } = require('./web');
++
++const DEFAULT_SCOPES = ['network'];
++const MAX_RESULTS = 100;
++const MAX_EXPLAIN = 20;
++const MAX_CITATIONS = 20;
++const FRESHNESS_MS = { day: 86400000, week: 7 * 86400000, month: 30 * 86400000, year: 365 * 86400000 };
++
++/**
++ * The vertical scopes and the document kinds Search actually indexes for them. A scope with no entry
++ * (images and people today) has nothing to narrow yet and says so instead of guessing. Several
++ * matchers are allowed where one product has two kinds (a recorded stream and an uploaded video).
++ */
++const VERTICALS = {
++ news: [{ owner: 'news', type: 'story' }],
++ video: [{ owner: 'live', type: 'vod' }, { owner: 'media-hub', type: 'video' }],
++ products: [{ owner: 'reviews', type: 'entity' }],
++ places: [{ owner: 'rent', type: 'city' }, { owner: 'rent', type: 'listing' }, { owner: 'reviews', type: 'entity' }],
++ code: [{ owner: 'codes' }],
++};
++
++function clip(value, max) {
++ return String(value == null ? '' : value).slice(0, max);
++}
++
++function msSince(start, now) {
++ return Math.max(0, Math.round(now() - start));
++}
++
++/** URL identity for de-duplication: no #fragment and no trailing slash. */
++function urlKey(url) {
++ try {
++ const u = new URL(String(url));
++ u.hash = '';
++ const s = u.toString();
++ return s.endsWith('/') ? s.slice(0, -1) : s;
++ } catch {
++ const s = String(url).split('#')[0];
++ return s.endsWith('/') ? s.slice(0, -1) : s;
++ }
++}
++
++function hostMatches(host, site) {
++ const h = String(host || '').toLowerCase();
++ const s = String(site || '').toLowerCase();
++ return Boolean(h && s) && (h === s || h.endsWith(`.${s}`));
++}
++
++function isFresh(publishedAt, freshness, at) {
++ const window = FRESHNESS_MS[freshness];
++ if (!window || !publishedAt) return true;
++ const t = Date.parse(publishedAt);
++ return !Number.isFinite(t) || t >= at - window;
++}
++
++/** One document from Search's index as a fabric result. `scope` decides the group it belongs to. */
++function documentResult(doc, scope) {
++ if (!doc || !doc.canonical_url) return null;
++ const url = clip(doc.canonical_url, 2048);
++ const snippet = doc.snippet_html ? plainText(doc.snippet_html, 600)
++ : doc.summary ? plainText(doc.summary, 600) : null;
++ const site = hostOf(url);
++ return {
++ scope,
++ kind: 'document',
++ title: clip(doc.title || url, 300),
++ url,
++ snippet: snippet || null,
++ site: site ? clip(site, 253) : null,
++ image: null,
++ published_at: doc.published_at ? clip(doc.published_at, 40) : null,
++ retrieved_at: null,
++ provider: null,
++ owner: doc.owner ? clip(doc.owner, 32) : null,
++ type: doc.type ? clip(doc.type, 32) : null,
++ id: doc.id ? clip(doc.id, 200) : null,
++ score: null,
++ };
++}
++
++/** An open-index document as a web page: it is in Search's index, but it cites like any web source. */
++function openIndexResult(doc, at) {
++ const r = documentResult(doc, 'web');
++ if (!r) return null;
++ return {
++ ...r,
++ kind: 'page',
++ retrieved_at: new Date(at).toISOString(),
++ provider: 'open_index',
++ owner: null, type: null, id: null,
++ };
++}
++
++/** Cache TTLs by freshness: a day's news is stale in an hour, the week's in six, the rest in a day. */
++function cacheTtl(freshness) {
++ if (freshness === 'day') return 3600000;
++ if (freshness === 'week') return 6 * 3600000;
++ return 24 * 3600000;
++}
++
++function createFabric({ config, searcher, cache, web, db, now = () => Date.now(), log = console }) {
++ const fabric = config.fabric || {};
++ const st = {
++ stat: db.prepare('SELECT calls, answered, ms_total FROM query_provider_stats WHERE day = ? AND provider = ?'),
++ bump: db.prepare(`INSERT INTO query_provider_stats (day, provider, calls, answered, ms_total)
++ VALUES (@day, @provider, 1, @answered, @ms_total)
++ ON CONFLICT (day, provider) DO UPDATE SET
++ calls = query_provider_stats.calls + 1,
++ answered = query_provider_stats.answered + EXCLUDED.answered,
++ ms_total = query_provider_stats.ms_total + EXCLUDED.ms_total`),
++ };
++
++ const day = () => new Date(now()).toISOString().slice(0, 10);
++ const dailyCap = () => {
++ const cap = Number(fabric.dailyCap);
++ return Number.isFinite(cap) ? cap : 500;
++ };
++
++ async function providerStats(provider) {
++ const row = await st.stat.get(day(), provider);
++ return {
++ calls: row ? Number(row.calls) : 0,
++ answered: row ? Number(row.answered) : 0,
++ ms_total: row ? Number(row.ms_total) : 0,
++ };
++ }
++
++ /** One provider call: counted whether or not it answered; `answered` only when it returned results. */
++ async function recordProvider(provider, r) {
++ await st.bump.run({
++ day: day(), provider,
++ answered: r.ok && r.results.length ? 1 : 0,
++ ms_total: Math.max(0, Math.round(Number(r.ms) || 0)),
++ });
++ }
++
++ function networkFilters(requested) {
++ const filters = { facets: [] };
++ if (requested.owner) filters.owner = requested.owner;
++ if (requested.type) filters.type = requested.type;
++ if (requested.lang) filters.language = requested.lang;
++ return filters;
++ }
++
++ function runNetwork({ q, viewer, limit, cursor, requested }) {
++ return (async () => {
++ const started = now();
++ const page = await searcher.run({ text: q, filters: networkFilters(requested), viewer, limit, cursor });
++ const results = [];
++ for (const doc of page.results || []) {
++ const r = documentResult(doc, 'network');
++ if (r) results.push(r);
++ if (results.length >= limit) break;
++ }
++ return {
++ results,
++ next_cursor: page.next_cursor || null,
++ explain: [{ scope: 'network', step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],
++ };
++ })();
++ }
++
++ function runMine({ q, viewer, limit, cursor, requested }) {
++ return (async () => {
++ if (!viewer.subject) {
++ return {
++ results: [], next_cursor: null,
++ explain: [{ scope: 'mine', step: 'index', ok: false, count: 0, skipped: 'needs a signed-in person' }],
++ };
++ }
++ const started = now();
++ const page = await searcher.run({ text: q, filters: networkFilters(requested), viewer, limit, cursor });
++ const results = [];
++ for (const doc of page.results || []) {
++ // The scope is the caller's own private material; public hits are the network scope's.
++ if (doc.visibility === 'public') continue;
++ const r = documentResult(doc, 'mine');
++ if (r) results.push(r);
++ if (results.length >= limit) break;
++ }
++ return {
++ results,
++ next_cursor: page.next_cursor || null,
++ explain: [{ scope: 'mine', step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],
++ };
++ })();
++ }
++
++ function runVertical({ scope, q, viewer, limit }) {
++ const matchers = VERTICALS[scope];
++ if (!matchers || !matchers.length) {
++ return Promise.resolve({
++ results: [], next_cursor: null,
++ explain: [{ scope, step: 'index', ok: false, count: 0, skipped: 'not indexed yet' }],
++ });
++ }
++ return (async () => {
++ const started = now();
++ const pages = await Promise.all(matchers.map((m) => searcher.run({
++ text: q, filters: { facets: [], ...m }, viewer, limit,
++ })));
++ const results = [];
++ const seen = new Set();
++ for (const page of pages) {
++ for (const doc of page.results || []) {
++ const key = `${doc.owner}/${doc.type}/${doc.id}`;
++ if (seen.has(key)) continue;
++ seen.add(key);
++ const r = documentResult(doc, scope);
++ if (r) results.push(r);
++ if (results.length >= limit) break;
++ }
++ if (results.length >= limit) break;
++ }
++ return {
++ results, next_cursor: null,
++ explain: [{ scope, step: 'index', ok: true, count: results.length, ms: msSince(started, now) }],
++ };
++ })();
++ }
++
++ /** The web ladder and its merged, de-duplicated results (one explain row per step). */
++ async function runWeb({ q, viewer, limit, freshness, site, depth }) {
++ const explain = [];
++ const merged = [];
++ const seen = new Set();
++ const threshold = Math.min(limit, 3);
++ // standard stops climbing once earlier steps found enough; deep runs every step.
++ const enough = () => depth !== 'deep' && merged.length >= threshold;
++
++ function add(results) {
++ const added = [];
++ for (const r of results) {
++ if (!r || !r.url) continue;
++ const key = urlKey(r.url);
++ if (!key || seen.has(key)) continue;
++ seen.add(key);
++ added.push(r);
++ merged.push(r);
++ }
++ return added;
++ }
++
++ function row(step, fields) {
++ explain.push({ scope: 'web', step, ...fields });
++ }
++
++ // 1. OpenVibe's own open index: free and always first, for anonymous and signed-in callers alike.
++ {
++ const started = now();
++ const owners = (Array.isArray(fabric.openIndexOwners) ? fabric.openIndexOwners : ['sources'])
++ .filter((o) => typeof o === 'string' && o);
++ const pages = await Promise.all(owners.map((owner) => searcher.run({
++ text: q, filters: { facets: [], owner }, viewer, limit,
++ })));
++ const stepResults = [];
++ const at = now();
++ for (const page of pages) {
++ for (const doc of page.results || []) {
++ const r = openIndexResult(doc, at);
++ if (r) stepResults.push(r);
++ if (stepResults.length >= limit) break;
++ }
++ if (stepResults.length >= limit) break;
++ }
++ const added = add(stepResults);
++ row('open_index', { ok: true, count: added.length, ms: msSince(started, now) });
++ }
++
++ async function ladder(step, providers, { paid }) {
++ const started = now();
++ const skip = (why) => row(step, { ok: false, count: 0, skipped: why });
++ if (enough()) return skip('earlier steps answered');
++
++ // A cached answer costs nothing and is served to anyone the earlier steps did not satisfy.
++ const key = cacheKey([q, 'web', step, freshness, site]);
++ let cached = null;
++ try { cached = await cache.get(key); } catch (err) { log.warn(`[fabric] cache read: ${err.message}`); }
++ if (cached) {
++ const results = (Array.isArray(cached.results) ? cached.results : []).map((r) => ({ ...r, scope: 'web' }));
++ const added = add(results);
++ return row(step, { ok: true, count: added.length, ms: msSince(started, now), cached: true, cost_usd: 0 });
++ }
++
++ if (depth === 'quick') return skip('quick search reads indexes only');
++ if (paid && !(viewer.subject || viewer.kind === 'service')) return skip('paid providers need a signed-in person');
++ if (fabric.web === false) return skip('web search is off');
++ const availability = web.available();
++ if (!availability.configured) return skip(availability.reason || 'not configured');
++ if (paid) {
++ const stats = await providerStats('deepseek');
++ if (stats.calls >= dailyCap()) return skip('daily cap reached');
++ }
++
++ const r = await web.research(q, { providers, limit });
++ await recordProvider(step, r);
++ if (!r.ok) return row(step, { ok: false, count: 0, ms: r.ms });
++ const results = r.results.slice(0, limit);
++ try {
++ await cache.put(key, { scope: 'web', step, results, cost_usd: r.cost_usd }, cacheTtl(freshness));
++ } catch (err) {
++ log.warn(`[fabric] cache write: ${err.message}`);
++ }
++ add(results);
++ return row(step, { ok: true, count: results.length, ms: r.ms, cost_usd: r.cost_usd });
++ }
++
++ await ladder('deepseek', ['deepseek'], { paid: true });
++ await ladder('wikipedia', ['wikipedia'], { paid: false });
++ return { results: merged.slice(0, limit), next_cursor: null, explain };
++ }
++
++ /**
++ * Answer one request. Validation, defaults and QueryError are the API's contract; this function
++ * never throws for a scope the caller may not use (that is a route.explain row).
++ */
++ async function query(body, viewer) {
++ viewer = viewer || { kind: 'anonymous', subject: null };
++ const checked = contracts.validate('search.fabric-request@1', body);
++ if (!checked.valid) {
++ const first = checked.errors[0];
++ const detail = first ? `${first.path === '/' ? '' : `${first.path}: `}${first.message}` : 'body does not match search.fabric-request@1';
++ throw new QueryError('search.bad_query', detail);
++ }
++
++ const q = body.q;
++ const scopes = body.scopes && body.scopes.length ? body.scopes : DEFAULT_SCOPES;
++ const freshness = body.freshness || 'any';
++ const depth = body.depth || 'standard';
++ const limit = body.limit || 20;
++ const cursor = body.cursor;
++ const requested = body.filters || {};
++ const site = requested.site || null;
++ const askedForAnswer = body.answer === true;
++
++ const runs = await Promise.all(scopes.map((scope) => {
++ if (scope === 'network') return runNetwork({ q, viewer, limit, cursor, requested });
++ if (scope === 'mine') return runMine({ q, viewer, limit, cursor, requested });
++ if (scope === 'web') return runWeb({ q, viewer, limit, freshness, site, depth });
++ return runVertical({ scope, q, viewer, limit });
++ }));
++
++ let results = [];
++ let explain = [];
++ const cursors = {};
++ for (let i = 0; i < scopes.length; i++) {
++ const run = runs[i];
++ results.push(...run.results);
++ explain.push(...run.explain);
++ if (run.next_cursor) cursors[scopes[i]] = run.next_cursor;
++ }
++
++ // Freshness is a property of the result, whichever scope found it; a site filter only narrows web pages.
++ const at = now();
++ if (freshness !== 'any') results = results.filter((r) => isFresh(r.published_at, freshness, at));
++ if (site) results = results.filter((r) => r.scope !== 'web' || hostMatches(r.site, site));
++ results = results.slice(0, MAX_RESULTS);
++
++ const groups = {};
++ for (const scope of scopes) groups[scope] = results.filter((r) => r.scope === scope).length;
++
++ if (askedForAnswer) {
++ explain.push({ scope: scopes[0], step: 'answer', ok: false, skipped: 'answers are not on yet' });
++ }
++
++ const citations = results.filter((r) => r.scope === 'web').slice(0, MAX_CITATIONS).map((r, i) => ({
++ n: i + 1,
++ url: r.url,
++ title: r.title,
++ provider: r.provider,
++ retrieved_at: r.retrieved_at,
++ }));
++
++ const cost_usd = explain.reduce((sum, r) => sum + (Number(r.cost_usd) || 0), 0);
++ const answer = {
++ results,
++ groups,
++ answer: null,
++ citations,
++ route: { explain: explain.slice(0, MAX_EXPLAIN), cost_usd },
++ next_cursor: scopes.length === 1 && (scopes[0] === 'network' || scopes[0] === 'mine')
++ ? (cursors[scopes[0]] || null)
++ : null,
++ };
++
++ // Development and tests catch a schema drift before it ships; production keeps answering.
++ if (!config.isProduction) {
++ const valid = contracts.validate('search.fabric-result@1', answer);
++ if (!valid.valid) {
++ const first = valid.errors[0];
++ log.error(`[fabric] answer violates search.fabric-result@1: ${first ? `${first.path} ${first.message}` : 'unknown'}`);
++ }
++ }
++ return answer;
++ }
++
++ /** GET /api/v1/query/providers: the ladder's offers, with today's paid-provider usage. */
++ async function providers() {
++ const checkedAt = new Date(now()).toISOString();
++ const stats = await providerStats('deepseek');
++ const cap = dailyCap();
++ const availability = web.available();
++ const reason = !availability.configured ? availability.reason
++ : fabric.web === false ? 'web search is off'
++ : stats.calls >= cap ? 'daily cap reached' : null;
++ return [
++ {
++ provider: 'open_index', scopes: ['web'], price_usd_per_call: 0,
++ independent_index: true, available: true, checked_at: checkedAt,
++ },
++ {
++ provider: 'deepseek', scopes: ['web', 'news'], price_usd_per_call: PRICE_USD,
++ independent_index: false, available: !reason, reason: reason || null,
++ daily_cap: cap, used_today: stats.calls,
++ latency_ms_p50: stats.calls ? Math.round(stats.ms_total / stats.calls) : null,
++ answered_rate: stats.calls ? stats.answered / stats.calls : null,
++ terms_url: 'https://api-docs.deepseek.com/guides/search',
++ checked_at: checkedAt,
++ },
++ {
++ provider: 'wikipedia', scopes: ['web'], price_usd_per_call: 0,
++ independent_index: true, available: true,
++ terms_url: 'https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use',
++ checked_at: checkedAt,
++ },
++ ];
++ }
++
++ return { query, providers };
++}
++
++module.exports = { createFabric, VERTICALS, urlKey, cacheTtl };
ddiff --git a/server/fabric/web.js b/server/fabric/web.js
nnew file mode 100644
iindex 0000000..629dc65
---- /dev/null
++++ b/server/fabric/web.js
@@@ -0,0 +1,145 @@
++'use strict';
++/**
++ * The web ladder's paid/free providers, through OpenVibe.AI's research workflow:
++ *
++ * POST <config.fabric.aiUrl>/api/v1/research?wait=15000
++ * { query, providers, images: false, max_sources, content_chars }
++ *
++ * AI answers { run }; run.output is the evidence pack (sources, providers, usage). Each source becomes
++ * one Search Fabric page result with its provider, URL and dates. research() never throws: a network
++ * failure, a non-2xx answer or a run that did not succeed comes back as { ok: false, error } so a web
++ * step can be reported in route.explain without failing the whole query.
++ *
++ * The token is the SDK's service-token client for audience openvibe.ai; it is fetched lazily and only
++ * when the provider is configured, so development and the tests boot without AI.
++ */
++const { createServiceTokenClient } = require('openvibe-sdk/auth');
++
++const PRICE_USD = 0.00075;
++const TIMEOUT_MS = 20000;
++
++const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', '#39': '\'', '#x27': '\'', apos: '\'' };
++
++/** Plain text for a snippet: tags out, the five common entities decoded, whitespace collapsed. */
++function plainText(value, max = 600) {
++ return String(value == null ? '' : value)
++ .replace(/<[^>]*>/g, ' ')
++ .replace(/&(amp|lt|gt|quot|#39|#x27|apos);/g, (_m, e) => ENTITIES[e])
++ .replace(/\s+/g, ' ')
++ .trim()
++ .slice(0, max);
++}
++
++/** A URL's host, lowercased, or null when it does not parse. */
++function hostOf(url) {
++ try { return new URL(String(url)).hostname.toLowerCase() || null; } catch { return null; }
++}
++
++/** Short, safe error text for route.explain/status; never a token or a full response body. */
++function shortError(err) {
++ const text = String((err && (err.message || err.name)) || 'request failed').replace(/\s+/g, ' ').trim();
++ return text.slice(0, 160) || 'request failed';
++}
++
++function createWebProviders({ config, fetchImpl = globalThis.fetch, now = () => Date.now(), log = console }) {
++ const fabric = config.fabric || {};
++ const oauth = config.oauth || {};
++ const aiUrl = String(fabric.aiUrl || '').replace(/\/$/, '');
++ let tokens = null;
++
++ function available() {
++ const missing = [];
++ if (!aiUrl) missing.push('OV_AI_INTERNAL_URL');
++ if (!oauth.clientSecret) missing.push('OV_OAUTH_CLIENT_SECRET');
++ return { configured: missing.length === 0, reason: missing.length ? `${missing.join(' and ')} unset` : null };
++ }
++
++ function tokenClient() {
++ if (!tokens) {
++ tokens = createServiceTokenClient({
++ network: config.networkInternalUrl,
++ clientId: oauth.clientId,
++ clientSecret: oauth.clientSecret,
++ audience: 'openvibe.ai',
++ fetch: fetchImpl,
++ now,
++ });
++ }
++ return tokens;
++ }
++
++ /** One AI research call: { results, ok, ms, cost_usd, error }. */
++ async function research(q, { providers = [], limit = 20, signal = null } = {}) {
++ const started = now();
++ const ms = () => Math.max(0, Math.round(now() - started));
++ const fail = (error) => ({ results: [], ok: false, ms: ms(), cost_usd: 0, error });
++
++ const availability = available();
++ if (!availability.configured) return fail(availability.reason);
++ if (!Array.isArray(providers) || !providers.length) return fail('no providers asked for');
++
++ const body = {
++ query: String(q).slice(0, 200),
++ providers,
++ images: false,
++ max_sources: Math.min(Math.max(1, Number(limit) || 20), 20),
++ content_chars: 500,
++ };
++
++ try {
++ const token = await tokenClient().getToken({ audience: 'openvibe.ai' });
++ const timeout = AbortSignal.timeout(TIMEOUT_MS);
++ const res = await fetchImpl(`${aiUrl}/api/v1/research?wait=15000`, {
++ method: 'POST',
++ headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json', Accept: 'application/json' },
++ body: JSON.stringify(body),
++ signal: signal ? AbortSignal.any([signal, timeout]) : timeout,
++ });
++ if (!res.ok) return fail(`AI answered ${res.status}`);
++ let json;
++ try { json = await res.json(); } catch { return fail('AI answered no JSON'); }
++ const run = json && json.run;
++ if (!run) return fail('AI answered no run');
++ if (run.status !== 'succeeded' && run.status !== 'cached') return fail(`AI run ${run.status || 'did not finish'}`);
++ const output = run.output;
++ if (!output || !Array.isArray(output.sources)) return fail('AI run has no sources');
++
++ const mapped = [];
++ for (const source of output.sources) {
++ if (!source || !source.url) continue;
++ const url = String(source.url).slice(0, 2048);
++ const site = hostOf(url);
++ if (!site) continue;
++ const provider = source.provider || (source.provenance && source.provenance.provider) || providers[0] || null;
++ const published = source.published_at || source.publishedAt || source.date || null;
++ mapped.push({
++ scope: 'web',
++ kind: 'page',
++ title: plainText(source.title || site, 300) || site,
++ url,
++ snippet: plainText(source.snippet || source.summary || source.excerpt || source.content, 600) || null,
++ site: site.slice(0, 253),
++ image: null,
++ published_at: published ? String(published).slice(0, 40) : null,
++ retrieved_at: String(source.retrieved_at || output.retrieved_at || new Date(now()).toISOString()).slice(0, 40),
++ provider: provider ? String(provider).slice(0, 40) : null,
++ owner: null, type: null, id: null,
++ score: null,
++ });
++ }
++ // 1 for the first source, falling evenly to 1/n for the last: order is the provider's rank.
++ mapped.forEach((r, i) => { r.score = 1 - i / mapped.length; });
++
++ const deepseek = output.providers && output.providers.deepseek;
++ const billed = providers.includes('deepseek') && deepseek && deepseek.ok === true && Number(deepseek.count) > 0;
++ return { results: mapped, ok: true, ms: ms(), cost_usd: billed ? PRICE_USD : 0, error: null };
++ } catch (err) {
++ log.warn(`[fabric] AI research: ${shortError(err)}`);
++ return fail(shortError(err));
++ }
++ }
++
++ return { research, available };
++}
++
++module.exports = { createWebProviders, plainText, hostOf, PRICE_USD };
ddiff --git a/server/index.js b/server/index.js
iindex ca9f08b..a3dc589 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -54,7 +54,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
config, saved, searcher, now, log,
network: networkPush || createNetworkPush({ config, fetchImpl }),
});
-- const app = createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher, notifier, log, now });
++ const app = createApp({ config, db, store, engine, auth, outbox, purges, purger, saved, searcher, notifier, fetchImpl, log, now });
// The background refresher is not started under the test runtime (the tests stub the fetch and load
// the keys once below); a real boot refreshes on an unref'd timer. keyLoaded settles the first load.
@@@ -65,6 +65,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
notifier.start();
const pruneTimer = setInterval(async () => { try { await outbox.outbox.prune(); } catch (err) { log.error(`[outbox] prune: ${err.message}`); } }, 6 * 3600 * 1000);
pruneTimer.unref?.();
++ // Expired Search Fabric answers are cheap to keep but pointless to serve; drop them hourly.
++ const queryCache = app.locals.queryCache;
++ const cachePruneTimer = setInterval(async () => { try { await queryCache.prune(); } catch (err) { log.error(`[fabric] cache prune: ${err.message}`); } }, 3600 * 1000);
++ cachePruneTimer.unref?.();
let server = null;
if (listen) {
@@@ -77,6 +81,7 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
async function close() {
clearInterval(pruneTimer);
++ clearInterval(cachePruneTimer);
jwks.stop();
await outbox.stop();
await purger.stop();
@@@ -88,7 +93,10 @@ async function start({ config, db: givenDb = null, now = () => Date.now(), fetch
if (!givenDb) await db.close();
}
-- return { config, db, engine, store, outbox, purges, purger, saved, searcher, notifier, jwks, keyLoaded, auth, app, server, close };
++ return {
++ config, db, engine, store, outbox, purges, purger, saved, searcher, notifier, jwks, keyLoaded, auth, app, server, close,
++ queryCache, fabric: app.locals.fabric,
++ };
}
if (require.main === module) {
ddiff --git a/test/fabric.test.js b/test/fabric.test.js
nnew file mode 100644
iindex 0000000..f3507f4
---- /dev/null
++++ b/test/fabric.test.js
@@@ -0,0 +1,242 @@
++'use strict';
++/**
++ * Search Fabric (server/fabric, POST/GET /api/v1/query and /api/v1/query/providers):
++ * network visibility, mine only for a person, the web ladder (open index, paid DeepSeek, free
++ * Wikipedia), cache hits, quick depth, provider failures that do not sink the network scope, the
++ * daily cap, a bad request, GET/POST parity and CORS, provider offers, and freshness/site filters.
++ */
++const assert = require('assert');
++const contracts = require('openvibe-contracts');
++const { ids } = require('openvibe-contracts');
++const { boot, request, userToken, doc, suite } = require('./helpers');
++
++const t = suite('fabric');
++const alice = ids.newId('user');
++const asAlice = { token: userToken({ subjectId: alice, aud: ['openvibe.search'] }) };
++const AI_ENV = { OV_AI_INTERNAL_URL: 'http://ai.test', OV_OAUTH_CLIENT_SECRET: 'test-ai-secret', SEARCH_WEB_DAILY: '500' };
++const RETRIEVED = '2026-10-28T12:00:00.000Z';
++
++let svc;
++const call = (method, p, opts = {}) => request(svc.base, method, p, opts);
++
++let aiCalls = [];
++let research = null;
++
++function response(status, json) {
++ return { ok: status >= 200 && status < 300, status, json: async () => json, text: async () => JSON.stringify(json) };
++}
++
++function runWith(sources, providers = {}, retrievedAt = RETRIEVED) {
++ return { run: { status: 'succeeded', output: { sources, providers, usage: {}, retrieved_at: retrievedAt } } };
++}
++
++function wikiSource(overrides = {}) {
++ return {
++ source_type: 'wikipedia', url: 'https://en.wikipedia.org/wiki/Test', title: 'Test — Wikipedia',
++ snippet: '<p>An <b>article</b> &amp; its summary.</p>', retrieved_at: RETRIEVED,
++ provenance: { provider: 'wikipedia' }, ...overrides,
++ };
++}
++
++function webSource(overrides = {}) {
++ return {
++ source_type: 'web', url: 'https://deep.example/quasar', title: 'Quasar result',
++ snippet: 'A web snippet', retrieved_at: RETRIEVED, published_at: '2026-10-27T00:00:00Z',
++ provenance: { provider: 'deepseek' }, ...overrides,
++ };
++}
++
++function defaultResearch(body) {
++ if (body.providers.includes('deepseek')) {
++ return runWith([webSource()], { deepseek: { ok: true, count: 1, ms: 10 } });
++ }
++ return runWith([wikiSource()], { wikipedia: { ok: true, count: 1, ms: 10 } });
++}
++
++async function fetchImpl(url, opts = {}) {
++ const u = String(url);
++ if (u.includes('/oauth/token')) return response(200, { access_token: 'test-ai-token', token_type: 'Bearer', expires_in: 300 });
++ if (u.includes('/api/v1/research')) {
++ const body = opts.body ? JSON.parse(opts.body) : {};
++ aiCalls.push(body);
++ if (research) return research(body, opts);
++ return response(200, defaultResearch(body));
++ }
++ throw new Error(`unexpected fetch ${u}`);
++}
++
++t('boot with the web ladder configured and the fixtures indexed', async () => {
++ svc = await boot({ env: AI_ENV, fetchImpl });
++ await svc.store.apply(doc({ id: 'solstice-public', title: 'Solstice migration public', body: 'solstice public body', canonical_url: 'https://openvibe.wiki/p/solstice' }));
++ await svc.store.apply(doc({
++ id: 'solstice-private', visibility: 'private', acl: { subjects: [alice] },
++ title: 'Solstice migration private', body: 'solstice private body', canonical_url: 'https://openvibe.wiki/p/solstice-private',
++ }));
++ await svc.store.apply(doc({ owner: 'sources', type: 'item', id: 'src_pulsar', title: 'Pulsar source page', body: 'pulsar', canonical_url: 'https://sources.example/pulsar' }));
++ await svc.store.apply(doc({ id: 'failure-public', title: 'Failure report public', body: 'failure', canonical_url: 'https://openvibe.wiki/p/failure' }));
++});
++
++t('anonymous network: public published documents only, as document results, and the answer validates', async () => {
++ const r = await call('POST', '/api/v1/query', { body: { q: 'solstice', scopes: ['network'] } });
++ assert.strictEqual(r.status, 200, r.text);
++ assert.deepStrictEqual(r.body.results.map((x) => x.id), ['solstice-public']);
++ assert.ok(r.body.results.every((x) => x.scope === 'network' && x.kind === 'document' && x.owner === 'wiki' && x.type === 'page'));
++ assert.strictEqual(r.body.groups.network, 1);
++ assert.ok(!('q' in r.body), 'the answer never echoes the query');
++ const v = contracts.validate('search.fabric-result@1', r.body);
++ assert.ok(v.valid, JSON.stringify(v.errors));
++});
++
++t('mine: anonymous is skipped, a signed-in person gets their private document', async () => {
++ const anon = await call('POST', '/api/v1/query', { body: { q: 'solstice', scopes: ['mine'] } });
++ assert.strictEqual(anon.status, 200);
++ assert.deepStrictEqual(anon.body.results, []);
++ assert.strictEqual(anon.body.route.explain.find((e) => e.scope === 'mine').skipped, 'needs a signed-in person');
++
++ const mine = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'solstice', scopes: ['mine'] } });
++ assert.strictEqual(mine.status, 200, mine.text);
++ assert.deepStrictEqual(mine.body.results.map((x) => x.id), ['solstice-private']);
++ assert.ok(mine.body.results.every((x) => x.scope === 'mine' && x.kind === 'document'));
++});
++
++t('web anonymous: the open index runs, DeepSeek is skipped, Wikipedia answers and citations are numbered', async () => {
++ aiCalls = [];
++ const r = await call('POST', '/api/v1/query', { body: { q: 'pulsar', scopes: ['web'] } });
++ assert.strictEqual(r.status, 200, r.text);
++ const explain = r.body.route.explain;
++ assert.strictEqual(explain.find((e) => e.step === 'open_index').ok, true);
++ assert.strictEqual(explain.find((e) => e.step === 'deepseek').skipped, 'paid providers need a signed-in person');
++ assert.strictEqual(explain.find((e) => e.step === 'wikipedia').ok, true);
++ assert.deepStrictEqual(aiCalls.map((b) => b.providers), [['wikipedia']], 'only the free provider is called');
++ assert.ok(r.body.results.some((x) => x.provider === 'wikipedia'), r.text);
++ assert.strictEqual(r.body.citations[0].n, 1);
++ assert.deepStrictEqual(r.body.citations.map((c) => c.n), r.body.citations.map((_c, i) => i + 1));
++ assert.strictEqual(r.body.citations[0].url, r.body.results.find((x) => x.scope === 'web').url);
++});
++
++t('web signed in: DeepSeek answers and costs $0.00075; the same query again is a free cache hit', async () => {
++ aiCalls = [];
++ const first = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quasar', scopes: ['web'] } });
++ assert.strictEqual(first.status, 200, first.text);
++ assert.deepStrictEqual(aiCalls[0].providers, ['deepseek']);
++ assert.strictEqual(first.body.route.cost_usd, 0.00075);
++ assert.strictEqual(first.body.route.explain.find((e) => e.step === 'deepseek').cost_usd, 0.00075);
++ assert.ok(aiCalls.length >= 1);
++
++ aiCalls = [];
++ const again = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quasar', scopes: ['web'] } });
++ assert.strictEqual(again.status, 200, again.text);
++ const deep = again.body.route.explain.find((e) => e.step === 'deepseek');
++ assert.deepStrictEqual([deep.cached, deep.cost_usd, deep.ok], [true, 0, true]);
++ assert.strictEqual(again.body.route.cost_usd, 0);
++ assert.strictEqual(aiCalls.length, 0, 'a cache hit never reaches AI');
++});
++
++t('depth quick reads indexes and caches only: AI is never called', async () => {
++ aiCalls = [];
++ const r = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'quick only', scopes: ['web'], depth: 'quick' } });
++ assert.strictEqual(r.status, 200, r.text);
++ assert.strictEqual(aiCalls.length, 0);
++ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'deepseek').skipped, 'quick search reads indexes only');
++ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'wikipedia').skipped, 'quick search reads indexes only');
++});
++
++t('AI answers 500: that step is not ok, while the network scope in the same request still answers', async () => {
++ research = () => response(500, { error: 'boom' });
++ try {
++ const r = await call('POST', '/api/v1/query', { ...asAlice, body: { q: 'failure', scopes: ['network', 'web'] } });
++ assert.strictEqual(r.status, 200, r.text);
++ assert.strictEqual(r.body.route.explain.find((e) => e.step === 'deepseek').ok, false);
++ assert.deepStrictEqual(r.body.results.filter((x) => x.scope === 'network').map((x) => x.id), ['failure-public']);
++ } finally {
++ research = null;
++ }
++});
++
++t('daily cap: a second distinct signed-in query skips DeepSeek with daily cap reached', async () => {
++ const calls = [];
++ const capFetch = async (url, opts = {}) => {
++ const u = String(url);
++ if (u.includes('/oauth/token')) return response(200, { access_token: 'cap-token', token_type: 'Bearer', expires_in: 300 });
++ if (u.includes('/api/v1/research')) {
++ const body = opts.body ? JSON.parse(opts.body) : {};
++ calls.push(body);
++ return response(200, defaultResearch(body));
++ }
++ throw new Error(`unexpected fetch ${u}`);
++ };
++ const capSvc = await boot({ env: { ...AI_ENV, SEARCH_WEB_DAILY: '1' }, fetchImpl: capFetch });
++ try {
++ const one = await request(capSvc.base, 'POST', '/api/v1/query', { ...asAlice, body: { q: 'cap one', scopes: ['web'] } });
++ assert.strictEqual(one.status, 200, one.text);
++ assert.strictEqual(one.body.route.explain.find((e) => e.step === 'deepseek').ok, true);
++ const two = await request(capSvc.base, 'POST', '/api/v1/query', { ...asAlice, body: { q: 'cap two', scopes: ['web'] } });
++ assert.strictEqual(two.status, 200, two.text);
++ assert.strictEqual(two.body.route.explain.find((e) => e.step === 'deepseek').skipped, 'daily cap reached');
++ } finally {
++ await capSvc.stop();
++ }
++});
++
++t('a bad request is 400 search.bad_query', async () => {
++ const r = await call('POST', '/api/v1/query', { body: { q: 'x', scopes: ['everything'] } });
++ assert.strictEqual(r.status, 400, r.text);
++ assert.strictEqual(r.body.code, 'search.bad_query');
++});
++
++t('GET /api/v1/query equals the POST answer in shape and results; CORS is on GET only', async () => {
++ const posted = await call('POST', '/api/v1/query', { body: { q: 'both', scopes: ['network', 'web'] } });
++ assert.strictEqual(posted.status, 200, posted.text);
++ assert.strictEqual(posted.headers.get('access-control-allow-origin'), null, 'POST is not cross-site readable');
++
++ const got = await call('GET', '/api/v1/query?q=both&scopes=network,web');
++ assert.strictEqual(got.status, 200, got.text);
++ assert.strictEqual(got.headers.get('access-control-allow-origin'), '*');
++ assert.deepStrictEqual(Object.keys(got.body).sort(), Object.keys(posted.body).sort());
++ assert.deepStrictEqual(got.body.groups, posted.body.groups);
++ assert.deepStrictEqual(got.body.results, posted.body.results, 'the second answer is the same cached page');
++});
++
++t('provider offers validate against search.provider-offer@1', async () => {
++ const r = await call('GET', '/api/v1/query/providers');
++ assert.strictEqual(r.status, 200, r.text);
++ assert.deepStrictEqual(r.body.providers.map((p) => p.provider), ['open_index', 'deepseek', 'wikipedia']);
++ for (const offer of r.body.providers) {
++ const v = contracts.validate('search.provider-offer@1', offer);
++ assert.ok(v.valid, `${offer.provider}: ${JSON.stringify(v.errors)}`);
++ }
++ const deep = r.body.providers.find((p) => p.provider === 'deepseek');
++ assert.strictEqual(deep.daily_cap, 500);
++ assert.ok(Number.isInteger(deep.used_today) && deep.used_today >= 1);
++ assert.strictEqual(deep.terms_url, 'https://api-docs.deepseek.com/guides/search');
++});
++
++t('freshness day drops an old dated result; filters.site keeps only that host', async () => {
++ research = (body) => {
++ if (body.query === 'fresh check') {
++ return response(200, runWith([
++ wikiSource({ url: 'https://example.com/old', title: 'Old', published_at: '2020-01-01T00:00:00Z' }),
++ wikiSource({ url: 'https://example.com/new', title: 'New', published_at: new Date().toISOString() }),
++ ], { wikipedia: { ok: true, count: 2, ms: 5 } }));
++ }
++ return response(200, runWith([
++ wikiSource({ url: 'https://example.com/kept', title: 'Kept' }),
++ wikiSource({ url: 'https://other.test/dropped', title: 'Dropped' }),
++ ], { wikipedia: { ok: true, count: 2, ms: 5 } }));
++ };
++ try {
++ const fresh = await call('POST', '/api/v1/query', { body: { q: 'fresh check', scopes: ['web'], freshness: 'day' } });
++ assert.strictEqual(fresh.status, 200, fresh.text);
++ assert.deepStrictEqual(fresh.body.results.map((x) => x.url), ['https://example.com/new']);
++
++ const site = await call('POST', '/api/v1/query', { body: { q: 'site check', scopes: ['web'], filters: { site: 'example.com' } } });
++ assert.strictEqual(site.status, 200, site.text);
++ assert.deepStrictEqual(site.body.results.map((x) => x.url), ['https://example.com/kept']);
++ } finally {
++ research = null;
++ }
++});
++
++t('stop', async () => { await svc.stop(); });
++
++t.run();