openvibers/openvibe.ai Commits
AI research searches through the Search Fabric, with citations and a fallback (#185)
Changes
@@@ -175,8 +175,15 @@ OV_MEDIA_URL=https://openvibe.media#AI_FETCH_MAX_IMAGE_BYTES=12582912#AI_FETCH_TIMEOUT_MS=120000--# MegaSource research (server/research/, research.gather): DeepSeek's paid web search joins the free sources when its key--# resolves. The key is read by name; set DEEPSEEK_API_KEY (or point AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).++# MegaSource research (server/research/, research.gather): web search asks the Search Fabric's web scope first++# (OpenVibe.Search POST /api/v1/query) as AI's service principal, and falls back to DeepSeek when the fabric is off,++# refuses or answers nothing. Network must grant `ai` search.query.delegate on audience openvibe.search (Search's++# viewer requires it for a service token); the route's public capability is search.fabric.query.++#OV_SEARCH_INTERNAL_URL=http://127.0.0.1:4710++#AI_RESEARCH_SEARCH=1++#AI_RESEARCH_SEARCH_TIMEOUT_MS=15000++# DeepSeek's paid search is the fallback. The key is read by name; set DEEPSEEK_API_KEY (or point++# AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).#AI_RESEARCH_DEEPSEEK=1#AI_RESEARCH_DEEPSEEK_KEY_REF=env:DEEPSEEK_API_KEY#AI_RESEARCH_DEEPSEEK_DAILY=1500@@@ -463,7 +463,7 @@ the service boots without Valkey; with it on, production refuses to boot without`research.gather` (namespace `research`, `server/research/`, `server/workflows/research.js`) answers one question frommany sources at once and returns one ranked, de-duplicated evidence pack the product workflows cite by number. No model--is called; a run is recorded like a provider call (provider `research`, model `megasource`), so DeepSeek's per-call++is called; a run is recorded like a provider call (provider `research`, model `megasource`), so the web route's per-callprice counts against the cost quotas and the daily usage.| Source | What it gives | Terms |@@@ -471,20 +471,28 @@ price counts against the cost quotas and the daily usage.| Wikipedia | the best articles' full text, their pictures (each with its licence; an article's own non-free files are marked) and the Wikidata id | CC BY-SA 4.0 || Fandom | the topic's own community wiki when it has one (from the web results' hosts, else the topic as a subdomain, confirmed by the wiki's name): pages read through its MediaWiki API, infobox rows kept as "Label: value" | CC BY-SA 3.0; pictures marked not free || Wikidata | the top article's key statements as labelled facts, and its picture | CC0 |--| DeepSeek | web search with page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each |++| Search Fabric | the web through OpenVibe.Search's web scope (`POST /api/v1/query`, capability `search.fabric.query`) first: the open index, then the providers Search routes to; every page keeps its provider, URL and dates | per provider Search used |++| DeepSeek | the fallback web search when the fabric is off, refuses or answers nothing: page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each || Commons, Openverse | freely licensed pictures (CC0, public domain, CC BY, CC BY-SA; NonCommercial and NoDerivatives are not counted as free), kept only when their title or description names the subject | per picture |Input: `query` (required), `topic` (the subject it belongs to: finds the topic's wiki), `focus` (one subject inside thetopic), `lang`, `max_sources` (≤ 20), `max_images` (≤ 24), `content_chars` (≤ 12,000), `providers` (a subset). Output:--`sources` (each with `url`, `title`, `content`, `retrieved_at` and `provenance.provider|site|license`), `images` (each with--`license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per source: ok, ms, count, or--why it was skipped or failed) and `usage` (requests, DeepSeek calls, `cost_usd`). Cached for a day per calling service.----Every source is optional and fails on its own. DeepSeek is used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`,--default `env:DEEPSEEK_API_KEY`; `AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's--run reads the free sources), within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own--answers: 402 (balance used up) for 30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web--(`test/fixtures/research-fetch.js`); none reaches the internet.++`sources` (each with `url`, `title`, `content`, `published_at`, `retrieved_at` and `provenance.provider|site|via|license`),++`images` (each with `license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per route:++ok, ms, count, cost, or why it was skipped or failed) and `usage` (requests, DeepSeek calls, fabric calls and the fabric's++`search_fabric_cost_usd`; `cost_usd` is this run's own DeepSeek calls, so a paid fabric step is not billed twice —++Search and its nested AI run pay it). Cached for a day per calling service.++++Web search asks the Search Fabric first, as AI's own service principal (audience `openvibe.search`). Search's viewer++requires `search.query.delegate` of a service token, so Network must grant `ai` that capability on `openvibe.search`++(the route's `search.fabric.query` is public). The token is requested with exactly that scope; without++`OV_OAUTH_CLIENT_SECRET` or with `AI_RESEARCH_SEARCH=0` the fabric is skipped. When Search's own web ladder calls this++endpoint for DeepSeek or Wikipedia, research keeps DeepSeek direct, so Search is never asked back in a loop. DeepSeek is++used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`, default `env:DEEPSEEK_API_KEY`;++`AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's run reads the free sources),++within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own answers: 402 (balance used up) for++30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web (`test/fixtures/research-fetch.js`); none reaches++the internet.OpenVibe.Wiki's "vibe a wiki" uses it with `wiki.generate_space` (plans the pages: main page first, each with a kind,an outline and a search phrase) and `wiki.generate_page` (writes one page: lead, sections, infobox, `[[links]]` to the@@@ -115,14 +115,24 @@ function load(env = process.env) {// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },-- // MegaSource (server/research/, workflow research.gather): DeepSeek's paid web search joins the free sources when-- // its key resolves (a secret reference, read by name) and stays within AI_RESEARCH_DEEPSEEK_DAILY calls a day.++ // MegaSource (server/research/, workflow research.gather): web search asks the Search Fabric's web scope++ // first (OpenVibe.Search POST /api/v1/query, capability search.fabric.query) as AI's service principal;++ // DeepSeek's paid search stays as the fallback when the fabric is off, refuses or answers nothing. The++ // service token needs Search's viewer capability search.query.delegate (Network grant: ai on++ // openvibe.search); Search's own ladder calls this service back, so that caller keeps DeepSeek directly.research: {deepseek: bool(env.AI_RESEARCH_DEEPSEEK, true),deepseekKeyRef: String(env.AI_RESEARCH_DEEPSEEK_KEY_REF || 'env:DEEPSEEK_API_KEY').trim(),deepseekUrl: trimUrl(env.AI_RESEARCH_DEEPSEEK_URL || 'https://api.deepseek.com'),deepseekDaily: int(env.AI_RESEARCH_DEEPSEEK_DAILY, 1500),timeoutMs: int(env.AI_RESEARCH_TIMEOUT_MS, 15000),++ fabric: {++ enabled: bool(env.AI_RESEARCH_SEARCH, true),++ url: trimUrl(env.OV_SEARCH_INTERNAL_URL || env.AI_RESEARCH_SEARCH_URL || 'http://127.0.0.1:4710'),++ networkUrl: trimUrl(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),++ clientId: String(env.OV_OAUTH_CLIENT_ID || 'ai').trim(),++ timeoutMs: int(env.AI_RESEARCH_SEARCH_TIMEOUT_MS, 15000),++ },},// media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),// the download timeout, the scene-change threshold (scdet, 0-100) and the tools.@@@ -6,8 +6,11 @@* wikipedia articles for the query: full text of the best ones, lead pictures, the Wikidata id* fandom the topic's own community wiki, when it has one (characters, episodes, places, lore)* wikidata the top article's key statements as labelled facts (an infobox's raw material) and its picture++ * search_fabric the web through OpenVibe.Search's web scope (the Search Fabric) first: the open index, then the++ * providers Search routes to, every page with its provider, URL and dates* deepseek web search with page text (DeepSeek's Search API) and pages it cannot read in full (its Fetch API);-- * paid per call, so only for first-party callers and within its daily cap++ * paid per call, so only for first-party callers and within its daily cap. It is the fallback when the++ * Search Fabric is not configured, refuses or answers nothing* commons freely licensed pictures for the query* openverse more openly licensed pictures*@@@ -20,6 +23,7 @@ const { createWikimedia } = require('./wikimedia');const { createFandom } = require('./fandom');const { createOpenverse } = require('./openverse');const { createDeepseek } = require('./deepseek');++const { createSearchFabric } = require('./search-fabric');const { clip, canonicalUrl, hostOf } = require('./text');const { resolveSecret } = require('../util');@@@ -30,15 +34,19 @@ const int = (v, lo, hi, d) => { const n = Math.round(Number(v)); return Number.iconst oneLine = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);const short = (err) => oneLine(err && err.message ? err.message : err, 160) || 'failed';--function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console } = {}) {++function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console, fabric = null } = {}) {const http = createHttp({ ...(fetchImpl ? { fetchImpl } : {}), timeoutMs: config.timeoutMs || 15000 });const wikimedia = createWikimedia({ http });const fandom = createFandom({ http });const openverse = createOpenverse({ http });const deepseek = createDeepseek({ http, key: config.deepseekKey || (config.deepseek !== false ? resolveSecret(config.deepseekKeyRef, env) : ''), base: config.deepseekUrl || 'https://api.deepseek.com', dailyCap: config.deepseekDaily || 1500, now, log });++ const search = fabric || createSearchFabric({ config: config.fabric || {}, env, fetchImpl, now });++ const searchAvailable = () => (search && typeof search.available === 'function'++ ? search.available()++ : { configured: Boolean(search && search.search), reason: search ? null : 'search fabric is off' });function status() {-- return { providers: PROVIDERS, deepseek: deepseek.status() };++ return { providers: PROVIDERS, deepseek: deepseek.status(), search: search && typeof search.status === 'function' ? search.status() : searchAvailable() };}/**@@@ -46,9 +54,10 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>* query what to research ("Kim Possible", "Ron Stoppable Kim Possible")* topic the subject the query belongs to, used to find the topic's own wiki (defaults to the query)* focus one subject inside the topic (a character, an episode): searched for inside the topic's wiki-- * opts: { signal, allowPaid } — allowPaid false (a developer app) leaves DeepSeek out.++ * opts: { signal, allowPaid, allowFabric } — allowPaid false (a developer app) leaves paid DeepSeek out; allowFabric++ * false keeps Search's own ladder from asking Search back (the caller is Search).*/-- async function gather(input = {}, { signal = null, allowPaid = true } = {}) {++ async function gather(input = {}, { signal = null, allowPaid = true, allowFabric = true } = {}) {const query = oneLine(input.query, 200);if (!query) throw Object.assign(new Error('research needs a query'), { status: 422 });const topic = oneLine(input.topic, 200) || query;@@@ -61,7 +70,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>const searchText = focus && !query.toLowerCase().includes(focus.toLowerCase()) ? `${focus} ${query}` : query;const retrievedAt = new Date(now()).toISOString();const providers = {};-- const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, cost_usd: 0 };++ const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, search_fabric: 0, search_fabric_cost_usd: 0, cost_usd: 0 };const startRequests = http.count();async function timed(name, fn) {@@@ -79,21 +88,66 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>}const skip = (name, reason) => { providers[name] = { ok: false, skipped: reason }; return null; };++ const wantWeb = want.has('deepseek');++ const fabricReady = wantWeb && allowFabric && searchAvailable().configured;++++ /**++ * The web evidence: the Search Fabric first, DeepSeek direct only when the fabric is not configured,++ * fails or answers nothing. `allowPaid: false` keeps the fabric to its free and cached steps (quick)++ * and never reaches DeepSeek.++ */++ async function webEvidence() {++ if (!wantWeb) return null;++ if (fabricReady) {++ const t0 = now();++ try {++ const page = await search.search(searchText, { limit: 8, depth: allowPaid ? 'standard' : 'quick', signal });++ const cost = Number(page.route && page.route.cost_usd) || 0;++ const results = page.results || [];++ // A page without a snippet cannot become a source; leave it to DeepSeek rather than cite a stub.++ const usable = results.filter((r) => r && (r.snippet || r.content));++ providers.search_fabric = { ok: usable.length > 0, ms: now() - t0, count: usable.length, cost_usd: cost };++ if (page.route && Array.isArray(page.route.explain)) providers.search_fabric.explain = page.route.explain.slice(0, 20);++ if (usable.length) {++ usage.search_fabric += 1;++ usage.search_fabric_cost_usd += cost;++ return { results: usable, via: 'search_fabric' };++ }++ providers.search_fabric.skipped = results.length ? 'no result text' : 'no results';++ } catch (err) {++ if (signal && signal.aborted) throw err;++ providers.search_fabric = { ok: false, ms: now() - t0, error: short(err) };++ }++ }++ if (!allowPaid) {++ skip('deepseek', 'first-party callers only');++ return null;++ }++ if (!deepseek.available()) {++ skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured');++ return null;++ }++ const results = await timed('deepseek', async () => {++ usage.deepseek_search += 1;++ return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });++ });++ return results && results.length ? { results, via: 'deepseek' } : null;++ }++// Round one, all at once: Wikipedia, web search and the picture libraries.-- const useDeepseek = want.has('deepseek') && (allowPaid ? (deepseek.available() || skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured')) : skip('deepseek', 'first-party callers only'));-- const [wiki, web, commons, openv] = await Promise.all([++ const webTask = webEvidence();++ const [wiki, webOut, commons, openv] = await Promise.all([want.has('wikipedia') ? timed('wikipedia', async () => {const hits = await wikimedia.searchWikipedia(searchText, { lang, limit: focus ? 4 : 5, signal });const list = await wikimedia.articles(hits.map((h) => h.title), { lang, full: focus ? 2 : 3, chars, signal });return list.filter((a) => !a.disambiguation && a.text);}) : null,-- useDeepseek ? timed('deepseek', async () => {-- usage.deepseek_search += 1;-- return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });-- }) : null,++ webTask,maxImages && want.has('commons') ? timed('commons', () => wikimedia.commonsSearch(searchText, { limit: 8, signal })) : null,maxImages && want.has('openverse') ? timed('openverse', () => openverse.search(searchText, { limit: 6, signal })) : null,]);++ const web = webOut ? webOut.results : null;++ const webVia = webOut ? webOut.via : null;// Round two: the topic's own wiki, the top article's facts and pictures, and full text for web results that came back short.const top = wiki && wiki[0];@@@ -108,7 +162,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>}) : null,want.has('wikidata') && top && top.qid ? timed('wikidata', () => wikimedia.wikidataFacts(top.qid, { lang, signal })) : null,maxImages && want.has('wikipedia') && top ? timed('wikipedia_images', () => wikimedia.articleImages(top.title, { lang, limit: 10, signal })) : null,-- useDeepseek && web ? (async () => {++ webVia === 'deepseek' && web ? (async () => {// A short result is read in full, unless it is a copy of an encyclopedia page already read.const read = new Set((wiki || []).map((a) => canonicalUrl(a.url)));const thin = web.filter((r) => r.score >= 1 && r.content.length < 400 && !read.has(canonicalUrl(r.url))).slice(0, 2);@@@ -127,8 +181,9 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>maxImages && leadName && !leadListed ? wikimedia.fileInfo([leadName], { lang, allowSvg: true, signal }).catch(() => null) : null,]);-- // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by DeepSeek's score.++ // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by its route's order.const scored = [];++ const fromFabric = webVia === 'search_fabric';const focusLc = (focus || '').toLowerCase();(fan && fan.pages || []).forEach((p, i) => {const exact = focusLc ? p.title.toLowerCase() === focusLc : i === 0;@@@ -141,7 +196,25 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>(web || []).forEach((r) => {const content = clip(r.content || r.summary || r.snippet, chars);if (!content) return;-- scored.push({ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08, s: { source_type: 'web', url: r.url, title: r.title || hostOf(r.url), snippet: r.snippet || clip(r.summary, 300), content, published_at: r.date || undefined, retrieved_at: retrievedAt, provenance: { provider: 'deepseek', site: r.site || hostOf(r.url), authority: r.authority, score: r.score } } });++ scored.push({++ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08,++ s: {++ source_type: 'web',++ url: r.url,++ title: r.title || hostOf(r.url),++ snippet: r.snippet || clip(r.summary, 300),++ content,++ published_at: r.date || undefined,++ retrieved_at: r.retrieved_at || retrievedAt,++ provenance: {++ provider: fromFabric ? (r.provider || 'search_fabric') : 'deepseek',++ site: r.site || hostOf(r.url),++ authority: r.authority,++ score: r.score,++ ...(fromFabric ? { via: 'search_fabric' } : {}),++ },++ },++ });});if (facts && facts.facts.length) {scored.push({ score: 2.0, s: { source_type: 'wikidata', url: facts.url, title: `${facts.name} — Wikidata`, snippet: facts.description || `${facts.facts.length} facts`, content: facts.facts.map((f) => `${f.label}: ${f.value}`).join('\n'), retrieved_at: retrievedAt, provenance: { provider: 'wikidata', site: 'wikidata.org', license: LICENSE.wikidata } } });@@@ -183,6 +256,8 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>}usage.requests = http.count() - startRequests;++ // Only calls this run made are its cost. A paid Search Fabric step is Search's (and, for DeepSeek, a nested++ // AI run as `search`), so its route cost is reported separately and never billed to this run twice.usage.cost_usd = Number(((usage.deepseek_search + usage.deepseek_fetch) * deepseek.PRICE_PER_CALL_USD).toFixed(5));return {query, topic, focus, lang, retrieved_at: retrievedAt,nnew file mode 100644@@@ -0,0 +1,147 @@++'use strict';++/**++ * The Search Fabric's web scope, for MegaSource (OpenVibe.Search, plan 2026-10-28 §4.5):++ *++ * POST <OV_SEARCH_INTERNAL_URL>/api/v1/query++ * { q, scopes: ["web"], depth, limit, freshness }++ *++ * Each page keeps the provider that found it, its URL and its dates, so a research citation can++ * name all three. Search's viewer requires `search.query.delegate` of any service token++ * (server/auth.js CAPS.delegate), so that is the capability AI asks Network for:++ *++ * grant `ai` `search.query.delegate` on audience `openvibe.search`.++ *++ * `search.fabric.query` is the route's public capability in openvibe-contracts; Search does not++ * check it for a service token today. AI asks only for the web scope. A Search that is down,++ * refuses the token or answers nothing never fails a research run: the caller falls back to++ * DeepSeek (server/research/deepseek.js).++ */++const { createServiceTokenClient } = require('openvibe-sdk/auth');++++const AUDIENCE = 'openvibe.search';++const CAPABILITY = 'search.query.delegate';++const TIMEOUT_MS = 15000;++const MAX_RESULTS = 20;++++function clip(value, max) {++ return String(value == null ? '' : value).slice(0, max);++}++++function hostOf(url) {++ try { return new URL(String(url)).hostname.toLowerCase() || null; } catch { return null; }++}++++function shortError(err) {++ const text = String((err && (err.message || err.name)) || 'request failed').replace(/\s+/g, ' ').trim();++ return text.slice(0, 160) || 'request failed';++}++++/**++ * Search Fabric web results as MegaSource's web shape. A citation needs url, title, published_at,++ * retrieved_at and provider: all survive here. `score` follows the ladder's order (Search says its++ * scores are not comparable across scopes), so the first page ranks first.++ */++function mapResults(body, at = Date.now()) {++ const list = Array.isArray(body && body.results) ? body.results : [];++ const web = list.filter((r) => r && r.url && (r.scope === 'web' || r.kind === 'page'));++ return web.slice(0, MAX_RESULTS).map((r, i) => ({++ url: clip(r.url, 2048),++ title: clip(r.title || hostOf(r.url) || r.url, 300),++ site: r.site ? clip(r.site, 253) : hostOf(r.url),++ snippet: r.snippet == null ? '' : clip(r.snippet, 600),++ summary: '',++ content: r.snippet == null ? '' : clip(r.snippet, 600),++ date: r.published_at ? clip(r.published_at, 40) : null,++ retrieved_at: r.retrieved_at ? clip(r.retrieved_at, 40) : new Date(at).toISOString(),++ provider: r.provider ? clip(r.provider, 40) : 'search_fabric',++ authority: -1,++ score: Math.max(0.1, 2.2 - i * 0.1),++ }));++}++++function createSearchFabric({ config = {}, env = process.env, fetchImpl = globalThis.fetch, now = () => Date.now(), tokens: givenTokens = null } = {}) {++ const url = String(config.url || '').replace(/\/+$/, '');++ const networkUrl = String(config.networkUrl || env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000').replace(/\/+$/, '');++ const clientId = String(config.clientId || env.OV_OAUTH_CLIENT_ID || 'ai').trim();++ const secret = env.OV_OAUTH_CLIENT_SECRET || '';++ const enabled = config.enabled !== false;++ let tokens = givenTokens;++++ function available() {++ if (!enabled) return { configured: false, reason: 'search fabric is off' };++ if (!url) return { configured: false, reason: 'OV_SEARCH_INTERNAL_URL is unset' };++ if (!tokens && !secret) return { configured: false, reason: 'OV_OAUTH_CLIENT_SECRET is unset' };++ return { configured: true, reason: null };++ }++++ function tokenClient() {++ if (!tokens) {++ tokens = createServiceTokenClient({++ tokenUrl: `${networkUrl}/oauth/token`,++ clientId,++ clientSecret: secret,++ scope: { [AUDIENCE]: CAPABILITY },++ fetch: fetchImpl,++ now,++ });++ }++ return tokens;++ }++++ /** One web query: { results, groups, citations, route, next_cursor }. */++ async function search(q, { limit = 8, depth = 'standard', freshness = 'any', signal = null } = {}) {++ const availability = available();++ if (!availability.configured) throw Object.assign(new Error(availability.reason), { code: 'search.unavailable' });++ const token = await tokenClient().getToken({ audience: AUDIENCE });++ const timeoutMs = Number(config.timeoutMs) > 0 ? Number(config.timeoutMs) : TIMEOUT_MS;++ const timeout = AbortSignal.timeout(timeoutMs);++ const combined = signal ? AbortSignal.any([signal, timeout]) : timeout;++ let res;++ try {++ res = await fetchImpl(`${url}/api/v1/query`, {++ method: 'POST',++ headers: {++ Authorization: `Bearer ${token}`,++ Accept: 'application/json',++ 'Content-Type': 'application/json',++ 'User-Agent': 'OpenVibeResearch/1.0 (+https://ai.openvibe.services; [email protected])',++ },++ body: JSON.stringify({++ q: clip(q, 200),++ scopes: ['web'],++ depth,++ freshness,++ limit: Math.min(Math.max(1, Number(limit) || 8), MAX_RESULTS),++ }),++ signal: combined,++ });++ } catch (err) {++ if (signal && signal.aborted) throw err;++ throw Object.assign(new Error(`OpenVibe.Search did not answer: ${shortError(err)}`), { code: 'search.unavailable' });++ }++ const text = await res.text().catch(() => '');++ let body = null;++ try { body = text ? JSON.parse(text) : null; } catch { body = null; }++ if (!res.ok) {++ const code = body && body.code ? ` (${clip(body.code, 80)})` : '';++ throw Object.assign(new Error(`OpenVibe.Search answered ${res.status}${code}`), { status: res.status, code: 'search.refused' });++ }++ if (!body || !Array.isArray(body.results)) throw Object.assign(new Error('OpenVibe.Search answered no results'), { code: 'search.bad_answer' });++ return {++ results: mapResults(body, now()),++ groups: body.groups && typeof body.groups === 'object' ? body.groups : {},++ citations: Array.isArray(body.citations) ? body.citations : [],++ route: body.route && typeof body.route === 'object' ? body.route : { explain: [], cost_usd: 0 },++ next_cursor: body.next_cursor || null,++ };++ }++++ function status() {++ const a = available();++ return { configured: a.configured, reason: a.reason, url: url || null, audience: AUDIENCE, capability: CAPABILITY };++ }++++ return { search, available, status };++}++++module.exports = { createSearchFabric, mapResults, AUDIENCE, CAPABILITY };@@@ -404,6 +404,8 @@ function createRuns({ db, registry, engine, cache, quotas, config, clock = { now// limited to free and local capacity when it is a sandbox run or when no tier budget governs its// project (AI_GOVERN_TIERS off): metered capacity is only ever spent under a project budget.appRun: Boolean(ctx.appRun), freeOnly: Boolean(ctx.appRun && (ctx.sandbox || !(tiers && tiers.enabled))),++ // Who asked, for a step that must not call the caller back (research: Search's own web ladder).++ requesterType: ctx.requesterType, requesterId: ctx.requesterId,...(ctx.routePin ? { routePin: ctx.routePin } : {}),...(credential ? { credential, role: input && typeof input.role === 'string' ? input.role : null } : {}),...(typeof ctx.onDelta === 'function' ? { onDelta: ctx.onDelta } : {}) });@@@ -256,7 +256,10 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spaasync function researchStep(step, input, wf, run, ctx) {let output;try {-- output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun });++ // Search's own web ladder calls this endpoint for DeepSeek and Wikipedia. It must not be asked back++ // (Search -> AI research -> Search): every other caller gets the fabric first, Search keeps DeepSeek.++ const fromSearch = ctx.requesterType === 'service' && ctx.requesterId === 'search';++ output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun, allowFabric: !fromSearch });} catch (err) {if (err && err.status === 422) throw new AiError(422, 'input.invalid', err.message);throw err;@@@ -24,7 +24,7 @@ const IMAGE = {const workflows = [{key: 'research.gather', name: 'Research a topic (MegaSource)', namespace: 'research',-- description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, DeepSeek web search, Commons and Openverse.',++ description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, the Search Fabric web scope (DeepSeek as its fallback), Commons and Openverse.',input_schema: {type: 'object', additionalProperties: false, required: ['query'],properties: {nnew file mode 100644@@@ -0,0 +1,233 @@++'use strict';++// SF-4: MegaSource asks the Search Fabric's web scope first, as AI's service principal, and keeps each page's++// provider, date and URL for the citation. DeepSeek remains the fallback when Search is off, refuses or answers++// nothing, and Search's own ladder never makes research ask Search back in a loop.++++const assert = require('assert');++const http = require('http');++const { createResearch } = require('../server/research');++const { createSearchFabric } = require('../server/research/search-fabric');++const { createResearchFetch } = require('./fixtures/research-fetch');++const { boot, request, token, suite } = require('./helpers');++++const t = suite('research fabric');++const quiet = { log() {}, warn() {}, error() {} };++++t.test('configuration: the fabric is on by default, pointed by OV_SEARCH_INTERNAL_URL, and AI_RESEARCH_SEARCH=0 turns it off', () => {++ const { load } = require('../server/config');++ const on = load({ NODE_ENV: 'test', OV_SEARCH_INTERNAL_URL: 'http://search.test:4710' }).research.fabric;++ assert.deepStrictEqual([on.enabled, on.url, on.clientId], [true, 'http://search.test:4710', 'ai']);++ assert.strictEqual(load({ NODE_ENV: 'test', AI_RESEARCH_SEARCH: '0' }).research.fabric.enabled, false);++});++++async function listen(handler) {++ const server = http.createServer(handler);++ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));++ return {++ url: `http://127.0.0.1:${server.address().port}`,++ close: () => new Promise((resolve) => { server.closeAllConnections?.(); server.close(() => resolve()); }),++ };++}++++function readBody(req) {++ return new Promise((resolve) => {++ let raw = '';++ req.on('data', (chunk) => { raw += chunk; });++ req.on('end', () => {++ let json = null;++ try { json = raw ? JSON.parse(raw) : null; } catch { json = null; }++ resolve({ raw, json });++ });++ });++}++++function json(res, status, body) {++ res.writeHead(status, { 'content-type': 'application/json' });++ res.end(JSON.stringify(body));++}++++t.test('the fabric client asks POST /api/v1/query for the web scope, as a service token, and keeps provider/date/url', async () => {++ const seen = [];++ const search = await listen(async (req, res) => {++ const { json: body } = await readBody(req);++ seen.push({ method: req.method, url: req.url, auth: req.headers.authorization, body });++ json(res, 200, {++ results: [++ {++ scope: 'web', kind: 'page', title: 'Fabric page', url: 'https://example.org/fabric', site: 'example.org',++ snippet: 'A fabric snippet', published_at: '2026-10-01T00:00:00Z', retrieved_at: '2026-10-28T12:00:00Z',++ provider: 'brave', owner: null, type: null, id: null, image: null, score: null,++ },++ {++ scope: 'web', kind: 'page', title: 'No dates', url: 'https://example.org/nodate', site: 'example.org',++ snippet: 'x', published_at: null, retrieved_at: null, provider: 'deepseek',++ },++ ],++ groups: { web: 2 },++ citations: [],++ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },++ next_cursor: null,++ });++ });++ const fabric = createSearchFabric({++ config: { url: search.url, timeoutMs: 2000, networkUrl: 'http://127.0.0.1:9' },++ env: { OV_OAUTH_CLIENT_SECRET: 'x'.repeat(40) },++ fetchImpl: globalThis.fetch,++ tokens: { getToken: async () => 'svc-search' },++ });++ try {++ const page = await fabric.search('solar eclipse', { limit: 8 });++ assert.deepStrictEqual([seen[0].method, seen[0].url, seen[0].auth], ['POST', '/api/v1/query', 'Bearer svc-search']);++ assert.strictEqual(seen[0].body.q, 'solar eclipse');++ assert.deepStrictEqual(seen[0].body.scopes, ['web']);++ assert.strictEqual(seen[0].body.depth, 'standard');++ assert.strictEqual(seen[0].body.limit, 8);++ assert.strictEqual(page.results[0].provider, 'brave');++ assert.strictEqual(page.results[0].date, '2026-10-01T00:00:00Z');++ assert.strictEqual(page.results[0].retrieved_at, '2026-10-28T12:00:00Z');++ assert.strictEqual(page.results[0].url, 'https://example.org/fabric');++ assert.ok(page.results[1].retrieved_at, 'a missing read date is filled at read time');++ assert.strictEqual(page.route.cost_usd, 0.0004);++ } finally { await search.close(); }++});++++t.test('research uses the fabric first, and its citation keeps the provider, date and URL', async () => {++ const calls = [];++ const fabric = {++ available: () => ({ configured: true }),++ status: () => ({ configured: true }),++ search: async (q, opts) => {++ calls.push({ q, opts });++ return {++ results: [{++ url: 'https://example.org/fabric', title: 'Fabric page', site: 'example.org',++ snippet: 'A fabric snippet', summary: '', content: 'A fabric snippet',++ date: '2026-10-02T03:04:05Z', retrieved_at: '2026-10-28T06:07:08Z',++ provider: 'brave', authority: -1, score: 2.2,++ }],++ groups: { web: 1 },++ citations: [],++ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },++ next_cursor: null,++ };++ },++ };++ const web = createResearchFetch();++ const r = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });++ const out = await r.gather({ query: 'SQLite', max_images: 0 });++ assert.strictEqual(calls.length, 1);++ assert.strictEqual(calls[0].opts.depth, 'standard');++ const source = out.sources.find((s) => s.url === 'https://example.org/fabric');++ assert.ok(source, JSON.stringify(out.sources.map((s) => s.url)));++ assert.strictEqual(source.published_at, '2026-10-02T03:04:05Z');++ assert.strictEqual(source.retrieved_at, '2026-10-28T06:07:08Z');++ assert.strictEqual(source.provenance.provider, 'brave');++ assert.strictEqual(source.provenance.via, 'search_fabric');++ assert.strictEqual(out.providers.search_fabric.ok, true);++ assert.strictEqual(out.providers.search_fabric.cost_usd, 0.0004);++ assert.strictEqual(out.usage.search_fabric, 1);++ assert.strictEqual(out.usage.search_fabric_cost_usd, 0.0004, 'the fabric route cost is reported');++ assert.strictEqual(out.usage.cost_usd, 0, 'a paid fabric step is Search\'s cost, not billed to this run twice');++ assert.ok(!web.calls.some((c) => c.host === 'api.deepseek.com'), 'DeepSeek is not called when the fabric answers');++});++++t.test('DeepSeek is the fallback when the fabric refuses or answers nothing', async () => {++ for (const [label, search] of [++ ['refuses', async () => { throw new Error('Search is down'); }],++ ['answers nothing', async () => ({ results: [], groups: { web: 0 }, citations: [], route: { explain: [], cost_usd: 0 }, next_cursor: null })],++ ]) {++ const fabric = { available: () => ({ configured: true }), status: () => ({ configured: true }), search };++ const web = createResearchFetch();++ const r = createResearch({ config: { deepseekKey: 'sk-test' }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });++ const out = await r.gather({ query: `SQLite ${label}`, max_images: 0 });++ assert.strictEqual(out.providers.search_fabric.ok, false, label);++ assert.strictEqual(out.providers.deepseek.ok, true, label);++ assert.ok(out.sources.some((s) => s.provenance.provider === 'deepseek' && !s.provenance.via), `${label}: a direct DeepSeek source`);++ assert.strictEqual(out.usage.deepseek_search, 1, label);++ assert.ok(out.usage.cost_usd >= 0.0015, `${label}: the fallback cost is counted`);++ }++});++++t.test('POST /api/v1/research: the token scope is search.query.delegate and the stored citation keeps provider/date/url', async () => {++ let tokenRequest = null;++ const network = await listen(async (req, res) => {++ const { raw } = await readBody(req);++ tokenRequest = Object.fromEntries(new URLSearchParams(raw));++ json(res, 200, { access_token: 'svc-ai-search', token_type: 'Bearer', expires_in: 300, scope: 'search.query.delegate' });++ });++ let query = null;++ const search = await listen(async (req, res) => {++ const { json: body } = await readBody(req);++ query = { method: req.method, url: req.url, auth: req.headers.authorization, body };++ json(res, 200, {++ results: [{++ scope: 'web', kind: 'page', title: 'Fabric page', url: 'https://example.org/fabric-e2e', site: 'example.org',++ snippet: 'A fabric snippet', published_at: '2026-10-03T00:00:00Z', retrieved_at: '2026-10-28T01:02:03Z',++ provider: 'brave', owner: null, type: null, id: null, image: null, score: null,++ }],++ groups: { web: 1 },++ citations: [],++ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },++ next_cursor: null,++ });++ });++ const fixture = createResearchFetch();++ const fetchImpl = async (url, init) => {++ const s = String(url);++ if (s.startsWith(network.url) || s.startsWith(search.url)) return fetch(s, init);++ return fixture.fetchImpl(url, init);++ };++ const researcher = createResearch({++ config: {++ timeoutMs: 15000,++ deepseek: false,++ fabric: { enabled: true, url: search.url, networkUrl: network.url, clientId: 'ai', timeoutMs: 2000 },++ },++ env: { OV_OAUTH_CLIENT_SECRET: 'x'.repeat(40) },++ fetchImpl,++ log: quiet,++ });++ const h = await boot({ research: researcher });++ try {++ const tok = token('wiki', ['ai.run.create'], { ns: ['research.*'] });++ const res = await request(h.base, 'POST', '/api/v1/research?wait=15000', { tok, body: { query: 'SQLite', max_images: 0 } });++ assert.strictEqual(res.status, 201, res.text);++ assert.strictEqual(res.body.run.status, 'succeeded', JSON.stringify(res.body.run.error));++ assert.deepStrictEqual([tokenRequest.grant_type, tokenRequest.client_id, tokenRequest.audience, tokenRequest.scope],++ ['client_credentials', 'ai', 'openvibe.search', 'search.query.delegate']);++ assert.deepStrictEqual([query.method, query.url, query.auth], ['POST', '/api/v1/query', 'Bearer svc-ai-search']);++ assert.deepStrictEqual(query.body.scopes, ['web']);++ // research.gather's output sources are the numbered citations its callers use.++ const citation = res.body.run.output.sources.find((s) => s.url === 'https://example.org/fabric-e2e');++ assert.ok(citation, JSON.stringify(res.body.run.output.sources.map((s) => s.url)));++ assert.strictEqual(citation.published_at, '2026-10-03T00:00:00Z');++ assert.strictEqual(citation.retrieved_at, '2026-10-28T01:02:03Z');++ assert.strictEqual(citation.provenance.provider, 'brave');++ assert.strictEqual(citation.provenance.via, 'search_fabric');++ } finally {++ await h.stop();++ await network.close();++ await search.close();++ }++});++++t.test('Search asking AI for a web source keeps DeepSeek and does not ask Search back', async () => {++ let fabricCalls = 0;++ const fabric = {++ available: () => ({ configured: true }),++ status: () => ({ configured: true }),++ search: async () => { fabricCalls += 1; return { results: [], route: { explain: [], cost_usd: 0 } }; },++ };++ const web = createResearchFetch();++ const researcher = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });++ const h = await boot({ research: researcher });++ try {++ const tok = token('search', ['ai.run.create'], { ns: ['research.*'] });++ const res = await request(h.base, 'POST', '/api/v1/research?wait=15000', { tok, body: { query: 'SQLite', max_images: 0 } });++ assert.strictEqual(res.status, 201, res.text);++ assert.strictEqual(fabricCalls, 0, 'the fabric is not asked back');++ assert.ok(res.body.run.output.sources.some((s) => s.provenance.provider === 'deepseek' && !s.provenance.via), 'DeepSeek answered directly');++ } finally { await h.stop(); }++});++++t.run();
@@ -175,8 +175,15 @@ OV_MEDIA_URL=https://openvibe.media
@@ -175,8 +175,15 @@ OV_MEDIA_URL=https://openvibe.media
#AI_FETCH_MAX_IMAGE_BYTES=12582912
#AI_FETCH_MAX_IMAGE_BYTES=12582912
#AI_FETCH_TIMEOUT_MS=120000
#AI_FETCH_TIMEOUT_MS=120000
-# MegaSource research (server/research/, research.gather): DeepSeek's paid web search joins the free sources when its key
-# resolves. The key is read by name; set DEEPSEEK_API_KEY (or point AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).
+# MegaSource research (server/research/, research.gather): web search asks the Search Fabric's web scope first
+# (OpenVibe.Search POST /api/v1/query) as AI's service principal, and falls back to DeepSeek when the fabric is off,
+# refuses or answers nothing. Network must grant `ai` search.query.delegate on audience openvibe.search (Search's
+# viewer requires it for a service token); the route's public capability is search.fabric.query.
+#OV_SEARCH_INTERNAL_URL=http://127.0.0.1:4710
+#AI_RESEARCH_SEARCH=1
+#AI_RESEARCH_SEARCH_TIMEOUT_MS=15000
+# DeepSeek's paid search is the fallback. The key is read by name; set DEEPSEEK_API_KEY (or point
+# AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).
#AI_RESEARCH_DEEPSEEK=1
#AI_RESEARCH_DEEPSEEK=1
#AI_RESEARCH_DEEPSEEK_KEY_REF=env:DEEPSEEK_API_KEY
#AI_RESEARCH_DEEPSEEK_KEY_REF=env:DEEPSEEK_API_KEY
#AI_RESEARCH_DEEPSEEK_DAILY=1500
#AI_RESEARCH_DEEPSEEK_DAILY=1500
@@ -463,7 +463,7 @@ the service boots without Valkey; with it on, production refuses to boot without
@@ -463,7 +463,7 @@ the service boots without Valkey; with it on, production refuses to boot without
`research.gather` (namespace `research`, `server/research/`, `server/workflows/research.js`) answers one question from
`research.gather` (namespace `research`, `server/research/`, `server/workflows/research.js`) answers one question from
many sources at once and returns one ranked, de-duplicated evidence pack the product workflows cite by number. No model
many sources at once and returns one ranked, de-duplicated evidence pack the product workflows cite by number. No model
-is called; a run is recorded like a provider call (provider `research`, model `megasource`), so DeepSeek's per-call
+is called; a run is recorded like a provider call (provider `research`, model `megasource`), so the web route's per-call
price counts against the cost quotas and the daily usage.
price counts against the cost quotas and the daily usage.
| Source | What it gives | Terms |
| Source | What it gives | Terms |
@@ -471,20 +471,28 @@ price counts against the cost quotas and the daily usage.
@@ -471,20 +471,28 @@ price counts against the cost quotas and the daily usage.
| Wikipedia | the best articles' full text, their pictures (each with its licence; an article's own non-free files are marked) and the Wikidata id | CC BY-SA 4.0 |
| Wikipedia | the best articles' full text, their pictures (each with its licence; an article's own non-free files are marked) and the Wikidata id | CC BY-SA 4.0 |
| Fandom | the topic's own community wiki when it has one (from the web results' hosts, else the topic as a subdomain, confirmed by the wiki's name): pages read through its MediaWiki API, infobox rows kept as "Label: value" | CC BY-SA 3.0; pictures marked not free |
| Fandom | the topic's own community wiki when it has one (from the web results' hosts, else the topic as a subdomain, confirmed by the wiki's name): pages read through its MediaWiki API, infobox rows kept as "Label: value" | CC BY-SA 3.0; pictures marked not free |
| Wikidata | the top article's key statements as labelled facts, and its picture | CC0 |
| Wikidata | the top article's key statements as labelled facts, and its picture | CC0 |
-| DeepSeek | web search with page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each |
+| Search Fabric | the web through OpenVibe.Search's web scope (`POST /api/v1/query`, capability `search.fabric.query`) first: the open index, then the providers Search routes to; every page keeps its provider, URL and dates | per provider Search used |
+| DeepSeek | the fallback web search when the fabric is off, refuses or answers nothing: page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each |
| Commons, Openverse | freely licensed pictures (CC0, public domain, CC BY, CC BY-SA; NonCommercial and NoDerivatives are not counted as free), kept only when their title or description names the subject | per picture |
| Commons, Openverse | freely licensed pictures (CC0, public domain, CC BY, CC BY-SA; NonCommercial and NoDerivatives are not counted as free), kept only when their title or description names the subject | per picture |
Input: `query` (required), `topic` (the subject it belongs to: finds the topic's wiki), `focus` (one subject inside the
Input: `query` (required), `topic` (the subject it belongs to: finds the topic's wiki), `focus` (one subject inside the
topic), `lang`, `max_sources` (≤ 20), `max_images` (≤ 24), `content_chars` (≤ 12,000), `providers` (a subset). Output:
topic), `lang`, `max_sources` (≤ 20), `max_images` (≤ 24), `content_chars` (≤ 12,000), `providers` (a subset). Output:
-`sources` (each with `url`, `title`, `content`, `retrieved_at` and `provenance.provider|site|license`), `images` (each with
-`license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per source: ok, ms, count, or
-why it was skipped or failed) and `usage` (requests, DeepSeek calls, `cost_usd`). Cached for a day per calling service.
-
-Every source is optional and fails on its own. DeepSeek is used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`,
-default `env:DEEPSEEK_API_KEY`; `AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's
-run reads the free sources), within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own
-answers: 402 (balance used up) for 30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web
-(`test/fixtures/research-fetch.js`); none reaches the internet.
+`sources` (each with `url`, `title`, `content`, `published_at`, `retrieved_at` and `provenance.provider|site|via|license`),
+`images` (each with `license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per route:
+ok, ms, count, cost, or why it was skipped or failed) and `usage` (requests, DeepSeek calls, fabric calls and the fabric's
+`search_fabric_cost_usd`; `cost_usd` is this run's own DeepSeek calls, so a paid fabric step is not billed twice —
+Search and its nested AI run pay it). Cached for a day per calling service.
+
+Web search asks the Search Fabric first, as AI's own service principal (audience `openvibe.search`). Search's viewer
+requires `search.query.delegate` of a service token, so Network must grant `ai` that capability on `openvibe.search`
+(the route's `search.fabric.query` is public). The token is requested with exactly that scope; without
+`OV_OAUTH_CLIENT_SECRET` or with `AI_RESEARCH_SEARCH=0` the fabric is skipped. When Search's own web ladder calls this
+endpoint for DeepSeek or Wikipedia, research keeps DeepSeek direct, so Search is never asked back in a loop. DeepSeek is
+used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`, default `env:DEEPSEEK_API_KEY`;
+`AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's run reads the free sources),
+within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own answers: 402 (balance used up) for
+30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web (`test/fixtures/research-fetch.js`); none reaches
+the internet.
OpenVibe.Wiki's "vibe a wiki" uses it with `wiki.generate_space` (plans the pages: main page first, each with a kind,
OpenVibe.Wiki's "vibe a wiki" uses it with `wiki.generate_space` (plans the pages: main page first, each with a kind,
an outline and a search phrase) and `wiki.generate_page` (writes one page: lead, sections, infobox, `[[links]]` to the
an outline and a search phrase) and `wiki.generate_page` (writes one page: lead, sections, infobox, `[[links]]` to the
@@ -115,14 +115,24 @@ function load(env = process.env) {
@@ -115,14 +115,24 @@ function load(env = process.env) {
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
- // MegaSource (server/research/, workflow research.gather): DeepSeek's paid web search joins the free sources when
- // its key resolves (a secret reference, read by name) and stays within AI_RESEARCH_DEEPSEEK_DAILY calls a day.
+ // MegaSource (server/research/, workflow research.gather): web search asks the Search Fabric's web scope
+ // first (OpenVibe.Search POST /api/v1/query, capability search.fabric.query) as AI's service principal;
+ // DeepSeek's paid search stays as the fallback when the fabric is off, refuses or answers nothing. The
+ // service token needs Search's viewer capability search.query.delegate (Network grant: ai on
+ // openvibe.search); Search's own ladder calls this service back, so that caller keeps DeepSeek directly.
research: {
research: {
deepseek: bool(env.AI_RESEARCH_DEEPSEEK, true),
deepseek: bool(env.AI_RESEARCH_DEEPSEEK, true),
deepseekKeyRef: String(env.AI_RESEARCH_DEEPSEEK_KEY_REF || 'env:DEEPSEEK_API_KEY').trim(),
deepseekKeyRef: String(env.AI_RESEARCH_DEEPSEEK_KEY_REF || 'env:DEEPSEEK_API_KEY').trim(),
deepseekUrl: trimUrl(env.AI_RESEARCH_DEEPSEEK_URL || 'https://api.deepseek.com'),
deepseekUrl: trimUrl(env.AI_RESEARCH_DEEPSEEK_URL || 'https://api.deepseek.com'),
deepseekDaily: int(env.AI_RESEARCH_DEEPSEEK_DAILY, 1500),
deepseekDaily: int(env.AI_RESEARCH_DEEPSEEK_DAILY, 1500),
timeoutMs: int(env.AI_RESEARCH_TIMEOUT_MS, 15000),
timeoutMs: int(env.AI_RESEARCH_TIMEOUT_MS, 15000),
+ fabric: {
+ enabled: bool(env.AI_RESEARCH_SEARCH, true),
+ url: trimUrl(env.OV_SEARCH_INTERNAL_URL || env.AI_RESEARCH_SEARCH_URL || 'http://127.0.0.1:4710'),
+ networkUrl: trimUrl(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),
+ clientId: String(env.OV_OAUTH_CLIENT_ID || 'ai').trim(),
+ timeoutMs: int(env.AI_RESEARCH_SEARCH_TIMEOUT_MS, 15000),
+ },
},
},
// media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),
// media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),
// the download timeout, the scene-change threshold (scdet, 0-100) and the tools.
// the download timeout, the scene-change threshold (scdet, 0-100) and the tools.
@@ -6,8 +6,11 @@
@@ -6,8 +6,11 @@
* wikipedia articles for the query: full text of the best ones, lead pictures, the Wikidata id
* wikipedia articles for the query: full text of the best ones, lead pictures, the Wikidata id
* fandom the topic's own community wiki, when it has one (characters, episodes, places, lore)
* fandom the topic's own community wiki, when it has one (characters, episodes, places, lore)
* wikidata the top article's key statements as labelled facts (an infobox's raw material) and its picture
* wikidata the top article's key statements as labelled facts (an infobox's raw material) and its picture
+ * search_fabric the web through OpenVibe.Search's web scope (the Search Fabric) first: the open index, then the
+ * providers Search routes to, every page with its provider, URL and dates
* deepseek web search with page text (DeepSeek's Search API) and pages it cannot read in full (its Fetch API);
* deepseek web search with page text (DeepSeek's Search API) and pages it cannot read in full (its Fetch API);
- * paid per call, so only for first-party callers and within its daily cap
+ * paid per call, so only for first-party callers and within its daily cap. It is the fallback when the
+ * Search Fabric is not configured, refuses or answers nothing
* commons freely licensed pictures for the query
* commons freely licensed pictures for the query
* openverse more openly licensed pictures
* openverse more openly licensed pictures
*
*
@@ -20,6 +23,7 @@ const { createWikimedia } = require('./wikimedia');
@@ -20,6 +23,7 @@ const { createWikimedia } = require('./wikimedia');
const { createFandom } = require('./fandom');
const { createFandom } = require('./fandom');
const { createOpenverse } = require('./openverse');
const { createOpenverse } = require('./openverse');
const { createDeepseek } = require('./deepseek');
const { createDeepseek } = require('./deepseek');
+const { createSearchFabric } = require('./search-fabric');
const { clip, canonicalUrl, hostOf } = require('./text');
const { clip, canonicalUrl, hostOf } = require('./text');
const { resolveSecret } = require('../util');
const { resolveSecret } = require('../util');
@@ -30,15 +34,19 @@ const int = (v, lo, hi, d) => { const n = Math.round(Number(v)); return Number.i
@@ -30,15 +34,19 @@ const int = (v, lo, hi, d) => { const n = Math.round(Number(v)); return Number.i
const oneLine = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);
const oneLine = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);
const short = (err) => oneLine(err && err.message ? err.message : err, 160) || 'failed';
const short = (err) => oneLine(err && err.message ? err.message : err, 160) || 'failed';
-function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console } = {}) {
+function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console, fabric = null } = {}) {
const http = createHttp({ ...(fetchImpl ? { fetchImpl } : {}), timeoutMs: config.timeoutMs || 15000 });
const http = createHttp({ ...(fetchImpl ? { fetchImpl } : {}), timeoutMs: config.timeoutMs || 15000 });
const wikimedia = createWikimedia({ http });
const wikimedia = createWikimedia({ http });
const fandom = createFandom({ http });
const fandom = createFandom({ http });
const openverse = createOpenverse({ http });
const openverse = createOpenverse({ http });
const deepseek = createDeepseek({ http, key: config.deepseekKey || (config.deepseek !== false ? resolveSecret(config.deepseekKeyRef, env) : ''), base: config.deepseekUrl || 'https://api.deepseek.com', dailyCap: config.deepseekDaily || 1500, now, log });
const deepseek = createDeepseek({ http, key: config.deepseekKey || (config.deepseek !== false ? resolveSecret(config.deepseekKeyRef, env) : ''), base: config.deepseekUrl || 'https://api.deepseek.com', dailyCap: config.deepseekDaily || 1500, now, log });
+ const search = fabric || createSearchFabric({ config: config.fabric || {}, env, fetchImpl, now });
+ const searchAvailable = () => (search && typeof search.available === 'function'
+ ? search.available()
+ : { configured: Boolean(search && search.search), reason: search ? null : 'search fabric is off' });
function status() {
function status() {
- return { providers: PROVIDERS, deepseek: deepseek.status() };
+ return { providers: PROVIDERS, deepseek: deepseek.status(), search: search && typeof search.status === 'function' ? search.status() : searchAvailable() };
}
}
/**
/**
@@ -46,9 +54,10 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
@@ -46,9 +54,10 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
* query what to research ("Kim Possible", "Ron Stoppable Kim Possible")
* query what to research ("Kim Possible", "Ron Stoppable Kim Possible")
* topic the subject the query belongs to, used to find the topic's own wiki (defaults to the query)
* topic the subject the query belongs to, used to find the topic's own wiki (defaults to the query)
* focus one subject inside the topic (a character, an episode): searched for inside the topic's wiki
* focus one subject inside the topic (a character, an episode): searched for inside the topic's wiki
- * opts: { signal, allowPaid } — allowPaid false (a developer app) leaves DeepSeek out.
+ * opts: { signal, allowPaid, allowFabric } — allowPaid false (a developer app) leaves paid DeepSeek out; allowFabric
+ * false keeps Search's own ladder from asking Search back (the caller is Search).
*/
*/
- async function gather(input = {}, { signal = null, allowPaid = true } = {}) {
+ async function gather(input = {}, { signal = null, allowPaid = true, allowFabric = true } = {}) {
const query = oneLine(input.query, 200);
const query = oneLine(input.query, 200);
if (!query) throw Object.assign(new Error('research needs a query'), { status: 422 });
if (!query) throw Object.assign(new Error('research needs a query'), { status: 422 });
const topic = oneLine(input.topic, 200) || query;
const topic = oneLine(input.topic, 200) || query;
@@ -61,7 +70,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
@@ -61,7 +70,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
const searchText = focus && !query.toLowerCase().includes(focus.toLowerCase()) ? `${focus} ${query}` : query;
const searchText = focus && !query.toLowerCase().includes(focus.toLowerCase()) ? `${focus} ${query}` : query;
const retrievedAt = new Date(now()).toISOString();
const retrievedAt = new Date(now()).toISOString();
const providers = {};
const providers = {};
- const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, cost_usd: 0 };
+ const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, search_fabric: 0, search_fabric_cost_usd: 0, cost_usd: 0 };
const startRequests = http.count();
const startRequests = http.count();
async function timed(name, fn) {
async function timed(name, fn) {
@@ -79,21 +88,66 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
@@ -79,21 +88,66 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
}
}
const skip = (name, reason) => { providers[name] = { ok: false, skipped: reason }; return null; };
const skip = (name, reason) => { providers[name] = { ok: false, skipped: reason }; return null; };
+ const wantWeb = want.has('deepseek');
+ const fabricReady = wantWeb && allowFabric && searchAvailable().configured;
+
+ /**
+ * The web evidence: the Search Fabric first, DeepSeek direct only when the fabric is not configured,
+ * fails or answers nothing. `allowPaid: false` keeps the fabric to its free and cached steps (quick)
+ * and never reaches DeepSeek.
+ */
+ async function webEvidence() {
+ if (!wantWeb) return null;
+ if (fabricReady) {
+ const t0 = now();
+ try {
+ const page = await search.search(searchText, { limit: 8, depth: allowPaid ? 'standard' : 'quick', signal });
+ const cost = Number(page.route && page.route.cost_usd) || 0;
+ const results = page.results || [];
+ // A page without a snippet cannot become a source; leave it to DeepSeek rather than cite a stub.
+ const usable = results.filter((r) => r && (r.snippet || r.content));
+ providers.search_fabric = { ok: usable.length > 0, ms: now() - t0, count: usable.length, cost_usd: cost };
+ if (page.route && Array.isArray(page.route.explain)) providers.search_fabric.explain = page.route.explain.slice(0, 20);
+ if (usable.length) {
+ usage.search_fabric += 1;
+ usage.search_fabric_cost_usd += cost;
+ return { results: usable, via: 'search_fabric' };
+ }
+ providers.search_fabric.skipped = results.length ? 'no result text' : 'no results';
+ } catch (err) {
+ if (signal && signal.aborted) throw err;
+ providers.search_fabric = { ok: false, ms: now() - t0, error: short(err) };
+ }
+ }
+ if (!allowPaid) {
+ skip('deepseek', 'first-party callers only');
+ return null;
+ }
+ if (!deepseek.available()) {
+ skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured');
+ return null;
+ }
+ const results = await timed('deepseek', async () => {
+ usage.deepseek_search += 1;
+ return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });
+ });
+ return results && results.length ? { results, via: 'deepseek' } : null;
+ }
+
// Round one, all at once: Wikipedia, web search and the picture libraries.
// Round one, all at once: Wikipedia, web search and the picture libraries.
- const useDeepseek = want.has('deepseek') && (allowPaid ? (deepseek.available() || skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured')) : skip('deepseek', 'first-party callers only'));
- const [wiki, web, commons, openv] = await Promise.all([
+ const webTask = webEvidence();
+ const [wiki, webOut, commons, openv] = await Promise.all([
want.has('wikipedia') ? timed('wikipedia', async () => {
want.has('wikipedia') ? timed('wikipedia', async () => {
const hits = await wikimedia.searchWikipedia(searchText, { lang, limit: focus ? 4 : 5, signal });
const hits = await wikimedia.searchWikipedia(searchText, { lang, limit: focus ? 4 : 5, signal });
const list = await wikimedia.articles(hits.map((h) => h.title), { lang, full: focus ? 2 : 3, chars, signal });
const list = await wikimedia.articles(hits.map((h) => h.title), { lang, full: focus ? 2 : 3, chars, signal });
return list.filter((a) => !a.disambiguation && a.text);
return list.filter((a) => !a.disambiguation && a.text);
}) : null,
}) : null,
- useDeepseek ? timed('deepseek', async () => {
- usage.deepseek_search += 1;
- return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });
- }) : null,
+ webTask,
maxImages && want.has('commons') ? timed('commons', () => wikimedia.commonsSearch(searchText, { limit: 8, signal })) : null,
maxImages && want.has('commons') ? timed('commons', () => wikimedia.commonsSearch(searchText, { limit: 8, signal })) : null,
maxImages && want.has('openverse') ? timed('openverse', () => openverse.search(searchText, { limit: 6, signal })) : null,
maxImages && want.has('openverse') ? timed('openverse', () => openverse.search(searchText, { limit: 6, signal })) : null,
]);
]);
+ const web = webOut ? webOut.results : null;
+ const webVia = webOut ? webOut.via : null;
// Round two: the topic's own wiki, the top article's facts and pictures, and full text for web results that came back short.
// Round two: the topic's own wiki, the top article's facts and pictures, and full text for web results that came back short.
const top = wiki && wiki[0];
const top = wiki && wiki[0];
@@ -108,7 +162,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
@@ -108,7 +162,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
}) : null,
}) : null,
want.has('wikidata') && top && top.qid ? timed('wikidata', () => wikimedia.wikidataFacts(top.qid, { lang, signal })) : null,
want.has('wikidata') && top && top.qid ? timed('wikidata', () => wikimedia.wikidataFacts(top.qid, { lang, signal })) : null,
maxImages && want.has('wikipedia') && top ? timed('wikipedia_images', () => wikimedia.articleImages(top.title, { lang, limit: 10, signal })) : null,
maxImages && want.has('wikipedia') && top ? timed('wikipedia_images', () => wikimedia.articleImages(top.title, { lang, limit: 10, signal })) : null,
- useDeepseek && web ? (async () => {
+ webVia === 'deepseek' && web ? (async () => {
// A short result is read in full, unless it is a copy of an encyclopedia page already read.
// A short result is read in full, unless it is a copy of an encyclopedia page already read.
const read = new Set((wiki || []).map((a) => canonicalUrl(a.url)));
const read = new Set((wiki || []).map((a) => canonicalUrl(a.url)));
const thin = web.filter((r) => r.score >= 1 && r.content.length < 400 && !read.has(canonicalUrl(r.url))).slice(0, 2);
const thin = web.filter((r) => r.score >= 1 && r.content.length < 400 && !read.has(canonicalUrl(r.url))).slice(0, 2);
@@ -127,8 +181,9 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
@@ -127,8 +181,9 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
maxImages && leadName && !leadListed ? wikimedia.fileInfo([leadName], { lang, allowSvg: true, signal }).catch(() => null) : null,
maxImages && leadName && !leadListed ? wikimedia.fileInfo([leadName], { lang, allowSvg: true, signal }).catch(() => null) : null,
]);
]);
- // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by DeepSeek's score.
+ // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by its route's order.
const scored = [];
const scored = [];
+ const fromFabric = webVia === 'search_fabric';
const focusLc = (focus || '').toLowerCase();
const focusLc = (focus || '').toLowerCase();
(fan && fan.pages || []).forEach((p, i) => {
(fan && fan.pages || []).forEach((p, i) => {
const exact = focusLc ? p.title.toLowerCase() === focusLc : i === 0;
const exact = focusLc ? p.title.toLowerCase() === focusLc : i === 0;
@@ -141,7 +196,25 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
@@ -141,7 +196,25 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
(web || []).forEach((r) => {
(web || []).forEach((r) => {
const content = clip(r.content || r.summary || r.snippet, chars);
const content = clip(r.content || r.summary || r.snippet, chars);
if (!content) return;
if (!content) return;
- scored.push({ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08, s: { source_type: 'web', url: r.url, title: r.title || hostOf(r.url), snippet: r.snippet || clip(r.summary, 300), content, published_at: r.date || undefined, retrieved_at: retrievedAt, provenance: { provider: 'deepseek', site: r.site || hostOf(r.url), authority: r.authority, score: r.score } } });
+ scored.push({
+ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08,
+ s: {
+ source_type: 'web',
+ url: r.url,
+ title: r.title || hostOf(r.url),
+ snippet: r.snippet || clip(r.summary, 300),
+ content,
+ published_at: r.date || undefined,
+ retrieved_at: r.retrieved_at || retrievedAt,
+ provenance: {
+ provider: fromFabric ? (r.provider || 'search_fabric') : 'deepseek',
+ site: r.site || hostOf(r.url),
+ authority: r.authority,
+ score: r.score,
+ ...(fromFabric ? { via: 'search_fabric' } : {}),
+ },
+ },
+ });
});
});
if (facts && facts.facts.length) {
if (facts && facts.facts.length) {
scored.push({ score: 2.0, s: { source_type: 'wikidata', url: facts.url, title: `${facts.name} — Wikidata`, snippet: facts.description || `${facts.facts.length} facts`, content: facts.facts.map((f) => `${f.label}: ${f.value}`).join('\n'), retrieved_at: retrievedAt, provenance: { provider: 'wikidata', site: 'wikidata.org', license: LICENSE.wikidata } } });
scored.push({ score: 2.0, s: { source_type: 'wikidata', url: facts.url, title: `${facts.name} — Wikidata`, snippet: facts.description || `${facts.facts.length} facts`, content: facts.facts.map((f) => `${f.label}: ${f.value}`).join('\n'), retrieved_at: retrievedAt, provenance: { provider: 'wikidata', site: 'wikidata.org', license: LICENSE.wikidata } } });
@@ -183,6 +256,8 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
@@ -183,6 +256,8 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
}
}
usage.requests = http.count() - startRequests;
usage.requests = http.count() - startRequests;
+ // Only calls this run made are its cost. A paid Search Fabric step is Search's (and, for DeepSeek, a nested
+ // AI run as `search`), so its route cost is reported separately and never billed to this run twice.
usage.cost_usd = Number(((usage.deepseek_search + usage.deepseek_fetch) * deepseek.PRICE_PER_CALL_USD).toFixed(5));
usage.cost_usd = Number(((usage.deepseek_search + usage.deepseek_fetch) * deepseek.PRICE_PER_CALL_USD).toFixed(5));
return {
return {
query, topic, focus, lang, retrieved_at: retrievedAt,
query, topic, focus, lang, retrieved_at: retrievedAt,
new file mode 100644
new file mode 100644
@@ -0,0 +1,147 @@
@@ -0,0 +1,147 @@
+'use strict';
+/**
+ * The Search Fabric's web scope, for MegaSource (OpenVibe.Search, plan 2026-10-28 §4.5):
+ *
+ * POST <OV_SEARCH_INTERNAL_URL>/api/v1/query
+ * { q, scopes: ["web"], depth, limit, freshness }
+ *
+ * Each page keeps the provider that found it, its URL and its dates, so a research citation can
+ * name all three. Search's viewer requires `search.query.delegate` of any service token
+ * (server/auth.js CAPS.delegate), so that is the capability AI asks Network for:
+ *
+ * grant `ai` `search.query.delegate` on audience `openvibe.search`.
+ *
+ * `search.fabric.query` is the route's public capability in openvibe-contracts; Search does not
+ * check it for a service token today. AI asks only for the web scope. A Search that is down,
+ * refuses the token or answers nothing never fails a research run: the caller falls back to
+ * DeepSeek (server/research/deepseek.js).
+ */
+const { createServiceTokenClient } = require('openvibe-sdk/auth');
+
+const AUDIENCE = 'openvibe.search';
+const CAPABILITY = 'search.query.delegate';
+const TIMEOUT_MS = 15000;
+const MAX_RESULTS = 20;
+
+function clip(value, max) {
+ return String(value == null ? '' : value).slice(0, max);
+}
+
+function hostOf(url) {
+ try { return new URL(String(url)).hostname.toLowerCase() || null; } catch { return null; }
+}
+
+function shortError(err) {
+ const text = String((err && (err.message || err.name)) || 'request failed').replace(/\s+/g, ' ').trim();
+ return text.slice(0, 160) || 'request failed';
+}
+
+/**
+ * Search Fabric web results as MegaSource's web shape. A citation needs url, title, published_at,
+ * retrieved_at and provider: all survive here. `score` follows the ladder's order (Search says its
+ * scores are not comparable across scopes), so the first page ranks first.
+ */
+function mapResults(body, at = Date.now()) {
+ const list = Array.isArray(body && body.results) ? body.results : [];
+ const web = list.filter((r) => r && r.url && (r.scope === 'web' || r.kind === 'page'));
+ return web.slice(0, MAX_RESULTS).map((r, i) => ({
+ url: clip(r.url, 2048),
+ title: clip(r.title || hostOf(r.url) || r.url, 300),
+ site: r.site ? clip(r.site, 253) : hostOf(r.url),
+ snippet: r.snippet == null ? '' : clip(r.snippet, 600),
+ summary: '',
+ content: r.snippet == null ? '' : clip(r.snippet, 600),
+ date: r.published_at ? clip(r.published_at, 40) : null,
+ retrieved_at: r.retrieved_at ? clip(r.retrieved_at, 40) : new Date(at).toISOString(),
+ provider: r.provider ? clip(r.provider, 40) : 'search_fabric',
+ authority: -1,
+ score: Math.max(0.1, 2.2 - i * 0.1),
+ }));
+}
+
+function createSearchFabric({ config = {}, env = process.env, fetchImpl = globalThis.fetch, now = () => Date.now(), tokens: givenTokens = null } = {}) {
+ const url = String(config.url || '').replace(/\/+$/, '');
+ const networkUrl = String(config.networkUrl || env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000').replace(/\/+$/, '');
+ const clientId = String(config.clientId || env.OV_OAUTH_CLIENT_ID || 'ai').trim();
+ const secret = env.OV_OAUTH_CLIENT_SECRET || '';
+ const enabled = config.enabled !== false;
+ let tokens = givenTokens;
+
+ function available() {
+ if (!enabled) return { configured: false, reason: 'search fabric is off' };
+ if (!url) return { configured: false, reason: 'OV_SEARCH_INTERNAL_URL is unset' };
+ if (!tokens && !secret) return { configured: false, reason: 'OV_OAUTH_CLIENT_SECRET is unset' };
+ return { configured: true, reason: null };
+ }
+
+ function tokenClient() {
+ if (!tokens) {
+ tokens = createServiceTokenClient({
+ tokenUrl: `${networkUrl}/oauth/token`,
+ clientId,
+ clientSecret: secret,
+ scope: { [AUDIENCE]: CAPABILITY },
+ fetch: fetchImpl,
+ now,
+ });
+ }
+ return tokens;
+ }
+
+ /** One web query: { results, groups, citations, route, next_cursor }. */
+ async function search(q, { limit = 8, depth = 'standard', freshness = 'any', signal = null } = {}) {
+ const availability = available();
+ if (!availability.configured) throw Object.assign(new Error(availability.reason), { code: 'search.unavailable' });
+ const token = await tokenClient().getToken({ audience: AUDIENCE });
+ const timeoutMs = Number(config.timeoutMs) > 0 ? Number(config.timeoutMs) : TIMEOUT_MS;
+ const timeout = AbortSignal.timeout(timeoutMs);
+ const combined = signal ? AbortSignal.any([signal, timeout]) : timeout;
+ let res;
+ try {
+ res = await fetchImpl(`${url}/api/v1/query`, {
+ method: 'POST',
+ headers: {
+ Authorization: `Bearer ${token}`,
+ Accept: 'application/json',
+ 'Content-Type': 'application/json',
+ 'User-Agent': 'OpenVibeResearch/1.0 (+https://ai.openvibe.services; [email protected])',
+ },
+ body: JSON.stringify({
+ q: clip(q, 200),
+ scopes: ['web'],
+ depth,
+ freshness,
+ limit: Math.min(Math.max(1, Number(limit) || 8), MAX_RESULTS),
+ }),
+ signal: combined,
+ });
+ } catch (err) {
+ if (signal && signal.aborted) throw err;
+ throw Object.assign(new Error(`OpenVibe.Search did not answer: ${shortError(err)}`), { code: 'search.unavailable' });
+ }
+ const text = await res.text().catch(() => '');
+ let body = null;
+ try { body = text ? JSON.parse(text) : null; } catch { body = null; }
+ if (!res.ok) {
+ const code = body && body.code ? ` (${clip(body.code, 80)})` : '';
+ throw Object.assign(new Error(`OpenVibe.Search answered ${res.status}${code}`), { status: res.status, code: 'search.refused' });
+ }
+ if (!body || !Array.isArray(body.results)) throw Object.assign(new Error('OpenVibe.Search answered no results'), { code: 'search.bad_answer' });
+ return {
+ results: mapResults(body, now()),
+ groups: body.groups && typeof body.groups === 'object' ? body.groups : {},
+ citations: Array.isArray(body.citations) ? body.citations : [],
+ route: body.route && typeof body.route === 'object' ? body.route : { explain: [], cost_usd: 0 },
+ next_cursor: body.next_cursor || null,
+ };
+ }
+
+ function status() {
+ const a = available();
+ return { configured: a.configured, reason: a.reason, url: url || null, audience: AUDIENCE, capability: CAPABILITY };
+ }
+
+ return { search, available, status };
+}
+
+module.exports = { createSearchFabric, mapResults, AUDIENCE, CAPABILITY };
@@ -404,6 +404,8 @@ function createRuns({ db, registry, engine, cache, quotas, config, clock = { now
@@ -404,6 +404,8 @@ function createRuns({ db, registry, engine, cache, quotas, config, clock = { now
// limited to free and local capacity when it is a sandbox run or when no tier budget governs its
// limited to free and local capacity when it is a sandbox run or when no tier budget governs its
// project (AI_GOVERN_TIERS off): metered capacity is only ever spent under a project budget.
// project (AI_GOVERN_TIERS off): metered capacity is only ever spent under a project budget.
appRun: Boolean(ctx.appRun), freeOnly: Boolean(ctx.appRun && (ctx.sandbox || !(tiers && tiers.enabled))),
appRun: Boolean(ctx.appRun), freeOnly: Boolean(ctx.appRun && (ctx.sandbox || !(tiers && tiers.enabled))),
+ // Who asked, for a step that must not call the caller back (research: Search's own web ladder).
+ requesterType: ctx.requesterType, requesterId: ctx.requesterId,
...(ctx.routePin ? { routePin: ctx.routePin } : {}),
...(ctx.routePin ? { routePin: ctx.routePin } : {}),
...(credential ? { credential, role: input && typeof input.role === 'string' ? input.role : null } : {}),
...(credential ? { credential, role: input && typeof input.role === 'string' ? input.role : null } : {}),
...(typeof ctx.onDelta === 'function' ? { onDelta: ctx.onDelta } : {}) });
...(typeof ctx.onDelta === 'function' ? { onDelta: ctx.onDelta } : {}) });
@@ -256,7 +256,10 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
@@ -256,7 +256,10 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
async function researchStep(step, input, wf, run, ctx) {
async function researchStep(step, input, wf, run, ctx) {
let output;
let output;
try {
try {
- output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun });
+ // Search's own web ladder calls this endpoint for DeepSeek and Wikipedia. It must not be asked back
+ // (Search -> AI research -> Search): every other caller gets the fabric first, Search keeps DeepSeek.
+ const fromSearch = ctx.requesterType === 'service' && ctx.requesterId === 'search';
+ output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun, allowFabric: !fromSearch });
} catch (err) {
} catch (err) {
if (err && err.status === 422) throw new AiError(422, 'input.invalid', err.message);
if (err && err.status === 422) throw new AiError(422, 'input.invalid', err.message);
throw err;
throw err;
@@ -24,7 +24,7 @@ const IMAGE = {
@@ -24,7 +24,7 @@ const IMAGE = {
const workflows = [
const workflows = [
{
{
key: 'research.gather', name: 'Research a topic (MegaSource)', namespace: 'research',
key: 'research.gather', name: 'Research a topic (MegaSource)', namespace: 'research',
- description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, DeepSeek web search, Commons and Openverse.',
+ description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, the Search Fabric web scope (DeepSeek as its fallback), Commons and Openverse.',
input_schema: {
input_schema: {
type: 'object', additionalProperties: false, required: ['query'],
type: 'object', additionalProperties: false, required: ['query'],
properties: {
properties: {
new file mode 100644
new file mode 100644
@@ -0,0 +1,233 @@
@@ -0,0 +1,233 @@
+'use strict';
+// SF-4: MegaSource asks the Search Fabric's web scope first, as AI's service principal, and keeps each page's
+// provider, date and URL for the citation. DeepSeek remains the fallback when Search is off, refuses or answers
+// nothing, and Search's own ladder never makes research ask Search back in a loop.
+
+const assert = require('assert');
+const http = require('http');
+const { createResearch } = require('../server/research');
+const { createSearchFabric } = require('../server/research/search-fabric');
+const { createResearchFetch } = require('./fixtures/research-fetch');
+const { boot, request, token, suite } = require('./helpers');
+
+const t = suite('research fabric');
+const quiet = { log() {}, warn() {}, error() {} };
+
+t.test('configuration: the fabric is on by default, pointed by OV_SEARCH_INTERNAL_URL, and AI_RESEARCH_SEARCH=0 turns it off', () => {
+ const { load } = require('../server/config');
+ const on = load({ NODE_ENV: 'test', OV_SEARCH_INTERNAL_URL: 'http://search.test:4710' }).research.fabric;
+ assert.deepStrictEqual([on.enabled, on.url, on.clientId], [true, 'http://search.test:4710', 'ai']);
+ assert.strictEqual(load({ NODE_ENV: 'test', AI_RESEARCH_SEARCH: '0' }).research.fabric.enabled, false);
+});
+
+async function listen(handler) {
+ const server = http.createServer(handler);
+ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
+ return {
+ url: `http://127.0.0.1:${server.address().port}`,
+ close: () => new Promise((resolve) => { server.closeAllConnections?.(); server.close(() => resolve()); }),
+ };
+}
+
+function readBody(req) {
+ return new Promise((resolve) => {
+ let raw = '';
+ req.on('data', (chunk) => { raw += chunk; });
+ req.on('end', () => {
+ let json = null;
+ try { json = raw ? JSON.parse(raw) : null; } catch { json = null; }
+ resolve({ raw, json });
+ });
+ });
+}
+
+function json(res, status, body) {
+ res.writeHead(status, { 'content-type': 'application/json' });
+ res.end(JSON.stringify(body));
+}
+
+t.test('the fabric client asks POST /api/v1/query for the web scope, as a service token, and keeps provider/date/url', async () => {
+ const seen = [];
+ const search = await listen(async (req, res) => {
+ const { json: body } = await readBody(req);
+ seen.push({ method: req.method, url: req.url, auth: req.headers.authorization, body });
+ json(res, 200, {
+ results: [
+ {
+ scope: 'web', kind: 'page', title: 'Fabric page', url: 'https://example.org/fabric', site: 'example.org',
+ snippet: 'A fabric snippet', published_at: '2026-10-01T00:00:00Z', retrieved_at: '2026-10-28T12:00:00Z',
+ provider: 'brave', owner: null, type: null, id: null, image: null, score: null,
+ },
+ {
+ scope: 'web', kind: 'page', title: 'No dates', url: 'https://example.org/nodate', site: 'example.org',
+ snippet: 'x', published_at: null, retrieved_at: null, provider: 'deepseek',
+ },
+ ],
+ groups: { web: 2 },
+ citations: [],
+ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },
+ next_cursor: null,
+ });
+ });
+ const fabric = createSearchFabric({
+ config: { url: search.url, timeoutMs: 2000, networkUrl: 'http://127.0.0.1:9' },
+ env: { OV_OAUTH_CLIENT_SECRET: 'x'.repeat(40) },
+ fetchImpl: globalThis.fetch,
+ tokens: { getToken: async () => 'svc-search' },
+ });
+ try {
+ const page = await fabric.search('solar eclipse', { limit: 8 });
+ assert.deepStrictEqual([seen[0].method, seen[0].url, seen[0].auth], ['POST', '/api/v1/query', 'Bearer svc-search']);
+ assert.strictEqual(seen[0].body.q, 'solar eclipse');
+ assert.deepStrictEqual(seen[0].body.scopes, ['web']);
+ assert.strictEqual(seen[0].body.depth, 'standard');
+ assert.strictEqual(seen[0].body.limit, 8);
+ assert.strictEqual(page.results[0].provider, 'brave');
+ assert.strictEqual(page.results[0].date, '2026-10-01T00:00:00Z');
+ assert.strictEqual(page.results[0].retrieved_at, '2026-10-28T12:00:00Z');
+ assert.strictEqual(page.results[0].url, 'https://example.org/fabric');
+ assert.ok(page.results[1].retrieved_at, 'a missing read date is filled at read time');
+ assert.strictEqual(page.route.cost_usd, 0.0004);
+ } finally { await search.close(); }
+});
+
+t.test('research uses the fabric first, and its citation keeps the provider, date and URL', async () => {
+ const calls = [];
+ const fabric = {
+ available: () => ({ configured: true }),
+ status: () => ({ configured: true }),
+ search: async (q, opts) => {
+ calls.push({ q, opts });
+ return {
+ results: [{
+ url: 'https://example.org/fabric', title: 'Fabric page', site: 'example.org',
+ snippet: 'A fabric snippet', summary: '', content: 'A fabric snippet',
+ date: '2026-10-02T03:04:05Z', retrieved_at: '2026-10-28T06:07:08Z',
+ provider: 'brave', authority: -1, score: 2.2,
+ }],
+ groups: { web: 1 },
+ citations: [],
+ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },
+ next_cursor: null,
+ };
+ },
+ };
+ const web = createResearchFetch();
+ const r = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });
+ const out = await r.gather({ query: 'SQLite', max_images: 0 });
+ assert.strictEqual(calls.length, 1);
+ assert.strictEqual(calls[0].opts.depth, 'standard');
+ const source = out.sources.find((s) => s.url === 'https://example.org/fabric');
+ assert.ok(source, JSON.stringify(out.sources.map((s) => s.url)));
+ assert.strictEqual(source.published_at, '2026-10-02T03:04:05Z');
+ assert.strictEqual(source.retrieved_at, '2026-10-28T06:07:08Z');
+ assert.strictEqual(source.provenance.provider, 'brave');
+ assert.strictEqual(source.provenance.via, 'search_fabric');
+ assert.strictEqual(out.providers.search_fabric.ok, true);
+ assert.strictEqual(out.providers.search_fabric.cost_usd, 0.0004);
+ assert.strictEqual(out.usage.search_fabric, 1);
+ assert.strictEqual(out.usage.search_fabric_cost_usd, 0.0004, 'the fabric route cost is reported');
+ assert.strictEqual(out.usage.cost_usd, 0, 'a paid fabric step is Search\'s cost, not billed to this run twice');
+ assert.ok(!web.calls.some((c) => c.host === 'api.deepseek.com'), 'DeepSeek is not called when the fabric answers');
+});
+
+t.test('DeepSeek is the fallback when the fabric refuses or answers nothing', async () => {
+ for (const [label, search] of [
+ ['refuses', async () => { throw new Error('Search is down'); }],
+ ['answers nothing', async () => ({ results: [], groups: { web: 0 }, citations: [], route: { explain: [], cost_usd: 0 }, next_cursor: null })],
+ ]) {
+ const fabric = { available: () => ({ configured: true }), status: () => ({ configured: true }), search };
+ const web = createResearchFetch();
+ const r = createResearch({ config: { deepseekKey: 'sk-test' }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });
+ const out = await r.gather({ query: `SQLite ${label}`, max_images: 0 });
+ assert.strictEqual(out.providers.search_fabric.ok, false, label);
+ assert.strictEqual(out.providers.deepseek.ok, true, label);
+ assert.ok(out.sources.some((s) => s.provenance.provider === 'deepseek' && !s.provenance.via), `${label}: a direct DeepSeek source`);
+ assert.strictEqual(out.usage.deepseek_search, 1, label);
+ assert.ok(out.usage.cost_usd >= 0.0015, `${label}: the fallback cost is counted`);
+ }
+});
+
+t.test('POST /api/v1/research: the token scope is search.query.delegate and the stored citation keeps provider/date/url', async () => {
+ let tokenRequest = null;
+ const network = await listen(async (req, res) => {
+ const { raw } = await readBody(req);
+ tokenRequest = Object.fromEntries(new URLSearchParams(raw));
+ json(res, 200, { access_token: 'svc-ai-search', token_type: 'Bearer', expires_in: 300, scope: 'search.query.delegate' });
+ });
+ let query = null;
+ const search = await listen(async (req, res) => {
+ const { json: body } = await readBody(req);
+ query = { method: req.method, url: req.url, auth: req.headers.authorization, body };
+ json(res, 200, {
+ results: [{
+ scope: 'web', kind: 'page', title: 'Fabric page', url: 'https://example.org/fabric-e2e', site: 'example.org',
+ snippet: 'A fabric snippet', published_at: '2026-10-03T00:00:00Z', retrieved_at: '2026-10-28T01:02:03Z',
+ provider: 'brave', owner: null, type: null, id: null, image: null, score: null,
+ }],
+ groups: { web: 1 },
+ citations: [],
+ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },
+ next_cursor: null,
+ });
+ });
+ const fixture = createResearchFetch();
+ const fetchImpl = async (url, init) => {
+ const s = String(url);
+ if (s.startsWith(network.url) || s.startsWith(search.url)) return fetch(s, init);
+ return fixture.fetchImpl(url, init);
+ };
+ const researcher = createResearch({
+ config: {
+ timeoutMs: 15000,
+ deepseek: false,
+ fabric: { enabled: true, url: search.url, networkUrl: network.url, clientId: 'ai', timeoutMs: 2000 },
+ },
+ env: { OV_OAUTH_CLIENT_SECRET: 'x'.repeat(40) },
+ fetchImpl,
+ log: quiet,
+ });
+ const h = await boot({ research: researcher });
+ try {
+ const tok = token('wiki', ['ai.run.create'], { ns: ['research.*'] });
+ const res = await request(h.base, 'POST', '/api/v1/research?wait=15000', { tok, body: { query: 'SQLite', max_images: 0 } });
+ assert.strictEqual(res.status, 201, res.text);
+ assert.strictEqual(res.body.run.status, 'succeeded', JSON.stringify(res.body.run.error));
+ assert.deepStrictEqual([tokenRequest.grant_type, tokenRequest.client_id, tokenRequest.audience, tokenRequest.scope],
+ ['client_credentials', 'ai', 'openvibe.search', 'search.query.delegate']);
+ assert.deepStrictEqual([query.method, query.url, query.auth], ['POST', '/api/v1/query', 'Bearer svc-ai-search']);
+ assert.deepStrictEqual(query.body.scopes, ['web']);
+ // research.gather's output sources are the numbered citations its callers use.
+ const citation = res.body.run.output.sources.find((s) => s.url === 'https://example.org/fabric-e2e');
+ assert.ok(citation, JSON.stringify(res.body.run.output.sources.map((s) => s.url)));
+ assert.strictEqual(citation.published_at, '2026-10-03T00:00:00Z');
+ assert.strictEqual(citation.retrieved_at, '2026-10-28T01:02:03Z');
+ assert.strictEqual(citation.provenance.provider, 'brave');
+ assert.strictEqual(citation.provenance.via, 'search_fabric');
+ } finally {
+ await h.stop();
+ await network.close();
+ await search.close();
+ }
+});
+
+t.test('Search asking AI for a web source keeps DeepSeek and does not ask Search back', async () => {
+ let fabricCalls = 0;
+ const fabric = {
+ available: () => ({ configured: true }),
+ status: () => ({ configured: true }),
+ search: async () => { fabricCalls += 1; return { results: [], route: { explain: [], cost_usd: 0 } }; },
+ };
+ const web = createResearchFetch();
+ const researcher = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });
+ const h = await boot({ research: researcher });
+ try {
+ const tok = token('search', ['ai.run.create'], { ns: ['research.*'] });
+ const res = await request(h.base, 'POST', '/api/v1/research?wait=15000', { tok, body: { query: 'SQLite', max_images: 0 } });
+ assert.strictEqual(res.status, 201, res.text);
+ assert.strictEqual(fabricCalls, 0, 'the fabric is not asked back');
+ assert.ok(res.body.run.output.sources.some((s) => s.provenance.provider === 'deepseek' && !s.provenance.via), 'DeepSeek answered directly');
+ } finally { await h.stop(); }
+});
+
+t.run();