openvibers/openvibe.ai Commits

AI research searches through the Search Fabric, with citations and a fallback (#185)

OP OpenVibers committed 16 days ago · 6635c93a8a · 9 files, +517 −32

Changes

ddiff --git a/.env.example b/.env.example
iindex 2aa4c10..54e6e92 100644
---- a/.env.example
++++ b/.env.example
@@@ -175,8 +175,15 @@ OV_MEDIA_URL=https://openvibe.media
#AI_FETCH_MAX_IMAGE_BYTES=12582912
#AI_FETCH_TIMEOUT_MS=120000
--# MegaSource research (server/research/, research.gather): DeepSeek's paid web search joins the free sources when its key
--# resolves. The key is read by name; set DEEPSEEK_API_KEY (or point AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).
++# MegaSource research (server/research/, research.gather): web search asks the Search Fabric's web scope first
++# (OpenVibe.Search POST /api/v1/query) as AI's service principal, and falls back to DeepSeek when the fabric is off,
++# refuses or answers nothing. Network must grant `ai` search.query.delegate on audience openvibe.search (Search's
++# viewer requires it for a service token); the route's public capability is search.fabric.query.
++#OV_SEARCH_INTERNAL_URL=http://127.0.0.1:4710
++#AI_RESEARCH_SEARCH=1
++#AI_RESEARCH_SEARCH_TIMEOUT_MS=15000
++# DeepSeek's paid search is the fallback. The key is read by name; set DEEPSEEK_API_KEY (or point
++# AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).
#AI_RESEARCH_DEEPSEEK=1
#AI_RESEARCH_DEEPSEEK_KEY_REF=env:DEEPSEEK_API_KEY
#AI_RESEARCH_DEEPSEEK_DAILY=1500
ddiff --git a/README.md b/README.md
iindex 582e904..c3759da 100644
---- a/README.md
++++ b/README.md
@@@ -463,7 +463,7 @@ the service boots without Valkey; with it on, production refuses to boot without
`research.gather` (namespace `research`, `server/research/`, `server/workflows/research.js`) answers one question from
many sources at once and returns one ranked, de-duplicated evidence pack the product workflows cite by number. No model
--is called; a run is recorded like a provider call (provider `research`, model `megasource`), so DeepSeek's per-call
++is called; a run is recorded like a provider call (provider `research`, model `megasource`), so the web route's per-call
price counts against the cost quotas and the daily usage.
| Source | What it gives | Terms |
@@@ -471,20 +471,28 @@ price counts against the cost quotas and the daily usage.
| Wikipedia | the best articles' full text, their pictures (each with its licence; an article's own non-free files are marked) and the Wikidata id | CC BY-SA 4.0 |
| Fandom | the topic's own community wiki when it has one (from the web results' hosts, else the topic as a subdomain, confirmed by the wiki's name): pages read through its MediaWiki API, infobox rows kept as "Label: value" | CC BY-SA 3.0; pictures marked not free |
| Wikidata | the top article's key statements as labelled facts, and its picture | CC0 |
--| DeepSeek | web search with page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each |
++| Search Fabric | the web through OpenVibe.Search's web scope (`POST /api/v1/query`, capability `search.fabric.query`) first: the open index, then the providers Search routes to; every page keeps its provider, URL and dates | per provider Search used |
++| DeepSeek | the fallback web search when the fabric is off, refuses or answers nothing: page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each |
| Commons, Openverse | freely licensed pictures (CC0, public domain, CC BY, CC BY-SA; NonCommercial and NoDerivatives are not counted as free), kept only when their title or description names the subject | per picture |
Input: `query` (required), `topic` (the subject it belongs to: finds the topic's wiki), `focus` (one subject inside the
topic), `lang`, `max_sources` (≤ 20), `max_images` (≤ 24), `content_chars` (≤ 12,000), `providers` (a subset). Output:
--`sources` (each with `url`, `title`, `content`, `retrieved_at` and `provenance.provider|site|license`), `images` (each with
--`license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per source: ok, ms, count, or
--why it was skipped or failed) and `usage` (requests, DeepSeek calls, `cost_usd`). Cached for a day per calling service.
--
--Every source is optional and fails on its own. DeepSeek is used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`,
--default `env:DEEPSEEK_API_KEY`; `AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's
--run reads the free sources), within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own
--answers: 402 (balance used up) for 30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web
--(`test/fixtures/research-fetch.js`); none reaches the internet.
++`sources` (each with `url`, `title`, `content`, `published_at`, `retrieved_at` and `provenance.provider|site|via|license`),
++`images` (each with `license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per route:
++ok, ms, count, cost, or why it was skipped or failed) and `usage` (requests, DeepSeek calls, fabric calls and the fabric's
++`search_fabric_cost_usd`; `cost_usd` is this run's own DeepSeek calls, so a paid fabric step is not billed twice —
++Search and its nested AI run pay it). Cached for a day per calling service.
++
++Web search asks the Search Fabric first, as AI's own service principal (audience `openvibe.search`). Search's viewer
++requires `search.query.delegate` of a service token, so Network must grant `ai` that capability on `openvibe.search`
++(the route's `search.fabric.query` is public). The token is requested with exactly that scope; without
++`OV_OAUTH_CLIENT_SECRET` or with `AI_RESEARCH_SEARCH=0` the fabric is skipped. When Search's own web ladder calls this
++endpoint for DeepSeek or Wikipedia, research keeps DeepSeek direct, so Search is never asked back in a loop. DeepSeek is
++used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`, default `env:DEEPSEEK_API_KEY`;
++`AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's run reads the free sources),
++within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own answers: 402 (balance used up) for
++30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web (`test/fixtures/research-fetch.js`); none reaches
++the internet.
OpenVibe.Wiki's "vibe a wiki" uses it with `wiki.generate_space` (plans the pages: main page first, each with a kind,
an outline and a search phrase) and `wiki.generate_page` (writes one page: lead, sections, infobox, `[[links]]` to the
ddiff --git a/server/config.js b/server/config.js
iindex bf7b2f2..06e443a 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -115,14 +115,24 @@ function load(env = process.env) {
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
-- // MegaSource (server/research/, workflow research.gather): DeepSeek's paid web search joins the free sources when
-- // its key resolves (a secret reference, read by name) and stays within AI_RESEARCH_DEEPSEEK_DAILY calls a day.
++ // MegaSource (server/research/, workflow research.gather): web search asks the Search Fabric's web scope
++ // first (OpenVibe.Search POST /api/v1/query, capability search.fabric.query) as AI's service principal;
++ // DeepSeek's paid search stays as the fallback when the fabric is off, refuses or answers nothing. The
++ // service token needs Search's viewer capability search.query.delegate (Network grant: ai on
++ // openvibe.search); Search's own ladder calls this service back, so that caller keeps DeepSeek directly.
research: {
deepseek: bool(env.AI_RESEARCH_DEEPSEEK, true),
deepseekKeyRef: String(env.AI_RESEARCH_DEEPSEEK_KEY_REF || 'env:DEEPSEEK_API_KEY').trim(),
deepseekUrl: trimUrl(env.AI_RESEARCH_DEEPSEEK_URL || 'https://api.deepseek.com'),
deepseekDaily: int(env.AI_RESEARCH_DEEPSEEK_DAILY, 1500),
timeoutMs: int(env.AI_RESEARCH_TIMEOUT_MS, 15000),
++ fabric: {
++ enabled: bool(env.AI_RESEARCH_SEARCH, true),
++ url: trimUrl(env.OV_SEARCH_INTERNAL_URL || env.AI_RESEARCH_SEARCH_URL || 'http://127.0.0.1:4710'),
++ networkUrl: trimUrl(env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000'),
++ clientId: String(env.OV_OAUTH_CLIENT_ID || 'ai').trim(),
++ timeoutMs: int(env.AI_RESEARCH_SEARCH_TIMEOUT_MS, 15000),
++ },
},
// media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),
// the download timeout, the scene-change threshold (scdet, 0-100) and the tools.
ddiff --git a/server/research/index.js b/server/research/index.js
iindex a84dd25..c5249b1 100644
---- a/server/research/index.js
++++ b/server/research/index.js
@@@ -6,8 +6,11 @@
* wikipedia articles for the query: full text of the best ones, lead pictures, the Wikidata id
* fandom the topic's own community wiki, when it has one (characters, episodes, places, lore)
* wikidata the top article's key statements as labelled facts (an infobox's raw material) and its picture
++ * search_fabric the web through OpenVibe.Search's web scope (the Search Fabric) first: the open index, then the
++ * providers Search routes to, every page with its provider, URL and dates
* deepseek web search with page text (DeepSeek's Search API) and pages it cannot read in full (its Fetch API);
-- * paid per call, so only for first-party callers and within its daily cap
++ * paid per call, so only for first-party callers and within its daily cap. It is the fallback when the
++ * Search Fabric is not configured, refuses or answers nothing
* commons freely licensed pictures for the query
* openverse more openly licensed pictures
*
@@@ -20,6 +23,7 @@ const { createWikimedia } = require('./wikimedia');
const { createFandom } = require('./fandom');
const { createOpenverse } = require('./openverse');
const { createDeepseek } = require('./deepseek');
++const { createSearchFabric } = require('./search-fabric');
const { clip, canonicalUrl, hostOf } = require('./text');
const { resolveSecret } = require('../util');
@@@ -30,15 +34,19 @@ const int = (v, lo, hi, d) => { const n = Math.round(Number(v)); return Number.i
const oneLine = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);
const short = (err) => oneLine(err && err.message ? err.message : err, 160) || 'failed';
--function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console } = {}) {
++function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console, fabric = null } = {}) {
const http = createHttp({ ...(fetchImpl ? { fetchImpl } : {}), timeoutMs: config.timeoutMs || 15000 });
const wikimedia = createWikimedia({ http });
const fandom = createFandom({ http });
const openverse = createOpenverse({ http });
const deepseek = createDeepseek({ http, key: config.deepseekKey || (config.deepseek !== false ? resolveSecret(config.deepseekKeyRef, env) : ''), base: config.deepseekUrl || 'https://api.deepseek.com', dailyCap: config.deepseekDaily || 1500, now, log });
++ const search = fabric || createSearchFabric({ config: config.fabric || {}, env, fetchImpl, now });
++ const searchAvailable = () => (search && typeof search.available === 'function'
++ ? search.available()
++ : { configured: Boolean(search && search.search), reason: search ? null : 'search fabric is off' });
function status() {
-- return { providers: PROVIDERS, deepseek: deepseek.status() };
++ return { providers: PROVIDERS, deepseek: deepseek.status(), search: search && typeof search.status === 'function' ? search.status() : searchAvailable() };
}
/**
@@@ -46,9 +54,10 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
* query what to research ("Kim Possible", "Ron Stoppable Kim Possible")
* topic the subject the query belongs to, used to find the topic's own wiki (defaults to the query)
* focus one subject inside the topic (a character, an episode): searched for inside the topic's wiki
-- * opts: { signal, allowPaid } — allowPaid false (a developer app) leaves DeepSeek out.
++ * opts: { signal, allowPaid, allowFabric } — allowPaid false (a developer app) leaves paid DeepSeek out; allowFabric
++ * false keeps Search's own ladder from asking Search back (the caller is Search).
*/
-- async function gather(input = {}, { signal = null, allowPaid = true } = {}) {
++ async function gather(input = {}, { signal = null, allowPaid = true, allowFabric = true } = {}) {
const query = oneLine(input.query, 200);
if (!query) throw Object.assign(new Error('research needs a query'), { status: 422 });
const topic = oneLine(input.topic, 200) || query;
@@@ -61,7 +70,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
const searchText = focus && !query.toLowerCase().includes(focus.toLowerCase()) ? `${focus} ${query}` : query;
const retrievedAt = new Date(now()).toISOString();
const providers = {};
-- const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, cost_usd: 0 };
++ const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, search_fabric: 0, search_fabric_cost_usd: 0, cost_usd: 0 };
const startRequests = http.count();
async function timed(name, fn) {
@@@ -79,21 +88,66 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
}
const skip = (name, reason) => { providers[name] = { ok: false, skipped: reason }; return null; };
++ const wantWeb = want.has('deepseek');
++ const fabricReady = wantWeb && allowFabric && searchAvailable().configured;
++
++ /**
++ * The web evidence: the Search Fabric first, DeepSeek direct only when the fabric is not configured,
++ * fails or answers nothing. `allowPaid: false` keeps the fabric to its free and cached steps (quick)
++ * and never reaches DeepSeek.
++ */
++ async function webEvidence() {
++ if (!wantWeb) return null;
++ if (fabricReady) {
++ const t0 = now();
++ try {
++ const page = await search.search(searchText, { limit: 8, depth: allowPaid ? 'standard' : 'quick', signal });
++ const cost = Number(page.route && page.route.cost_usd) || 0;
++ const results = page.results || [];
++ // A page without a snippet cannot become a source; leave it to DeepSeek rather than cite a stub.
++ const usable = results.filter((r) => r && (r.snippet || r.content));
++ providers.search_fabric = { ok: usable.length > 0, ms: now() - t0, count: usable.length, cost_usd: cost };
++ if (page.route && Array.isArray(page.route.explain)) providers.search_fabric.explain = page.route.explain.slice(0, 20);
++ if (usable.length) {
++ usage.search_fabric += 1;
++ usage.search_fabric_cost_usd += cost;
++ return { results: usable, via: 'search_fabric' };
++ }
++ providers.search_fabric.skipped = results.length ? 'no result text' : 'no results';
++ } catch (err) {
++ if (signal && signal.aborted) throw err;
++ providers.search_fabric = { ok: false, ms: now() - t0, error: short(err) };
++ }
++ }
++ if (!allowPaid) {
++ skip('deepseek', 'first-party callers only');
++ return null;
++ }
++ if (!deepseek.available()) {
++ skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured');
++ return null;
++ }
++ const results = await timed('deepseek', async () => {
++ usage.deepseek_search += 1;
++ return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });
++ });
++ return results && results.length ? { results, via: 'deepseek' } : null;
++ }
++
// Round one, all at once: Wikipedia, web search and the picture libraries.
-- const useDeepseek = want.has('deepseek') && (allowPaid ? (deepseek.available() || skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured')) : skip('deepseek', 'first-party callers only'));
-- const [wiki, web, commons, openv] = await Promise.all([
++ const webTask = webEvidence();
++ const [wiki, webOut, commons, openv] = await Promise.all([
want.has('wikipedia') ? timed('wikipedia', async () => {
const hits = await wikimedia.searchWikipedia(searchText, { lang, limit: focus ? 4 : 5, signal });
const list = await wikimedia.articles(hits.map((h) => h.title), { lang, full: focus ? 2 : 3, chars, signal });
return list.filter((a) => !a.disambiguation && a.text);
}) : null,
-- useDeepseek ? timed('deepseek', async () => {
-- usage.deepseek_search += 1;
-- return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });
-- }) : null,
++ webTask,
maxImages && want.has('commons') ? timed('commons', () => wikimedia.commonsSearch(searchText, { limit: 8, signal })) : null,
maxImages && want.has('openverse') ? timed('openverse', () => openverse.search(searchText, { limit: 6, signal })) : null,
]);
++ const web = webOut ? webOut.results : null;
++ const webVia = webOut ? webOut.via : null;
// Round two: the topic's own wiki, the top article's facts and pictures, and full text for web results that came back short.
const top = wiki && wiki[0];
@@@ -108,7 +162,7 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
}) : null,
want.has('wikidata') && top && top.qid ? timed('wikidata', () => wikimedia.wikidataFacts(top.qid, { lang, signal })) : null,
maxImages && want.has('wikipedia') && top ? timed('wikipedia_images', () => wikimedia.articleImages(top.title, { lang, limit: 10, signal })) : null,
-- useDeepseek && web ? (async () => {
++ webVia === 'deepseek' && web ? (async () => {
// A short result is read in full, unless it is a copy of an encyclopedia page already read.
const read = new Set((wiki || []).map((a) => canonicalUrl(a.url)));
const thin = web.filter((r) => r.score >= 1 && r.content.length < 400 && !read.has(canonicalUrl(r.url))).slice(0, 2);
@@@ -127,8 +181,9 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
maxImages && leadName && !leadListed ? wikimedia.fileInfo([leadName], { lang, allowSvg: true, signal }).catch(() => null) : null,
]);
-- // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by DeepSeek's score.
++ // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by its route's order.
const scored = [];
++ const fromFabric = webVia === 'search_fabric';
const focusLc = (focus || '').toLowerCase();
(fan && fan.pages || []).forEach((p, i) => {
const exact = focusLc ? p.title.toLowerCase() === focusLc : i === 0;
@@@ -141,7 +196,25 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
(web || []).forEach((r) => {
const content = clip(r.content || r.summary || r.snippet, chars);
if (!content) return;
-- scored.push({ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08, s: { source_type: 'web', url: r.url, title: r.title || hostOf(r.url), snippet: r.snippet || clip(r.summary, 300), content, published_at: r.date || undefined, retrieved_at: retrievedAt, provenance: { provider: 'deepseek', site: r.site || hostOf(r.url), authority: r.authority, score: r.score } } });
++ scored.push({
++ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08,
++ s: {
++ source_type: 'web',
++ url: r.url,
++ title: r.title || hostOf(r.url),
++ snippet: r.snippet || clip(r.summary, 300),
++ content,
++ published_at: r.date || undefined,
++ retrieved_at: r.retrieved_at || retrievedAt,
++ provenance: {
++ provider: fromFabric ? (r.provider || 'search_fabric') : 'deepseek',
++ site: r.site || hostOf(r.url),
++ authority: r.authority,
++ score: r.score,
++ ...(fromFabric ? { via: 'search_fabric' } : {}),
++ },
++ },
++ });
});
if (facts && facts.facts.length) {
scored.push({ score: 2.0, s: { source_type: 'wikidata', url: facts.url, title: `${facts.name} — Wikidata`, snippet: facts.description || `${facts.facts.length} facts`, content: facts.facts.map((f) => `${f.label}: ${f.value}`).join('\n'), retrieved_at: retrievedAt, provenance: { provider: 'wikidata', site: 'wikidata.org', license: LICENSE.wikidata } } });
@@@ -183,6 +256,8 @@ function createResearch({ config = {}, env = process.env, fetchImpl, now = () =>
}
usage.requests = http.count() - startRequests;
++ // Only calls this run made are its cost. A paid Search Fabric step is Search's (and, for DeepSeek, a nested
++ // AI run as `search`), so its route cost is reported separately and never billed to this run twice.
usage.cost_usd = Number(((usage.deepseek_search + usage.deepseek_fetch) * deepseek.PRICE_PER_CALL_USD).toFixed(5));
return {
query, topic, focus, lang, retrieved_at: retrievedAt,
ddiff --git a/server/research/search-fabric.js b/server/research/search-fabric.js
nnew file mode 100644
iindex 0000000..d96f95a
---- /dev/null
++++ b/server/research/search-fabric.js
@@@ -0,0 +1,147 @@
++'use strict';
++/**
++ * The Search Fabric's web scope, for MegaSource (OpenVibe.Search, plan 2026-10-28 §4.5):
++ *
++ * POST <OV_SEARCH_INTERNAL_URL>/api/v1/query
++ * { q, scopes: ["web"], depth, limit, freshness }
++ *
++ * Each page keeps the provider that found it, its URL and its dates, so a research citation can
++ * name all three. Search's viewer requires `search.query.delegate` of any service token
++ * (server/auth.js CAPS.delegate), so that is the capability AI asks Network for:
++ *
++ * grant `ai` `search.query.delegate` on audience `openvibe.search`.
++ *
++ * `search.fabric.query` is the route's public capability in openvibe-contracts; Search does not
++ * check it for a service token today. AI asks only for the web scope. A Search that is down,
++ * refuses the token or answers nothing never fails a research run: the caller falls back to
++ * DeepSeek (server/research/deepseek.js).
++ */
++const { createServiceTokenClient } = require('openvibe-sdk/auth');
++
++const AUDIENCE = 'openvibe.search';
++const CAPABILITY = 'search.query.delegate';
++const TIMEOUT_MS = 15000;
++const MAX_RESULTS = 20;
++
++function clip(value, max) {
++ return String(value == null ? '' : value).slice(0, max);
++}
++
++function hostOf(url) {
++ try { return new URL(String(url)).hostname.toLowerCase() || null; } catch { return null; }
++}
++
++function shortError(err) {
++ const text = String((err && (err.message || err.name)) || 'request failed').replace(/\s+/g, ' ').trim();
++ return text.slice(0, 160) || 'request failed';
++}
++
++/**
++ * Search Fabric web results as MegaSource's web shape. A citation needs url, title, published_at,
++ * retrieved_at and provider: all survive here. `score` follows the ladder's order (Search says its
++ * scores are not comparable across scopes), so the first page ranks first.
++ */
++function mapResults(body, at = Date.now()) {
++ const list = Array.isArray(body && body.results) ? body.results : [];
++ const web = list.filter((r) => r && r.url && (r.scope === 'web' || r.kind === 'page'));
++ return web.slice(0, MAX_RESULTS).map((r, i) => ({
++ url: clip(r.url, 2048),
++ title: clip(r.title || hostOf(r.url) || r.url, 300),
++ site: r.site ? clip(r.site, 253) : hostOf(r.url),
++ snippet: r.snippet == null ? '' : clip(r.snippet, 600),
++ summary: '',
++ content: r.snippet == null ? '' : clip(r.snippet, 600),
++ date: r.published_at ? clip(r.published_at, 40) : null,
++ retrieved_at: r.retrieved_at ? clip(r.retrieved_at, 40) : new Date(at).toISOString(),
++ provider: r.provider ? clip(r.provider, 40) : 'search_fabric',
++ authority: -1,
++ score: Math.max(0.1, 2.2 - i * 0.1),
++ }));
++}
++
++function createSearchFabric({ config = {}, env = process.env, fetchImpl = globalThis.fetch, now = () => Date.now(), tokens: givenTokens = null } = {}) {
++ const url = String(config.url || '').replace(/\/+$/, '');
++ const networkUrl = String(config.networkUrl || env.OV_NETWORK_INTERNAL_URL || 'http://127.0.0.1:4000').replace(/\/+$/, '');
++ const clientId = String(config.clientId || env.OV_OAUTH_CLIENT_ID || 'ai').trim();
++ const secret = env.OV_OAUTH_CLIENT_SECRET || '';
++ const enabled = config.enabled !== false;
++ let tokens = givenTokens;
++
++ function available() {
++ if (!enabled) return { configured: false, reason: 'search fabric is off' };
++ if (!url) return { configured: false, reason: 'OV_SEARCH_INTERNAL_URL is unset' };
++ if (!tokens && !secret) return { configured: false, reason: 'OV_OAUTH_CLIENT_SECRET is unset' };
++ return { configured: true, reason: null };
++ }
++
++ function tokenClient() {
++ if (!tokens) {
++ tokens = createServiceTokenClient({
++ tokenUrl: `${networkUrl}/oauth/token`,
++ clientId,
++ clientSecret: secret,
++ scope: { [AUDIENCE]: CAPABILITY },
++ fetch: fetchImpl,
++ now,
++ });
++ }
++ return tokens;
++ }
++
++ /** One web query: { results, groups, citations, route, next_cursor }. */
++ async function search(q, { limit = 8, depth = 'standard', freshness = 'any', signal = null } = {}) {
++ const availability = available();
++ if (!availability.configured) throw Object.assign(new Error(availability.reason), { code: 'search.unavailable' });
++ const token = await tokenClient().getToken({ audience: AUDIENCE });
++ const timeoutMs = Number(config.timeoutMs) > 0 ? Number(config.timeoutMs) : TIMEOUT_MS;
++ const timeout = AbortSignal.timeout(timeoutMs);
++ const combined = signal ? AbortSignal.any([signal, timeout]) : timeout;
++ let res;
++ try {
++ res = await fetchImpl(`${url}/api/v1/query`, {
++ method: 'POST',
++ headers: {
++ Authorization: `Bearer ${token}`,
++ Accept: 'application/json',
++ 'Content-Type': 'application/json',
++ 'User-Agent': 'OpenVibeResearch/1.0 (+https://ai.openvibe.services; [email protected])',
++ },
++ body: JSON.stringify({
++ q: clip(q, 200),
++ scopes: ['web'],
++ depth,
++ freshness,
++ limit: Math.min(Math.max(1, Number(limit) || 8), MAX_RESULTS),
++ }),
++ signal: combined,
++ });
++ } catch (err) {
++ if (signal && signal.aborted) throw err;
++ throw Object.assign(new Error(`OpenVibe.Search did not answer: ${shortError(err)}`), { code: 'search.unavailable' });
++ }
++ const text = await res.text().catch(() => '');
++ let body = null;
++ try { body = text ? JSON.parse(text) : null; } catch { body = null; }
++ if (!res.ok) {
++ const code = body && body.code ? ` (${clip(body.code, 80)})` : '';
++ throw Object.assign(new Error(`OpenVibe.Search answered ${res.status}${code}`), { status: res.status, code: 'search.refused' });
++ }
++ if (!body || !Array.isArray(body.results)) throw Object.assign(new Error('OpenVibe.Search answered no results'), { code: 'search.bad_answer' });
++ return {
++ results: mapResults(body, now()),
++ groups: body.groups && typeof body.groups === 'object' ? body.groups : {},
++ citations: Array.isArray(body.citations) ? body.citations : [],
++ route: body.route && typeof body.route === 'object' ? body.route : { explain: [], cost_usd: 0 },
++ next_cursor: body.next_cursor || null,
++ };
++ }
++
++ function status() {
++ const a = available();
++ return { configured: a.configured, reason: a.reason, url: url || null, audience: AUDIENCE, capability: CAPABILITY };
++ }
++
++ return { search, available, status };
++}
++
++module.exports = { createSearchFabric, mapResults, AUDIENCE, CAPABILITY };
ddiff --git a/server/runs.js b/server/runs.js
iindex 53e81bf..d71bce3 100644
---- a/server/runs.js
++++ b/server/runs.js
@@@ -404,6 +404,8 @@ function createRuns({ db, registry, engine, cache, quotas, config, clock = { now
// limited to free and local capacity when it is a sandbox run or when no tier budget governs its
// project (AI_GOVERN_TIERS off): metered capacity is only ever spent under a project budget.
appRun: Boolean(ctx.appRun), freeOnly: Boolean(ctx.appRun && (ctx.sandbox || !(tiers && tiers.enabled))),
++ // Who asked, for a step that must not call the caller back (research: Search's own web ladder).
++ requesterType: ctx.requesterType, requesterId: ctx.requesterId,
...(ctx.routePin ? { routePin: ctx.routePin } : {}),
...(credential ? { credential, role: input && typeof input.role === 'string' ? input.role : null } : {}),
...(typeof ctx.onDelta === 'function' ? { onDelta: ctx.onDelta } : {}) });
ddiff --git a/server/workflows/engine.js b/server/workflows/engine.js
iindex 67ab3ba..f8049a5 100644
---- a/server/workflows/engine.js
++++ b/server/workflows/engine.js
@@@ -256,7 +256,10 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
async function researchStep(step, input, wf, run, ctx) {
let output;
try {
-- output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun });
++ // Search's own web ladder calls this endpoint for DeepSeek and Wikipedia. It must not be asked back
++ // (Search -> AI research -> Search): every other caller gets the fabric first, Search keeps DeepSeek.
++ const fromSearch = ctx.requesterType === 'service' && ctx.requesterId === 'search';
++ output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun, allowFabric: !fromSearch });
} catch (err) {
if (err && err.status === 422) throw new AiError(422, 'input.invalid', err.message);
throw err;
ddiff --git a/server/workflows/research.js b/server/workflows/research.js
iindex 66575ae..c22bea3 100644
---- a/server/workflows/research.js
++++ b/server/workflows/research.js
@@@ -24,7 +24,7 @@ const IMAGE = {
const workflows = [
{
key: 'research.gather', name: 'Research a topic (MegaSource)', namespace: 'research',
-- description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, DeepSeek web search, Commons and Openverse.',
++ description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, the Search Fabric web scope (DeepSeek as its fallback), Commons and Openverse.',
input_schema: {
type: 'object', additionalProperties: false, required: ['query'],
properties: {
ddiff --git a/test/research-fabric.test.js b/test/research-fabric.test.js
nnew file mode 100644
iindex 0000000..43d4ca7
---- /dev/null
++++ b/test/research-fabric.test.js
@@@ -0,0 +1,233 @@
++'use strict';
++// SF-4: MegaSource asks the Search Fabric's web scope first, as AI's service principal, and keeps each page's
++// provider, date and URL for the citation. DeepSeek remains the fallback when Search is off, refuses or answers
++// nothing, and Search's own ladder never makes research ask Search back in a loop.
++
++const assert = require('assert');
++const http = require('http');
++const { createResearch } = require('../server/research');
++const { createSearchFabric } = require('../server/research/search-fabric');
++const { createResearchFetch } = require('./fixtures/research-fetch');
++const { boot, request, token, suite } = require('./helpers');
++
++const t = suite('research fabric');
++const quiet = { log() {}, warn() {}, error() {} };
++
++t.test('configuration: the fabric is on by default, pointed by OV_SEARCH_INTERNAL_URL, and AI_RESEARCH_SEARCH=0 turns it off', () => {
++ const { load } = require('../server/config');
++ const on = load({ NODE_ENV: 'test', OV_SEARCH_INTERNAL_URL: 'http://search.test:4710' }).research.fabric;
++ assert.deepStrictEqual([on.enabled, on.url, on.clientId], [true, 'http://search.test:4710', 'ai']);
++ assert.strictEqual(load({ NODE_ENV: 'test', AI_RESEARCH_SEARCH: '0' }).research.fabric.enabled, false);
++});
++
++async function listen(handler) {
++ const server = http.createServer(handler);
++ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
++ return {
++ url: `http://127.0.0.1:${server.address().port}`,
++ close: () => new Promise((resolve) => { server.closeAllConnections?.(); server.close(() => resolve()); }),
++ };
++}
++
++function readBody(req) {
++ return new Promise((resolve) => {
++ let raw = '';
++ req.on('data', (chunk) => { raw += chunk; });
++ req.on('end', () => {
++ let json = null;
++ try { json = raw ? JSON.parse(raw) : null; } catch { json = null; }
++ resolve({ raw, json });
++ });
++ });
++}
++
++function json(res, status, body) {
++ res.writeHead(status, { 'content-type': 'application/json' });
++ res.end(JSON.stringify(body));
++}
++
++t.test('the fabric client asks POST /api/v1/query for the web scope, as a service token, and keeps provider/date/url', async () => {
++ const seen = [];
++ const search = await listen(async (req, res) => {
++ const { json: body } = await readBody(req);
++ seen.push({ method: req.method, url: req.url, auth: req.headers.authorization, body });
++ json(res, 200, {
++ results: [
++ {
++ scope: 'web', kind: 'page', title: 'Fabric page', url: 'https://example.org/fabric', site: 'example.org',
++ snippet: 'A fabric snippet', published_at: '2026-10-01T00:00:00Z', retrieved_at: '2026-10-28T12:00:00Z',
++ provider: 'brave', owner: null, type: null, id: null, image: null, score: null,
++ },
++ {
++ scope: 'web', kind: 'page', title: 'No dates', url: 'https://example.org/nodate', site: 'example.org',
++ snippet: 'x', published_at: null, retrieved_at: null, provider: 'deepseek',
++ },
++ ],
++ groups: { web: 2 },
++ citations: [],
++ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },
++ next_cursor: null,
++ });
++ });
++ const fabric = createSearchFabric({
++ config: { url: search.url, timeoutMs: 2000, networkUrl: 'http://127.0.0.1:9' },
++ env: { OV_OAUTH_CLIENT_SECRET: 'x'.repeat(40) },
++ fetchImpl: globalThis.fetch,
++ tokens: { getToken: async () => 'svc-search' },
++ });
++ try {
++ const page = await fabric.search('solar eclipse', { limit: 8 });
++ assert.deepStrictEqual([seen[0].method, seen[0].url, seen[0].auth], ['POST', '/api/v1/query', 'Bearer svc-search']);
++ assert.strictEqual(seen[0].body.q, 'solar eclipse');
++ assert.deepStrictEqual(seen[0].body.scopes, ['web']);
++ assert.strictEqual(seen[0].body.depth, 'standard');
++ assert.strictEqual(seen[0].body.limit, 8);
++ assert.strictEqual(page.results[0].provider, 'brave');
++ assert.strictEqual(page.results[0].date, '2026-10-01T00:00:00Z');
++ assert.strictEqual(page.results[0].retrieved_at, '2026-10-28T12:00:00Z');
++ assert.strictEqual(page.results[0].url, 'https://example.org/fabric');
++ assert.ok(page.results[1].retrieved_at, 'a missing read date is filled at read time');
++ assert.strictEqual(page.route.cost_usd, 0.0004);
++ } finally { await search.close(); }
++});
++
++t.test('research uses the fabric first, and its citation keeps the provider, date and URL', async () => {
++ const calls = [];
++ const fabric = {
++ available: () => ({ configured: true }),
++ status: () => ({ configured: true }),
++ search: async (q, opts) => {
++ calls.push({ q, opts });
++ return {
++ results: [{
++ url: 'https://example.org/fabric', title: 'Fabric page', site: 'example.org',
++ snippet: 'A fabric snippet', summary: '', content: 'A fabric snippet',
++ date: '2026-10-02T03:04:05Z', retrieved_at: '2026-10-28T06:07:08Z',
++ provider: 'brave', authority: -1, score: 2.2,
++ }],
++ groups: { web: 1 },
++ citations: [],
++ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },
++ next_cursor: null,
++ };
++ },
++ };
++ const web = createResearchFetch();
++ const r = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });
++ const out = await r.gather({ query: 'SQLite', max_images: 0 });
++ assert.strictEqual(calls.length, 1);
++ assert.strictEqual(calls[0].opts.depth, 'standard');
++ const source = out.sources.find((s) => s.url === 'https://example.org/fabric');
++ assert.ok(source, JSON.stringify(out.sources.map((s) => s.url)));
++ assert.strictEqual(source.published_at, '2026-10-02T03:04:05Z');
++ assert.strictEqual(source.retrieved_at, '2026-10-28T06:07:08Z');
++ assert.strictEqual(source.provenance.provider, 'brave');
++ assert.strictEqual(source.provenance.via, 'search_fabric');
++ assert.strictEqual(out.providers.search_fabric.ok, true);
++ assert.strictEqual(out.providers.search_fabric.cost_usd, 0.0004);
++ assert.strictEqual(out.usage.search_fabric, 1);
++ assert.strictEqual(out.usage.search_fabric_cost_usd, 0.0004, 'the fabric route cost is reported');
++ assert.strictEqual(out.usage.cost_usd, 0, 'a paid fabric step is Search\'s cost, not billed to this run twice');
++ assert.ok(!web.calls.some((c) => c.host === 'api.deepseek.com'), 'DeepSeek is not called when the fabric answers');
++});
++
++t.test('DeepSeek is the fallback when the fabric refuses or answers nothing', async () => {
++ for (const [label, search] of [
++ ['refuses', async () => { throw new Error('Search is down'); }],
++ ['answers nothing', async () => ({ results: [], groups: { web: 0 }, citations: [], route: { explain: [], cost_usd: 0 }, next_cursor: null })],
++ ]) {
++ const fabric = { available: () => ({ configured: true }), status: () => ({ configured: true }), search };
++ const web = createResearchFetch();
++ const r = createResearch({ config: { deepseekKey: 'sk-test' }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });
++ const out = await r.gather({ query: `SQLite ${label}`, max_images: 0 });
++ assert.strictEqual(out.providers.search_fabric.ok, false, label);
++ assert.strictEqual(out.providers.deepseek.ok, true, label);
++ assert.ok(out.sources.some((s) => s.provenance.provider === 'deepseek' && !s.provenance.via), `${label}: a direct DeepSeek source`);
++ assert.strictEqual(out.usage.deepseek_search, 1, label);
++ assert.ok(out.usage.cost_usd >= 0.0015, `${label}: the fallback cost is counted`);
++ }
++});
++
++t.test('POST /api/v1/research: the token scope is search.query.delegate and the stored citation keeps provider/date/url', async () => {
++ let tokenRequest = null;
++ const network = await listen(async (req, res) => {
++ const { raw } = await readBody(req);
++ tokenRequest = Object.fromEntries(new URLSearchParams(raw));
++ json(res, 200, { access_token: 'svc-ai-search', token_type: 'Bearer', expires_in: 300, scope: 'search.query.delegate' });
++ });
++ let query = null;
++ const search = await listen(async (req, res) => {
++ const { json: body } = await readBody(req);
++ query = { method: req.method, url: req.url, auth: req.headers.authorization, body };
++ json(res, 200, {
++ results: [{
++ scope: 'web', kind: 'page', title: 'Fabric page', url: 'https://example.org/fabric-e2e', site: 'example.org',
++ snippet: 'A fabric snippet', published_at: '2026-10-03T00:00:00Z', retrieved_at: '2026-10-28T01:02:03Z',
++ provider: 'brave', owner: null, type: null, id: null, image: null, score: null,
++ }],
++ groups: { web: 1 },
++ citations: [],
++ route: { explain: [{ scope: 'web', step: 'brave', ok: true, count: 1 }], cost_usd: 0.0004 },
++ next_cursor: null,
++ });
++ });
++ const fixture = createResearchFetch();
++ const fetchImpl = async (url, init) => {
++ const s = String(url);
++ if (s.startsWith(network.url) || s.startsWith(search.url)) return fetch(s, init);
++ return fixture.fetchImpl(url, init);
++ };
++ const researcher = createResearch({
++ config: {
++ timeoutMs: 15000,
++ deepseek: false,
++ fabric: { enabled: true, url: search.url, networkUrl: network.url, clientId: 'ai', timeoutMs: 2000 },
++ },
++ env: { OV_OAUTH_CLIENT_SECRET: 'x'.repeat(40) },
++ fetchImpl,
++ log: quiet,
++ });
++ const h = await boot({ research: researcher });
++ try {
++ const tok = token('wiki', ['ai.run.create'], { ns: ['research.*'] });
++ const res = await request(h.base, 'POST', '/api/v1/research?wait=15000', { tok, body: { query: 'SQLite', max_images: 0 } });
++ assert.strictEqual(res.status, 201, res.text);
++ assert.strictEqual(res.body.run.status, 'succeeded', JSON.stringify(res.body.run.error));
++ assert.deepStrictEqual([tokenRequest.grant_type, tokenRequest.client_id, tokenRequest.audience, tokenRequest.scope],
++ ['client_credentials', 'ai', 'openvibe.search', 'search.query.delegate']);
++ assert.deepStrictEqual([query.method, query.url, query.auth], ['POST', '/api/v1/query', 'Bearer svc-ai-search']);
++ assert.deepStrictEqual(query.body.scopes, ['web']);
++ // research.gather's output sources are the numbered citations its callers use.
++ const citation = res.body.run.output.sources.find((s) => s.url === 'https://example.org/fabric-e2e');
++ assert.ok(citation, JSON.stringify(res.body.run.output.sources.map((s) => s.url)));
++ assert.strictEqual(citation.published_at, '2026-10-03T00:00:00Z');
++ assert.strictEqual(citation.retrieved_at, '2026-10-28T01:02:03Z');
++ assert.strictEqual(citation.provenance.provider, 'brave');
++ assert.strictEqual(citation.provenance.via, 'search_fabric');
++ } finally {
++ await h.stop();
++ await network.close();
++ await search.close();
++ }
++});
++
++t.test('Search asking AI for a web source keeps DeepSeek and does not ask Search back', async () => {
++ let fabricCalls = 0;
++ const fabric = {
++ available: () => ({ configured: true }),
++ status: () => ({ configured: true }),
++ search: async () => { fabricCalls += 1; return { results: [], route: { explain: [], cost_usd: 0 } }; },
++ };
++ const web = createResearchFetch();
++ const researcher = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, fabric, log: quiet });
++ const h = await boot({ research: researcher });
++ try {
++ const tok = token('search', ['ai.run.create'], { ns: ['research.*'] });
++ const res = await request(h.base, 'POST', '/api/v1/research?wait=15000', { tok, body: { query: 'SQLite', max_images: 0 } });
++ assert.strictEqual(res.status, 201, res.text);
++ assert.strictEqual(fabricCalls, 0, 'the fabric is not asked back');
++ assert.ok(res.body.run.output.sources.some((s) => s.provenance.provider === 'deepseek' && !s.provenance.via), 'DeepSeek answered directly');
++ } finally { await h.stop(); }
++});
++
++t.run();