openvibers/openvibe.ai Commits
MegaSource research (research.gather, POST /api/v1/research): Wikipedia, the topic's Fandom wiki, Wikidata, DeepSeek web search, Commons and Openverse in one ranked evidence pack with licensed pictures; the wiki planner and page writer for vibe-a-wiki on route wiki.write (#149)
Changes
@@@ -171,3 +171,10 @@ OV_MEDIA_URL=https://openvibe.media#AI_FETCH_MAX_BYTES=536870912#AI_FETCH_MAX_IMAGE_BYTES=12582912#AI_FETCH_TIMEOUT_MS=120000++++# MegaSource research (server/research/, research.gather): DeepSeek's paid web search joins the free sources when its key++# resolves. The key is read by name; set DEEPSEEK_API_KEY (or point AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).++#AI_RESEARCH_DEEPSEEK=1++#AI_RESEARCH_DEEPSEEK_KEY_REF=env:DEEPSEEK_API_KEY++#AI_RESEARCH_DEEPSEEK_DAILY=1500++#AI_RESEARCH_TIMEOUT_MS=15000@@@ -100,6 +100,7 @@ idempotent replay):|---|---|| `POST /api/v1/runs` (`?wait=ms`, `Idempotency-Key`), `POST /runs/:id/cancel`, `POST /runs/:id/retry`, `POST /runs/:id/citations` | `ai.run.create` || `POST /api/v1/{chat,generate,summarize,classify,extract,enrich,embed,image}` — each a run of workflow `ai.<op>` | `ai.run.create` |++| `POST /api/v1/research` — a run of `research.gather` ([MegaSource](#megasource-research)) | `ai.run.create` (namespace `research.*`) or `ai.app.run` (free sources only) || `POST /api/v1/{chat,generate,summarize,classify,extract,embed,image}` (no `enrich`), `GET /runs/:id` (the app's own runs) | `ai.app.run` for a developer-app token ([Developer apps](#developer-apps)) || `GET /api/v1/runs`, `GET /runs/:id` (+ citations, request-log metadata), `GET /runs/:id/citations` | `ai.run.read` (own runs) || `GET /api/v1/workflows\|templates\|routes[/:key]` | `ai.run.create`, `ai.workflow.manage` or `ai.usage.read` |@@@ -374,7 +375,7 @@ the service boots without Valkey; with it on, production refuses to boot without## Workflows--- **Seed product workflows** — `wiki.generate_space`, `wiki.generate_page`, `blog.draft_post`,++- **Seed product workflows** — `wiki.generate_space`, `wiki.generate_page` (both on route `wiki.write`, see [MegaSource](#megasource-research)), `blog.draft_post`,`news.summarize_story`, `news.compare_perspectives`, `reviews.summarize_entity`,`deals.enrich_deal`, `coupons.extract_coupon`, `trade.summarize_market_context`,`codes.generate_docs`, `games.generate_lore`, `moderation.classify`, `food.suggest_recipes`,@@@ -432,6 +433,38 @@ the service boots without Valkey; with it on, production refuses to boot without`media.analyze`.- **Direct operations** — `ai.chat|generate|summarize|classify|extract|enrich|embed|image`.++## MegaSource research++++`research.gather` (namespace `research`, `server/research/`, `server/workflows/research.js`) answers one question from++many sources at once and returns one ranked, de-duplicated evidence pack the product workflows cite by number. No model++is called; a run is recorded like a provider call (provider `research`, model `megasource`), so DeepSeek's per-call++price counts against the cost quotas and the daily usage.++++| Source | What it gives | Terms |++|---|---|---|++| Wikipedia | the best articles' full text, their pictures (each with its licence; an article's own non-free files are marked) and the Wikidata id | CC BY-SA 4.0 |++| Fandom | the topic's own community wiki when it has one (from the web results' hosts, else the topic as a subdomain, confirmed by the wiki's name): pages read through its MediaWiki API, infobox rows kept as "Label: value" | CC BY-SA 3.0; pictures marked not free |++| Wikidata | the top article's key statements as labelled facts, and its picture | CC0 |++| DeepSeek | web search with page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each |++| Commons, Openverse | freely licensed pictures (CC0, public domain, CC BY, CC BY-SA; NonCommercial and NoDerivatives are not counted as free), kept only when their title or description names the subject | per picture |++++Input: `query` (required), `topic` (the subject it belongs to: finds the topic's wiki), `focus` (one subject inside the++topic), `lang`, `max_sources` (≤ 20), `max_images` (≤ 24), `content_chars` (≤ 12,000), `providers` (a subset). Output:++`sources` (each with `url`, `title`, `content`, `retrieved_at` and `provenance.provider|site|license`), `images` (each with++`license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per source: ok, ms, count, or++why it was skipped or failed) and `usage` (requests, DeepSeek calls, `cost_usd`). Cached for a day per calling service.++++Every source is optional and fails on its own. DeepSeek is used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`,++default `env:DEEPSEEK_API_KEY`; `AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's++run reads the free sources), within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own++answers: 402 (balance used up) for 30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web++(`test/fixtures/research-fetch.js`); none reaches the internet.++++OpenVibe.Wiki's "vibe a wiki" uses it with `wiki.generate_space` (plans the pages: main page first, each with a kind,++an outline and a search phrase) and `wiki.generate_page` (writes one page: lead, sections, infobox, `[[links]]` to the++planned pages, and up to three of the numbered pictures by heading), both on route `wiki.write` (the creative model,++7,000 output tokens, 170 s, past the provider's chat timeout).++## Image generationOff unless `AI_IMAGE_MODEL` names the shared provider's image model (`dall-e-3`, `gpt-image-1`; OpenAI-shaped@@@ -142,6 +142,17 @@ function runsRouter({ runs, registry, auth, config, log = console }) {});}++ // MegaSource research (workflow research.gather): first-party services and developer apps alike; an app's run leaves++ // DeepSeek's paid web search out and reads only the free sources.++ r.post('/api/v1/research', createApp, async (req, res) => {++ try {++ const body = req.body || {};++ const { idempotency_key: idem, target, attribution, on_behalf_of: obo, options, ...input } = body;++ const wait = req.query.wait != null ? Math.max(0, Math.min(Number(req.query.wait) || 0, config.runs.maxWaitMs)) : config.runs.maxWaitMs;++ await createAndRespond(req, res, { workflow: 'research.gather', input, idempotency_key: idem, target, attribution, on_behalf_of: obo, options }, wait);++ } catch (err) { sendError(res, err, req.ov, log); }++ });++return r;}@@@ -24,7 +24,7 @@ const APP_SUB_RE = new RegExp(`^app:app_(${ULID})$`);const PROJECT_RE = new RegExp(`^prj_(${ULID})$`);const ENVS = ['sandbox', 'production'];/** The operations ai.app.run grants by route, as workflow keys (POST /api/v1/<op> is a run of ai.<op>). */--const DIRECT_OPS = Object.freeze(['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'embed', 'image'].map((op) => `ai.${op}`));++const DIRECT_OPS = Object.freeze([...['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'embed', 'image'].map((op) => `ai.${op}`), 'research.gather']);/** 'prj_01JAB…' -> 'p01jab…' (null for anything that is not a project id). */function projectKey(projectId) {@@@ -108,6 +108,15 @@ function load(env = process.env) {// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },++ // MegaSource (server/research/, workflow research.gather): DeepSeek's paid web search joins the free sources when++ // its key resolves (a secret reference, read by name) and stays within AI_RESEARCH_DEEPSEEK_DAILY calls a day.++ research: {++ deepseek: bool(env.AI_RESEARCH_DEEPSEEK, true),++ deepseekKeyRef: String(env.AI_RESEARCH_DEEPSEEK_KEY_REF || 'env:DEEPSEEK_API_KEY').trim(),++ deepseekUrl: trimUrl(env.AI_RESEARCH_DEEPSEEK_URL || 'https://api.deepseek.com'),++ deepseekDaily: int(env.AI_RESEARCH_DEEPSEEK_DAILY, 1500),++ timeoutMs: int(env.AI_RESEARCH_TIMEOUT_MS, 15000),++ },// media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),// the download timeout, the scene-change threshold (scdet, 0-100) and the tools.mediaAnalysis: {@@@ -26,7 +26,7 @@ const { createApp } = require('./app');const { gracefulStop, telemetry } = require('openvibe-sdk/service');const { createPgTelemetrySink, startTelemetryRetention } = require('openvibe-sdk/telemetry');--async function start({ config, db: givenDb = null, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, listen = true, credentialFetch = null, valkey = null } = {}) {++async function start({ config, db: givenDb = null, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, listen = true, credentialFetch = null, valkey = null, research = null } = {}) {config = config || load(env);schemas.configure(config.schemaCache);// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.@@@ -41,7 +41,8 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowconst quotas = createQuotas(db, { clock, registry, freeAllowance, tiers });const cache = createCache(db, { clock });const fetcher = createFetcher(config);-- const engine = createEngine({ registry, pool, fetcher, quotas, config, log });++ // MegaSource (server/research/): the DeepSeek key resolves by name from this environment; tests hand in their own.++ const engine = createEngine({ registry, pool, fetcher, quotas, config, log, research: research || require('./research').createResearch({ config: config.research, env, log }) });// ai.run.* to OpenVibe.Events through the outbox (server/events.js); before recovery, so its failures are announced.require('./events').init(db, { log });// platform.usage-sample@1 readings to OpenVibe.Billing through their own outbox (server/usage-samples.js).@@@ -340,7 +340,7 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.eschemas.assertSchema(w.output_schema, 'output_schema');if (!Array.isArray(w.steps) || !w.steps.length) throw new AiError(422, 'ai.invalid', 'workflow needs at least one step');for (const s of w.steps) {-- if (!s || !['llm', 'passthrough', 'complete', 'transcribe', 'embed', 'image', 'media_analysis'].includes(s.kind)) throw new AiError(422, 'ai.invalid', 'step kind must be llm, passthrough, complete, transcribe, embed, image or media_analysis');++ if (!s || !['llm', 'passthrough', 'complete', 'transcribe', 'embed', 'image', 'media_analysis', 'research'].includes(s.kind)) throw new AiError(422, 'ai.invalid', 'step kind must be llm, passthrough, complete, transcribe, embed, image, media_analysis or research');if ((s.kind === 'llm' || (s.kind === 'media_analysis' && s.template)) && !(s.template && await latestAny('templates', s.template))) throw new AiError(422, 'ai.invalid', `step references unknown template ${s.template}`);}if (!['none', 'private', 'service'].includes(w.cache_mode)) throw new AiError(422, 'ai.invalid', 'cache_mode must be none, private or service');nnew file mode 100644@@@ -0,0 +1,74 @@++'use strict';++/**++ * DeepSeek's Search and Fetch APIs (api-docs.deepseek.com/guides/search): web search with page summaries and text,++ * and any page as Markdown, fetched by DeepSeek (never from this host). $0.75 per 1,000 successful calls each, from++ * the same balance as the account's tokens; an empty search or a failed call is not billed.++ *++ * Spend is bounded twice: a daily cap on calls (AI_RESEARCH_DEEPSEEK_DAILY) and the provider standing down on its own++ * answers: 402 (balance used up) for 30 minutes, 401 (key refused) for an hour, 429 for a minute.++ */++const { clip, canonicalUrl } = require('./text');++++const PRICE_PER_CALL_USD = 0.00075;++++function createDeepseek({ http, key, base = 'https://api.deepseek.com', dailyCap = 1500, now = () => Date.now(), log = console }) {++ const state = { day: '', used: 0, downUntil: 0, reason: null };++ const today = () => new Date(now()).toISOString().slice(0, 10);++++ function available() {++ if (!key) return false;++ if (state.day !== today()) { state.day = today(); state.used = 0; }++ return now() >= state.downUntil && state.used < dailyCap;++ }++++ function status() {++ if (!key) return { configured: false };++ available();++ return { configured: true, used_today: state.used, daily_cap: dailyCap, down_until: state.downUntil > now() ? new Date(state.downUntil).toISOString() : null, reason: state.downUntil > now() ? state.reason : null };++ }++++ function standDown(ms, reason) {++ state.downUntil = now() + ms;++ state.reason = reason;++ log.warn && log.warn(`[research] deepseek stands down for ${Math.round(ms / 1000)} s: ${reason}`);++ }++++ async function call(path, body, signal) {++ if (!available()) throw Object.assign(new Error(state.reason || 'daily cap reached'), { status: 503 });++ state.used += 1;++ const r = await http.request(`${base}${path}`, { method: 'POST', body, headers: { Authorization: `Bearer ${key}` }, signal, timeout: 20000 });++ if (r.status === 402) { standDown(30 * 60000, 'balance used up'); throw Object.assign(new Error('DeepSeek balance used up'), { status: 402 }); }++ if (r.status === 401) { standDown(60 * 60000, 'key refused'); throw Object.assign(new Error('DeepSeek refused the key'), { status: 401 }); }++ if (r.status === 429) { standDown(60000, 'rate limited'); throw Object.assign(new Error('DeepSeek rate limit'), { status: 429 }); }++ if (r.status !== 200 || !r.json) throw Object.assign(new Error(`DeepSeek answered ${r.status}`), { status: r.status });++ return r.json;++ }++++ /** Web search: [{ url, title, site, snippet, summary, content, date, authority, score }]. */++ async function search(query, { count = 8, needContent = true, question = null, signal } = {}) {++ const body = { query: String(query).slice(0, 128), count, need_content: needContent };++ if (question) body.ori_question = String(question).slice(0, 128);++ const j = await call('/v1/tools/search', body, signal);++ return (j.results || []).filter((r) => r && canonicalUrl(r.url)).map((r) => ({++ url: r.url,++ title: String(r.title || '').slice(0, 300),++ site: r.site_name || null,++ snippet: String(r.snippet || '').slice(0, 400),++ summary: String(r.summary || ''),++ content: String(r.content || ''),++ date: r.date || null,++ authority: Number.isInteger(r.authority) ? r.authority : -1,++ score: Number(r.rerank_score) || 0,++ }));++ }++++ /** One page as Markdown, fetched by DeepSeek. */++ async function fetchPage(url, { chars = 6000, signal } = {}) {++ const j = await call('/v1/tools/fetch', { url }, signal);++ return { title: String(j.title || ''), content: clip(String(j.content || ''), chars) };++ }++++ return { available, status, search, fetchPage, PRICE_PER_CALL_USD };++}++++module.exports = { createDeepseek, PRICE_PER_CALL_USD };nnew file mode 100644@@@ -0,0 +1,88 @@++'use strict';++/**++ * Fandom: the community wiki dedicated to a topic (a show, a game, a franchise), through its public MediaWiki API.++ * Finding it: the hosts web search returned, else the topic's name as a subdomain (kimpossible.fandom.com), checked++ * by asking the wiki its name, which must share a word with the topic. Text is CC BY-SA 3.0 (Fandom's licence); a page's++ * picture is the wiki's own and is marked as not freely licensed.++ */++const { articleText, clip, hostOf } = require('./text');++++const qs = (params) => Object.entries({ format: 'json', formatversion: '2', ...params }).map(([k, v]) => `${k}=${encodeURIComponent(v)}`).join('&');++const STOP = new Set(['the', 'a', 'an', 'of', 'and', 'in', 'on', 'wiki', 'fandom', 'tv', 'series', 'show', 'game', 'film', 'movie']);++++function words(s) {++ return String(s || '').toLowerCase().normalize('NFKD').replace(/[̀-ͯ]/g, '').split(/[^a-z0-9]+/).filter((w) => w && !STOP.has(w));++}++++/** Subdomains to try for a topic, most specific first. */++function slugsFor(topic) {++ const all = String(topic || '').toLowerCase().normalize('NFKD').replace(/[̀-ͯ]/g, '').replace(/\([^)]*\)/g, ' ');++ const w = all.split(/[^a-z0-9]+/).filter(Boolean);++ const content = w.filter((x) => !STOP.has(x));++ const out = [w.join(''), content.join(''), content.slice(0, 2).join(''), content[0] && content[0].length >= 4 ? content[0] : ''];++ return [...new Set(out.filter((s) => /^[a-z0-9]{3,60}$/.test(s)))].slice(0, 4);++}++++function createFandom({ http }) {++ async function probe(base, topic, signal) {++ const r = await http.request(`${base}/api.php?${qs({ action: 'query', meta: 'siteinfo', siprop: 'general' })}`, { signal, timeout: 8000 });++ const g = r.status === 200 && r.json && r.json.query && r.json.query.general;++ if (!g || !g.sitename) return null;++ const topicWords = new Set(words(topic));++ const overlap = words(g.sitename).some((x) => topicWords.has(x)) || words(new URL(base).hostname.split('.')[0]).some((x) => topicWords.has(x));++ return overlap ? { base, name: String(g.sitename), lang: g.lang || 'en' } : null;++ }++++ /** The topic's wiki: { base, name } or null. hints are URLs a web search returned. */++ async function findWiki(topic, hints = [], { lang = 'en', signal } = {}) {++ const hosts = [];++ for (const u of hints) {++ const h = hostOf(u);++ const m = /^([a-z0-9-]{2,60})\.fandom\.com$/.exec(h);++ if (m && m[1] !== 'community' && m[1] !== 'www' && !hosts.includes(m[1])) hosts.push(m[1]);++ }++ const path = lang && lang !== 'en' ? `/${lang}` : '';++ for (const slug of [...hosts.slice(0, 2), ...slugsFor(topic)].filter((s, i, a) => a.indexOf(s) === i).slice(0, 5)) {++ try {++ const found = await probe(`https://${slug}.fandom.com${path}`, topic, signal);++ if (found) return found;++ } catch { /* try the next one */ }++ }++ return null;++ }++++ async function search(base, query, { limit = 4, signal } = {}) {++ const j = await http.getJson(`${base}/api.php?${qs({ action: 'query', list: 'search', srsearch: query, srlimit: String(limit), srnamespace: '0' })}`, { signal });++ return ((j.query && j.query.search) || []).map((s) => s.title).filter((t) => !/\/(Gallery|Transcript|Quotes|LAM)$/i.test(t));++ }++++ /** One article: readable text and its picture. */++ async function page(base, title, { chars = 6000, signal } = {}) {++ const [parsed, images] = await Promise.all([++ http.getJson(`${base}/api.php?${qs({ action: 'parse', page: title, prop: 'text|displaytitle', redirects: '1', disableeditsection: '1', disabletoc: '1', disablelimitreport: '1' })}`, { signal }),++ http.getJson(`${base}/api.php?${qs({ action: 'query', titles: title, prop: 'pageimages', piprop: 'original', redirects: '1' })}`, { signal }).catch(() => null),++ ]);++ const p = parsed && parsed.parse;++ if (!p || !p.text) return null;++ const name = p.title || title;++ const url = `${base}/wiki/${encodeURIComponent(name.replace(/ /g, '_'))}`;++ const text = clip(articleText(typeof p.text === 'string' ? p.text : p.text['*']), chars);++ const pg = images && images.query && images.query.pages && images.query.pages[0];++ const original = pg && pg.original;++ const image = original && original.source && original.width >= 160 ? {++ url: String(original.source).replace(/\/revision\/latest(\?|$)/, '/revision/latest/scale-to-width-down/1024$1'),++ original: original.source,++ width: Math.min(original.width, 1024),++ height: original.width > 1024 ? Math.round(original.height * 1024 / original.width) : original.height,++ mime: /\.png/i.test(original.source) ? 'image/png' : /\.webp/i.test(original.source) ? 'image/webp' : /\.gif/i.test(original.source) ? 'image/gif' : 'image/jpeg',++ title: name, description: null, author: null,++ license: 'Fandom wiki picture (copyright of its owners; used for identification)', license_url: null, free: false,++ source_url: url, provider: 'fandom',++ } : null;++ return text ? { title: name, url, text, image } : null;++ }++++ return { findWiki, search, page };++}++++module.exports = { createFandom, slugsFor };nnew file mode 100644@@@ -0,0 +1,50 @@++'use strict';++/**++ * HTTPS for the research providers: a descriptive User-Agent with a contact (Wikimedia's API policy asks for one), a++ * per-call timeout, the run's cancellation, and the guarded fetch (public addresses only, no redirects, 8 MB cap).++ */++const { guardedFetch } = require('../providers/guarded-fetch');++++const USER_AGENT = 'OpenVibeResearch/1.0 (+https://ai.openvibe.services; [email protected])';++++class ProviderError extends Error {++ constructor(status, message) { super(message); this.status = status; }++}++++function createHttp({ fetchImpl = guardedFetch, timeoutMs = 15000, userAgent = USER_AGENT } = {}) {++ let requests = 0;++ async function request(url, { method = 'GET', body = null, headers = {}, signal = null, timeout = timeoutMs } = {}) {++ const ctrl = new AbortController();++ const timer = setTimeout(() => ctrl.abort(), timeout);++ const onAbort = () => ctrl.abort();++ if (signal) { if (signal.aborted) ctrl.abort(); else signal.addEventListener('abort', onAbort, { once: true }); }++ requests += 1;++ try {++ const res = await fetchImpl(url, {++ method,++ headers: { 'User-Agent': userAgent, Accept: 'application/json', ...(body != null ? { 'Content-Type': 'application/json' } : {}), ...headers },++ body: body == null ? undefined : JSON.stringify(body),++ signal: ctrl.signal,++ });++ const text = await res.text();++ let json = null;++ try { json = text ? JSON.parse(text) : null; } catch { /* not JSON: the caller decides */ }++ return { status: res.status, json, text };++ } catch (err) {++ if (ctrl.signal.aborted && !(signal && signal.aborted)) throw new ProviderError(504, `timed out after ${timeout} ms`);++ throw err;++ } finally {++ clearTimeout(timer);++ if (signal) signal.removeEventListener('abort', onAbort);++ }++ }++ /** GET a JSON API; anything but a 200 with JSON is an error. */++ async function getJson(url, opts) {++ const r = await request(url, opts);++ if (r.status !== 200 || !r.json) throw new ProviderError(r.status, `${new URL(url).hostname} answered ${r.status}`);++ return r.json;++ }++ return { request, getJson, count: () => requests };++}++++module.exports = { createHttp, ProviderError, USER_AGENT };nnew file mode 100644@@@ -0,0 +1,199 @@++'use strict';++/**++ * MegaSource: one research call over many sources, merged into one ranked, de-duplicated evidence pack that the++ * product workflows cite by number (workflow research.gather, POST /api/v1/research).++ *++ * wikipedia articles for the query: full text of the best ones, lead pictures, the Wikidata id++ * fandom the topic's own community wiki, when it has one (characters, episodes, places, lore)++ * wikidata the top article's key statements as labelled facts (an infobox's raw material) and its picture++ * deepseek web search with page text (DeepSeek's Search API) and pages it cannot read in full (its Fetch API);++ * paid per call, so only for first-party callers and within its daily cap++ * commons freely licensed pictures for the query++ * openverse more openly licensed pictures++ *++ * Every provider is optional and fails on its own: a provider that errors, times out or stands down is reported in++ * `providers` and the rest still answer. Nothing is invented: a source is a page that was read, with its URL, when it++ * was read and its licence; a picture carries its own licence and `free` says whether anyone may reuse it.++ */++const { createHttp } = require('./http');++const { createWikimedia } = require('./wikimedia');++const { createFandom } = require('./fandom');++const { createOpenverse } = require('./openverse');++const { createDeepseek } = require('./deepseek');++const { clip, canonicalUrl, hostOf } = require('./text');++const { resolveSecret } = require('../util');++++const PROVIDERS = ['wikipedia', 'fandom', 'wikidata', 'deepseek', 'commons', 'openverse'];++const LICENSE = { wikipedia: 'CC BY-SA 4.0 (Wikipedia)', fandom: 'CC BY-SA 3.0 (Fandom)', wikidata: 'CC0 (Wikidata)' };++++const int = (v, lo, hi, d) => { const n = Math.round(Number(v)); return Number.isFinite(n) ? Math.min(hi, Math.max(lo, n)) : d; };++const oneLine = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);++const short = (err) => oneLine(err && err.message ? err.message : err, 160) || 'failed';++++function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console } = {}) {++ const http = createHttp({ ...(fetchImpl ? { fetchImpl } : {}), timeoutMs: config.timeoutMs || 15000 });++ const wikimedia = createWikimedia({ http });++ const fandom = createFandom({ http });++ const openverse = createOpenverse({ http });++ const deepseek = createDeepseek({ http, key: config.deepseekKey || (config.deepseek !== false ? resolveSecret(config.deepseekKeyRef, env) : ''), base: config.deepseekUrl || 'https://api.deepseek.com', dailyCap: config.deepseekDaily || 1500, now, log });++++ function status() {++ return { providers: PROVIDERS, deepseek: deepseek.status() };++ }++++ /**++ * input: { query, topic?, focus?, question?, lang?, max_sources?, max_images?, images?, content_chars?, providers? }++ * query what to research ("Kim Possible", "Ron Stoppable Kim Possible")++ * topic the subject the query belongs to, used to find the topic's own wiki (defaults to the query)++ * focus one subject inside the topic (a character, an episode): searched for inside the topic's wiki++ * opts: { signal, allowPaid } — allowPaid false (a developer app) leaves DeepSeek out.++ */++ async function gather(input = {}, { signal = null, allowPaid = true } = {}) {++ const query = oneLine(input.query, 200);++ if (!query) throw Object.assign(new Error('research needs a query'), { status: 422 });++ const topic = oneLine(input.topic, 200) || query;++ const focus = oneLine(input.focus, 200) || null;++ const lang = /^[a-z]{2,3}$/.test(input.lang || '') ? input.lang : 'en';++ const maxSources = int(input.max_sources, 1, 20, 10);++ const maxImages = input.images === false ? 0 : int(input.max_images, 0, 24, 10);++ const chars = int(input.content_chars, 500, 12000, 5000);++ const want = new Set(Array.isArray(input.providers) && input.providers.length ? input.providers.filter((p) => PROVIDERS.includes(p)) : PROVIDERS);++ const searchText = focus && !query.toLowerCase().includes(focus.toLowerCase()) ? `${focus} ${query}` : query;++ const retrievedAt = new Date(now()).toISOString();++ const providers = {};++ const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, cost_usd: 0 };++ const startRequests = http.count();++++ async function timed(name, fn) {++ const t0 = now();++ try {++ const value = await fn();++ const count = value == null ? 0 : Array.isArray(value) ? value.length : (value.count != null ? value.count : 1);++ providers[name] = { ok: true, ms: now() - t0, count };++ return value;++ } catch (err) {++ if (signal && signal.aborted) throw err;++ providers[name] = { ok: false, ms: now() - t0, error: short(err) };++ return null;++ }++ }++ const skip = (name, reason) => { providers[name] = { ok: false, skipped: reason }; return null; };++++ // Round one, all at once: Wikipedia, web search and the picture libraries.++ const useDeepseek = want.has('deepseek') && (allowPaid ? (deepseek.available() || skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured')) : skip('deepseek', 'first-party callers only'));++ const [wiki, web, commons, openv] = await Promise.all([++ want.has('wikipedia') ? timed('wikipedia', async () => {++ const hits = await wikimedia.searchWikipedia(searchText, { lang, limit: focus ? 4 : 5, signal });++ const list = await wikimedia.articles(hits.map((h) => h.title), { lang, full: focus ? 2 : 3, chars, signal });++ return list.filter((a) => !a.disambiguation && a.text);++ }) : null,++ useDeepseek ? timed('deepseek', async () => {++ usage.deepseek_search += 1;++ return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });++ }) : null,++ maxImages && want.has('commons') ? timed('commons', () => wikimedia.commonsSearch(searchText, { limit: 8, signal })) : null,++ maxImages && want.has('openverse') ? timed('openverse', () => openverse.search(searchText, { limit: 6, signal })) : null,++ ]);++++ // Round two: the topic's own wiki, the top article's facts and pictures, and full text for web results that came back short.++ const top = wiki && wiki[0];++ const hints = (web || []).map((r) => r.url);++ const [fan, facts, articlePictures, fetched] = await Promise.all([++ want.has('fandom') ? timed('fandom', async () => {++ const site = await fandom.findWiki(topic, hints, { lang, signal });++ if (!site) return null;++ const titles = await fandom.search(site.base, focus || topic, { limit: 4, signal });++ const pages = (await Promise.all(titles.slice(0, focus ? 2 : 3).map((t) => fandom.page(site.base, t, { chars, signal }).catch(() => null)))).filter(Boolean);++ return { site, pages, count: pages.length };++ }) : null,++ want.has('wikidata') && top && top.qid ? timed('wikidata', () => wikimedia.wikidataFacts(top.qid, { lang, signal })) : null,++ maxImages && want.has('wikipedia') && top ? timed('wikipedia_images', () => wikimedia.articleImages(top.title, { lang, limit: 10, signal })) : null,++ useDeepseek && web ? (async () => {++ // A short result is read in full, unless it is a copy of an encyclopedia page already read.++ const read = new Set((wiki || []).map((a) => canonicalUrl(a.url)));++ const thin = web.filter((r) => r.score >= 1 && r.content.length < 400 && !read.has(canonicalUrl(r.url))).slice(0, 2);++ return Promise.all(thin.map(async (r) => {++ try { usage.deepseek_fetch += 1; const page = await deepseek.fetchPage(r.url, { chars, signal }); r.content = page.content || r.content; } catch { /* the summary stays */ }++ }));++ })() : null,++ ]);++ void fetched;++ // The Wikidata item's picture, and the article's lead picture when the article's own list left it out (a logo++ // drawn as SVG), each with its licence from the file's page.++ const leadName = top && top.lead ? decodeURIComponent(String(top.lead.original).split('/').pop() || '') : '';++ const leadListed = leadName && (articlePictures || []).some((i) => String(i.original).endsWith(`/${encodeURIComponent(leadName).replace(/%20/g, '_')}`) || String(i.original).endsWith(`/${leadName}`));++ const [factPicture, leadPicture] = await Promise.all([++ maxImages && facts && facts.image ? timed('wikidata_image', () => wikimedia.commonsFiles([facts.image], { signal })) : null,++ maxImages && leadName && !leadListed ? wikimedia.fileInfo([leadName], { lang, allowSvg: true, signal }).catch(() => null) : null,++ ]);++++ // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by DeepSeek's score.++ const scored = [];++ const focusLc = (focus || '').toLowerCase();++ (fan && fan.pages || []).forEach((p, i) => {++ const exact = focusLc ? p.title.toLowerCase() === focusLc : i === 0;++ scored.push({ score: (exact ? 3.3 : 2.4) - i * 0.15, s: { source_type: 'fandom', url: p.url, title: `${p.title} — ${fan.site.name}`, snippet: clip(p.text, 300), content: p.text, retrieved_at: retrievedAt, provenance: { provider: 'fandom', site: fan.site.name, license: LICENSE.fandom } } });++ });++ (wiki || []).forEach((a, i) => {++ const exact = a.title.toLowerCase() === (focusLc || query.toLowerCase());++ scored.push({ score: (exact || i === 0 ? 3.0 : 1.7) - i * 0.12, s: { source_type: 'wikipedia', url: a.url, title: `${a.title} — Wikipedia`, snippet: a.description || clip(a.text, 300), content: a.text, published_at: a.revisedAt || undefined, retrieved_at: retrievedAt, provenance: { provider: 'wikipedia', site: `${lang}.wikipedia.org`, license: LICENSE.wikipedia } } });++ });++ (web || []).forEach((r) => {++ const content = clip(r.content || r.summary || r.snippet, chars);++ if (!content) return;++ scored.push({ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08, s: { source_type: 'web', url: r.url, title: r.title || hostOf(r.url), snippet: r.snippet || clip(r.summary, 300), content, published_at: r.date || undefined, retrieved_at: retrievedAt, provenance: { provider: 'deepseek', site: r.site || hostOf(r.url), authority: r.authority, score: r.score } } });++ });++ if (facts && facts.facts.length) {++ scored.push({ score: 2.0, s: { source_type: 'wikidata', url: facts.url, title: `${facts.name} — Wikidata`, snippet: facts.description || `${facts.facts.length} facts`, content: facts.facts.map((f) => `${f.label}: ${f.value}`).join('\n'), retrieved_at: retrievedAt, provenance: { provider: 'wikidata', site: 'wikidata.org', license: LICENSE.wikidata } } });++ }++ scored.sort((a, b) => b.score - a.score);++ const seen = new Set();++ const sources = [];++ for (const { s } of scored) {++ const key = canonicalUrl(s.url);++ if (!key || seen.has(key)) continue;++ seen.add(key);++ for (const k of Object.keys(s)) if (s[k] === undefined) delete s[k];++ sources.push(s);++ if (sources.length >= maxSources) break;++ }++++ // Pictures: the subject's own (Wikidata, the article, the topic's wiki) before library search results.++ const images = [];++ const seenImg = new Set();++ const lead = (leadPicture || []).map((i) => ({ ...i, title: top.title, description: i.description || top.description || null, source_url: i.source_url || top.url }));++ const fanPics = (fan && fan.pages || []).map((p) => p.image).filter(Boolean);++ // Library search is loose ("Ron Stoppable" also finds "whistle stop" photos): a library picture must name every++ // word of the focus, or most of the query's, in its title or description.++ const significant = (s) => String(s || '').toLowerCase().normalize('NFKD').replace(/[\u0300-\u036f]/g, '').split(/[^a-z0-9]+/).filter((w) => w.length > 2 && !/^(the|and|for|with|from|wiki)$/.test(w));++ const named = (img) => {++ const text = significant(`${img.title} ${img.description || ''}`).join(' ');++ const need = significant(focus || query);++ const hits = need.filter((w) => text.includes(w)).length;++ return need.length ? (focus ? hits === need.length : hits >= Math.ceil(need.length / 2)) : true;++ };++ const library = [...(commons || []), ...(openv || [])].filter(named);++ for (const img of [...(factPicture || []), ...lead, ...(articlePictures || []), ...fanPics, ...library]) {++ if (!img || !img.url || !/^https:\/\//.test(img.url) || /\.svg(\?|$)/i.test(img.url)) continue;++ const key = String(img.original || img.url).replace(/\/(scale-to-width-down|thumb)\/.*$/, '');++ if (seenImg.has(key)) continue;++ seenImg.add(key);++ images.push({ ...img });++ if (images.length >= maxImages) break;++ }++++ usage.requests = http.count() - startRequests;++ usage.cost_usd = Number(((usage.deepseek_search + usage.deepseek_fetch) * deepseek.PRICE_PER_CALL_USD).toFixed(5));++ return {++ query, topic, focus, lang, retrieved_at: retrievedAt,++ sources, images,++ facts: facts ? facts.facts : [],++ topic_wiki: fan && fan.site ? { name: fan.site.name, url: fan.site.base } : null,++ providers, usage,++ };++ }++++ return { gather, status };++}++++module.exports = { createResearch, PROVIDERS };nnew file mode 100644@@@ -0,0 +1,32 @@++'use strict';++/**++ * Openverse (api.openverse.org): openly licensed pictures from Flickr, museums and other collections, anonymous use.++ * Only licences that allow commercial reuse are asked for (CC0, public domain, CC BY, CC BY-SA).++ */++++const NAMES = { cc0: 'CC0', pdm: 'Public domain', by: 'CC BY', 'by-sa': 'CC BY-SA' };++++function createOpenverse({ http }) {++ async function search(query, { limit = 8, signal } = {}) {++ const url = `https://api.openverse.org/v1/images/?q=${encodeURIComponent(query)}&page_size=${limit}&license_type=commercial&mature=false`;++ const j = await http.getJson(url, { signal, timeout: 10000 });++ return (j.results || []).filter((r) => r.url && /^https:/.test(r.url) && (r.width || 0) >= 320 && NAMES[r.license]).map((r) => ({++ url: r.url,++ original: r.url,++ width: r.width || null,++ height: r.height || null,++ mime: null,++ title: String(r.title || '').slice(0, 200) || query,++ description: null,++ author: r.creator ? String(r.creator).slice(0, 160) : null,++ license: `${NAMES[r.license]}${r.license_version && !/^(cc0|pdm)$/.test(r.license) ? ` ${r.license_version}` : ''}`,++ license_url: r.license_url || null,++ free: true,++ source_url: r.foreign_landing_url || null,++ provider: `openverse${r.source ? `/${r.source}` : ''}`,++ }));++ }++ return { search };++}++++module.exports = { createOpenverse };nnew file mode 100644@@@ -0,0 +1,96 @@++'use strict';++/** Text helpers for research: HTML to readable text, clipping at a boundary, canonical URLs, image licences. */++++const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: '\'', nbsp: ' ', ndash: '–', mdash: '—', hellip: '…', rsquo: '’', lsquo: '‘', rdquo: '”', ldquo: '“', middot: '·', times: '×' };++++function decodeEntities(s) {++ return String(s).replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {++ if (e[0] === '#') {++ const n = e[1].toLowerCase() === 'x' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);++ return Number.isFinite(n) && n > 0 && n < 0x110000 ? String.fromCodePoint(n) : '';++ }++ return ENTITIES[e.toLowerCase()] ?? m;++ });++}++++function stripTags(html) {++ return decodeEntities(String(html || '').replace(/<[^>]*>/g, ' ')).replace(/\s+/g, ' ').trim();++}++++/**++ * MediaWiki article HTML (Fandom's parse output) to readable text: infobox rows become "Label: value" lines, headings++ * become "## Heading", list items "- item"; scripts, references, navigation boxes and tables of contents are dropped.++ */++function articleText(html) {++ let h = String(html || '');++ h = h.replace(/<(script|style|noscript)\b[\s\S]*?<\/\1>/gi, ' ')++ .replace(/<sup\b[^>]*class="[^"]*reference[^"]*"[\s\S]*?<\/sup>/gi, '')++ .replace(/<(table|div|nav)\b[^>]*class="[^"]*\b(navbox|toc|mw-references-wrap|references|printfooter|catlinks|mbox)\b[^"]*"[\s\S]*?<\/\1>/gi, ' ')++ .replace(/<h[23]\b[^>]*class="[^"]*pi-(?:data-label|secondary-heading)[^"]*"[^>]*>([\s\S]*?)<\/h[23]>\s*<div\b[^>]*class="[^"]*pi-data-value[^"]*"[^>]*>([\s\S]*?)<\/div>/gi,++ (_m, label, value) => `\n${stripTags(label)}: ${stripTags(value.replace(/<br\s*\/?>/gi, ', ').replace(/<\/li>/gi, ', ')).replace(/,\s*$/, '')}\n`)++ .replace(/<h2\b[^>]*class="[^"]*pi-title[^"]*"[^>]*>([\s\S]*?)<\/h2>/gi, (_m, t) => `\n${stripTags(t)}\n`)++ .replace(/<h([1-6])\b[^>]*>([\s\S]*?)<\/h\1>/gi, (_m, n, t) => `\n\n${'#'.repeat(Math.min(Math.max(Number(n), 2), 4))} ${stripTags(t)}\n`)++ .replace(/<li\b[^>]*>/gi, '\n- ')++ .replace(/<\/(p|div|tr|ul|ol|table|aside|figure|blockquote|dd|dt)>/gi, '\n')++ .replace(/<br\s*\/?>/gi, '\n')++ .replace(/<\/t[dh]>/gi, ' · ');++ const lines = decodeEntities(h.replace(/<[^>]*>/g, '')).split('\n').map((l) => l.replace(/[ \t ]+/g, ' ').replace(/(?: · )+$/, '').trim());++ const out = [];++ for (const l of lines) {++ if (!l) { if (out.length && out[out.length - 1] !== '') out.push(''); continue; }++ if (/^#{2,4} (?:References|Notes|External links|See also|Gallery|Navigation|Sources|Appearances)$/i.test(l)) { out.push(l); continue; }++ out.push(l);++ }++ // Drop the trailing reference-ish sections entirely: they are lists of links, not facts.++ const text = out.join('\n').replace(/\n#{2,4} (?:References|Notes|External links|Navigation|Sources)\n[\s\S]*?(?=\n#{2,4} |$)/gi, '\n');++ return text.replace(/\n{3,}/g, '\n\n').trim();++}++++/** MediaWiki plain extract with exsectionformat=wiki ("== Heading ==") to "## Heading" Markdown-ish text. */++function extractText(text) {++ return String(text || '')++ .replace(/^(={2,5})\s*(.+?)\s*\1\s*$/gm, (_m, eq, t) => `${'#'.repeat(Math.min(eq.length, 4))} ${t}`)++ .replace(/\n#{2,4} (?:References|Notes|External links|Further reading|Citations|Sources|Bibliography)\n[\s\S]*?(?=\n#{2,4} |$)/gi, '\n')++ .replace(/\n{3,}/g, '\n\n')++ .trim();++}++++/** Cut text at about n characters, at the last paragraph or sentence end before it. */++function clip(text, n) {++ const t = String(text || '').trim();++ if (t.length <= n) return t;++ const cut = t.slice(0, n);++ const para = cut.lastIndexOf('\n\n');++ if (para > n * 0.6) return cut.slice(0, para).trim();++ const sentence = Math.max(cut.lastIndexOf('. '), cut.lastIndexOf('.\n'), cut.lastIndexOf('! '), cut.lastIndexOf('? '));++ if (sentence > n * 0.6) return cut.slice(0, sentence + 1).trim();++ return `${cut.replace(/\s+\S*$/, '')}…`;++}++++/** One spelling per page: https, lowercase host without www. or a Wikipedia mobile prefix, no fragment or tracking query. */++function canonicalUrl(u) {++ try {++ const url = new URL(String(u));++ if (url.protocol !== 'https:' && url.protocol !== 'http:') return null;++ url.protocol = 'https:';++ url.hash = '';++ url.hostname = url.hostname.toLowerCase().replace(/^www\./, '').replace(/^([a-z-]+)\.m\.(wikipedia|wiktionary|wikiquote)\.org$/, '$1.$2.org');++ for (const k of [...url.searchParams.keys()]) if (/^(utm_|fbclid$|gclid$|ref$|ref_src$)/i.test(k)) url.searchParams.delete(k);++ return url.toString().replace(/\/$/, '');++ } catch { return null; }++}++++function hostOf(u) {++ try { return new URL(String(u)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return ''; }++}++++// Licences anyone may reuse with credit (commercial use and unchanged copies included): CC0, public domain, CC BY,++// CC BY-SA, GFDL and the like. NonCommercial and NoDerivatives licences are not counted as free here.++const FREE_LICENSE = /^(cc0|cc[- ]?zero|public domain|pd(?:m|-|\b)|no restrictions|attribution|cc[- ]by(?:-sa)?(?![- ]?n[cd])\b|gfdl|fal\b|free art|ogl|copyrighted free use)/i;++++function isFreeLicense(name) {++ const n = String(name || '').trim();++ return Boolean(n) && FREE_LICENSE.test(n) && !/\b(nc|nd|non-?commercial|no ?deriv)/i.test(n) && !/fair use|non-?free/i.test(n);++}++++module.exports = { decodeEntities, stripTags, articleText, extractText, clip, canonicalUrl, hostOf, isFreeLicense };nnew file mode 100644@@@ -0,0 +1,209 @@++'use strict';++/**++ * Wikipedia, Wikimedia Commons and Wikidata, through their public APIs (no key; a descriptive User-Agent).++ *++ * searchWikipedia article titles for a query++ * articles intro, lead image, Wikidata id and description for several titles at once, and the full plain++ * text of the first few++ * articleImages the pictures an article uses, each with its licence (free files from Commons and an article's++ * own non-free ones, marked so)++ * commonsSearch freely licensed pictures on Commons for a query++ * commonsFiles licence and size for named files (a Wikidata item's image)++ * wikidataFacts a Wikidata item's key statements as labelled facts++ *++ * Text is CC BY-SA (Wikipedia) or CC0 (Wikidata); every picture carries its own licence from the file's metadata.++ */++const { stripTags, extractText, clip, isFreeLicense } = require('./text');++++const API = (lang) => `https://${lang}.wikipedia.org/w/api.php`;++const COMMONS = 'https://commons.wikimedia.org/w/api.php';++const WIKIDATA = 'https://www.wikidata.org/w/api.php';++const IMAGE_MIME = /^image\/(jpeg|png|webp|gif)$/;++// Interface pictures an article carries that are not about its subject.++const NOT_SUBJECT = /(^|[\s_-])(icon|symbol|flag of|logo of wikipedia|commons-logo|wiki letter|question book|edit-clear|ambox|padlock|crystal|nuvola|disambig|portal|stub|blank|placeholder)([\s_.-]|$)/i;++const META = 'LicenseShortName|LicenseUrl|Artist|ImageDescription|NonFree|UsageTerms|ObjectName';++++/** Wikimedia adds ?utm_source=… to file URLs; the picture is the same without it. */++const untracked = (u) => String(u || '').replace(/\?utm_[^#]*$/, '');++++const qs = (params) => Object.entries({ format: 'json', formatversion: '2', ...params }).map(([k, v]) => `${k}=${encodeURIComponent(v)}`).join('&');++++/** A file page with imageinfo (iiprop url|size|mime|extmetadata) → a research image, or null when unusable. */++function imageFrom(page, provider, fallbackSource, { allowSvg = false } = {}) {++ const ii = page && Array.isArray(page.imageinfo) ? page.imageinfo[0] : null;++ if (!ii) return null;++ // A drawing (SVG) only as its PNG rendering, and only when asked for (an article's lead picture, usually a logo).++ const svg = ii.mime === 'image/svg+xml' && allowSvg && /\.png(\?|$)/i.test(ii.thumburl || '');++ if (!svg && !IMAGE_MIME.test(ii.mime || '')) return null;++ const width = ii.width || 0;++ const height = ii.height || 0;++ if ((!svg && (width < 200 || height < 150)) || NOT_SUBJECT.test(page.title || '')) return null;++ const m = ii.extmetadata || {};++ const v = (k) => (m[k] && m[k].value != null ? String(m[k].value) : '');++ const license = stripTags(v('LicenseShortName')) || stripTags(v('UsageTerms')) || null;++ const nonFree = /^(true|1)$/i.test(v('NonFree').trim()) || /fair use|non-?free/i.test(license || '');++ return {++ url: untracked(ii.thumburl || ii.url),++ original: untracked(ii.url),++ width: ii.thumbwidth || width,++ height: ii.thumbheight || height,++ mime: svg ? 'image/png' : ii.mime,++ title: stripTags(v('ObjectName')) || String(page.title || '').replace(/^File:/, '').replace(/\.[a-z0-9]{2,5}$/i, '').replace(/_/g, ' '),++ description: clip(stripTags(v('ImageDescription')), 300) || null,++ author: clip(stripTags(v('Artist')), 160) || null,++ license: nonFree ? (license || 'Non-free (used for identification)') : license,++ license_url: v('LicenseUrl') || null,++ free: !nonFree && isFreeLicense(license),++ source_url: ii.descriptionurl || fallbackSource || null,++ provider,++ };++}++++// The statements worth an infobox, in the order shown.++const PROPS = [++ ['P31', 'Instance of'], ['P1476', 'Title'], ['P170', 'Created by'], ['P50', 'Author'], ['P57', 'Directed by'], ['P162', 'Produced by'],++ ['P86', 'Music by'], ['P175', 'Performer'], ['P161', 'Cast member'], ['P725', 'Voice actor'], ['P449', 'Original network'], ['P272', 'Production company'],++ ['P750', 'Distributed by'], ['P123', 'Publisher'], ['P178', 'Developer'], ['P400', 'Platform'], ['P136', 'Genre'], ['P495', 'Country of origin'],++ ['P364', 'Original language'], ['P571', 'Founded / created'], ['P577', 'Publication date'], ['P580', 'Start'], ['P582', 'End'], ['P1113', 'Episodes'],++ ['P2437', 'Seasons'], ['P569', 'Born'], ['P19', 'Place of birth'], ['P570', 'Died'], ['P27', 'Citizenship'], ['P106', 'Occupation'],++ ['P112', 'Founded by'], ['P159', 'Headquarters'], ['P452', 'Industry'], ['P17', 'Country'], ['P131', 'Located in'], ['P36', 'Capital'],++ ['P1082', 'Population'], ['P2046', 'Area'], ['P856', 'Official website'],++];++++function timeText(v) {++ const m = /^([+-])(\d{1,16})-(\d\d)-(\d\d)/.exec(String(v.time || ''));++ if (!m) return null;++ const year = `${m[1] === '-' ? '-' : ''}${Number(m[2])}`;++ if (v.precision >= 11 && m[3] !== '00' && m[4] !== '00') return `${year}-${m[3]}-${m[4]}`;++ if (v.precision === 10 && m[3] !== '00') return `${year}-${m[3]}`;++ return year;++}++++function createWikimedia({ http }) {++ async function searchWikipedia(query, { lang = 'en', limit = 5, signal } = {}) {++ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', list: 'search', srsearch: query, srlimit: String(limit), srprop: 'snippet', srenablerewrites: '1', maxlag: '5' })}`, { signal });++ return ((j.query && j.query.search) || []).map((s) => ({ title: s.title, snippet: stripTags(s.snippet) }));++ }++++ async function fullText(title, { lang, signal }) {++ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', prop: 'extracts', explaintext: '1', exsectionformat: 'wiki', redirects: '1', titles: title, maxlag: '5' })}`, { signal });++ const p = j.query && j.query.pages && j.query.pages[0];++ return p && p.extract ? extractText(p.extract) : '';++ }++++ /** Several articles in one call (intro, lead image, Wikidata id), with the full text of the first `full`. */++ async function articles(titles, { lang = 'en', full = 3, chars = 6000, signal } = {}) {++ if (!titles.length) return [];++ const j = await http.getJson(`${API(lang)}?${qs({++ action: 'query', prop: 'extracts|pageimages|pageprops|info|description', exintro: '1', explaintext: '1', exlimit: 'max',++ piprop: 'original|thumbnail', pithumbsize: '1280', ppprop: 'wikibase_item|disambiguation', inprop: 'url', redirects: '1',++ titles: titles.slice(0, 20).join('|'), maxlag: '5',++ })}`, { signal });++ const query = j.query || {};++ const rename = new Map();++ for (const r of [...(query.normalized || []), ...(query.redirects || [])]) rename.set(r.from, r.to);++ const byTitle = new Map((query.pages || []).filter((p) => !p.missing).map((p) => [p.title, p]));++ const ordered = [];++ const seen = new Set();++ for (const t of titles) {++ let name = t;++ for (let i = 0; i < 3 && rename.has(name); i++) name = rename.get(name);++ const p = byTitle.get(name);++ if (p && !seen.has(p.title)) { seen.add(p.title); ordered.push(p); }++ }++ const out = ordered.map((p) => ({++ title: p.title,++ url: p.fullurl || `https://${lang}.wikipedia.org/wiki/${encodeURIComponent(p.title.replace(/ /g, '_'))}`,++ description: p.description || null,++ text: p.extract || '',++ lead: p.original ? { url: (p.thumbnail && p.thumbnail.source) || p.original.source, original: p.original.source, width: (p.thumbnail && p.thumbnail.width) || p.original.width, height: (p.thumbnail && p.thumbnail.height) || p.original.height } : null,++ qid: (p.pageprops && p.pageprops.wikibase_item) || null,++ disambiguation: Boolean(p.pageprops && p.pageprops.disambiguation !== undefined),++ revisedAt: p.touched || null,++ }));++ const fulls = out.filter((a) => !a.disambiguation).slice(0, full);++ await Promise.all(fulls.map(async (a) => {++ try { const t = await fullText(a.title, { lang, signal }); if (t) a.text = t; } catch { /* the intro stays */ }++ }));++ for (const a of out) a.text = clip(a.text, chars);++ return out;++ }++++ function imageQuery(extra) {++ return { prop: 'imageinfo', iiprop: 'url|size|mime|extmetadata', iiurlwidth: '1280', iiextmetadatafilter: META, iiextmetadatalanguage: 'en', maxlag: '5', ...extra };++ }++++ async function articleImages(title, { lang = 'en', limit = 12, signal } = {}) {++ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', generator: 'images', titles: title, gimlimit: '40', redirects: '1', ...imageQuery() })}`, { signal });++ const source = `https://${lang}.wikipedia.org/wiki/${encodeURIComponent(title.replace(/ /g, '_'))}`;++ return ((j.query && j.query.pages) || []).map((p) => imageFrom(p, 'wikipedia', source)).filter(Boolean).slice(0, limit);++ }++++ async function commonsSearch(query, { limit = 10, signal } = {}) {++ const j = await http.getJson(`${COMMONS}?${qs({ action: 'query', generator: 'search', gsrsearch: `${query} filetype:bitmap`, gsrnamespace: '6', gsrlimit: String(limit), ...imageQuery() })}`, { signal });++ const pages = ((j.query && j.query.pages) || []).sort((a, b) => (a.index || 0) - (b.index || 0));++ return pages.map((p) => imageFrom(p, 'commons')).filter((i) => i && i.free);++ }++++ /** Licence and size for files by name on a Wikipedia (its own files and Commons files alike). */++ async function fileInfo(names, { lang = 'en', allowSvg = false, provider = 'wikipedia', signal } = {}) {++ if (!names.length) return [];++ const titles = names.slice(0, 20).map((n) => (/^File:/i.test(n) ? n : `File:${n}`)).join('|');++ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', titles, ...imageQuery() })}`, { signal });++ return ((j.query && j.query.pages) || []).map((p) => imageFrom(p, provider, null, { allowSvg })).filter(Boolean);++ }++++ async function commonsFiles(names, { signal } = {}) {++ if (!names.length) return [];++ const titles = names.slice(0, 20).map((n) => (/^File:/i.test(n) ? n : `File:${n}`)).join('|');++ const j = await http.getJson(`${COMMONS}?${qs({ action: 'query', titles, ...imageQuery() })}`, { signal });++ return ((j.query && j.query.pages) || []).map((p) => imageFrom(p, 'wikidata')).filter(Boolean);++ }++++ /** Labelled facts from a Wikidata item, plus its image file name (P18) when it has one. */++ async function wikidataFacts(qid, { lang = 'en', signal } = {}) {++ if (!/^Q\d{1,12}$/.test(qid || '')) return null;++ const j = await http.getJson(`${WIKIDATA}?action=wbgetentities&ids=${qid}&props=claims%7Clabels%7Cdescriptions&languages=${lang}%7Cen&format=json`, { signal });++ const e = j.entities && j.entities[qid];++ if (!e || !e.claims) return null;++ const pending = [];++ for (const [pid, label] of PROPS) {++ const claims = (e.claims[pid] || []).filter((c) => c.rank !== 'deprecated' && c.mainsnak && c.mainsnak.datavalue).slice(0, 8);++ if (!claims.length) continue;++ pending.push({ pid, label, values: claims.map((c) => c.mainsnak.datavalue) });++ }++ const ids = new Set();++ for (const f of pending) for (const dv of f.values) if (dv.type === 'wikibase-entityid' && dv.value && dv.value.id) ids.add(dv.value.id);++ const labels = new Map();++ const list = [...ids].slice(0, 50);++ if (list.length) {++ const lj = await http.getJson(`${WIKIDATA}?action=wbgetentities&ids=${list.join('%7C')}&props=labels&languages=${lang}%7Cen&format=json`, { signal });++ for (const [id, ent] of Object.entries(lj.entities || {})) {++ const l = ent.labels && (ent.labels[lang] || ent.labels.en);++ if (l) labels.set(id, l.value);++ }++ }++ const facts = [];++ for (const f of pending) {++ const vals = f.values.map((dv) => {++ switch (dv.type) {++ case 'wikibase-entityid': return labels.get(dv.value.id) || null;++ case 'time': return timeText(dv.value);++ case 'quantity': { const n = Number(dv.value.amount); return Number.isFinite(n) ? n.toLocaleString('en-US') : null; }++ case 'monolingualtext': return dv.value.text || null;++ case 'string': return typeof dv.value === 'string' ? dv.value : null;++ default: return null;++ }++ }).filter(Boolean);++ if (vals.length) facts.push({ label: f.label, value: [...new Set(vals)].join(', ').slice(0, 400) });++ }++ const image = e.claims.P18 && e.claims.P18[0] && e.claims.P18[0].mainsnak.datavalue ? String(e.claims.P18[0].mainsnak.datavalue.value) : null;++ const name = (e.labels && (e.labels[lang] || e.labels.en) || {}).value || qid;++ const description = (e.descriptions && (e.descriptions[lang] || e.descriptions.en) || {}).value || null;++ return { qid, name, description, url: `https://www.wikidata.org/wiki/${qid}`, facts: facts.slice(0, 30), image };++ }++++ return { searchWikipedia, articles, articleImages, fileInfo, commonsSearch, commonsFiles, wikidataFacts };++}++++module.exports = { createWikimedia, imageFrom, timeText, untracked };@@@ -12,6 +12,8 @@* embed embeddings* media_analysis local-first media analysis: FFmpeg signals, scenes, local speech-to-text, highlights and* an overview from a local model (or extractive; paid only with a budget). See media-analysis.js.++ * research MegaSource (server/research/): Wikipedia, the topic's Fandom wiki, Wikidata, DeepSeek web search,++ * Commons and Openverse, merged into numbered sources, licensed pictures and facts. No model call.** The final output is validated against the workflow's output schema. An empty or invalid answer* fails the run (output.empty / output.invalid); nothing is ever filled in. input.sources become@@@ -96,11 +98,13 @@ function normMessages(messages, user) {return out;}--function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spawnImpl, log = console }) {++function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spawnImpl, log = console, research = null }) {// One loopback reader for everything that streams media (server/media-proxy.js): transcripts and media.analyze.let proxy = null;const reader = () => proxy || (proxy = require('../media-proxy').createMediaProxy({ fetcher, log }));const mediaAnalysis = require('./media-analysis').createMediaAnalysis({ registry, pool, fetcher, quotas, config, spawnImpl, log, reader });++ let researcher = research;++ const megasource = () => researcher || (researcher = require('../research').createResearch({ config: config.research || {}, log }));async function routeFor(key) {const r = await registry.resolveRoute(key);if (!r) throw new AiError(503, 'route.unavailable', `no route ${key}`);@@@ -248,7 +252,29 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spareturn { output: { images: images.map(i => ({ b64: i.b64 || null, url: i.url || null })), count: images.length }, exec, route };}-- const STEP = { llm: llmStep, passthrough: passthroughStep, complete: completeStep, transcribe: transcribeStep, embed: embedStep, image: imageStep, media_analysis: mediaAnalysis.step };++ /** research: no model call. A developer app's run leaves the paid web search out (its DeepSeek calls are ours). */++ async function researchStep(step, input, wf, run, ctx) {++ let output;++ try {++ output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun });++ } catch (err) {++ if (err && err.status === 422) throw new AiError(422, 'input.invalid', err.message);++ throw err;++ }++ if (!output.sources.length && !output.images.length) throw new AiError(502, 'source.unavailable', `no source answered for "${output.query}"`, { providers: output.providers });++ // Recorded like a provider call: DeepSeek's per-call price counts against the cost quotas and the daily usage.++ const exec = { provider: 'research', model: 'megasource', usage: { input: 0, output: 0, cached: 0 }, cost: Number(output.usage && output.usage.cost_usd) || 0, synthetic: false, fallbackUsed: false, explain: null };++ return { output, exec };++ }++++ /** A research run's grounding: what it read is the output itself; the gaps are the sources that did not answer. */++ function researchGrounding(output) {++ const gaps = Object.entries((output && output.providers) || {}).filter(([, p]) => p && !p.ok)++ .map(([name, p]) => `${name}: ${p.skipped || p.error || 'did not answer'}`.slice(0, 300));++ return { cited: [], gaps: gaps.slice(0, 20), read: (output && output.sources ? output.sources.length : 0) };++ }++++ const STEP = { llm: llmStep, passthrough: passthroughStep, complete: completeStep, transcribe: transcribeStep, embed: embedStep, image: imageStep, media_analysis: mediaAnalysis.step, research: researchStep };/*** Run a workflow. ctx: { signal, logRequest, debugRaw, inputHash }@@@ -286,7 +312,7 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa}));return {output: last.output, citations, jsonSchema,-- grounding: groundingOf(last.output, sources.length, citeStep),++ grounding: wf.steps.some((s) => s.kind === 'research') ? researchGrounding(last.output) : groundingOf(last.output, sources.length, citeStep),synthetic: Boolean(exec && exec.synthetic),provider: exec ? exec.provider : null, model: exec ? exec.model : null, fallbackUsed: Boolean(exec && exec.fallbackUsed),usage, cost,@@@ -295,7 +321,7 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa};}-- return { execute, narrowCitations };++ return { execute, narrowCitations, research: () => megasource() };}module.exports = { createEngine, narrowCitations, collectCited, collectGaps, groundingOf };@@@ -271,6 +271,24 @@ const PREPARE = {sources(input) {return { vars: { ...input, sources_block: sourcesBlock(input.sources), source_count: (input.sources || []).length } };},++ // Wiki generation (OpenVibe.Wiki's "vibe a wiki"): the plan's page budget, and a page's link targets and pictures.++ 'wiki.space'(input) {++ return { vars: { ...input, max_pages: input.max_pages || 12, sources_block: sourcesBlock(input.sources), source_count: (input.sources || []).length } };++ },++ 'wiki.page'(input) {++ const pages = (Array.isArray(input.pages) ? input.pages : []).map((t) => clean(t, 200)).filter(Boolean).slice(0, 60);++ const images = (Array.isArray(input.images) ? input.images : []).slice(0, 16);++ return {++ vars: {++ ...input,++ topic: input.topic || input.space || input.title,++ pages_block: pages.map((t) => `- ${t}`).join('\n') || '(this is the only page)',++ images_block: images.map((im, i) => `[${i}] ${clean(im.title, 300)}${im.description ? ` — ${clean(im.description, 400)}` : ''}`).join('\n'),++ sources_block: sourcesBlock(input.sources),++ source_count: (input.sources || []).length,++ },++ };++ },'pics.describe_image'(input) {return { vars: { title: clean(input.title, 120) || '' } };},@@@ -17,6 +17,8 @@ const STR = (n) => ({ type: 'string', maxLength: n });const OBJ = (props, required) => ({ type: 'object', additionalProperties: false, required: required || Object.keys(props), properties: props });const ARR = (items, max = 20) => ({ type: 'array', items, maxItems: max });const CLAIM = OBJ({ text: STR(600), citations: CITES });++// What a wiki page is about (wiki.generate_space): the wiki's own page groups follow from it.++const PAGE_KINDS = ['overview', 'character', 'person', 'episode', 'season', 'work', 'place', 'group', 'object', 'concept', 'event', 'production', 'reception', 'list', 'other'];const RULES = 'You prepare a DRAFT evidence package for OpenVibe editors — never publication truth. Use ONLY the numbered sources provided. Every claim must cite the source numbers it rests on in its `citations` array (integers, e.g. [0, 2]). If the sources do not support something, leave it out and list it in `gaps`. Never invent facts, names, numbers, prices, ratings, dates or quotes. Output only JSON matching the schema.';@@@ -31,10 +33,10 @@ function productTemplate(key, name, task, route, extraUser = '') {}const templates = [-- productTemplate('wiki.generate_space', 'Wiki: propose a space', 'Propose a wiki space for the topic: a title, a one-paragraph summary and a page tree (page titles with short outlines), grounded in the sources.', 'wiki.generate',-- 'Topic: {{topic}}{{#audience}}\nAudience: {{audience}}{{/audience}}\n\n'),-- productTemplate('wiki.generate_page', 'Wiki: draft a page', 'Draft one wiki page: summary, sections and infobox values, each cited.', 'wiki.generate',-- 'Page title: {{title}}{{#space}}\nSpace: {{space}}{{/space}}{{#outline}}\nRequested outline: {{json outline}}{{/outline}}\n\n'),++ productTemplate('wiki.generate_space', 'Wiki: plan a wiki', 'Plan a wiki about the topic, written like an encyclopedia or a well-kept fan wiki, from the numbered sources. `title`: the wiki\'s name (usually the topic\'s name). `summary`: one paragraph saying what the topic is, cited. `pages`: the main page first, titled exactly as the topic is best known, with kind overview; then the subjects a reader would look up, as many as the sources say enough about, up to {{max_pages}} pages in all: the people or characters, works (episodes, seasons, films, books, games, albums), places, groups, objects, concepts, events, and its production and reception when the sources cover them. Each page: `title` (its usual name; add the topic in brackets only to tell two subjects apart), `kind`, `parent` (the title of another page in this list it belongs under, else an empty string; the main page has none), `outline` (three to six short section headings this page should have), `query` (a short web search phrase that finds this subject, such as "Ron Stoppable Kim Possible"), and `citations`. Only plan subjects the sources name; leave out anything they do not support.', 'wiki.write',++ 'Topic: {{topic}}{{#audience}}\nAudience: {{audience}}{{/audience}}\nPages at most: {{max_pages}}\n\n'),++ productTemplate('wiki.generate_page', 'Wiki: write a page', 'Write one page of the wiki from the numbered sources, encyclopedic and engaging, the way a well-kept wiki reads. `summary`: one or two plain sentences saying what the subject is. `lead`: the opening paragraph or two, in Markdown. `sections`: follow the requested outline where the sources support it (skip or merge headings they do not), each with a `heading` and a Markdown `body` of one or more full paragraphs (a list where it reads better), each citing its sources; about 500 to 1500 words in all when the sources allow, shorter when they do not. Link other pages of this wiki, and only titles from the list of pages, as [[Exact Title]] the first time each is mentioned. `infobox`: up to twelve key facts as `label` and `value` (for example First appearance, Voiced by, Created by, Release date, Location), each cited; the Wikidata facts are good material. `images`: up to three pictures from the numbered list of pictures that clearly show this page\'s subject, judged by their title and description (never a picture of something or someone else, and none when no picture fits), each with its `index`, a short factual `caption`, and `section`: the heading it illustrates, or lead for the page\'s main picture. Never write source numbers or "[n]" in the text. Never describe a real person\'s looks, body, clothing or age.', 'wiki.write',++ 'Page title: {{title}}\nWiki topic: {{topic}}{{#outline}}\nRequested outline: {{json outline}}{{/outline}}\n\nPages of this wiki (link only these, as [[Title]]):\n{{pages_block}}\n\n{{#images_block}}Pictures (by number):\n{{images_block}}\n\n{{/images_block}}'),productTemplate('blog.draft_post', 'Blog: draft a post', 'Draft a blog post in Markdown for the brief. Claims of fact must cite sources; opinion and framing need none. If there are no sources, write only what needs no facts and list the facts you would need in gaps.', 'blog.draft','Topic: {{topic}}{{#brief}}\nBrief: {{brief}}{{/brief}}{{#tone}}\nTone: {{tone}}{{/tone}}{{#audience}}\nAudience: {{audience}}{{/audience}}\n\n'),productTemplate('blog.write_digest', 'Blog: write a desk digest', 'Write one digest post for an OpenVibe.Blog desk from the numbered sources (the items its sources published in the period). Pick the items a reader of this desk would most want to know about, about twelve at most, and skip routine, minor or duplicate ones. Group them into two to five sections, most important first. Each section is a list of short paragraphs, one story per paragraph, two to four plain sentences saying what happened and why it matters to this desk\'s readers, exactly as the sources state it; each paragraph lists the numbers of the sources it rests on in its `citations`. Never write source numbers, "(Source n)" or brackets in the text itself. Neutral and readable: no opinion stated as fact, no hype, no advice, no predictions. Name prices, dates and numbers only as the sources give them. List in gaps only what a reader would miss (an item you could not make sense of, a missing date or price). title: a headline naming the one or two biggest stories, at most 90 characters, without the words digest, desk, roundup or snapshot and without a date. dek: one sentence about the period.', 'blog.draft',@@@ -67,16 +69,31 @@ const templates = [const workflows = [{-- key: 'wiki.generate_space', name: 'Propose a wiki space', namespace: 'wiki',-- input_schema: OBJ({ topic: STR(300), audience: STR(300), sources: SOURCES(1) }, ['topic', 'sources']),-- output_schema: OBJ({ title: STR(200), summary: STR(2000), pages: ARR(OBJ({ title: STR(200), slug_hint: STR(120), outline: ARR(STR(300), 20), citations: CITES }), 40), citations: CITES, gaps: GAPS }),-- default_route: 'wiki.generate',++ key: 'wiki.generate_space', name: 'Plan a wiki', namespace: 'wiki',++ input_schema: OBJ({ topic: STR(300), audience: STR(300), max_pages: { type: 'integer', minimum: 1, maximum: 40 }, sources: SOURCES(1) }, ['topic', 'sources']),++ output_schema: OBJ({++ title: STR(200), summary: STR(2000),++ pages: ARR(OBJ({ title: STR(200), kind: { enum: PAGE_KINDS }, parent: STR(200), outline: ARR(STR(120), 8), query: STR(160), citations: CITES }), 40),++ citations: CITES, gaps: GAPS,++ }),++ default_route: 'wiki.write',++ steps: [{ kind: 'llm', template: 'wiki.generate_space', operation: 'generate', output: 'json', prepare: 'wiki.space', cite: true, params: { max_tokens: 4000, temperature: 0.3, timeout_ms: 150000 } }],},{-- key: 'wiki.generate_page', name: 'Draft a wiki page', namespace: 'wiki',-- input_schema: OBJ({ title: STR(300), space: STR(300), outline: ARR(STR(300), 30), sources: SOURCES(1) }, ['title', 'sources']),-- output_schema: OBJ({ title: STR(300), summary: STR(2000), sections: ARR(OBJ({ heading: STR(200), body: STR(20000), citations: CITES }), 40), infobox: ARR(OBJ({ key: STR(100), value: STR(500), citations: CITES }), 30), citations: CITES, gaps: GAPS }),-- default_route: 'wiki.generate',++ key: 'wiki.generate_page', name: 'Write a wiki page', namespace: 'wiki',++ input_schema: OBJ({++ title: STR(300), topic: STR(300), space: STR(300), outline: ARR(STR(300), 30), pages: ARR(STR(200), 60),++ images: ARR(OBJ({ title: STR(300), description: STR(400) }, ['title']), 16), sources: SOURCES(1),++ }, ['title', 'sources']),++ output_schema: OBJ({++ title: STR(300), summary: STR(500), lead: STR(6000),++ sections: ARR(OBJ({ heading: STR(160), body: STR(12000), citations: CITES }), 14),++ infobox: ARR(OBJ({ label: STR(60), value: STR(300), citations: CITES }), 14),++ images: ARR(OBJ({ index: { type: 'integer', minimum: 0, maximum: 15 }, caption: STR(300), section: STR(160) }), 3),++ citations: CITES, gaps: GAPS,++ }),++ default_route: 'wiki.write',++ steps: [{ kind: 'llm', template: 'wiki.generate_page', operation: 'generate', output: 'json', prepare: 'wiki.page', cite: true, params: { max_tokens: 7000, temperature: 0.4, timeout_ms: 170000 } }],},{key: 'blog.draft_post', name: 'Draft a blog post', namespace: 'blog',@@@ -215,7 +232,7 @@ const workflows = [].map(w => ({cache_mode: 'private',...w,-- steps: [{ kind: 'llm', template: w.key, operation: w.key === 'moderation.classify' || w.key === 'rent.screen_listing' ? 'classify' : 'generate', output: 'json', prepare: 'sources', cite: true, params: { max_tokens: 2400, temperature: 0.3 } }],++ steps: w.steps || [{ kind: 'llm', template: w.key, operation: w.key === 'moderation.classify' || w.key === 'rent.screen_listing' ? 'classify' : 'generate', output: 'json', prepare: 'sources', cite: true, params: { max_tokens: 2400, temperature: 0.3 } }],metadata: { draft: true, seed: true },}));nnew file mode 100644@@@ -0,0 +1,56 @@++'use strict';++/**++ * MegaSource (server/research/): one research call over Wikipedia, the topic's Fandom wiki, Wikidata, DeepSeek web++ * search, Commons and Openverse. No model call and no template: the step reads, ranks and returns the evidence, and the++ * product workflows (wiki.generate_space, wiki.generate_page, …) cite it by number. Cached for a day per calling++ * service: the same question asked twice reads the web once.++ */++const { SOURCE } = require('./common-schemas');++++const STR = (n) => ({ type: 'string', maxLength: n });++const INT = (lo, hi) => ({ type: 'integer', minimum: lo, maximum: hi });++const NULLABLE = (n) => ({ type: ['string', 'null'], maxLength: n });++const PROVIDERS = ['wikipedia', 'fandom', 'wikidata', 'deepseek', 'commons', 'openverse'];++++const IMAGE = {++ type: 'object', additionalProperties: false, required: ['url', 'free', 'provider'],++ properties: {++ url: STR(2048), original: NULLABLE(2048), width: { type: ['integer', 'null'] }, height: { type: ['integer', 'null'] }, mime: NULLABLE(40),++ title: NULLABLE(300), description: NULLABLE(400), author: NULLABLE(200), license: NULLABLE(200), license_url: NULLABLE(2048),++ free: { type: 'boolean' }, source_url: NULLABLE(2048), provider: STR(80),++ },++};++++const workflows = [++ {++ key: 'research.gather', name: 'Research a topic (MegaSource)', namespace: 'research',++ description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, DeepSeek web search, Commons and Openverse.',++ input_schema: {++ type: 'object', additionalProperties: false, required: ['query'],++ properties: {++ query: STR(200), topic: STR(200), focus: STR(200), question: STR(200),++ lang: { type: 'string', pattern: '^[a-z]{2,3}$' },++ max_sources: INT(1, 20), max_images: INT(0, 24), images: { type: 'boolean' }, content_chars: INT(500, 12000),++ providers: { type: 'array', maxItems: PROVIDERS.length, items: { enum: PROVIDERS } },++ },++ },++ output_schema: {++ type: 'object', additionalProperties: false, required: ['query', 'sources', 'images', 'facts', 'providers', 'usage'],++ properties: {++ query: STR(200), topic: STR(200), focus: NULLABLE(200), lang: STR(3), retrieved_at: STR(40),++ sources: { type: 'array', maxItems: 20, items: SOURCE },++ images: { type: 'array', maxItems: 24, items: IMAGE },++ facts: { type: 'array', maxItems: 40, items: { type: 'object', additionalProperties: false, required: ['label', 'value'], properties: { label: STR(80), value: STR(400) } } },++ topic_wiki: { type: ['object', 'null'] },++ providers: { type: 'object' },++ usage: { type: 'object' },++ },++ },++ steps: [{ kind: 'research' }],++ default_route: null,++ cache_mode: 'service', cache_ttl_sec: 86400,++ metadata: { seed: true },++ },++];++++module.exports = { templates: [], workflows, PROVIDERS };@@@ -20,6 +20,7 @@ const media = require('./media');const core = require('./core');const products = require('./products');const chat = require('./chat');++const research = require('./research');const { resolveSecret } = require('../util');const ROLE_TIMEOUT = { chat: 20000, vision: 30000, director: 25000, summary: 30000, legacy: 30000 };@@@ -155,15 +156,18 @@ async function seed({ registry, quotas, config, env = process.env, db }) {// Space's spaces (server/workflows/space.js): the builder writes whole apps, so it gets the creative model and room to// finish; a space's own AI is conversation text, an alias of the chat pool.await registry.seedVersioned('route', 'space.build', pool(180000, { max_output_tokens: 16000, options: { outlast_provider_timeout: true }, ...(creativeModel ? { constraints: { objective: 'correctness', latency_class: 'interactive', model: creativeModel, authority: `shared:${creativeModel}` } } : {}) }));++ // Wiki generation (wiki.generate_space, wiki.generate_page): long cited pages people read, so the creative model, room++ // to write (a page is several thousand tokens) and time to finish past the provider's chat timeout.++ await registry.seedVersioned('route', 'wiki.write', pool(170000, { max_output_tokens: 7000, response_format: 'json', options: { outlast_provider_timeout: true, temperature: 0.4 }, ...(creativeModel ? { constraints: { objective: 'correctness', latency_class: 'interactive', model: creativeModel, authority: `shared:${creativeModel}` } } : {}) }));await registry.seedVersioned('route', 'space.ask', { primary: { provider: 'shared', model: null }, fallbacks: [], options: {}, max_output_tokens: 800, response_format: 'text', timeout_ms: null, alias_of: 'default.chat' });await registry.seedVersioned('route', 'chat.member', { primary: { provider: 'shared', model: null }, fallbacks: [], options: {}, max_output_tokens: 600, response_format: 'text', timeout_ms: null, alias_of: 'default.chat' });// Templates, then workflows (workflows reference templates)-- for (const t of [...live.templates, ...core.templates, ...products.templates, ...media.templates, ...chat.templates, ...space.templates]) {++ for (const t of [...live.templates, ...core.templates, ...products.templates, ...media.templates, ...chat.templates, ...space.templates, ...research.templates]) {const { key, ...def } = t;await registry.seedVersioned('template', key, { description: null, default_route: null, owner: 'ai', visibility: 'internal', metadata: {}, ...def });}-- for (const w of [...live.workflows, ...core.workflows, ...products.workflows, ...media.workflows, ...chat.workflows, ...space.workflows]) {++ for (const w of [...live.workflows, ...core.workflows, ...products.workflows, ...media.workflows, ...chat.workflows, ...space.workflows, ...research.workflows]) {const { key, ...def } = w;await registry.seedVersioned('workflow', key, { description: null, default_route: null, cache_ttl_sec: null, metadata: {}, ...def });}nnew file mode 100644@@@ -0,0 +1,64 @@++'use strict';++/**++ * A canned web for MegaSource tests (no network): Wikipedia, Commons, Wikidata, Fandom, Openverse and DeepSeek answer++ * from these fixtures by URL. `calls` records every request; `deepseek` scripts DeepSeek's status.++ */++const json = (obj, status = 200) => new Response(JSON.stringify(obj), { status, headers: { 'content-type': 'application/json' } });++const meta = (license, extra = {}) => ({ LicenseShortName: { value: license }, Artist: { value: '<a href="x">Jane Doe</a>' }, ImageDescription: { value: 'The <b>SQLite</b> logo' }, ...extra });++const file = (title, mime, w, h, license, extra) => ({ title, imageinfo: [{ url: `https://upload.wikimedia.org/wikipedia/commons/a/ab/${encodeURIComponent(title.slice(5))}?utm_source=x`, thumburl: `https://upload.wikimedia.org/thumb/${encodeURIComponent(title.slice(5))}.png`, thumbwidth: Math.min(w, 1280), thumbheight: h, width: w, height: h, mime, descriptionurl: `https://commons.wikimedia.org/wiki/${encodeURIComponent(title)}`, extmetadata: meta(license, extra) }] });++++function createResearchFetch({ deepseek = { status: 200 }, fandom = false } = {}) {++ const calls = [];++ async function fetchImpl(url, init = {}) {++ const u = new URL(url);++ const p = u.searchParams;++ calls.push({ host: u.hostname, path: u.pathname, params: Object.fromEntries(p), method: init.method || 'GET', body: init.body || null });++ if (u.hostname.endsWith('.wikipedia.org')) {++ if (p.get('list') === 'search') return json({ query: { search: [{ title: 'SQLite', snippet: 'an embedded <span>database</span>' }, { title: 'SQL', snippet: 'a language' }, { title: 'SQLite (disambiguation)', snippet: '' }] } });++ if (p.get('generator') === 'images') return json({ query: { pages: [file('File:SQLite diagram.png', 'image/png', 1600, 900, 'CC BY-SA 4.0'), file('File:SQLite promo.jpg', 'image/jpeg', 800, 600, 'Fair use', { NonFree: { value: 'true' } }), file('File:Symbol category class.svg', 'image/svg+xml', 180, 180, 'Public domain')] } });++ if (p.get('prop') === 'extracts') return json({ query: { pages: [{ title: p.get('titles'), extract: 'SQLite is a database engine written in C.\n\n== History ==\nD. Richard Hipp designed it in 2000.\n\n== References ==\nA list of links.' }] } });++ if (p.get('prop') === 'imageinfo') return json({ query: { pages: [file(`File:${p.get('titles').replace(/^File:/, '')}`, 'image/svg+xml', 400, 200, 'Public domain')] } });++ return json({ query: {++ normalized: [{ from: 'SQLite (disambiguation)', to: 'SQLite (disambiguation)' }],++ pages: [++ { title: 'SQL', fullurl: 'https://en.wikipedia.org/wiki/SQL', extract: 'SQL is a query language.', description: 'Query language', pageprops: { wikibase_item: 'Q47607' } },++ { title: 'SQLite', fullurl: 'https://en.wikipedia.org/wiki/SQLite', extract: 'SQLite is a database engine.', description: 'Embedded database', pageprops: { wikibase_item: 'Q319417' }, original: { source: 'https://upload.wikimedia.org/wikipedia/commons/3/38/SQLite370.svg', width: 400, height: 200 }, thumbnail: { source: 'https://upload.wikimedia.org/thumb/SQLite370.svg.png', width: 400, height: 200 } },++ { title: 'SQLite (disambiguation)', fullurl: 'https://en.wikipedia.org/wiki/SQLite_(disambiguation)', extract: 'may refer to', pageprops: { disambiguation: '' } },++ ],++ } });++ }++ if (u.hostname === 'commons.wikimedia.org') {++ if (p.get('generator') === 'search') return json({ query: { pages: [{ index: 2, ...file('File:Whistle stop tour.jpg', 'image/jpeg', 1200, 800, 'CC BY 2.0', { ImageDescription: { value: 'A whistle stop campaign tour' } }) }, { index: 1, ...file('File:SQLite logo photo.jpg', 'image/jpeg', 1200, 800, 'CC0') }, { index: 3, ...file('File:SQLite poster.jpg', 'image/jpeg', 1200, 800, 'CC BY-NC 2.0') }] } });++ return json({ query: { pages: [file('File:SQLite at work.jpg', 'image/jpeg', 2000, 1500, 'CC BY 4.0')] } });++ }++ if (u.hostname === 'www.wikidata.org') {++ if (p.get('ids') === 'Q319417' && /claims/.test(p.get('props'))) return json({ entities: { Q319417: { labels: { en: { value: 'SQLite' } }, descriptions: { en: { value: 'relational database engine' } }, claims: {++ P178: [{ rank: 'normal', mainsnak: { datavalue: { type: 'wikibase-entityid', value: { id: 'Q92743' } } } }],++ P571: [{ rank: 'normal', mainsnak: { datavalue: { type: 'time', value: { time: '+2000-08-17T00:00:00Z', precision: 11 } } } }],++ P18: [{ rank: 'normal', mainsnak: { datavalue: { type: 'string', value: 'SQLite at work.jpg' } } }],++ } } } });++ return json({ entities: { Q92743: { labels: { en: { value: 'D. Richard Hipp' } } } } });++ }++ if (u.hostname.endsWith('.fandom.com')) {++ if (!fandom) return json({ error: 'no wiki' }, 404);++ if (p.get('meta') === 'siteinfo') return json({ query: { general: { sitename: 'SQLite Wiki', lang: 'en' } } });++ if (p.get('list') === 'search') return json({ query: { search: [{ title: 'WAL mode' }, { title: 'WAL mode/Gallery' }] } });++ if (p.get('action') === 'parse') return json({ parse: { title: p.get('page'), text: '<aside class="portable-infobox"><h3 class="pi-data-label">Introduced</h3><div class="pi-data-value">2010</div></aside><p>Write-ahead logging keeps readers going.</p><h2>Use</h2><ul><li>Set the journal mode.</li></ul><h2>References</h2><ol><li>link</li></ol>' } });++ return json({ query: { pages: [{ title: p.get('titles'), original: { source: 'https://static.wikia.nocookie.net/sqlite/images/a/ab/Wal.png/revision/latest?cb=1', width: 600, height: 400 } }] } });++ }++ if (u.hostname === 'api.openverse.org') return json({ results: [{ url: 'https://live.staticflickr.com/1/sqlite.jpg', title: 'SQLite sticker', width: 1024, height: 768, license: 'by', license_version: '2.0', creator: 'Sam', foreign_landing_url: 'https://www.flickr.com/photos/1', source: 'flickr' }] });++ if (u.hostname === 'api.deepseek.com') {++ if (deepseek.status !== 200) return json({ error: { message: 'no' } }, deepseek.status);++ if (u.pathname === '/v1/tools/search') return json({ results: [++ { title: 'SQLite Home Page', url: 'https://sqlite.org/index.html?utm_source=a', snippet: 'Small. Fast. Reliable.', summary: 'SQLite is a C library.', content: 'SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured SQL database engine. '.repeat(6), site_name: 'SQLite', date: '2026-01-01T00:00:00Z', authority: 4, rerank_score: 2.8 },++ { title: 'SQLite — Wikipedia mirror', url: 'https://en.m.wikipedia.org/wiki/SQLite', snippet: 'dup', summary: 'dup', content: 'duplicate of the encyclopedia page', authority: 3, rerank_score: 2.0 },++ { title: 'A thin page', url: 'https://example.com/thin', snippet: 'short', summary: '', content: '', authority: 1, rerank_score: 1.2 },++ ] });++ if (u.pathname === '/v1/tools/fetch') return json({ title: 'A thin page', content: 'The full text DeepSeek fetched for the thin page.' });++ }++ return json({ error: 'not in the fixture web' }, 404);++ }++ return { fetchImpl, calls };++}++++module.exports = { createResearchFetch };@@@ -45,7 +45,7 @@ function tmpDir() {// A boot on a dir used before is a restart on the same database: its database stays open.const restartable = new Map();--async function boot({ env = {}, dir, clock, fetchImpl, credentialFetch } = {}) {++async function boot({ env = {}, dir, clock, fetchImpl, credentialFetch, research } = {}) {const reuse = dir !== undefined;if (dir === undefined) dir = tmpDir();const fullEnv = {@@@ -57,7 +57,9 @@ async function boot({ env = {}, dir, clock, fetchImpl, credentialFetch } = {}) {// One database per boot (PGlite, or AI_TEST_STORE=pg: the containers), dropped when the boot stops.const testdb = (reuse && restartable.get(dir)) || await require('./db').testDb();if (reuse) restartable.set(dir, testdb);-- const h = await start({ config, db: testdb.db, env: fullEnv, log: silent, clock, fetchImpl, credentialFetch });++ // Research reads a canned web (test/fixtures/research-fetch.js) unless a test brings its own: no test reaches the internet.++ const researcher = research || require('../server/research').createResearch({ config: config.research, env: {}, fetchImpl: require('./fixtures/research-fetch').createResearchFetch().fetchImpl, log: silent });++ const h = await start({ config, db: testdb.db, env: fullEnv, log: silent, clock, fetchImpl, credentialFetch, research: researcher });const base = `http://127.0.0.1:${h.server.address().port}`;return { ...h, base, dir, env: fullEnv, testdb, stop: async () => { await h.close(); if (!reuse) await testdb.close(); } };}nnew file mode 100644@@@ -0,0 +1,141 @@++'use strict';++// MegaSource (server/research/): every provider merged into one ranked, de-duplicated pack against a canned web; pictures++// carry their licences and library results must name the subject; DeepSeek is used only with a key, only for++// first-party callers, and stands down on 402; the research.gather workflow and POST /api/v1/research answer.++++const assert = require('assert');++const { createResearch } = require('../server/research');++const { articleText, extractText, canonicalUrl, isFreeLicense, clip } = require('../server/research/text');++const { slugsFor } = require('../server/research/fandom');++const { createResearchFetch } = require('./fixtures/research-fetch');++const { boot, request, token, suite } = require('./helpers');++++const t = suite('research');++const quiet = { log() {}, warn() {}, error() {} };++++t.test('text helpers: article HTML, extracts, URLs, licences', () => {++ const text = articleText('<aside class="portable-infobox"><h3 class="pi-data-label">Voiced by</h3><div class="pi-data-value">Will Friedle<br>someone</div></aside><p>Ron is <b>loyal</b>.<sup class="reference">[1]</sup></p><h2>Personality</h2><ul><li>Brave</li></ul><h2>References</h2><ol><li>x</li></ol><script>evil()</script>');++ assert.match(text, /^Voiced by: Will Friedle, someone$/m);++ assert.match(text, /Ron is loyal\./);++ assert.match(text, /^## Personality$/m);++ assert.match(text, /^- Brave$/m);++ assert.ok(!/References|evil|\[1\]/.test(text), 'references, scripts and markers are gone');++ assert.strictEqual(extractText('Intro.\n\n== History ==\nOld.\n\n== References ==\nlinks'), 'Intro.\n\n## History\nOld.');++ assert.strictEqual(canonicalUrl('http://en.m.wikipedia.org/wiki/SQLite?utm_source=x#top'), 'https://en.wikipedia.org/wiki/SQLite');++ assert.strictEqual(canonicalUrl('javascript:alert(1)'), null);++ for (const free of ['CC BY-SA 4.0', 'CC0', 'Public domain', 'CC BY 2.0', 'GFDL']) assert.ok(isFreeLicense(free), free);++ for (const not of ['CC BY-NC 2.0', 'CC BY-ND 4.0', 'Fair use', '', null, 'All rights reserved']) assert.ok(!isFreeLicense(not), String(not));++ assert.ok(clip('One. Two. Three. Four.', 12).endsWith('.'));++ assert.deepStrictEqual(slugsFor('The Legend of Zelda'), ['thelegendofzelda', 'legendzelda', 'legend']);++ assert.deepStrictEqual(slugsFor('Kim Possible'), ['kimpossible']);++});++++t.test('gather: free sources only without a key, ranked and de-duplicated, pictures with licences', async () => {++ const web = createResearchFetch();++ const r = createResearch({ config: {}, env: {}, fetchImpl: web.fetchImpl, log: quiet });++ const out = await r.gather({ query: 'SQLite', max_images: 10 });++ assert.strictEqual(out.providers.deepseek.skipped, 'not configured');++ assert.ok(!web.calls.some((c) => c.host === 'api.deepseek.com'), 'no paid call without a key');++ assert.strictEqual(out.sources[0].url, 'https://en.wikipedia.org/wiki/SQLite', 'the exact article leads');++ assert.match(out.sources[0].content, /## History/);++ assert.ok(!/References/.test(out.sources[0].content));++ assert.ok(!out.sources.some((s) => /disambiguation/.test(s.url)), 'disambiguation pages are left out');++ const wd = out.sources.find((s) => s.source_type === 'wikidata');++ assert.match(wd.content, /Developer: D\. Richard Hipp/);++ assert.match(wd.content, /Founded \/ created: 2000-08-17/);++ assert.deepStrictEqual(out.facts.map((f) => f.label), ['Developer', 'Founded / created']);++ for (const s of out.sources) { assert.ok(s.retrieved_at && s.url && s.title); assert.ok(s.provenance.provider); }++ const byTitle = Object.fromEntries(out.images.map((i) => [i.title, i]));++ assert.strictEqual(out.images[0].provider, 'wikidata', 'the item\'s own picture first');++ assert.strictEqual(byTitle['SQLite diagram'].free, true);++ assert.strictEqual(byTitle['SQLite diagram'].license, 'CC BY-SA 4.0');++ assert.strictEqual(byTitle['SQLite diagram'].author, 'Jane Doe', 'credit is plain text');++ assert.ok(!/utm_/.test(byTitle['SQLite diagram'].original), 'tracking query removed');++ assert.strictEqual(byTitle['SQLite promo'].free, false, 'an article\'s non-free picture is marked');++ assert.ok(!out.images.some((i) => /Symbol category/.test(i.title)), 'interface icons are left out');++ assert.ok(byTitle['SQLite logo photo'], 'a library picture naming the subject is kept');++ assert.ok(!out.images.some((i) => /Whistle/.test(i.title)), 'a library picture of something else is left out');++ assert.ok(!out.images.some((i) => /poster/.test(i.title)), 'NonCommercial licences are not free');++ assert.ok(out.images.some((i) => i.provider === 'openverse/flickr'));++ assert.strictEqual(out.topic_wiki, null);++ assert.strictEqual(out.usage.cost_usd, 0);++});++++t.test('gather: the topic\'s Fandom wiki ranks first for a focus page; DeepSeek joins with a key', async () => {++ const web = createResearchFetch({ fandom: true });++ const r = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, log: quiet });++ const out = await r.gather({ query: 'WAL mode SQLite', topic: 'SQLite', focus: 'WAL mode', max_images: 6 });++ assert.strictEqual(out.sources[0].source_type, 'fandom');++ assert.strictEqual(out.sources[0].url, 'https://sqlite.fandom.com/wiki/WAL_mode');++ assert.match(out.sources[0].content, /^Introduced: 2010$/m);++ assert.deepStrictEqual(out.topic_wiki, { name: 'SQLite Wiki', url: 'https://sqlite.fandom.com' });++ assert.ok(!web.calls.some((c) => c.params.page === 'WAL mode/Gallery'), 'gallery subpages are not read');++ const urls = out.sources.map((s) => s.url);++ assert.ok(urls.includes('https://sqlite.org/index.html?utm_source=a'), 'a web result');++ assert.ok(!urls.includes('https://en.m.wikipedia.org/wiki/SQLite'), 'a mirror of a page already read is dropped');++ const thin = out.sources.find((s) => s.url === 'https://example.com/thin');++ assert.match(thin.content, /full text DeepSeek fetched/, 'a thin result is read in full through Fetch');++ assert.strictEqual(out.usage.deepseek_search, 1);++ assert.strictEqual(out.usage.deepseek_fetch, 1);++ assert.strictEqual(out.usage.cost_usd, 0.0015);++ const ds = web.calls.find((c) => c.path === '/v1/tools/search');++ assert.strictEqual(JSON.parse(ds.body).need_content, true);++ const fan = out.images.find((i) => i.provider === 'fandom');++ assert.strictEqual(fan.free, false);++ assert.match(fan.url, /scale-to-width-down\/1024/);++ // A developer app's run reads only the free sources.++ const before = web.calls.length;++ const app = await r.gather({ query: 'SQLite' }, { allowPaid: false });++ assert.strictEqual(app.providers.deepseek.skipped, 'first-party callers only');++ assert.ok(!web.calls.slice(before).some((c) => c.host === 'api.deepseek.com'));++});++++t.test('gather: DeepSeek stands down on 402 and the free sources still answer', async () => {++ const web = createResearchFetch({ deepseek: { status: 402 } });++ const r = createResearch({ config: { deepseekKey: 'sk-test' }, env: {}, fetchImpl: web.fetchImpl, log: quiet });++ const one = await r.gather({ query: 'SQLite' });++ assert.strictEqual(one.providers.deepseek.ok, false);++ assert.match(one.providers.deepseek.error, /balance/);++ assert.ok(one.sources.length >= 2);++ const two = await r.gather({ query: 'SQLite 2' });++ assert.strictEqual(two.providers.deepseek.skipped, 'balance used up', 'no second paid call while standing down');++ assert.strictEqual(web.calls.filter((c) => c.host === 'api.deepseek.com').length, 1);++ assert.strictEqual(r.status().deepseek.reason, 'balance used up');++});++++t.test('gather: the key comes from the environment by name, and a daily cap', async () => {++ const web = createResearchFetch();++ const r = createResearch({ config: { deepseekKeyRef: 'env:MY_DS', deepseekDaily: 1 }, env: { MY_DS: 'sk-env' }, fetchImpl: web.fetchImpl, log: quiet });++ await r.gather({ query: 'SQLite' });++ const auth = web.calls.find((c) => c.host === 'api.deepseek.com');++ assert.ok(auth, 'the key resolved');++ const again = await r.gather({ query: 'SQLite again' });++ assert.strictEqual(again.providers.deepseek.skipped, 'daily cap reached');++ const off = createResearch({ config: { deepseek: false, deepseekKeyRef: 'env:MY_DS' }, env: { MY_DS: 'sk-env' }, fetchImpl: web.fetchImpl, log: quiet });++ assert.strictEqual(off.status().deepseek.configured, false);++ await assert.rejects(r.gather({ query: ' ' }), /needs a query/);++});++++t.test('research.gather runs as a workflow and through POST /api/v1/research', async () => {++ const h = await boot();++ try {++ const tok = token('wiki', ['ai.run.create'], { ns: ['research.*'] });++ const res = await request(h.base, 'POST', '/api/v1/research?wait=10000', { tok, body: { query: 'SQLite', max_images: 3 } });++ assert.strictEqual(res.status, 201, JSON.stringify(res.body));++ const run = res.body.run;++ assert.strictEqual(run.status, 'succeeded');++ assert.strictEqual(run.provenance.workflow, 'research.gather');++ assert.ok(run.output.sources.length >= 2);++ assert.ok(run.output.images.length <= 3);++ // Cached per calling service for a day: the second ask reads nothing new.++ const again = await request(h.base, 'POST', '/api/v1/research?wait=10000', { tok, body: { query: 'SQLite', max_images: 3 } });++ assert.ok([200, 201].includes(again.status));++ assert.strictEqual(again.body.run.status, 'cached');++ const denied = await request(h.base, 'POST', '/api/v1/research', { tok: token('games', ['ai.run.create'], { ns: ['games.*'] }), body: { query: 'SQLite' } });++ assert.strictEqual(denied.status, 403);++ const bad = await request(h.base, 'POST', '/api/v1/research', { tok, body: { query: 'x', providers: ['google'] } });++ assert.strictEqual(bad.status, 422);++ } finally { await h.stop(); }++});++++t.run();@@@ -34,8 +34,9 @@ const EXAMPLES = {'ai.enrich': { record: { name: 'Widget' }, instructions: 'normalise the name' },'ai.embed': { input: ['one', 'two'] },'ai.image': { prompt: 'A goose eating croutons, watercolour.', size: '1024x1024', n: 1 },-- 'wiki.generate_space': { topic: 'SQLite', sources: SRC(2) },-- 'wiki.generate_page': { title: 'WAL mode', sources: SRC(3) },++ 'wiki.generate_space': { topic: 'SQLite', max_pages: 6, sources: SRC(2) },++ 'wiki.generate_page': { title: 'WAL mode', topic: 'SQLite', pages: ['SQLite', 'WAL mode', 'Journal'], images: [{ title: 'WAL diagram', description: 'How WAL writes' }], sources: SRC(3) },++ 'research.gather': { query: 'SQLite', max_images: 4 },'blog.draft_post': { topic: 'Why we self-host', sources: SRC(1) },'blog.write_digest': { desk: 'News digest', about: 'Government and space agency announcements', period: '14 October 2026', sources: SRC(3) },'blog.write_satire': { topic: 'Is it safe to let your air fryer vote?', angle: 'a consumer safety explainer' },@@@ -136,11 +137,14 @@ t.test('every workflow runs end to end on the stub and returns schema-valid outpassert.strictEqual(r.status, 201, `${w.key}: ${r.status} ${r.text.slice(0, 300)}`);const run = r.body.run;assert.strictEqual(run.status, 'succeeded', `${w.key}: ${JSON.stringify(run.error)}`);-- assert.strictEqual(run.synthetic, true, `${w.key} should be marked synthetic`);++ // Research calls no model: it reads (here, the canned web of test/fixtures/research-fetch.js) and is never synthetic.++ const reads = w.steps.some((s) => s.kind === 'research');++ assert.strictEqual(run.synthetic, !reads, `${w.key} should ${reads ? 'not ' : ''}be marked synthetic`);assert.strictEqual(run.provenance.origin, 'ai');assert.strictEqual(run.provenance.workflow, w.key);assert.ok(run.provenance.model, `${w.key} has a model`);assert.ok(schemas.validate(w.output_schema, run.output).valid, `${w.key} output invalid`);++ if (reads) { assert.ok(run.output.sources.length && Array.isArray(run.grounding.gaps)); continue; }// WS-O task 3: every output says what backs it: cited sources, or explicit gaps.assert.ok(run.grounding && Array.isArray(run.grounding.cited) && Array.isArray(run.grounding.gaps), `${w.key} has grounding`);assert.ok(run.grounding.cited.length || run.grounding.gaps.length, `${w.key}: citations or explicit gaps`);
@@ -171,3 +171,10 @@ OV_MEDIA_URL=https://openvibe.media
@@ -171,3 +171,10 @@ OV_MEDIA_URL=https://openvibe.media
#AI_FETCH_MAX_BYTES=536870912
#AI_FETCH_MAX_BYTES=536870912
#AI_FETCH_MAX_IMAGE_BYTES=12582912
#AI_FETCH_MAX_IMAGE_BYTES=12582912
#AI_FETCH_TIMEOUT_MS=120000
#AI_FETCH_TIMEOUT_MS=120000
+
+# MegaSource research (server/research/, research.gather): DeepSeek's paid web search joins the free sources when its key
+# resolves. The key is read by name; set DEEPSEEK_API_KEY (or point AI_RESEARCH_DEEPSEEK_KEY_REF elsewhere).
+#AI_RESEARCH_DEEPSEEK=1
+#AI_RESEARCH_DEEPSEEK_KEY_REF=env:DEEPSEEK_API_KEY
+#AI_RESEARCH_DEEPSEEK_DAILY=1500
+#AI_RESEARCH_TIMEOUT_MS=15000
@@ -100,6 +100,7 @@ idempotent replay):
@@ -100,6 +100,7 @@ idempotent replay):
|---|---|
|---|---|
| `POST /api/v1/runs` (`?wait=ms`, `Idempotency-Key`), `POST /runs/:id/cancel`, `POST /runs/:id/retry`, `POST /runs/:id/citations` | `ai.run.create` |
| `POST /api/v1/runs` (`?wait=ms`, `Idempotency-Key`), `POST /runs/:id/cancel`, `POST /runs/:id/retry`, `POST /runs/:id/citations` | `ai.run.create` |
| `POST /api/v1/{chat,generate,summarize,classify,extract,enrich,embed,image}` — each a run of workflow `ai.<op>` | `ai.run.create` |
| `POST /api/v1/{chat,generate,summarize,classify,extract,enrich,embed,image}` — each a run of workflow `ai.<op>` | `ai.run.create` |
+| `POST /api/v1/research` — a run of `research.gather` ([MegaSource](#megasource-research)) | `ai.run.create` (namespace `research.*`) or `ai.app.run` (free sources only) |
| `POST /api/v1/{chat,generate,summarize,classify,extract,embed,image}` (no `enrich`), `GET /runs/:id` (the app's own runs) | `ai.app.run` for a developer-app token ([Developer apps](#developer-apps)) |
| `POST /api/v1/{chat,generate,summarize,classify,extract,embed,image}` (no `enrich`), `GET /runs/:id` (the app's own runs) | `ai.app.run` for a developer-app token ([Developer apps](#developer-apps)) |
| `GET /api/v1/runs`, `GET /runs/:id` (+ citations, request-log metadata), `GET /runs/:id/citations` | `ai.run.read` (own runs) |
| `GET /api/v1/runs`, `GET /runs/:id` (+ citations, request-log metadata), `GET /runs/:id/citations` | `ai.run.read` (own runs) |
| `GET /api/v1/workflows\|templates\|routes[/:key]` | `ai.run.create`, `ai.workflow.manage` or `ai.usage.read` |
| `GET /api/v1/workflows\|templates\|routes[/:key]` | `ai.run.create`, `ai.workflow.manage` or `ai.usage.read` |
@@ -374,7 +375,7 @@ the service boots without Valkey; with it on, production refuses to boot without
@@ -374,7 +375,7 @@ the service boots without Valkey; with it on, production refuses to boot without
## Workflows
## Workflows
-- **Seed product workflows** — `wiki.generate_space`, `wiki.generate_page`, `blog.draft_post`,
+- **Seed product workflows** — `wiki.generate_space`, `wiki.generate_page` (both on route `wiki.write`, see [MegaSource](#megasource-research)), `blog.draft_post`,
`news.summarize_story`, `news.compare_perspectives`, `reviews.summarize_entity`,
`news.summarize_story`, `news.compare_perspectives`, `reviews.summarize_entity`,
`deals.enrich_deal`, `coupons.extract_coupon`, `trade.summarize_market_context`,
`deals.enrich_deal`, `coupons.extract_coupon`, `trade.summarize_market_context`,
`codes.generate_docs`, `games.generate_lore`, `moderation.classify`, `food.suggest_recipes`,
`codes.generate_docs`, `games.generate_lore`, `moderation.classify`, `food.suggest_recipes`,
@@ -432,6 +433,38 @@ the service boots without Valkey; with it on, production refuses to boot without
@@ -432,6 +433,38 @@ the service boots without Valkey; with it on, production refuses to boot without
`media.analyze`.
`media.analyze`.
- **Direct operations** — `ai.chat|generate|summarize|classify|extract|enrich|embed|image`.
- **Direct operations** — `ai.chat|generate|summarize|classify|extract|enrich|embed|image`.
+## MegaSource research
+
+`research.gather` (namespace `research`, `server/research/`, `server/workflows/research.js`) answers one question from
+many sources at once and returns one ranked, de-duplicated evidence pack the product workflows cite by number. No model
+is called; a run is recorded like a provider call (provider `research`, model `megasource`), so DeepSeek's per-call
+price counts against the cost quotas and the daily usage.
+
+| Source | What it gives | Terms |
+|---|---|---|
+| Wikipedia | the best articles' full text, their pictures (each with its licence; an article's own non-free files are marked) and the Wikidata id | CC BY-SA 4.0 |
+| Fandom | the topic's own community wiki when it has one (from the web results' hosts, else the topic as a subdomain, confirmed by the wiki's name): pages read through its MediaWiki API, infobox rows kept as "Label: value" | CC BY-SA 3.0; pictures marked not free |
+| Wikidata | the top article's key statements as labelled facts, and its picture | CC0 |
+| DeepSeek | web search with page text (Search API) and full text for short results (Fetch API), fetched by DeepSeek, never from this host | $0.75 per 1,000 successful calls each |
+| Commons, Openverse | freely licensed pictures (CC0, public domain, CC BY, CC BY-SA; NonCommercial and NoDerivatives are not counted as free), kept only when their title or description names the subject | per picture |
+
+Input: `query` (required), `topic` (the subject it belongs to: finds the topic's wiki), `focus` (one subject inside the
+topic), `lang`, `max_sources` (≤ 20), `max_images` (≤ 24), `content_chars` (≤ 12,000), `providers` (a subset). Output:
+`sources` (each with `url`, `title`, `content`, `retrieved_at` and `provenance.provider|site|license`), `images` (each with
+`license`, `free`, `author`, `source_url`, `provider`), `facts`, `topic_wiki`, `providers` (per source: ok, ms, count, or
+why it was skipped or failed) and `usage` (requests, DeepSeek calls, `cost_usd`). Cached for a day per calling service.
+
+Every source is optional and fails on its own. DeepSeek is used only when its key resolves (`AI_RESEARCH_DEEPSEEK_KEY_REF`,
+default `env:DEEPSEEK_API_KEY`; `AI_RESEARCH_DEEPSEEK=0` turns it off), only for first-party callers (a developer app's
+run reads the free sources), within `AI_RESEARCH_DEEPSEEK_DAILY` calls a day (1500), and it stands down on its own
+answers: 402 (balance used up) for 30 minutes, 401 for an hour, 429 for a minute. Tests read a canned web
+(`test/fixtures/research-fetch.js`); none reaches the internet.
+
+OpenVibe.Wiki's "vibe a wiki" uses it with `wiki.generate_space` (plans the pages: main page first, each with a kind,
+an outline and a search phrase) and `wiki.generate_page` (writes one page: lead, sections, infobox, `[[links]]` to the
+planned pages, and up to three of the numbered pictures by heading), both on route `wiki.write` (the creative model,
+7,000 output tokens, 170 s, past the provider's chat timeout).
+
## Image generation
## Image generation
Off unless `AI_IMAGE_MODEL` names the shared provider's image model (`dall-e-3`, `gpt-image-1`; OpenAI-shaped
Off unless `AI_IMAGE_MODEL` names the shared provider's image model (`dall-e-3`, `gpt-image-1`; OpenAI-shaped
@@ -142,6 +142,17 @@ function runsRouter({ runs, registry, auth, config, log = console }) {
@@ -142,6 +142,17 @@ function runsRouter({ runs, registry, auth, config, log = console }) {
});
});
}
}
+ // MegaSource research (workflow research.gather): first-party services and developer apps alike; an app's run leaves
+ // DeepSeek's paid web search out and reads only the free sources.
+ r.post('/api/v1/research', createApp, async (req, res) => {
+ try {
+ const body = req.body || {};
+ const { idempotency_key: idem, target, attribution, on_behalf_of: obo, options, ...input } = body;
+ const wait = req.query.wait != null ? Math.max(0, Math.min(Number(req.query.wait) || 0, config.runs.maxWaitMs)) : config.runs.maxWaitMs;
+ await createAndRespond(req, res, { workflow: 'research.gather', input, idempotency_key: idem, target, attribution, on_behalf_of: obo, options }, wait);
+ } catch (err) { sendError(res, err, req.ov, log); }
+ });
+
return r;
return r;
}
}
@@ -24,7 +24,7 @@ const APP_SUB_RE = new RegExp(`^app:app_(${ULID})$`);
@@ -24,7 +24,7 @@ const APP_SUB_RE = new RegExp(`^app:app_(${ULID})$`);
const PROJECT_RE = new RegExp(`^prj_(${ULID})$`);
const PROJECT_RE = new RegExp(`^prj_(${ULID})$`);
const ENVS = ['sandbox', 'production'];
const ENVS = ['sandbox', 'production'];
/** The operations ai.app.run grants by route, as workflow keys (POST /api/v1/<op> is a run of ai.<op>). */
/** The operations ai.app.run grants by route, as workflow keys (POST /api/v1/<op> is a run of ai.<op>). */
-const DIRECT_OPS = Object.freeze(['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'embed', 'image'].map((op) => `ai.${op}`));
+const DIRECT_OPS = Object.freeze([...['chat', 'complete', 'generate', 'summarize', 'classify', 'extract', 'embed', 'image'].map((op) => `ai.${op}`), 'research.gather']);
/** 'prj_01JAB…' -> 'p01jab…' (null for anything that is not a project id). */
/** 'prj_01JAB…' -> 'p01jab…' (null for anything that is not a project id). */
function projectKey(projectId) {
function projectKey(projectId) {
@@ -108,6 +108,15 @@ function load(env = process.env) {
@@ -108,6 +108,15 @@ function load(env = process.env) {
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
+ // MegaSource (server/research/, workflow research.gather): DeepSeek's paid web search joins the free sources when
+ // its key resolves (a secret reference, read by name) and stays within AI_RESEARCH_DEEPSEEK_DAILY calls a day.
+ research: {
+ deepseek: bool(env.AI_RESEARCH_DEEPSEEK, true),
+ deepseekKeyRef: String(env.AI_RESEARCH_DEEPSEEK_KEY_REF || 'env:DEEPSEEK_API_KEY').trim(),
+ deepseekUrl: trimUrl(env.AI_RESEARCH_DEEPSEEK_URL || 'https://api.deepseek.com'),
+ deepseekDaily: int(env.AI_RESEARCH_DEEPSEEK_DAILY, 1500),
+ timeoutMs: int(env.AI_RESEARCH_TIMEOUT_MS, 15000),
+ },
// media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),
// media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),
// the download timeout, the scene-change threshold (scdet, 0-100) and the tools.
// the download timeout, the scene-change threshold (scdet, 0-100) and the tools.
mediaAnalysis: {
mediaAnalysis: {
@@ -26,7 +26,7 @@ const { createApp } = require('./app');
@@ -26,7 +26,7 @@ const { createApp } = require('./app');
const { gracefulStop, telemetry } = require('openvibe-sdk/service');
const { gracefulStop, telemetry } = require('openvibe-sdk/service');
const { createPgTelemetrySink, startTelemetryRetention } = require('openvibe-sdk/telemetry');
const { createPgTelemetrySink, startTelemetryRetention } = require('openvibe-sdk/telemetry');
-async function start({ config, db: givenDb = null, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, listen = true, credentialFetch = null, valkey = null } = {}) {
+async function start({ config, db: givenDb = null, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, env = process.env, log = console, listen = true, credentialFetch = null, valkey = null, research = null } = {}) {
config = config || load(env);
config = config || load(env);
schemas.configure(config.schemaCache);
schemas.configure(config.schemaCache);
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
// PostgreSQL (ADR-035): opened and migrated here unless the caller (a test) hands in a migrated handle.
@@ -41,7 +41,8 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -41,7 +41,8 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
const quotas = createQuotas(db, { clock, registry, freeAllowance, tiers });
const quotas = createQuotas(db, { clock, registry, freeAllowance, tiers });
const cache = createCache(db, { clock });
const cache = createCache(db, { clock });
const fetcher = createFetcher(config);
const fetcher = createFetcher(config);
- const engine = createEngine({ registry, pool, fetcher, quotas, config, log });
+ // MegaSource (server/research/): the DeepSeek key resolves by name from this environment; tests hand in their own.
+ const engine = createEngine({ registry, pool, fetcher, quotas, config, log, research: research || require('./research').createResearch({ config: config.research, env, log }) });
// ai.run.* to OpenVibe.Events through the outbox (server/events.js); before recovery, so its failures are announced.
// ai.run.* to OpenVibe.Events through the outbox (server/events.js); before recovery, so its failures are announced.
require('./events').init(db, { log });
require('./events').init(db, { log });
// platform.usage-sample@1 readings to OpenVibe.Billing through their own outbox (server/usage-samples.js).
// platform.usage-sample@1 readings to OpenVibe.Billing through their own outbox (server/usage-samples.js).
@@ -340,7 +340,7 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
@@ -340,7 +340,7 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
schemas.assertSchema(w.output_schema, 'output_schema');
schemas.assertSchema(w.output_schema, 'output_schema');
if (!Array.isArray(w.steps) || !w.steps.length) throw new AiError(422, 'ai.invalid', 'workflow needs at least one step');
if (!Array.isArray(w.steps) || !w.steps.length) throw new AiError(422, 'ai.invalid', 'workflow needs at least one step');
for (const s of w.steps) {
for (const s of w.steps) {
- if (!s || !['llm', 'passthrough', 'complete', 'transcribe', 'embed', 'image', 'media_analysis'].includes(s.kind)) throw new AiError(422, 'ai.invalid', 'step kind must be llm, passthrough, complete, transcribe, embed, image or media_analysis');
+ if (!s || !['llm', 'passthrough', 'complete', 'transcribe', 'embed', 'image', 'media_analysis', 'research'].includes(s.kind)) throw new AiError(422, 'ai.invalid', 'step kind must be llm, passthrough, complete, transcribe, embed, image, media_analysis or research');
if ((s.kind === 'llm' || (s.kind === 'media_analysis' && s.template)) && !(s.template && await latestAny('templates', s.template))) throw new AiError(422, 'ai.invalid', `step references unknown template ${s.template}`);
if ((s.kind === 'llm' || (s.kind === 'media_analysis' && s.template)) && !(s.template && await latestAny('templates', s.template))) throw new AiError(422, 'ai.invalid', `step references unknown template ${s.template}`);
}
}
if (!['none', 'private', 'service'].includes(w.cache_mode)) throw new AiError(422, 'ai.invalid', 'cache_mode must be none, private or service');
if (!['none', 'private', 'service'].includes(w.cache_mode)) throw new AiError(422, 'ai.invalid', 'cache_mode must be none, private or service');
new file mode 100644
new file mode 100644
@@ -0,0 +1,74 @@
@@ -0,0 +1,74 @@
+'use strict';
+/**
+ * DeepSeek's Search and Fetch APIs (api-docs.deepseek.com/guides/search): web search with page summaries and text,
+ * and any page as Markdown, fetched by DeepSeek (never from this host). $0.75 per 1,000 successful calls each, from
+ * the same balance as the account's tokens; an empty search or a failed call is not billed.
+ *
+ * Spend is bounded twice: a daily cap on calls (AI_RESEARCH_DEEPSEEK_DAILY) and the provider standing down on its own
+ * answers: 402 (balance used up) for 30 minutes, 401 (key refused) for an hour, 429 for a minute.
+ */
+const { clip, canonicalUrl } = require('./text');
+
+const PRICE_PER_CALL_USD = 0.00075;
+
+function createDeepseek({ http, key, base = 'https://api.deepseek.com', dailyCap = 1500, now = () => Date.now(), log = console }) {
+ const state = { day: '', used: 0, downUntil: 0, reason: null };
+ const today = () => new Date(now()).toISOString().slice(0, 10);
+
+ function available() {
+ if (!key) return false;
+ if (state.day !== today()) { state.day = today(); state.used = 0; }
+ return now() >= state.downUntil && state.used < dailyCap;
+ }
+
+ function status() {
+ if (!key) return { configured: false };
+ available();
+ return { configured: true, used_today: state.used, daily_cap: dailyCap, down_until: state.downUntil > now() ? new Date(state.downUntil).toISOString() : null, reason: state.downUntil > now() ? state.reason : null };
+ }
+
+ function standDown(ms, reason) {
+ state.downUntil = now() + ms;
+ state.reason = reason;
+ log.warn && log.warn(`[research] deepseek stands down for ${Math.round(ms / 1000)} s: ${reason}`);
+ }
+
+ async function call(path, body, signal) {
+ if (!available()) throw Object.assign(new Error(state.reason || 'daily cap reached'), { status: 503 });
+ state.used += 1;
+ const r = await http.request(`${base}${path}`, { method: 'POST', body, headers: { Authorization: `Bearer ${key}` }, signal, timeout: 20000 });
+ if (r.status === 402) { standDown(30 * 60000, 'balance used up'); throw Object.assign(new Error('DeepSeek balance used up'), { status: 402 }); }
+ if (r.status === 401) { standDown(60 * 60000, 'key refused'); throw Object.assign(new Error('DeepSeek refused the key'), { status: 401 }); }
+ if (r.status === 429) { standDown(60000, 'rate limited'); throw Object.assign(new Error('DeepSeek rate limit'), { status: 429 }); }
+ if (r.status !== 200 || !r.json) throw Object.assign(new Error(`DeepSeek answered ${r.status}`), { status: r.status });
+ return r.json;
+ }
+
+ /** Web search: [{ url, title, site, snippet, summary, content, date, authority, score }]. */
+ async function search(query, { count = 8, needContent = true, question = null, signal } = {}) {
+ const body = { query: String(query).slice(0, 128), count, need_content: needContent };
+ if (question) body.ori_question = String(question).slice(0, 128);
+ const j = await call('/v1/tools/search', body, signal);
+ return (j.results || []).filter((r) => r && canonicalUrl(r.url)).map((r) => ({
+ url: r.url,
+ title: String(r.title || '').slice(0, 300),
+ site: r.site_name || null,
+ snippet: String(r.snippet || '').slice(0, 400),
+ summary: String(r.summary || ''),
+ content: String(r.content || ''),
+ date: r.date || null,
+ authority: Number.isInteger(r.authority) ? r.authority : -1,
+ score: Number(r.rerank_score) || 0,
+ }));
+ }
+
+ /** One page as Markdown, fetched by DeepSeek. */
+ async function fetchPage(url, { chars = 6000, signal } = {}) {
+ const j = await call('/v1/tools/fetch', { url }, signal);
+ return { title: String(j.title || ''), content: clip(String(j.content || ''), chars) };
+ }
+
+ return { available, status, search, fetchPage, PRICE_PER_CALL_USD };
+}
+
+module.exports = { createDeepseek, PRICE_PER_CALL_USD };
new file mode 100644
new file mode 100644
@@ -0,0 +1,88 @@
@@ -0,0 +1,88 @@
+'use strict';
+/**
+ * Fandom: the community wiki dedicated to a topic (a show, a game, a franchise), through its public MediaWiki API.
+ * Finding it: the hosts web search returned, else the topic's name as a subdomain (kimpossible.fandom.com), checked
+ * by asking the wiki its name, which must share a word with the topic. Text is CC BY-SA 3.0 (Fandom's licence); a page's
+ * picture is the wiki's own and is marked as not freely licensed.
+ */
+const { articleText, clip, hostOf } = require('./text');
+
+const qs = (params) => Object.entries({ format: 'json', formatversion: '2', ...params }).map(([k, v]) => `${k}=${encodeURIComponent(v)}`).join('&');
+const STOP = new Set(['the', 'a', 'an', 'of', 'and', 'in', 'on', 'wiki', 'fandom', 'tv', 'series', 'show', 'game', 'film', 'movie']);
+
+function words(s) {
+ return String(s || '').toLowerCase().normalize('NFKD').replace(/[̀-ͯ]/g, '').split(/[^a-z0-9]+/).filter((w) => w && !STOP.has(w));
+}
+
+/** Subdomains to try for a topic, most specific first. */
+function slugsFor(topic) {
+ const all = String(topic || '').toLowerCase().normalize('NFKD').replace(/[̀-ͯ]/g, '').replace(/\([^)]*\)/g, ' ');
+ const w = all.split(/[^a-z0-9]+/).filter(Boolean);
+ const content = w.filter((x) => !STOP.has(x));
+ const out = [w.join(''), content.join(''), content.slice(0, 2).join(''), content[0] && content[0].length >= 4 ? content[0] : ''];
+ return [...new Set(out.filter((s) => /^[a-z0-9]{3,60}$/.test(s)))].slice(0, 4);
+}
+
+function createFandom({ http }) {
+ async function probe(base, topic, signal) {
+ const r = await http.request(`${base}/api.php?${qs({ action: 'query', meta: 'siteinfo', siprop: 'general' })}`, { signal, timeout: 8000 });
+ const g = r.status === 200 && r.json && r.json.query && r.json.query.general;
+ if (!g || !g.sitename) return null;
+ const topicWords = new Set(words(topic));
+ const overlap = words(g.sitename).some((x) => topicWords.has(x)) || words(new URL(base).hostname.split('.')[0]).some((x) => topicWords.has(x));
+ return overlap ? { base, name: String(g.sitename), lang: g.lang || 'en' } : null;
+ }
+
+ /** The topic's wiki: { base, name } or null. hints are URLs a web search returned. */
+ async function findWiki(topic, hints = [], { lang = 'en', signal } = {}) {
+ const hosts = [];
+ for (const u of hints) {
+ const h = hostOf(u);
+ const m = /^([a-z0-9-]{2,60})\.fandom\.com$/.exec(h);
+ if (m && m[1] !== 'community' && m[1] !== 'www' && !hosts.includes(m[1])) hosts.push(m[1]);
+ }
+ const path = lang && lang !== 'en' ? `/${lang}` : '';
+ for (const slug of [...hosts.slice(0, 2), ...slugsFor(topic)].filter((s, i, a) => a.indexOf(s) === i).slice(0, 5)) {
+ try {
+ const found = await probe(`https://${slug}.fandom.com${path}`, topic, signal);
+ if (found) return found;
+ } catch { /* try the next one */ }
+ }
+ return null;
+ }
+
+ async function search(base, query, { limit = 4, signal } = {}) {
+ const j = await http.getJson(`${base}/api.php?${qs({ action: 'query', list: 'search', srsearch: query, srlimit: String(limit), srnamespace: '0' })}`, { signal });
+ return ((j.query && j.query.search) || []).map((s) => s.title).filter((t) => !/\/(Gallery|Transcript|Quotes|LAM)$/i.test(t));
+ }
+
+ /** One article: readable text and its picture. */
+ async function page(base, title, { chars = 6000, signal } = {}) {
+ const [parsed, images] = await Promise.all([
+ http.getJson(`${base}/api.php?${qs({ action: 'parse', page: title, prop: 'text|displaytitle', redirects: '1', disableeditsection: '1', disabletoc: '1', disablelimitreport: '1' })}`, { signal }),
+ http.getJson(`${base}/api.php?${qs({ action: 'query', titles: title, prop: 'pageimages', piprop: 'original', redirects: '1' })}`, { signal }).catch(() => null),
+ ]);
+ const p = parsed && parsed.parse;
+ if (!p || !p.text) return null;
+ const name = p.title || title;
+ const url = `${base}/wiki/${encodeURIComponent(name.replace(/ /g, '_'))}`;
+ const text = clip(articleText(typeof p.text === 'string' ? p.text : p.text['*']), chars);
+ const pg = images && images.query && images.query.pages && images.query.pages[0];
+ const original = pg && pg.original;
+ const image = original && original.source && original.width >= 160 ? {
+ url: String(original.source).replace(/\/revision\/latest(\?|$)/, '/revision/latest/scale-to-width-down/1024$1'),
+ original: original.source,
+ width: Math.min(original.width, 1024),
+ height: original.width > 1024 ? Math.round(original.height * 1024 / original.width) : original.height,
+ mime: /\.png/i.test(original.source) ? 'image/png' : /\.webp/i.test(original.source) ? 'image/webp' : /\.gif/i.test(original.source) ? 'image/gif' : 'image/jpeg',
+ title: name, description: null, author: null,
+ license: 'Fandom wiki picture (copyright of its owners; used for identification)', license_url: null, free: false,
+ source_url: url, provider: 'fandom',
+ } : null;
+ return text ? { title: name, url, text, image } : null;
+ }
+
+ return { findWiki, search, page };
+}
+
+module.exports = { createFandom, slugsFor };
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+'use strict';
+/**
+ * HTTPS for the research providers: a descriptive User-Agent with a contact (Wikimedia's API policy asks for one), a
+ * per-call timeout, the run's cancellation, and the guarded fetch (public addresses only, no redirects, 8 MB cap).
+ */
+const { guardedFetch } = require('../providers/guarded-fetch');
+
+const USER_AGENT = 'OpenVibeResearch/1.0 (+https://ai.openvibe.services; [email protected])';
+
+class ProviderError extends Error {
+ constructor(status, message) { super(message); this.status = status; }
+}
+
+function createHttp({ fetchImpl = guardedFetch, timeoutMs = 15000, userAgent = USER_AGENT } = {}) {
+ let requests = 0;
+ async function request(url, { method = 'GET', body = null, headers = {}, signal = null, timeout = timeoutMs } = {}) {
+ const ctrl = new AbortController();
+ const timer = setTimeout(() => ctrl.abort(), timeout);
+ const onAbort = () => ctrl.abort();
+ if (signal) { if (signal.aborted) ctrl.abort(); else signal.addEventListener('abort', onAbort, { once: true }); }
+ requests += 1;
+ try {
+ const res = await fetchImpl(url, {
+ method,
+ headers: { 'User-Agent': userAgent, Accept: 'application/json', ...(body != null ? { 'Content-Type': 'application/json' } : {}), ...headers },
+ body: body == null ? undefined : JSON.stringify(body),
+ signal: ctrl.signal,
+ });
+ const text = await res.text();
+ let json = null;
+ try { json = text ? JSON.parse(text) : null; } catch { /* not JSON: the caller decides */ }
+ return { status: res.status, json, text };
+ } catch (err) {
+ if (ctrl.signal.aborted && !(signal && signal.aborted)) throw new ProviderError(504, `timed out after ${timeout} ms`);
+ throw err;
+ } finally {
+ clearTimeout(timer);
+ if (signal) signal.removeEventListener('abort', onAbort);
+ }
+ }
+ /** GET a JSON API; anything but a 200 with JSON is an error. */
+ async function getJson(url, opts) {
+ const r = await request(url, opts);
+ if (r.status !== 200 || !r.json) throw new ProviderError(r.status, `${new URL(url).hostname} answered ${r.status}`);
+ return r.json;
+ }
+ return { request, getJson, count: () => requests };
+}
+
+module.exports = { createHttp, ProviderError, USER_AGENT };
new file mode 100644
new file mode 100644
@@ -0,0 +1,199 @@
@@ -0,0 +1,199 @@
+'use strict';
+/**
+ * MegaSource: one research call over many sources, merged into one ranked, de-duplicated evidence pack that the
+ * product workflows cite by number (workflow research.gather, POST /api/v1/research).
+ *
+ * wikipedia articles for the query: full text of the best ones, lead pictures, the Wikidata id
+ * fandom the topic's own community wiki, when it has one (characters, episodes, places, lore)
+ * wikidata the top article's key statements as labelled facts (an infobox's raw material) and its picture
+ * deepseek web search with page text (DeepSeek's Search API) and pages it cannot read in full (its Fetch API);
+ * paid per call, so only for first-party callers and within its daily cap
+ * commons freely licensed pictures for the query
+ * openverse more openly licensed pictures
+ *
+ * Every provider is optional and fails on its own: a provider that errors, times out or stands down is reported in
+ * `providers` and the rest still answer. Nothing is invented: a source is a page that was read, with its URL, when it
+ * was read and its licence; a picture carries its own licence and `free` says whether anyone may reuse it.
+ */
+const { createHttp } = require('./http');
+const { createWikimedia } = require('./wikimedia');
+const { createFandom } = require('./fandom');
+const { createOpenverse } = require('./openverse');
+const { createDeepseek } = require('./deepseek');
+const { clip, canonicalUrl, hostOf } = require('./text');
+const { resolveSecret } = require('../util');
+
+const PROVIDERS = ['wikipedia', 'fandom', 'wikidata', 'deepseek', 'commons', 'openverse'];
+const LICENSE = { wikipedia: 'CC BY-SA 4.0 (Wikipedia)', fandom: 'CC BY-SA 3.0 (Fandom)', wikidata: 'CC0 (Wikidata)' };
+
+const int = (v, lo, hi, d) => { const n = Math.round(Number(v)); return Number.isFinite(n) ? Math.min(hi, Math.max(lo, n)) : d; };
+const oneLine = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);
+const short = (err) => oneLine(err && err.message ? err.message : err, 160) || 'failed';
+
+function createResearch({ config = {}, env = process.env, fetchImpl, now = () => Date.now(), log = console } = {}) {
+ const http = createHttp({ ...(fetchImpl ? { fetchImpl } : {}), timeoutMs: config.timeoutMs || 15000 });
+ const wikimedia = createWikimedia({ http });
+ const fandom = createFandom({ http });
+ const openverse = createOpenverse({ http });
+ const deepseek = createDeepseek({ http, key: config.deepseekKey || (config.deepseek !== false ? resolveSecret(config.deepseekKeyRef, env) : ''), base: config.deepseekUrl || 'https://api.deepseek.com', dailyCap: config.deepseekDaily || 1500, now, log });
+
+ function status() {
+ return { providers: PROVIDERS, deepseek: deepseek.status() };
+ }
+
+ /**
+ * input: { query, topic?, focus?, question?, lang?, max_sources?, max_images?, images?, content_chars?, providers? }
+ * query what to research ("Kim Possible", "Ron Stoppable Kim Possible")
+ * topic the subject the query belongs to, used to find the topic's own wiki (defaults to the query)
+ * focus one subject inside the topic (a character, an episode): searched for inside the topic's wiki
+ * opts: { signal, allowPaid } — allowPaid false (a developer app) leaves DeepSeek out.
+ */
+ async function gather(input = {}, { signal = null, allowPaid = true } = {}) {
+ const query = oneLine(input.query, 200);
+ if (!query) throw Object.assign(new Error('research needs a query'), { status: 422 });
+ const topic = oneLine(input.topic, 200) || query;
+ const focus = oneLine(input.focus, 200) || null;
+ const lang = /^[a-z]{2,3}$/.test(input.lang || '') ? input.lang : 'en';
+ const maxSources = int(input.max_sources, 1, 20, 10);
+ const maxImages = input.images === false ? 0 : int(input.max_images, 0, 24, 10);
+ const chars = int(input.content_chars, 500, 12000, 5000);
+ const want = new Set(Array.isArray(input.providers) && input.providers.length ? input.providers.filter((p) => PROVIDERS.includes(p)) : PROVIDERS);
+ const searchText = focus && !query.toLowerCase().includes(focus.toLowerCase()) ? `${focus} ${query}` : query;
+ const retrievedAt = new Date(now()).toISOString();
+ const providers = {};
+ const usage = { requests: 0, deepseek_search: 0, deepseek_fetch: 0, cost_usd: 0 };
+ const startRequests = http.count();
+
+ async function timed(name, fn) {
+ const t0 = now();
+ try {
+ const value = await fn();
+ const count = value == null ? 0 : Array.isArray(value) ? value.length : (value.count != null ? value.count : 1);
+ providers[name] = { ok: true, ms: now() - t0, count };
+ return value;
+ } catch (err) {
+ if (signal && signal.aborted) throw err;
+ providers[name] = { ok: false, ms: now() - t0, error: short(err) };
+ return null;
+ }
+ }
+ const skip = (name, reason) => { providers[name] = { ok: false, skipped: reason }; return null; };
+
+ // Round one, all at once: Wikipedia, web search and the picture libraries.
+ const useDeepseek = want.has('deepseek') && (allowPaid ? (deepseek.available() || skip('deepseek', deepseek.status().configured ? (deepseek.status().reason || 'daily cap reached') : 'not configured')) : skip('deepseek', 'first-party callers only'));
+ const [wiki, web, commons, openv] = await Promise.all([
+ want.has('wikipedia') ? timed('wikipedia', async () => {
+ const hits = await wikimedia.searchWikipedia(searchText, { lang, limit: focus ? 4 : 5, signal });
+ const list = await wikimedia.articles(hits.map((h) => h.title), { lang, full: focus ? 2 : 3, chars, signal });
+ return list.filter((a) => !a.disambiguation && a.text);
+ }) : null,
+ useDeepseek ? timed('deepseek', async () => {
+ usage.deepseek_search += 1;
+ return deepseek.search(searchText, { count: 8, needContent: true, question: input.question || null, signal });
+ }) : null,
+ maxImages && want.has('commons') ? timed('commons', () => wikimedia.commonsSearch(searchText, { limit: 8, signal })) : null,
+ maxImages && want.has('openverse') ? timed('openverse', () => openverse.search(searchText, { limit: 6, signal })) : null,
+ ]);
+
+ // Round two: the topic's own wiki, the top article's facts and pictures, and full text for web results that came back short.
+ const top = wiki && wiki[0];
+ const hints = (web || []).map((r) => r.url);
+ const [fan, facts, articlePictures, fetched] = await Promise.all([
+ want.has('fandom') ? timed('fandom', async () => {
+ const site = await fandom.findWiki(topic, hints, { lang, signal });
+ if (!site) return null;
+ const titles = await fandom.search(site.base, focus || topic, { limit: 4, signal });
+ const pages = (await Promise.all(titles.slice(0, focus ? 2 : 3).map((t) => fandom.page(site.base, t, { chars, signal }).catch(() => null)))).filter(Boolean);
+ return { site, pages, count: pages.length };
+ }) : null,
+ want.has('wikidata') && top && top.qid ? timed('wikidata', () => wikimedia.wikidataFacts(top.qid, { lang, signal })) : null,
+ maxImages && want.has('wikipedia') && top ? timed('wikipedia_images', () => wikimedia.articleImages(top.title, { lang, limit: 10, signal })) : null,
+ useDeepseek && web ? (async () => {
+ // A short result is read in full, unless it is a copy of an encyclopedia page already read.
+ const read = new Set((wiki || []).map((a) => canonicalUrl(a.url)));
+ const thin = web.filter((r) => r.score >= 1 && r.content.length < 400 && !read.has(canonicalUrl(r.url))).slice(0, 2);
+ return Promise.all(thin.map(async (r) => {
+ try { usage.deepseek_fetch += 1; const page = await deepseek.fetchPage(r.url, { chars, signal }); r.content = page.content || r.content; } catch { /* the summary stays */ }
+ }));
+ })() : null,
+ ]);
+ void fetched;
+ // The Wikidata item's picture, and the article's lead picture when the article's own list left it out (a logo
+ // drawn as SVG), each with its licence from the file's page.
+ const leadName = top && top.lead ? decodeURIComponent(String(top.lead.original).split('/').pop() || '') : '';
+ const leadListed = leadName && (articlePictures || []).some((i) => String(i.original).endsWith(`/${encodeURIComponent(leadName).replace(/%20/g, '_')}`) || String(i.original).endsWith(`/${leadName}`));
+ const [factPicture, leadPicture] = await Promise.all([
+ maxImages && facts && facts.image ? timed('wikidata_image', () => wikimedia.commonsFiles([facts.image], { signal })) : null,
+ maxImages && leadName && !leadListed ? wikimedia.fileInfo([leadName], { lang, allowSvg: true, signal }).catch(() => null) : null,
+ ]);
+
+ // Sources, ranked: the topic's own pages first, then the encyclopedia, then the web by DeepSeek's score.
+ const scored = [];
+ const focusLc = (focus || '').toLowerCase();
+ (fan && fan.pages || []).forEach((p, i) => {
+ const exact = focusLc ? p.title.toLowerCase() === focusLc : i === 0;
+ scored.push({ score: (exact ? 3.3 : 2.4) - i * 0.15, s: { source_type: 'fandom', url: p.url, title: `${p.title} — ${fan.site.name}`, snippet: clip(p.text, 300), content: p.text, retrieved_at: retrievedAt, provenance: { provider: 'fandom', site: fan.site.name, license: LICENSE.fandom } } });
+ });
+ (wiki || []).forEach((a, i) => {
+ const exact = a.title.toLowerCase() === (focusLc || query.toLowerCase());
+ scored.push({ score: (exact || i === 0 ? 3.0 : 1.7) - i * 0.12, s: { source_type: 'wikipedia', url: a.url, title: `${a.title} — Wikipedia`, snippet: a.description || clip(a.text, 300), content: a.text, published_at: a.revisedAt || undefined, retrieved_at: retrievedAt, provenance: { provider: 'wikipedia', site: `${lang}.wikipedia.org`, license: LICENSE.wikipedia } } });
+ });
+ (web || []).forEach((r) => {
+ const content = clip(r.content || r.summary || r.snippet, chars);
+ if (!content) return;
+ scored.push({ score: Math.min(r.score, 3) * 0.9 + Math.max(r.authority, 0) * 0.08, s: { source_type: 'web', url: r.url, title: r.title || hostOf(r.url), snippet: r.snippet || clip(r.summary, 300), content, published_at: r.date || undefined, retrieved_at: retrievedAt, provenance: { provider: 'deepseek', site: r.site || hostOf(r.url), authority: r.authority, score: r.score } } });
+ });
+ if (facts && facts.facts.length) {
+ scored.push({ score: 2.0, s: { source_type: 'wikidata', url: facts.url, title: `${facts.name} — Wikidata`, snippet: facts.description || `${facts.facts.length} facts`, content: facts.facts.map((f) => `${f.label}: ${f.value}`).join('\n'), retrieved_at: retrievedAt, provenance: { provider: 'wikidata', site: 'wikidata.org', license: LICENSE.wikidata } } });
+ }
+ scored.sort((a, b) => b.score - a.score);
+ const seen = new Set();
+ const sources = [];
+ for (const { s } of scored) {
+ const key = canonicalUrl(s.url);
+ if (!key || seen.has(key)) continue;
+ seen.add(key);
+ for (const k of Object.keys(s)) if (s[k] === undefined) delete s[k];
+ sources.push(s);
+ if (sources.length >= maxSources) break;
+ }
+
+ // Pictures: the subject's own (Wikidata, the article, the topic's wiki) before library search results.
+ const images = [];
+ const seenImg = new Set();
+ const lead = (leadPicture || []).map((i) => ({ ...i, title: top.title, description: i.description || top.description || null, source_url: i.source_url || top.url }));
+ const fanPics = (fan && fan.pages || []).map((p) => p.image).filter(Boolean);
+ // Library search is loose ("Ron Stoppable" also finds "whistle stop" photos): a library picture must name every
+ // word of the focus, or most of the query's, in its title or description.
+ const significant = (s) => String(s || '').toLowerCase().normalize('NFKD').replace(/[\u0300-\u036f]/g, '').split(/[^a-z0-9]+/).filter((w) => w.length > 2 && !/^(the|and|for|with|from|wiki)$/.test(w));
+ const named = (img) => {
+ const text = significant(`${img.title} ${img.description || ''}`).join(' ');
+ const need = significant(focus || query);
+ const hits = need.filter((w) => text.includes(w)).length;
+ return need.length ? (focus ? hits === need.length : hits >= Math.ceil(need.length / 2)) : true;
+ };
+ const library = [...(commons || []), ...(openv || [])].filter(named);
+ for (const img of [...(factPicture || []), ...lead, ...(articlePictures || []), ...fanPics, ...library]) {
+ if (!img || !img.url || !/^https:\/\//.test(img.url) || /\.svg(\?|$)/i.test(img.url)) continue;
+ const key = String(img.original || img.url).replace(/\/(scale-to-width-down|thumb)\/.*$/, '');
+ if (seenImg.has(key)) continue;
+ seenImg.add(key);
+ images.push({ ...img });
+ if (images.length >= maxImages) break;
+ }
+
+ usage.requests = http.count() - startRequests;
+ usage.cost_usd = Number(((usage.deepseek_search + usage.deepseek_fetch) * deepseek.PRICE_PER_CALL_USD).toFixed(5));
+ return {
+ query, topic, focus, lang, retrieved_at: retrievedAt,
+ sources, images,
+ facts: facts ? facts.facts : [],
+ topic_wiki: fan && fan.site ? { name: fan.site.name, url: fan.site.base } : null,
+ providers, usage,
+ };
+ }
+
+ return { gather, status };
+}
+
+module.exports = { createResearch, PROVIDERS };
new file mode 100644
new file mode 100644
@@ -0,0 +1,32 @@
@@ -0,0 +1,32 @@
+'use strict';
+/**
+ * Openverse (api.openverse.org): openly licensed pictures from Flickr, museums and other collections, anonymous use.
+ * Only licences that allow commercial reuse are asked for (CC0, public domain, CC BY, CC BY-SA).
+ */
+
+const NAMES = { cc0: 'CC0', pdm: 'Public domain', by: 'CC BY', 'by-sa': 'CC BY-SA' };
+
+function createOpenverse({ http }) {
+ async function search(query, { limit = 8, signal } = {}) {
+ const url = `https://api.openverse.org/v1/images/?q=${encodeURIComponent(query)}&page_size=${limit}&license_type=commercial&mature=false`;
+ const j = await http.getJson(url, { signal, timeout: 10000 });
+ return (j.results || []).filter((r) => r.url && /^https:/.test(r.url) && (r.width || 0) >= 320 && NAMES[r.license]).map((r) => ({
+ url: r.url,
+ original: r.url,
+ width: r.width || null,
+ height: r.height || null,
+ mime: null,
+ title: String(r.title || '').slice(0, 200) || query,
+ description: null,
+ author: r.creator ? String(r.creator).slice(0, 160) : null,
+ license: `${NAMES[r.license]}${r.license_version && !/^(cc0|pdm)$/.test(r.license) ? ` ${r.license_version}` : ''}`,
+ license_url: r.license_url || null,
+ free: true,
+ source_url: r.foreign_landing_url || null,
+ provider: `openverse${r.source ? `/${r.source}` : ''}`,
+ }));
+ }
+ return { search };
+}
+
+module.exports = { createOpenverse };
new file mode 100644
new file mode 100644
@@ -0,0 +1,96 @@
@@ -0,0 +1,96 @@
+'use strict';
+/** Text helpers for research: HTML to readable text, clipping at a boundary, canonical URLs, image licences. */
+
+const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: '\'', nbsp: ' ', ndash: '–', mdash: '—', hellip: '…', rsquo: '’', lsquo: '‘', rdquo: '”', ldquo: '“', middot: '·', times: '×' };
+
+function decodeEntities(s) {
+ return String(s).replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
+ if (e[0] === '#') {
+ const n = e[1].toLowerCase() === 'x' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
+ return Number.isFinite(n) && n > 0 && n < 0x110000 ? String.fromCodePoint(n) : '';
+ }
+ return ENTITIES[e.toLowerCase()] ?? m;
+ });
+}
+
+function stripTags(html) {
+ return decodeEntities(String(html || '').replace(/<[^>]*>/g, ' ')).replace(/\s+/g, ' ').trim();
+}
+
+/**
+ * MediaWiki article HTML (Fandom's parse output) to readable text: infobox rows become "Label: value" lines, headings
+ * become "## Heading", list items "- item"; scripts, references, navigation boxes and tables of contents are dropped.
+ */
+function articleText(html) {
+ let h = String(html || '');
+ h = h.replace(/<(script|style|noscript)\b[\s\S]*?<\/\1>/gi, ' ')
+ .replace(/<sup\b[^>]*class="[^"]*reference[^"]*"[\s\S]*?<\/sup>/gi, '')
+ .replace(/<(table|div|nav)\b[^>]*class="[^"]*\b(navbox|toc|mw-references-wrap|references|printfooter|catlinks|mbox)\b[^"]*"[\s\S]*?<\/\1>/gi, ' ')
+ .replace(/<h[23]\b[^>]*class="[^"]*pi-(?:data-label|secondary-heading)[^"]*"[^>]*>([\s\S]*?)<\/h[23]>\s*<div\b[^>]*class="[^"]*pi-data-value[^"]*"[^>]*>([\s\S]*?)<\/div>/gi,
+ (_m, label, value) => `\n${stripTags(label)}: ${stripTags(value.replace(/<br\s*\/?>/gi, ', ').replace(/<\/li>/gi, ', ')).replace(/,\s*$/, '')}\n`)
+ .replace(/<h2\b[^>]*class="[^"]*pi-title[^"]*"[^>]*>([\s\S]*?)<\/h2>/gi, (_m, t) => `\n${stripTags(t)}\n`)
+ .replace(/<h([1-6])\b[^>]*>([\s\S]*?)<\/h\1>/gi, (_m, n, t) => `\n\n${'#'.repeat(Math.min(Math.max(Number(n), 2), 4))} ${stripTags(t)}\n`)
+ .replace(/<li\b[^>]*>/gi, '\n- ')
+ .replace(/<\/(p|div|tr|ul|ol|table|aside|figure|blockquote|dd|dt)>/gi, '\n')
+ .replace(/<br\s*\/?>/gi, '\n')
+ .replace(/<\/t[dh]>/gi, ' · ');
+ const lines = decodeEntities(h.replace(/<[^>]*>/g, '')).split('\n').map((l) => l.replace(/[ \t ]+/g, ' ').replace(/(?: · )+$/, '').trim());
+ const out = [];
+ for (const l of lines) {
+ if (!l) { if (out.length && out[out.length - 1] !== '') out.push(''); continue; }
+ if (/^#{2,4} (?:References|Notes|External links|See also|Gallery|Navigation|Sources|Appearances)$/i.test(l)) { out.push(l); continue; }
+ out.push(l);
+ }
+ // Drop the trailing reference-ish sections entirely: they are lists of links, not facts.
+ const text = out.join('\n').replace(/\n#{2,4} (?:References|Notes|External links|Navigation|Sources)\n[\s\S]*?(?=\n#{2,4} |$)/gi, '\n');
+ return text.replace(/\n{3,}/g, '\n\n').trim();
+}
+
+/** MediaWiki plain extract with exsectionformat=wiki ("== Heading ==") to "## Heading" Markdown-ish text. */
+function extractText(text) {
+ return String(text || '')
+ .replace(/^(={2,5})\s*(.+?)\s*\1\s*$/gm, (_m, eq, t) => `${'#'.repeat(Math.min(eq.length, 4))} ${t}`)
+ .replace(/\n#{2,4} (?:References|Notes|External links|Further reading|Citations|Sources|Bibliography)\n[\s\S]*?(?=\n#{2,4} |$)/gi, '\n')
+ .replace(/\n{3,}/g, '\n\n')
+ .trim();
+}
+
+/** Cut text at about n characters, at the last paragraph or sentence end before it. */
+function clip(text, n) {
+ const t = String(text || '').trim();
+ if (t.length <= n) return t;
+ const cut = t.slice(0, n);
+ const para = cut.lastIndexOf('\n\n');
+ if (para > n * 0.6) return cut.slice(0, para).trim();
+ const sentence = Math.max(cut.lastIndexOf('. '), cut.lastIndexOf('.\n'), cut.lastIndexOf('! '), cut.lastIndexOf('? '));
+ if (sentence > n * 0.6) return cut.slice(0, sentence + 1).trim();
+ return `${cut.replace(/\s+\S*$/, '')}…`;
+}
+
+/** One spelling per page: https, lowercase host without www. or a Wikipedia mobile prefix, no fragment or tracking query. */
+function canonicalUrl(u) {
+ try {
+ const url = new URL(String(u));
+ if (url.protocol !== 'https:' && url.protocol !== 'http:') return null;
+ url.protocol = 'https:';
+ url.hash = '';
+ url.hostname = url.hostname.toLowerCase().replace(/^www\./, '').replace(/^([a-z-]+)\.m\.(wikipedia|wiktionary|wikiquote)\.org$/, '$1.$2.org');
+ for (const k of [...url.searchParams.keys()]) if (/^(utm_|fbclid$|gclid$|ref$|ref_src$)/i.test(k)) url.searchParams.delete(k);
+ return url.toString().replace(/\/$/, '');
+ } catch { return null; }
+}
+
+function hostOf(u) {
+ try { return new URL(String(u)).hostname.toLowerCase().replace(/^www\./, ''); } catch { return ''; }
+}
+
+// Licences anyone may reuse with credit (commercial use and unchanged copies included): CC0, public domain, CC BY,
+// CC BY-SA, GFDL and the like. NonCommercial and NoDerivatives licences are not counted as free here.
+const FREE_LICENSE = /^(cc0|cc[- ]?zero|public domain|pd(?:m|-|\b)|no restrictions|attribution|cc[- ]by(?:-sa)?(?![- ]?n[cd])\b|gfdl|fal\b|free art|ogl|copyrighted free use)/i;
+
+function isFreeLicense(name) {
+ const n = String(name || '').trim();
+ return Boolean(n) && FREE_LICENSE.test(n) && !/\b(nc|nd|non-?commercial|no ?deriv)/i.test(n) && !/fair use|non-?free/i.test(n);
+}
+
+module.exports = { decodeEntities, stripTags, articleText, extractText, clip, canonicalUrl, hostOf, isFreeLicense };
new file mode 100644
new file mode 100644
@@ -0,0 +1,209 @@
@@ -0,0 +1,209 @@
+'use strict';
+/**
+ * Wikipedia, Wikimedia Commons and Wikidata, through their public APIs (no key; a descriptive User-Agent).
+ *
+ * searchWikipedia article titles for a query
+ * articles intro, lead image, Wikidata id and description for several titles at once, and the full plain
+ * text of the first few
+ * articleImages the pictures an article uses, each with its licence (free files from Commons and an article's
+ * own non-free ones, marked so)
+ * commonsSearch freely licensed pictures on Commons for a query
+ * commonsFiles licence and size for named files (a Wikidata item's image)
+ * wikidataFacts a Wikidata item's key statements as labelled facts
+ *
+ * Text is CC BY-SA (Wikipedia) or CC0 (Wikidata); every picture carries its own licence from the file's metadata.
+ */
+const { stripTags, extractText, clip, isFreeLicense } = require('./text');
+
+const API = (lang) => `https://${lang}.wikipedia.org/w/api.php`;
+const COMMONS = 'https://commons.wikimedia.org/w/api.php';
+const WIKIDATA = 'https://www.wikidata.org/w/api.php';
+const IMAGE_MIME = /^image\/(jpeg|png|webp|gif)$/;
+// Interface pictures an article carries that are not about its subject.
+const NOT_SUBJECT = /(^|[\s_-])(icon|symbol|flag of|logo of wikipedia|commons-logo|wiki letter|question book|edit-clear|ambox|padlock|crystal|nuvola|disambig|portal|stub|blank|placeholder)([\s_.-]|$)/i;
+const META = 'LicenseShortName|LicenseUrl|Artist|ImageDescription|NonFree|UsageTerms|ObjectName';
+
+/** Wikimedia adds ?utm_source=… to file URLs; the picture is the same without it. */
+const untracked = (u) => String(u || '').replace(/\?utm_[^#]*$/, '');
+
+const qs = (params) => Object.entries({ format: 'json', formatversion: '2', ...params }).map(([k, v]) => `${k}=${encodeURIComponent(v)}`).join('&');
+
+/** A file page with imageinfo (iiprop url|size|mime|extmetadata) → a research image, or null when unusable. */
+function imageFrom(page, provider, fallbackSource, { allowSvg = false } = {}) {
+ const ii = page && Array.isArray(page.imageinfo) ? page.imageinfo[0] : null;
+ if (!ii) return null;
+ // A drawing (SVG) only as its PNG rendering, and only when asked for (an article's lead picture, usually a logo).
+ const svg = ii.mime === 'image/svg+xml' && allowSvg && /\.png(\?|$)/i.test(ii.thumburl || '');
+ if (!svg && !IMAGE_MIME.test(ii.mime || '')) return null;
+ const width = ii.width || 0;
+ const height = ii.height || 0;
+ if ((!svg && (width < 200 || height < 150)) || NOT_SUBJECT.test(page.title || '')) return null;
+ const m = ii.extmetadata || {};
+ const v = (k) => (m[k] && m[k].value != null ? String(m[k].value) : '');
+ const license = stripTags(v('LicenseShortName')) || stripTags(v('UsageTerms')) || null;
+ const nonFree = /^(true|1)$/i.test(v('NonFree').trim()) || /fair use|non-?free/i.test(license || '');
+ return {
+ url: untracked(ii.thumburl || ii.url),
+ original: untracked(ii.url),
+ width: ii.thumbwidth || width,
+ height: ii.thumbheight || height,
+ mime: svg ? 'image/png' : ii.mime,
+ title: stripTags(v('ObjectName')) || String(page.title || '').replace(/^File:/, '').replace(/\.[a-z0-9]{2,5}$/i, '').replace(/_/g, ' '),
+ description: clip(stripTags(v('ImageDescription')), 300) || null,
+ author: clip(stripTags(v('Artist')), 160) || null,
+ license: nonFree ? (license || 'Non-free (used for identification)') : license,
+ license_url: v('LicenseUrl') || null,
+ free: !nonFree && isFreeLicense(license),
+ source_url: ii.descriptionurl || fallbackSource || null,
+ provider,
+ };
+}
+
+// The statements worth an infobox, in the order shown.
+const PROPS = [
+ ['P31', 'Instance of'], ['P1476', 'Title'], ['P170', 'Created by'], ['P50', 'Author'], ['P57', 'Directed by'], ['P162', 'Produced by'],
+ ['P86', 'Music by'], ['P175', 'Performer'], ['P161', 'Cast member'], ['P725', 'Voice actor'], ['P449', 'Original network'], ['P272', 'Production company'],
+ ['P750', 'Distributed by'], ['P123', 'Publisher'], ['P178', 'Developer'], ['P400', 'Platform'], ['P136', 'Genre'], ['P495', 'Country of origin'],
+ ['P364', 'Original language'], ['P571', 'Founded / created'], ['P577', 'Publication date'], ['P580', 'Start'], ['P582', 'End'], ['P1113', 'Episodes'],
+ ['P2437', 'Seasons'], ['P569', 'Born'], ['P19', 'Place of birth'], ['P570', 'Died'], ['P27', 'Citizenship'], ['P106', 'Occupation'],
+ ['P112', 'Founded by'], ['P159', 'Headquarters'], ['P452', 'Industry'], ['P17', 'Country'], ['P131', 'Located in'], ['P36', 'Capital'],
+ ['P1082', 'Population'], ['P2046', 'Area'], ['P856', 'Official website'],
+];
+
+function timeText(v) {
+ const m = /^([+-])(\d{1,16})-(\d\d)-(\d\d)/.exec(String(v.time || ''));
+ if (!m) return null;
+ const year = `${m[1] === '-' ? '-' : ''}${Number(m[2])}`;
+ if (v.precision >= 11 && m[3] !== '00' && m[4] !== '00') return `${year}-${m[3]}-${m[4]}`;
+ if (v.precision === 10 && m[3] !== '00') return `${year}-${m[3]}`;
+ return year;
+}
+
+function createWikimedia({ http }) {
+ async function searchWikipedia(query, { lang = 'en', limit = 5, signal } = {}) {
+ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', list: 'search', srsearch: query, srlimit: String(limit), srprop: 'snippet', srenablerewrites: '1', maxlag: '5' })}`, { signal });
+ return ((j.query && j.query.search) || []).map((s) => ({ title: s.title, snippet: stripTags(s.snippet) }));
+ }
+
+ async function fullText(title, { lang, signal }) {
+ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', prop: 'extracts', explaintext: '1', exsectionformat: 'wiki', redirects: '1', titles: title, maxlag: '5' })}`, { signal });
+ const p = j.query && j.query.pages && j.query.pages[0];
+ return p && p.extract ? extractText(p.extract) : '';
+ }
+
+ /** Several articles in one call (intro, lead image, Wikidata id), with the full text of the first `full`. */
+ async function articles(titles, { lang = 'en', full = 3, chars = 6000, signal } = {}) {
+ if (!titles.length) return [];
+ const j = await http.getJson(`${API(lang)}?${qs({
+ action: 'query', prop: 'extracts|pageimages|pageprops|info|description', exintro: '1', explaintext: '1', exlimit: 'max',
+ piprop: 'original|thumbnail', pithumbsize: '1280', ppprop: 'wikibase_item|disambiguation', inprop: 'url', redirects: '1',
+ titles: titles.slice(0, 20).join('|'), maxlag: '5',
+ })}`, { signal });
+ const query = j.query || {};
+ const rename = new Map();
+ for (const r of [...(query.normalized || []), ...(query.redirects || [])]) rename.set(r.from, r.to);
+ const byTitle = new Map((query.pages || []).filter((p) => !p.missing).map((p) => [p.title, p]));
+ const ordered = [];
+ const seen = new Set();
+ for (const t of titles) {
+ let name = t;
+ for (let i = 0; i < 3 && rename.has(name); i++) name = rename.get(name);
+ const p = byTitle.get(name);
+ if (p && !seen.has(p.title)) { seen.add(p.title); ordered.push(p); }
+ }
+ const out = ordered.map((p) => ({
+ title: p.title,
+ url: p.fullurl || `https://${lang}.wikipedia.org/wiki/${encodeURIComponent(p.title.replace(/ /g, '_'))}`,
+ description: p.description || null,
+ text: p.extract || '',
+ lead: p.original ? { url: (p.thumbnail && p.thumbnail.source) || p.original.source, original: p.original.source, width: (p.thumbnail && p.thumbnail.width) || p.original.width, height: (p.thumbnail && p.thumbnail.height) || p.original.height } : null,
+ qid: (p.pageprops && p.pageprops.wikibase_item) || null,
+ disambiguation: Boolean(p.pageprops && p.pageprops.disambiguation !== undefined),
+ revisedAt: p.touched || null,
+ }));
+ const fulls = out.filter((a) => !a.disambiguation).slice(0, full);
+ await Promise.all(fulls.map(async (a) => {
+ try { const t = await fullText(a.title, { lang, signal }); if (t) a.text = t; } catch { /* the intro stays */ }
+ }));
+ for (const a of out) a.text = clip(a.text, chars);
+ return out;
+ }
+
+ function imageQuery(extra) {
+ return { prop: 'imageinfo', iiprop: 'url|size|mime|extmetadata', iiurlwidth: '1280', iiextmetadatafilter: META, iiextmetadatalanguage: 'en', maxlag: '5', ...extra };
+ }
+
+ async function articleImages(title, { lang = 'en', limit = 12, signal } = {}) {
+ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', generator: 'images', titles: title, gimlimit: '40', redirects: '1', ...imageQuery() })}`, { signal });
+ const source = `https://${lang}.wikipedia.org/wiki/${encodeURIComponent(title.replace(/ /g, '_'))}`;
+ return ((j.query && j.query.pages) || []).map((p) => imageFrom(p, 'wikipedia', source)).filter(Boolean).slice(0, limit);
+ }
+
+ async function commonsSearch(query, { limit = 10, signal } = {}) {
+ const j = await http.getJson(`${COMMONS}?${qs({ action: 'query', generator: 'search', gsrsearch: `${query} filetype:bitmap`, gsrnamespace: '6', gsrlimit: String(limit), ...imageQuery() })}`, { signal });
+ const pages = ((j.query && j.query.pages) || []).sort((a, b) => (a.index || 0) - (b.index || 0));
+ return pages.map((p) => imageFrom(p, 'commons')).filter((i) => i && i.free);
+ }
+
+ /** Licence and size for files by name on a Wikipedia (its own files and Commons files alike). */
+ async function fileInfo(names, { lang = 'en', allowSvg = false, provider = 'wikipedia', signal } = {}) {
+ if (!names.length) return [];
+ const titles = names.slice(0, 20).map((n) => (/^File:/i.test(n) ? n : `File:${n}`)).join('|');
+ const j = await http.getJson(`${API(lang)}?${qs({ action: 'query', titles, ...imageQuery() })}`, { signal });
+ return ((j.query && j.query.pages) || []).map((p) => imageFrom(p, provider, null, { allowSvg })).filter(Boolean);
+ }
+
+ async function commonsFiles(names, { signal } = {}) {
+ if (!names.length) return [];
+ const titles = names.slice(0, 20).map((n) => (/^File:/i.test(n) ? n : `File:${n}`)).join('|');
+ const j = await http.getJson(`${COMMONS}?${qs({ action: 'query', titles, ...imageQuery() })}`, { signal });
+ return ((j.query && j.query.pages) || []).map((p) => imageFrom(p, 'wikidata')).filter(Boolean);
+ }
+
+ /** Labelled facts from a Wikidata item, plus its image file name (P18) when it has one. */
+ async function wikidataFacts(qid, { lang = 'en', signal } = {}) {
+ if (!/^Q\d{1,12}$/.test(qid || '')) return null;
+ const j = await http.getJson(`${WIKIDATA}?action=wbgetentities&ids=${qid}&props=claims%7Clabels%7Cdescriptions&languages=${lang}%7Cen&format=json`, { signal });
+ const e = j.entities && j.entities[qid];
+ if (!e || !e.claims) return null;
+ const pending = [];
+ for (const [pid, label] of PROPS) {
+ const claims = (e.claims[pid] || []).filter((c) => c.rank !== 'deprecated' && c.mainsnak && c.mainsnak.datavalue).slice(0, 8);
+ if (!claims.length) continue;
+ pending.push({ pid, label, values: claims.map((c) => c.mainsnak.datavalue) });
+ }
+ const ids = new Set();
+ for (const f of pending) for (const dv of f.values) if (dv.type === 'wikibase-entityid' && dv.value && dv.value.id) ids.add(dv.value.id);
+ const labels = new Map();
+ const list = [...ids].slice(0, 50);
+ if (list.length) {
+ const lj = await http.getJson(`${WIKIDATA}?action=wbgetentities&ids=${list.join('%7C')}&props=labels&languages=${lang}%7Cen&format=json`, { signal });
+ for (const [id, ent] of Object.entries(lj.entities || {})) {
+ const l = ent.labels && (ent.labels[lang] || ent.labels.en);
+ if (l) labels.set(id, l.value);
+ }
+ }
+ const facts = [];
+ for (const f of pending) {
+ const vals = f.values.map((dv) => {
+ switch (dv.type) {
+ case 'wikibase-entityid': return labels.get(dv.value.id) || null;
+ case 'time': return timeText(dv.value);
+ case 'quantity': { const n = Number(dv.value.amount); return Number.isFinite(n) ? n.toLocaleString('en-US') : null; }
+ case 'monolingualtext': return dv.value.text || null;
+ case 'string': return typeof dv.value === 'string' ? dv.value : null;
+ default: return null;
+ }
+ }).filter(Boolean);
+ if (vals.length) facts.push({ label: f.label, value: [...new Set(vals)].join(', ').slice(0, 400) });
+ }
+ const image = e.claims.P18 && e.claims.P18[0] && e.claims.P18[0].mainsnak.datavalue ? String(e.claims.P18[0].mainsnak.datavalue.value) : null;
+ const name = (e.labels && (e.labels[lang] || e.labels.en) || {}).value || qid;
+ const description = (e.descriptions && (e.descriptions[lang] || e.descriptions.en) || {}).value || null;
+ return { qid, name, description, url: `https://www.wikidata.org/wiki/${qid}`, facts: facts.slice(0, 30), image };
+ }
+
+ return { searchWikipedia, articles, articleImages, fileInfo, commonsSearch, commonsFiles, wikidataFacts };
+}
+
+module.exports = { createWikimedia, imageFrom, timeText, untracked };
@@ -12,6 +12,8 @@
@@ -12,6 +12,8 @@
* embed embeddings
* embed embeddings
* media_analysis local-first media analysis: FFmpeg signals, scenes, local speech-to-text, highlights and
* media_analysis local-first media analysis: FFmpeg signals, scenes, local speech-to-text, highlights and
* an overview from a local model (or extractive; paid only with a budget). See media-analysis.js.
* an overview from a local model (or extractive; paid only with a budget). See media-analysis.js.
+ * research MegaSource (server/research/): Wikipedia, the topic's Fandom wiki, Wikidata, DeepSeek web search,
+ * Commons and Openverse, merged into numbered sources, licensed pictures and facts. No model call.
*
*
* The final output is validated against the workflow's output schema. An empty or invalid answer
* The final output is validated against the workflow's output schema. An empty or invalid answer
* fails the run (output.empty / output.invalid); nothing is ever filled in. input.sources become
* fails the run (output.empty / output.invalid); nothing is ever filled in. input.sources become
@@ -96,11 +98,13 @@ function normMessages(messages, user) {
@@ -96,11 +98,13 @@ function normMessages(messages, user) {
return out;
return out;
}
}
-function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spawnImpl, log = console }) {
+function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spawnImpl, log = console, research = null }) {
// One loopback reader for everything that streams media (server/media-proxy.js): transcripts and media.analyze.
// One loopback reader for everything that streams media (server/media-proxy.js): transcripts and media.analyze.
let proxy = null;
let proxy = null;
const reader = () => proxy || (proxy = require('../media-proxy').createMediaProxy({ fetcher, log }));
const reader = () => proxy || (proxy = require('../media-proxy').createMediaProxy({ fetcher, log }));
const mediaAnalysis = require('./media-analysis').createMediaAnalysis({ registry, pool, fetcher, quotas, config, spawnImpl, log, reader });
const mediaAnalysis = require('./media-analysis').createMediaAnalysis({ registry, pool, fetcher, quotas, config, spawnImpl, log, reader });
+ let researcher = research;
+ const megasource = () => researcher || (researcher = require('../research').createResearch({ config: config.research || {}, log }));
async function routeFor(key) {
async function routeFor(key) {
const r = await registry.resolveRoute(key);
const r = await registry.resolveRoute(key);
if (!r) throw new AiError(503, 'route.unavailable', `no route ${key}`);
if (!r) throw new AiError(503, 'route.unavailable', `no route ${key}`);
@@ -248,7 +252,29 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
@@ -248,7 +252,29 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
return { output: { images: images.map(i => ({ b64: i.b64 || null, url: i.url || null })), count: images.length }, exec, route };
return { output: { images: images.map(i => ({ b64: i.b64 || null, url: i.url || null })), count: images.length }, exec, route };
}
}
- const STEP = { llm: llmStep, passthrough: passthroughStep, complete: completeStep, transcribe: transcribeStep, embed: embedStep, image: imageStep, media_analysis: mediaAnalysis.step };
+ /** research: no model call. A developer app's run leaves the paid web search out (its DeepSeek calls are ours). */
+ async function researchStep(step, input, wf, run, ctx) {
+ let output;
+ try {
+ output = await megasource().gather(input, { signal: ctx.signal, allowPaid: !ctx.appRun });
+ } catch (err) {
+ if (err && err.status === 422) throw new AiError(422, 'input.invalid', err.message);
+ throw err;
+ }
+ if (!output.sources.length && !output.images.length) throw new AiError(502, 'source.unavailable', `no source answered for "${output.query}"`, { providers: output.providers });
+ // Recorded like a provider call: DeepSeek's per-call price counts against the cost quotas and the daily usage.
+ const exec = { provider: 'research', model: 'megasource', usage: { input: 0, output: 0, cached: 0 }, cost: Number(output.usage && output.usage.cost_usd) || 0, synthetic: false, fallbackUsed: false, explain: null };
+ return { output, exec };
+ }
+
+ /** A research run's grounding: what it read is the output itself; the gaps are the sources that did not answer. */
+ function researchGrounding(output) {
+ const gaps = Object.entries((output && output.providers) || {}).filter(([, p]) => p && !p.ok)
+ .map(([name, p]) => `${name}: ${p.skipped || p.error || 'did not answer'}`.slice(0, 300));
+ return { cited: [], gaps: gaps.slice(0, 20), read: (output && output.sources ? output.sources.length : 0) };
+ }
+
+ const STEP = { llm: llmStep, passthrough: passthroughStep, complete: completeStep, transcribe: transcribeStep, embed: embedStep, image: imageStep, media_analysis: mediaAnalysis.step, research: researchStep };
/**
/**
* Run a workflow. ctx: { signal, logRequest, debugRaw, inputHash }
* Run a workflow. ctx: { signal, logRequest, debugRaw, inputHash }
@@ -286,7 +312,7 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
@@ -286,7 +312,7 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
}));
}));
return {
return {
output: last.output, citations, jsonSchema,
output: last.output, citations, jsonSchema,
- grounding: groundingOf(last.output, sources.length, citeStep),
+ grounding: wf.steps.some((s) => s.kind === 'research') ? researchGrounding(last.output) : groundingOf(last.output, sources.length, citeStep),
synthetic: Boolean(exec && exec.synthetic),
synthetic: Boolean(exec && exec.synthetic),
provider: exec ? exec.provider : null, model: exec ? exec.model : null, fallbackUsed: Boolean(exec && exec.fallbackUsed),
provider: exec ? exec.provider : null, model: exec ? exec.model : null, fallbackUsed: Boolean(exec && exec.fallbackUsed),
usage, cost,
usage, cost,
@@ -295,7 +321,7 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
@@ -295,7 +321,7 @@ function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spa
};
};
}
}
- return { execute, narrowCitations };
+ return { execute, narrowCitations, research: () => megasource() };
}
}
module.exports = { createEngine, narrowCitations, collectCited, collectGaps, groundingOf };
module.exports = { createEngine, narrowCitations, collectCited, collectGaps, groundingOf };
@@ -271,6 +271,24 @@ const PREPARE = {
@@ -271,6 +271,24 @@ const PREPARE = {
sources(input) {
sources(input) {
return { vars: { ...input, sources_block: sourcesBlock(input.sources), source_count: (input.sources || []).length } };
return { vars: { ...input, sources_block: sourcesBlock(input.sources), source_count: (input.sources || []).length } };
},
},
+ // Wiki generation (OpenVibe.Wiki's "vibe a wiki"): the plan's page budget, and a page's link targets and pictures.
+ 'wiki.space'(input) {
+ return { vars: { ...input, max_pages: input.max_pages || 12, sources_block: sourcesBlock(input.sources), source_count: (input.sources || []).length } };
+ },
+ 'wiki.page'(input) {
+ const pages = (Array.isArray(input.pages) ? input.pages : []).map((t) => clean(t, 200)).filter(Boolean).slice(0, 60);
+ const images = (Array.isArray(input.images) ? input.images : []).slice(0, 16);
+ return {
+ vars: {
+ ...input,
+ topic: input.topic || input.space || input.title,
+ pages_block: pages.map((t) => `- ${t}`).join('\n') || '(this is the only page)',
+ images_block: images.map((im, i) => `[${i}] ${clean(im.title, 300)}${im.description ? ` — ${clean(im.description, 400)}` : ''}`).join('\n'),
+ sources_block: sourcesBlock(input.sources),
+ source_count: (input.sources || []).length,
+ },
+ };
+ },
'pics.describe_image'(input) {
'pics.describe_image'(input) {
return { vars: { title: clean(input.title, 120) || '' } };
return { vars: { title: clean(input.title, 120) || '' } };
},
},
@@ -17,6 +17,8 @@ const STR = (n) => ({ type: 'string', maxLength: n });
@@ -17,6 +17,8 @@ const STR = (n) => ({ type: 'string', maxLength: n });
const OBJ = (props, required) => ({ type: 'object', additionalProperties: false, required: required || Object.keys(props), properties: props });
const OBJ = (props, required) => ({ type: 'object', additionalProperties: false, required: required || Object.keys(props), properties: props });
const ARR = (items, max = 20) => ({ type: 'array', items, maxItems: max });
const ARR = (items, max = 20) => ({ type: 'array', items, maxItems: max });
const CLAIM = OBJ({ text: STR(600), citations: CITES });
const CLAIM = OBJ({ text: STR(600), citations: CITES });
+// What a wiki page is about (wiki.generate_space): the wiki's own page groups follow from it.
+const PAGE_KINDS = ['overview', 'character', 'person', 'episode', 'season', 'work', 'place', 'group', 'object', 'concept', 'event', 'production', 'reception', 'list', 'other'];
const RULES = 'You prepare a DRAFT evidence package for OpenVibe editors — never publication truth. Use ONLY the numbered sources provided. Every claim must cite the source numbers it rests on in its `citations` array (integers, e.g. [0, 2]). If the sources do not support something, leave it out and list it in `gaps`. Never invent facts, names, numbers, prices, ratings, dates or quotes. Output only JSON matching the schema.';
const RULES = 'You prepare a DRAFT evidence package for OpenVibe editors — never publication truth. Use ONLY the numbered sources provided. Every claim must cite the source numbers it rests on in its `citations` array (integers, e.g. [0, 2]). If the sources do not support something, leave it out and list it in `gaps`. Never invent facts, names, numbers, prices, ratings, dates or quotes. Output only JSON matching the schema.';
@@ -31,10 +33,10 @@ function productTemplate(key, name, task, route, extraUser = '') {
@@ -31,10 +33,10 @@ function productTemplate(key, name, task, route, extraUser = '') {
}
}
const templates = [
const templates = [
- productTemplate('wiki.generate_space', 'Wiki: propose a space', 'Propose a wiki space for the topic: a title, a one-paragraph summary and a page tree (page titles with short outlines), grounded in the sources.', 'wiki.generate',
- 'Topic: {{topic}}{{#audience}}\nAudience: {{audience}}{{/audience}}\n\n'),
- productTemplate('wiki.generate_page', 'Wiki: draft a page', 'Draft one wiki page: summary, sections and infobox values, each cited.', 'wiki.generate',
- 'Page title: {{title}}{{#space}}\nSpace: {{space}}{{/space}}{{#outline}}\nRequested outline: {{json outline}}{{/outline}}\n\n'),
+ productTemplate('wiki.generate_space', 'Wiki: plan a wiki', 'Plan a wiki about the topic, written like an encyclopedia or a well-kept fan wiki, from the numbered sources. `title`: the wiki\'s name (usually the topic\'s name). `summary`: one paragraph saying what the topic is, cited. `pages`: the main page first, titled exactly as the topic is best known, with kind overview; then the subjects a reader would look up, as many as the sources say enough about, up to {{max_pages}} pages in all: the people or characters, works (episodes, seasons, films, books, games, albums), places, groups, objects, concepts, events, and its production and reception when the sources cover them. Each page: `title` (its usual name; add the topic in brackets only to tell two subjects apart), `kind`, `parent` (the title of another page in this list it belongs under, else an empty string; the main page has none), `outline` (three to six short section headings this page should have), `query` (a short web search phrase that finds this subject, such as "Ron Stoppable Kim Possible"), and `citations`. Only plan subjects the sources name; leave out anything they do not support.', 'wiki.write',
+ 'Topic: {{topic}}{{#audience}}\nAudience: {{audience}}{{/audience}}\nPages at most: {{max_pages}}\n\n'),
+ productTemplate('wiki.generate_page', 'Wiki: write a page', 'Write one page of the wiki from the numbered sources, encyclopedic and engaging, the way a well-kept wiki reads. `summary`: one or two plain sentences saying what the subject is. `lead`: the opening paragraph or two, in Markdown. `sections`: follow the requested outline where the sources support it (skip or merge headings they do not), each with a `heading` and a Markdown `body` of one or more full paragraphs (a list where it reads better), each citing its sources; about 500 to 1500 words in all when the sources allow, shorter when they do not. Link other pages of this wiki, and only titles from the list of pages, as [[Exact Title]] the first time each is mentioned. `infobox`: up to twelve key facts as `label` and `value` (for example First appearance, Voiced by, Created by, Release date, Location), each cited; the Wikidata facts are good material. `images`: up to three pictures from the numbered list of pictures that clearly show this page\'s subject, judged by their title and description (never a picture of something or someone else, and none when no picture fits), each with its `index`, a short factual `caption`, and `section`: the heading it illustrates, or lead for the page\'s main picture. Never write source numbers or "[n]" in the text. Never describe a real person\'s looks, body, clothing or age.', 'wiki.write',
+ 'Page title: {{title}}\nWiki topic: {{topic}}{{#outline}}\nRequested outline: {{json outline}}{{/outline}}\n\nPages of this wiki (link only these, as [[Title]]):\n{{pages_block}}\n\n{{#images_block}}Pictures (by number):\n{{images_block}}\n\n{{/images_block}}'),
productTemplate('blog.draft_post', 'Blog: draft a post', 'Draft a blog post in Markdown for the brief. Claims of fact must cite sources; opinion and framing need none. If there are no sources, write only what needs no facts and list the facts you would need in gaps.', 'blog.draft',
productTemplate('blog.draft_post', 'Blog: draft a post', 'Draft a blog post in Markdown for the brief. Claims of fact must cite sources; opinion and framing need none. If there are no sources, write only what needs no facts and list the facts you would need in gaps.', 'blog.draft',
'Topic: {{topic}}{{#brief}}\nBrief: {{brief}}{{/brief}}{{#tone}}\nTone: {{tone}}{{/tone}}{{#audience}}\nAudience: {{audience}}{{/audience}}\n\n'),
'Topic: {{topic}}{{#brief}}\nBrief: {{brief}}{{/brief}}{{#tone}}\nTone: {{tone}}{{/tone}}{{#audience}}\nAudience: {{audience}}{{/audience}}\n\n'),
productTemplate('blog.write_digest', 'Blog: write a desk digest', 'Write one digest post for an OpenVibe.Blog desk from the numbered sources (the items its sources published in the period). Pick the items a reader of this desk would most want to know about, about twelve at most, and skip routine, minor or duplicate ones. Group them into two to five sections, most important first. Each section is a list of short paragraphs, one story per paragraph, two to four plain sentences saying what happened and why it matters to this desk\'s readers, exactly as the sources state it; each paragraph lists the numbers of the sources it rests on in its `citations`. Never write source numbers, "(Source n)" or brackets in the text itself. Neutral and readable: no opinion stated as fact, no hype, no advice, no predictions. Name prices, dates and numbers only as the sources give them. List in gaps only what a reader would miss (an item you could not make sense of, a missing date or price). title: a headline naming the one or two biggest stories, at most 90 characters, without the words digest, desk, roundup or snapshot and without a date. dek: one sentence about the period.', 'blog.draft',
productTemplate('blog.write_digest', 'Blog: write a desk digest', 'Write one digest post for an OpenVibe.Blog desk from the numbered sources (the items its sources published in the period). Pick the items a reader of this desk would most want to know about, about twelve at most, and skip routine, minor or duplicate ones. Group them into two to five sections, most important first. Each section is a list of short paragraphs, one story per paragraph, two to four plain sentences saying what happened and why it matters to this desk\'s readers, exactly as the sources state it; each paragraph lists the numbers of the sources it rests on in its `citations`. Never write source numbers, "(Source n)" or brackets in the text itself. Neutral and readable: no opinion stated as fact, no hype, no advice, no predictions. Name prices, dates and numbers only as the sources give them. List in gaps only what a reader would miss (an item you could not make sense of, a missing date or price). title: a headline naming the one or two biggest stories, at most 90 characters, without the words digest, desk, roundup or snapshot and without a date. dek: one sentence about the period.', 'blog.draft',
@@ -67,16 +69,31 @@ const templates = [
@@ -67,16 +69,31 @@ const templates = [
const workflows = [
const workflows = [
{
{
- key: 'wiki.generate_space', name: 'Propose a wiki space', namespace: 'wiki',
- input_schema: OBJ({ topic: STR(300), audience: STR(300), sources: SOURCES(1) }, ['topic', 'sources']),
- output_schema: OBJ({ title: STR(200), summary: STR(2000), pages: ARR(OBJ({ title: STR(200), slug_hint: STR(120), outline: ARR(STR(300), 20), citations: CITES }), 40), citations: CITES, gaps: GAPS }),
- default_route: 'wiki.generate',
+ key: 'wiki.generate_space', name: 'Plan a wiki', namespace: 'wiki',
+ input_schema: OBJ({ topic: STR(300), audience: STR(300), max_pages: { type: 'integer', minimum: 1, maximum: 40 }, sources: SOURCES(1) }, ['topic', 'sources']),
+ output_schema: OBJ({
+ title: STR(200), summary: STR(2000),
+ pages: ARR(OBJ({ title: STR(200), kind: { enum: PAGE_KINDS }, parent: STR(200), outline: ARR(STR(120), 8), query: STR(160), citations: CITES }), 40),
+ citations: CITES, gaps: GAPS,
+ }),
+ default_route: 'wiki.write',
+ steps: [{ kind: 'llm', template: 'wiki.generate_space', operation: 'generate', output: 'json', prepare: 'wiki.space', cite: true, params: { max_tokens: 4000, temperature: 0.3, timeout_ms: 150000 } }],
},
},
{
{
- key: 'wiki.generate_page', name: 'Draft a wiki page', namespace: 'wiki',
- input_schema: OBJ({ title: STR(300), space: STR(300), outline: ARR(STR(300), 30), sources: SOURCES(1) }, ['title', 'sources']),
- output_schema: OBJ({ title: STR(300), summary: STR(2000), sections: ARR(OBJ({ heading: STR(200), body: STR(20000), citations: CITES }), 40), infobox: ARR(OBJ({ key: STR(100), value: STR(500), citations: CITES }), 30), citations: CITES, gaps: GAPS }),
- default_route: 'wiki.generate',
+ key: 'wiki.generate_page', name: 'Write a wiki page', namespace: 'wiki',
+ input_schema: OBJ({
+ title: STR(300), topic: STR(300), space: STR(300), outline: ARR(STR(300), 30), pages: ARR(STR(200), 60),
+ images: ARR(OBJ({ title: STR(300), description: STR(400) }, ['title']), 16), sources: SOURCES(1),
+ }, ['title', 'sources']),
+ output_schema: OBJ({
+ title: STR(300), summary: STR(500), lead: STR(6000),
+ sections: ARR(OBJ({ heading: STR(160), body: STR(12000), citations: CITES }), 14),
+ infobox: ARR(OBJ({ label: STR(60), value: STR(300), citations: CITES }), 14),
+ images: ARR(OBJ({ index: { type: 'integer', minimum: 0, maximum: 15 }, caption: STR(300), section: STR(160) }), 3),
+ citations: CITES, gaps: GAPS,
+ }),
+ default_route: 'wiki.write',
+ steps: [{ kind: 'llm', template: 'wiki.generate_page', operation: 'generate', output: 'json', prepare: 'wiki.page', cite: true, params: { max_tokens: 7000, temperature: 0.4, timeout_ms: 170000 } }],
},
},
{
{
key: 'blog.draft_post', name: 'Draft a blog post', namespace: 'blog',
key: 'blog.draft_post', name: 'Draft a blog post', namespace: 'blog',
@@ -215,7 +232,7 @@ const workflows = [
@@ -215,7 +232,7 @@ const workflows = [
].map(w => ({
].map(w => ({
cache_mode: 'private',
cache_mode: 'private',
...w,
...w,
- steps: [{ kind: 'llm', template: w.key, operation: w.key === 'moderation.classify' || w.key === 'rent.screen_listing' ? 'classify' : 'generate', output: 'json', prepare: 'sources', cite: true, params: { max_tokens: 2400, temperature: 0.3 } }],
+ steps: w.steps || [{ kind: 'llm', template: w.key, operation: w.key === 'moderation.classify' || w.key === 'rent.screen_listing' ? 'classify' : 'generate', output: 'json', prepare: 'sources', cite: true, params: { max_tokens: 2400, temperature: 0.3 } }],
metadata: { draft: true, seed: true },
metadata: { draft: true, seed: true },
}));
}));
new file mode 100644
new file mode 100644
@@ -0,0 +1,56 @@
@@ -0,0 +1,56 @@
+'use strict';
+/**
+ * MegaSource (server/research/): one research call over Wikipedia, the topic's Fandom wiki, Wikidata, DeepSeek web
+ * search, Commons and Openverse. No model call and no template: the step reads, ranks and returns the evidence, and the
+ * product workflows (wiki.generate_space, wiki.generate_page, …) cite it by number. Cached for a day per calling
+ * service: the same question asked twice reads the web once.
+ */
+const { SOURCE } = require('./common-schemas');
+
+const STR = (n) => ({ type: 'string', maxLength: n });
+const INT = (lo, hi) => ({ type: 'integer', minimum: lo, maximum: hi });
+const NULLABLE = (n) => ({ type: ['string', 'null'], maxLength: n });
+const PROVIDERS = ['wikipedia', 'fandom', 'wikidata', 'deepseek', 'commons', 'openverse'];
+
+const IMAGE = {
+ type: 'object', additionalProperties: false, required: ['url', 'free', 'provider'],
+ properties: {
+ url: STR(2048), original: NULLABLE(2048), width: { type: ['integer', 'null'] }, height: { type: ['integer', 'null'] }, mime: NULLABLE(40),
+ title: NULLABLE(300), description: NULLABLE(400), author: NULLABLE(200), license: NULLABLE(200), license_url: NULLABLE(2048),
+ free: { type: 'boolean' }, source_url: NULLABLE(2048), provider: STR(80),
+ },
+};
+
+const workflows = [
+ {
+ key: 'research.gather', name: 'Research a topic (MegaSource)', namespace: 'research',
+ description: 'Sources, licensed pictures and facts for a topic from Wikipedia, its Fandom wiki, Wikidata, DeepSeek web search, Commons and Openverse.',
+ input_schema: {
+ type: 'object', additionalProperties: false, required: ['query'],
+ properties: {
+ query: STR(200), topic: STR(200), focus: STR(200), question: STR(200),
+ lang: { type: 'string', pattern: '^[a-z]{2,3}$' },
+ max_sources: INT(1, 20), max_images: INT(0, 24), images: { type: 'boolean' }, content_chars: INT(500, 12000),
+ providers: { type: 'array', maxItems: PROVIDERS.length, items: { enum: PROVIDERS } },
+ },
+ },
+ output_schema: {
+ type: 'object', additionalProperties: false, required: ['query', 'sources', 'images', 'facts', 'providers', 'usage'],
+ properties: {
+ query: STR(200), topic: STR(200), focus: NULLABLE(200), lang: STR(3), retrieved_at: STR(40),
+ sources: { type: 'array', maxItems: 20, items: SOURCE },
+ images: { type: 'array', maxItems: 24, items: IMAGE },
+ facts: { type: 'array', maxItems: 40, items: { type: 'object', additionalProperties: false, required: ['label', 'value'], properties: { label: STR(80), value: STR(400) } } },
+ topic_wiki: { type: ['object', 'null'] },
+ providers: { type: 'object' },
+ usage: { type: 'object' },
+ },
+ },
+ steps: [{ kind: 'research' }],
+ default_route: null,
+ cache_mode: 'service', cache_ttl_sec: 86400,
+ metadata: { seed: true },
+ },
+];
+
+module.exports = { templates: [], workflows, PROVIDERS };
@@ -20,6 +20,7 @@ const media = require('./media');
@@ -20,6 +20,7 @@ const media = require('./media');
const core = require('./core');
const core = require('./core');
const products = require('./products');
const products = require('./products');
const chat = require('./chat');
const chat = require('./chat');
+const research = require('./research');
const { resolveSecret } = require('../util');
const { resolveSecret } = require('../util');
const ROLE_TIMEOUT = { chat: 20000, vision: 30000, director: 25000, summary: 30000, legacy: 30000 };
const ROLE_TIMEOUT = { chat: 20000, vision: 30000, director: 25000, summary: 30000, legacy: 30000 };
@@ -155,15 +156,18 @@ async function seed({ registry, quotas, config, env = process.env, db }) {
@@ -155,15 +156,18 @@ async function seed({ registry, quotas, config, env = process.env, db }) {
// Space's spaces (server/workflows/space.js): the builder writes whole apps, so it gets the creative model and room to
// Space's spaces (server/workflows/space.js): the builder writes whole apps, so it gets the creative model and room to
// finish; a space's own AI is conversation text, an alias of the chat pool.
// finish; a space's own AI is conversation text, an alias of the chat pool.
await registry.seedVersioned('route', 'space.build', pool(180000, { max_output_tokens: 16000, options: { outlast_provider_timeout: true }, ...(creativeModel ? { constraints: { objective: 'correctness', latency_class: 'interactive', model: creativeModel, authority: `shared:${creativeModel}` } } : {}) }));
await registry.seedVersioned('route', 'space.build', pool(180000, { max_output_tokens: 16000, options: { outlast_provider_timeout: true }, ...(creativeModel ? { constraints: { objective: 'correctness', latency_class: 'interactive', model: creativeModel, authority: `shared:${creativeModel}` } } : {}) }));
+ // Wiki generation (wiki.generate_space, wiki.generate_page): long cited pages people read, so the creative model, room
+ // to write (a page is several thousand tokens) and time to finish past the provider's chat timeout.
+ await registry.seedVersioned('route', 'wiki.write', pool(170000, { max_output_tokens: 7000, response_format: 'json', options: { outlast_provider_timeout: true, temperature: 0.4 }, ...(creativeModel ? { constraints: { objective: 'correctness', latency_class: 'interactive', model: creativeModel, authority: `shared:${creativeModel}` } } : {}) }));
await registry.seedVersioned('route', 'space.ask', { primary: { provider: 'shared', model: null }, fallbacks: [], options: {}, max_output_tokens: 800, response_format: 'text', timeout_ms: null, alias_of: 'default.chat' });
await registry.seedVersioned('route', 'space.ask', { primary: { provider: 'shared', model: null }, fallbacks: [], options: {}, max_output_tokens: 800, response_format: 'text', timeout_ms: null, alias_of: 'default.chat' });
await registry.seedVersioned('route', 'chat.member', { primary: { provider: 'shared', model: null }, fallbacks: [], options: {}, max_output_tokens: 600, response_format: 'text', timeout_ms: null, alias_of: 'default.chat' });
await registry.seedVersioned('route', 'chat.member', { primary: { provider: 'shared', model: null }, fallbacks: [], options: {}, max_output_tokens: 600, response_format: 'text', timeout_ms: null, alias_of: 'default.chat' });
// Templates, then workflows (workflows reference templates)
// Templates, then workflows (workflows reference templates)
- for (const t of [...live.templates, ...core.templates, ...products.templates, ...media.templates, ...chat.templates, ...space.templates]) {
+ for (const t of [...live.templates, ...core.templates, ...products.templates, ...media.templates, ...chat.templates, ...space.templates, ...research.templates]) {
const { key, ...def } = t;
const { key, ...def } = t;
await registry.seedVersioned('template', key, { description: null, default_route: null, owner: 'ai', visibility: 'internal', metadata: {}, ...def });
await registry.seedVersioned('template', key, { description: null, default_route: null, owner: 'ai', visibility: 'internal', metadata: {}, ...def });
}
}
- for (const w of [...live.workflows, ...core.workflows, ...products.workflows, ...media.workflows, ...chat.workflows, ...space.workflows]) {
+ for (const w of [...live.workflows, ...core.workflows, ...products.workflows, ...media.workflows, ...chat.workflows, ...space.workflows, ...research.workflows]) {
const { key, ...def } = w;
const { key, ...def } = w;
await registry.seedVersioned('workflow', key, { description: null, default_route: null, cache_ttl_sec: null, metadata: {}, ...def });
await registry.seedVersioned('workflow', key, { description: null, default_route: null, cache_ttl_sec: null, metadata: {}, ...def });
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,64 @@
@@ -0,0 +1,64 @@
+'use strict';
+/**
+ * A canned web for MegaSource tests (no network): Wikipedia, Commons, Wikidata, Fandom, Openverse and DeepSeek answer
+ * from these fixtures by URL. `calls` records every request; `deepseek` scripts DeepSeek's status.
+ */
+const json = (obj, status = 200) => new Response(JSON.stringify(obj), { status, headers: { 'content-type': 'application/json' } });
+const meta = (license, extra = {}) => ({ LicenseShortName: { value: license }, Artist: { value: '<a href="x">Jane Doe</a>' }, ImageDescription: { value: 'The <b>SQLite</b> logo' }, ...extra });
+const file = (title, mime, w, h, license, extra) => ({ title, imageinfo: [{ url: `https://upload.wikimedia.org/wikipedia/commons/a/ab/${encodeURIComponent(title.slice(5))}?utm_source=x`, thumburl: `https://upload.wikimedia.org/thumb/${encodeURIComponent(title.slice(5))}.png`, thumbwidth: Math.min(w, 1280), thumbheight: h, width: w, height: h, mime, descriptionurl: `https://commons.wikimedia.org/wiki/${encodeURIComponent(title)}`, extmetadata: meta(license, extra) }] });
+
+function createResearchFetch({ deepseek = { status: 200 }, fandom = false } = {}) {
+ const calls = [];
+ async function fetchImpl(url, init = {}) {
+ const u = new URL(url);
+ const p = u.searchParams;
+ calls.push({ host: u.hostname, path: u.pathname, params: Object.fromEntries(p), method: init.method || 'GET', body: init.body || null });
+ if (u.hostname.endsWith('.wikipedia.org')) {
+ if (p.get('list') === 'search') return json({ query: { search: [{ title: 'SQLite', snippet: 'an embedded <span>database</span>' }, { title: 'SQL', snippet: 'a language' }, { title: 'SQLite (disambiguation)', snippet: '' }] } });
+ if (p.get('generator') === 'images') return json({ query: { pages: [file('File:SQLite diagram.png', 'image/png', 1600, 900, 'CC BY-SA 4.0'), file('File:SQLite promo.jpg', 'image/jpeg', 800, 600, 'Fair use', { NonFree: { value: 'true' } }), file('File:Symbol category class.svg', 'image/svg+xml', 180, 180, 'Public domain')] } });
+ if (p.get('prop') === 'extracts') return json({ query: { pages: [{ title: p.get('titles'), extract: 'SQLite is a database engine written in C.\n\n== History ==\nD. Richard Hipp designed it in 2000.\n\n== References ==\nA list of links.' }] } });
+ if (p.get('prop') === 'imageinfo') return json({ query: { pages: [file(`File:${p.get('titles').replace(/^File:/, '')}`, 'image/svg+xml', 400, 200, 'Public domain')] } });
+ return json({ query: {
+ normalized: [{ from: 'SQLite (disambiguation)', to: 'SQLite (disambiguation)' }],
+ pages: [
+ { title: 'SQL', fullurl: 'https://en.wikipedia.org/wiki/SQL', extract: 'SQL is a query language.', description: 'Query language', pageprops: { wikibase_item: 'Q47607' } },
+ { title: 'SQLite', fullurl: 'https://en.wikipedia.org/wiki/SQLite', extract: 'SQLite is a database engine.', description: 'Embedded database', pageprops: { wikibase_item: 'Q319417' }, original: { source: 'https://upload.wikimedia.org/wikipedia/commons/3/38/SQLite370.svg', width: 400, height: 200 }, thumbnail: { source: 'https://upload.wikimedia.org/thumb/SQLite370.svg.png', width: 400, height: 200 } },
+ { title: 'SQLite (disambiguation)', fullurl: 'https://en.wikipedia.org/wiki/SQLite_(disambiguation)', extract: 'may refer to', pageprops: { disambiguation: '' } },
+ ],
+ } });
+ }
+ if (u.hostname === 'commons.wikimedia.org') {
+ if (p.get('generator') === 'search') return json({ query: { pages: [{ index: 2, ...file('File:Whistle stop tour.jpg', 'image/jpeg', 1200, 800, 'CC BY 2.0', { ImageDescription: { value: 'A whistle stop campaign tour' } }) }, { index: 1, ...file('File:SQLite logo photo.jpg', 'image/jpeg', 1200, 800, 'CC0') }, { index: 3, ...file('File:SQLite poster.jpg', 'image/jpeg', 1200, 800, 'CC BY-NC 2.0') }] } });
+ return json({ query: { pages: [file('File:SQLite at work.jpg', 'image/jpeg', 2000, 1500, 'CC BY 4.0')] } });
+ }
+ if (u.hostname === 'www.wikidata.org') {
+ if (p.get('ids') === 'Q319417' && /claims/.test(p.get('props'))) return json({ entities: { Q319417: { labels: { en: { value: 'SQLite' } }, descriptions: { en: { value: 'relational database engine' } }, claims: {
+ P178: [{ rank: 'normal', mainsnak: { datavalue: { type: 'wikibase-entityid', value: { id: 'Q92743' } } } }],
+ P571: [{ rank: 'normal', mainsnak: { datavalue: { type: 'time', value: { time: '+2000-08-17T00:00:00Z', precision: 11 } } } }],
+ P18: [{ rank: 'normal', mainsnak: { datavalue: { type: 'string', value: 'SQLite at work.jpg' } } }],
+ } } } });
+ return json({ entities: { Q92743: { labels: { en: { value: 'D. Richard Hipp' } } } } });
+ }
+ if (u.hostname.endsWith('.fandom.com')) {
+ if (!fandom) return json({ error: 'no wiki' }, 404);
+ if (p.get('meta') === 'siteinfo') return json({ query: { general: { sitename: 'SQLite Wiki', lang: 'en' } } });
+ if (p.get('list') === 'search') return json({ query: { search: [{ title: 'WAL mode' }, { title: 'WAL mode/Gallery' }] } });
+ if (p.get('action') === 'parse') return json({ parse: { title: p.get('page'), text: '<aside class="portable-infobox"><h3 class="pi-data-label">Introduced</h3><div class="pi-data-value">2010</div></aside><p>Write-ahead logging keeps readers going.</p><h2>Use</h2><ul><li>Set the journal mode.</li></ul><h2>References</h2><ol><li>link</li></ol>' } });
+ return json({ query: { pages: [{ title: p.get('titles'), original: { source: 'https://static.wikia.nocookie.net/sqlite/images/a/ab/Wal.png/revision/latest?cb=1', width: 600, height: 400 } }] } });
+ }
+ if (u.hostname === 'api.openverse.org') return json({ results: [{ url: 'https://live.staticflickr.com/1/sqlite.jpg', title: 'SQLite sticker', width: 1024, height: 768, license: 'by', license_version: '2.0', creator: 'Sam', foreign_landing_url: 'https://www.flickr.com/photos/1', source: 'flickr' }] });
+ if (u.hostname === 'api.deepseek.com') {
+ if (deepseek.status !== 200) return json({ error: { message: 'no' } }, deepseek.status);
+ if (u.pathname === '/v1/tools/search') return json({ results: [
+ { title: 'SQLite Home Page', url: 'https://sqlite.org/index.html?utm_source=a', snippet: 'Small. Fast. Reliable.', summary: 'SQLite is a C library.', content: 'SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured SQL database engine. '.repeat(6), site_name: 'SQLite', date: '2026-01-01T00:00:00Z', authority: 4, rerank_score: 2.8 },
+ { title: 'SQLite — Wikipedia mirror', url: 'https://en.m.wikipedia.org/wiki/SQLite', snippet: 'dup', summary: 'dup', content: 'duplicate of the encyclopedia page', authority: 3, rerank_score: 2.0 },
+ { title: 'A thin page', url: 'https://example.com/thin', snippet: 'short', summary: '', content: '', authority: 1, rerank_score: 1.2 },
+ ] });
+ if (u.pathname === '/v1/tools/fetch') return json({ title: 'A thin page', content: 'The full text DeepSeek fetched for the thin page.' });
+ }
+ return json({ error: 'not in the fixture web' }, 404);
+ }
+ return { fetchImpl, calls };
+}
+
+module.exports = { createResearchFetch };
@@ -45,7 +45,7 @@ function tmpDir() {
@@ -45,7 +45,7 @@ function tmpDir() {
// A boot on a dir used before is a restart on the same database: its database stays open.
// A boot on a dir used before is a restart on the same database: its database stays open.
const restartable = new Map();
const restartable = new Map();
-async function boot({ env = {}, dir, clock, fetchImpl, credentialFetch } = {}) {
+async function boot({ env = {}, dir, clock, fetchImpl, credentialFetch, research } = {}) {
const reuse = dir !== undefined;
const reuse = dir !== undefined;
if (dir === undefined) dir = tmpDir();
if (dir === undefined) dir = tmpDir();
const fullEnv = {
const fullEnv = {
@@ -57,7 +57,9 @@ async function boot({ env = {}, dir, clock, fetchImpl, credentialFetch } = {}) {
@@ -57,7 +57,9 @@ async function boot({ env = {}, dir, clock, fetchImpl, credentialFetch } = {}) {
// One database per boot (PGlite, or AI_TEST_STORE=pg: the containers), dropped when the boot stops.
// One database per boot (PGlite, or AI_TEST_STORE=pg: the containers), dropped when the boot stops.
const testdb = (reuse && restartable.get(dir)) || await require('./db').testDb();
const testdb = (reuse && restartable.get(dir)) || await require('./db').testDb();
if (reuse) restartable.set(dir, testdb);
if (reuse) restartable.set(dir, testdb);
- const h = await start({ config, db: testdb.db, env: fullEnv, log: silent, clock, fetchImpl, credentialFetch });
+ // Research reads a canned web (test/fixtures/research-fetch.js) unless a test brings its own: no test reaches the internet.
+ const researcher = research || require('../server/research').createResearch({ config: config.research, env: {}, fetchImpl: require('./fixtures/research-fetch').createResearchFetch().fetchImpl, log: silent });
+ const h = await start({ config, db: testdb.db, env: fullEnv, log: silent, clock, fetchImpl, credentialFetch, research: researcher });
const base = `http://127.0.0.1:${h.server.address().port}`;
const base = `http://127.0.0.1:${h.server.address().port}`;
return { ...h, base, dir, env: fullEnv, testdb, stop: async () => { await h.close(); if (!reuse) await testdb.close(); } };
return { ...h, base, dir, env: fullEnv, testdb, stop: async () => { await h.close(); if (!reuse) await testdb.close(); } };
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,141 @@
@@ -0,0 +1,141 @@
+'use strict';
+// MegaSource (server/research/): every provider merged into one ranked, de-duplicated pack against a canned web; pictures
+// carry their licences and library results must name the subject; DeepSeek is used only with a key, only for
+// first-party callers, and stands down on 402; the research.gather workflow and POST /api/v1/research answer.
+
+const assert = require('assert');
+const { createResearch } = require('../server/research');
+const { articleText, extractText, canonicalUrl, isFreeLicense, clip } = require('../server/research/text');
+const { slugsFor } = require('../server/research/fandom');
+const { createResearchFetch } = require('./fixtures/research-fetch');
+const { boot, request, token, suite } = require('./helpers');
+
+const t = suite('research');
+const quiet = { log() {}, warn() {}, error() {} };
+
+t.test('text helpers: article HTML, extracts, URLs, licences', () => {
+ const text = articleText('<aside class="portable-infobox"><h3 class="pi-data-label">Voiced by</h3><div class="pi-data-value">Will Friedle<br>someone</div></aside><p>Ron is <b>loyal</b>.<sup class="reference">[1]</sup></p><h2>Personality</h2><ul><li>Brave</li></ul><h2>References</h2><ol><li>x</li></ol><script>evil()</script>');
+ assert.match(text, /^Voiced by: Will Friedle, someone$/m);
+ assert.match(text, /Ron is loyal\./);
+ assert.match(text, /^## Personality$/m);
+ assert.match(text, /^- Brave$/m);
+ assert.ok(!/References|evil|\[1\]/.test(text), 'references, scripts and markers are gone');
+ assert.strictEqual(extractText('Intro.\n\n== History ==\nOld.\n\n== References ==\nlinks'), 'Intro.\n\n## History\nOld.');
+ assert.strictEqual(canonicalUrl('http://en.m.wikipedia.org/wiki/SQLite?utm_source=x#top'), 'https://en.wikipedia.org/wiki/SQLite');
+ assert.strictEqual(canonicalUrl('javascript:alert(1)'), null);
+ for (const free of ['CC BY-SA 4.0', 'CC0', 'Public domain', 'CC BY 2.0', 'GFDL']) assert.ok(isFreeLicense(free), free);
+ for (const not of ['CC BY-NC 2.0', 'CC BY-ND 4.0', 'Fair use', '', null, 'All rights reserved']) assert.ok(!isFreeLicense(not), String(not));
+ assert.ok(clip('One. Two. Three. Four.', 12).endsWith('.'));
+ assert.deepStrictEqual(slugsFor('The Legend of Zelda'), ['thelegendofzelda', 'legendzelda', 'legend']);
+ assert.deepStrictEqual(slugsFor('Kim Possible'), ['kimpossible']);
+});
+
+t.test('gather: free sources only without a key, ranked and de-duplicated, pictures with licences', async () => {
+ const web = createResearchFetch();
+ const r = createResearch({ config: {}, env: {}, fetchImpl: web.fetchImpl, log: quiet });
+ const out = await r.gather({ query: 'SQLite', max_images: 10 });
+ assert.strictEqual(out.providers.deepseek.skipped, 'not configured');
+ assert.ok(!web.calls.some((c) => c.host === 'api.deepseek.com'), 'no paid call without a key');
+ assert.strictEqual(out.sources[0].url, 'https://en.wikipedia.org/wiki/SQLite', 'the exact article leads');
+ assert.match(out.sources[0].content, /## History/);
+ assert.ok(!/References/.test(out.sources[0].content));
+ assert.ok(!out.sources.some((s) => /disambiguation/.test(s.url)), 'disambiguation pages are left out');
+ const wd = out.sources.find((s) => s.source_type === 'wikidata');
+ assert.match(wd.content, /Developer: D\. Richard Hipp/);
+ assert.match(wd.content, /Founded \/ created: 2000-08-17/);
+ assert.deepStrictEqual(out.facts.map((f) => f.label), ['Developer', 'Founded / created']);
+ for (const s of out.sources) { assert.ok(s.retrieved_at && s.url && s.title); assert.ok(s.provenance.provider); }
+ const byTitle = Object.fromEntries(out.images.map((i) => [i.title, i]));
+ assert.strictEqual(out.images[0].provider, 'wikidata', 'the item\'s own picture first');
+ assert.strictEqual(byTitle['SQLite diagram'].free, true);
+ assert.strictEqual(byTitle['SQLite diagram'].license, 'CC BY-SA 4.0');
+ assert.strictEqual(byTitle['SQLite diagram'].author, 'Jane Doe', 'credit is plain text');
+ assert.ok(!/utm_/.test(byTitle['SQLite diagram'].original), 'tracking query removed');
+ assert.strictEqual(byTitle['SQLite promo'].free, false, 'an article\'s non-free picture is marked');
+ assert.ok(!out.images.some((i) => /Symbol category/.test(i.title)), 'interface icons are left out');
+ assert.ok(byTitle['SQLite logo photo'], 'a library picture naming the subject is kept');
+ assert.ok(!out.images.some((i) => /Whistle/.test(i.title)), 'a library picture of something else is left out');
+ assert.ok(!out.images.some((i) => /poster/.test(i.title)), 'NonCommercial licences are not free');
+ assert.ok(out.images.some((i) => i.provider === 'openverse/flickr'));
+ assert.strictEqual(out.topic_wiki, null);
+ assert.strictEqual(out.usage.cost_usd, 0);
+});
+
+t.test('gather: the topic\'s Fandom wiki ranks first for a focus page; DeepSeek joins with a key', async () => {
+ const web = createResearchFetch({ fandom: true });
+ const r = createResearch({ config: { deepseekKey: 'sk-test', deepseekDaily: 10 }, env: {}, fetchImpl: web.fetchImpl, log: quiet });
+ const out = await r.gather({ query: 'WAL mode SQLite', topic: 'SQLite', focus: 'WAL mode', max_images: 6 });
+ assert.strictEqual(out.sources[0].source_type, 'fandom');
+ assert.strictEqual(out.sources[0].url, 'https://sqlite.fandom.com/wiki/WAL_mode');
+ assert.match(out.sources[0].content, /^Introduced: 2010$/m);
+ assert.deepStrictEqual(out.topic_wiki, { name: 'SQLite Wiki', url: 'https://sqlite.fandom.com' });
+ assert.ok(!web.calls.some((c) => c.params.page === 'WAL mode/Gallery'), 'gallery subpages are not read');
+ const urls = out.sources.map((s) => s.url);
+ assert.ok(urls.includes('https://sqlite.org/index.html?utm_source=a'), 'a web result');
+ assert.ok(!urls.includes('https://en.m.wikipedia.org/wiki/SQLite'), 'a mirror of a page already read is dropped');
+ const thin = out.sources.find((s) => s.url === 'https://example.com/thin');
+ assert.match(thin.content, /full text DeepSeek fetched/, 'a thin result is read in full through Fetch');
+ assert.strictEqual(out.usage.deepseek_search, 1);
+ assert.strictEqual(out.usage.deepseek_fetch, 1);
+ assert.strictEqual(out.usage.cost_usd, 0.0015);
+ const ds = web.calls.find((c) => c.path === '/v1/tools/search');
+ assert.strictEqual(JSON.parse(ds.body).need_content, true);
+ const fan = out.images.find((i) => i.provider === 'fandom');
+ assert.strictEqual(fan.free, false);
+ assert.match(fan.url, /scale-to-width-down\/1024/);
+ // A developer app's run reads only the free sources.
+ const before = web.calls.length;
+ const app = await r.gather({ query: 'SQLite' }, { allowPaid: false });
+ assert.strictEqual(app.providers.deepseek.skipped, 'first-party callers only');
+ assert.ok(!web.calls.slice(before).some((c) => c.host === 'api.deepseek.com'));
+});
+
+t.test('gather: DeepSeek stands down on 402 and the free sources still answer', async () => {
+ const web = createResearchFetch({ deepseek: { status: 402 } });
+ const r = createResearch({ config: { deepseekKey: 'sk-test' }, env: {}, fetchImpl: web.fetchImpl, log: quiet });
+ const one = await r.gather({ query: 'SQLite' });
+ assert.strictEqual(one.providers.deepseek.ok, false);
+ assert.match(one.providers.deepseek.error, /balance/);
+ assert.ok(one.sources.length >= 2);
+ const two = await r.gather({ query: 'SQLite 2' });
+ assert.strictEqual(two.providers.deepseek.skipped, 'balance used up', 'no second paid call while standing down');
+ assert.strictEqual(web.calls.filter((c) => c.host === 'api.deepseek.com').length, 1);
+ assert.strictEqual(r.status().deepseek.reason, 'balance used up');
+});
+
+t.test('gather: the key comes from the environment by name, and a daily cap', async () => {
+ const web = createResearchFetch();
+ const r = createResearch({ config: { deepseekKeyRef: 'env:MY_DS', deepseekDaily: 1 }, env: { MY_DS: 'sk-env' }, fetchImpl: web.fetchImpl, log: quiet });
+ await r.gather({ query: 'SQLite' });
+ const auth = web.calls.find((c) => c.host === 'api.deepseek.com');
+ assert.ok(auth, 'the key resolved');
+ const again = await r.gather({ query: 'SQLite again' });
+ assert.strictEqual(again.providers.deepseek.skipped, 'daily cap reached');
+ const off = createResearch({ config: { deepseek: false, deepseekKeyRef: 'env:MY_DS' }, env: { MY_DS: 'sk-env' }, fetchImpl: web.fetchImpl, log: quiet });
+ assert.strictEqual(off.status().deepseek.configured, false);
+ await assert.rejects(r.gather({ query: ' ' }), /needs a query/);
+});
+
+t.test('research.gather runs as a workflow and through POST /api/v1/research', async () => {
+ const h = await boot();
+ try {
+ const tok = token('wiki', ['ai.run.create'], { ns: ['research.*'] });
+ const res = await request(h.base, 'POST', '/api/v1/research?wait=10000', { tok, body: { query: 'SQLite', max_images: 3 } });
+ assert.strictEqual(res.status, 201, JSON.stringify(res.body));
+ const run = res.body.run;
+ assert.strictEqual(run.status, 'succeeded');
+ assert.strictEqual(run.provenance.workflow, 'research.gather');
+ assert.ok(run.output.sources.length >= 2);
+ assert.ok(run.output.images.length <= 3);
+ // Cached per calling service for a day: the second ask reads nothing new.
+ const again = await request(h.base, 'POST', '/api/v1/research?wait=10000', { tok, body: { query: 'SQLite', max_images: 3 } });
+ assert.ok([200, 201].includes(again.status));
+ assert.strictEqual(again.body.run.status, 'cached');
+ const denied = await request(h.base, 'POST', '/api/v1/research', { tok: token('games', ['ai.run.create'], { ns: ['games.*'] }), body: { query: 'SQLite' } });
+ assert.strictEqual(denied.status, 403);
+ const bad = await request(h.base, 'POST', '/api/v1/research', { tok, body: { query: 'x', providers: ['google'] } });
+ assert.strictEqual(bad.status, 422);
+ } finally { await h.stop(); }
+});
+
+t.run();
@@ -34,8 +34,9 @@ const EXAMPLES = {
@@ -34,8 +34,9 @@ const EXAMPLES = {
'ai.enrich': { record: { name: 'Widget' }, instructions: 'normalise the name' },
'ai.enrich': { record: { name: 'Widget' }, instructions: 'normalise the name' },
'ai.embed': { input: ['one', 'two'] },
'ai.embed': { input: ['one', 'two'] },
'ai.image': { prompt: 'A goose eating croutons, watercolour.', size: '1024x1024', n: 1 },
'ai.image': { prompt: 'A goose eating croutons, watercolour.', size: '1024x1024', n: 1 },
- 'wiki.generate_space': { topic: 'SQLite', sources: SRC(2) },
- 'wiki.generate_page': { title: 'WAL mode', sources: SRC(3) },
+ 'wiki.generate_space': { topic: 'SQLite', max_pages: 6, sources: SRC(2) },
+ 'wiki.generate_page': { title: 'WAL mode', topic: 'SQLite', pages: ['SQLite', 'WAL mode', 'Journal'], images: [{ title: 'WAL diagram', description: 'How WAL writes' }], sources: SRC(3) },
+ 'research.gather': { query: 'SQLite', max_images: 4 },
'blog.draft_post': { topic: 'Why we self-host', sources: SRC(1) },
'blog.draft_post': { topic: 'Why we self-host', sources: SRC(1) },
'blog.write_digest': { desk: 'News digest', about: 'Government and space agency announcements', period: '14 October 2026', sources: SRC(3) },
'blog.write_digest': { desk: 'News digest', about: 'Government and space agency announcements', period: '14 October 2026', sources: SRC(3) },
'blog.write_satire': { topic: 'Is it safe to let your air fryer vote?', angle: 'a consumer safety explainer' },
'blog.write_satire': { topic: 'Is it safe to let your air fryer vote?', angle: 'a consumer safety explainer' },
@@ -136,11 +137,14 @@ t.test('every workflow runs end to end on the stub and returns schema-valid outp
@@ -136,11 +137,14 @@ t.test('every workflow runs end to end on the stub and returns schema-valid outp
assert.strictEqual(r.status, 201, `${w.key}: ${r.status} ${r.text.slice(0, 300)}`);
assert.strictEqual(r.status, 201, `${w.key}: ${r.status} ${r.text.slice(0, 300)}`);
const run = r.body.run;
const run = r.body.run;
assert.strictEqual(run.status, 'succeeded', `${w.key}: ${JSON.stringify(run.error)}`);
assert.strictEqual(run.status, 'succeeded', `${w.key}: ${JSON.stringify(run.error)}`);
- assert.strictEqual(run.synthetic, true, `${w.key} should be marked synthetic`);
+ // Research calls no model: it reads (here, the canned web of test/fixtures/research-fetch.js) and is never synthetic.
+ const reads = w.steps.some((s) => s.kind === 'research');
+ assert.strictEqual(run.synthetic, !reads, `${w.key} should ${reads ? 'not ' : ''}be marked synthetic`);
assert.strictEqual(run.provenance.origin, 'ai');
assert.strictEqual(run.provenance.origin, 'ai');
assert.strictEqual(run.provenance.workflow, w.key);
assert.strictEqual(run.provenance.workflow, w.key);
assert.ok(run.provenance.model, `${w.key} has a model`);
assert.ok(run.provenance.model, `${w.key} has a model`);
assert.ok(schemas.validate(w.output_schema, run.output).valid, `${w.key} output invalid`);
assert.ok(schemas.validate(w.output_schema, run.output).valid, `${w.key} output invalid`);
+ if (reads) { assert.ok(run.output.sources.length && Array.isArray(run.grounding.gaps)); continue; }
// WS-O task 3: every output says what backs it: cited sources, or explicit gaps.
// WS-O task 3: every output says what backs it: cited sources, or explicit gaps.
assert.ok(run.grounding && Array.isArray(run.grounding.cited) && Array.isArray(run.grounding.gaps), `${w.key} has grounding`);
assert.ok(run.grounding && Array.isArray(run.grounding.cited) && Array.isArray(run.grounding.gaps), `${w.key} has grounding`);
assert.ok(run.grounding.cited.length || run.grounding.gaps.length, `${w.key}: citations or explicit gaps`);
assert.ok(run.grounding.cited.length || run.grounding.gaps.length, `${w.key}: citations or explicit gaps`);