openvibers/openvibe.search Commits
Search ranks titles above owners and drops repeats (#176)
Changes
@@@ -98,21 +98,32 @@ an empty shelf is left out, and without the numbers the sources fall back to the### Ranking--Full-text results are ordered by full-text relevance (title ×10, summary ×4, body ×1) multiplied by a--freshness boost:++Full-text results are ordered by a tier first, then full-text relevance (title ×10, summary ×4, body ×1)++multiplied by a freshness boost:```--score = relevance × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)++tier = site × 10 + field field: title 3 > summary 2 > body 1++score = tier × 100 + relevance × (1 + W × 2^(−age_days / H))++ W = SEARCH_FRESHNESS_WEIGHT (default 1)H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)```++A query naming a site ("openvibe live", "openvibe tools") puts that site's own records first: the site is++the host `openvibe.<word>` (subdomains too) or a help article under `/a/<host>/`, with Live's old++`openre.stream` name included. A word that matches a document's owner or type is not a text match (the++`owner=` filter is how an owner narrows a search), a title match always outranks a summary-only match, and++a summary-only match outranks a body-only one. Within a tier, a thin document (a username or a very short++title with no real summary: the recorded-stream VODs) ranks below one with a real title and summary, and++one page keeps one result per title and owner.++`age_days` counts from `published_at`, else `updated_at` (an edit does not make an old documentnew); a future date counts as today and a missing one gets no boost. With the defaults a documentpublished today counts ×2.0, 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125 and a year old ×1.0002:relevance still decides, and between comparably relevant documents the newer one ranks first.`W = 0` turns it off. Suggestions use the same score; browsing without `q` stays newest first. A--cursor carries its first page's clock, so later pages rank on the same scores--(`test/freshness.test.js`).++cursor carries its first page's clock, so later pages rank on the same scores. The hybrid path fuses the++keyword and vector top 100 by RRF and then applies the same tiers to the fused list++(`test/freshness.test.js`, `test/ranking.test.js`).### Saved searches@@@ -43,9 +43,11 @@ There is no corpus to measure: no product indexes anything today.- One database: backups, restores and moves are the same as every other service (PostgreSQL,ADR-035).--- Ranking is `ts_rank` with title/summary/body weights 10/4/1 times a freshness boost-- ([README "Ranking"](../README.md#ranking)), returned as minus the score so ascending keyset-- cursors work.++- Ranking is a tier (site 10 + title 3 / summary 2 / body 1) ahead of `ts_rank` with++ title/summary/body weights 10/4/1 times a freshness boost ([README "Ranking"](../README.md#ranking)),++ returned as minus the score so ascending keyset cursors work. The owner and type are not text++ fields, thin username-titled VODs sink inside a tier, and one page keeps one result per title and++ owner.- Text is folded (diacritics removed) and split into runs of letters and digits before the`simple` configuration sees it, so tokens are predictable across languages (Japanese/Chinese getno word segmentation; a later engine or ICU tokenizer fixes that) and a query and the index@@@ -29,6 +29,7 @@ const { AuthError } = require('../auth');const { OWNER_RE, TYPE_RE, ID_RE } = require('./documents');const { fuseRrf } = require('../fabric/rrf');const { createQueryEmbeddingCache } = require('../fabric/embed');++const { docTier, isThin, dedupeKey, siteHosts } = require('../ranking');const FACET_KEY_RE = /^[a-z][a-z0-9_]{0,39}$/;const LANG_RE = /^[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*$/;@@@ -162,6 +163,29 @@ async function dealsOrder(fused, { match, store }) {.map(({ hit }) => ({ ...hit, found: rows.get(hit.rid) }));}++// ── A plain query's order (11-01) ──────────────────────────────────────────────────────────────++// The keyword engine already ranks with the same tiers (server/engine/pg.js), but a hybrid page is++// the RRF order of two top-100 lists, and a strong vector hit could still lift a summary-only VOD++// above a help article whose title says what was asked. Re-apply the tiers to the fused list, so:++// - a query naming a site ("openvibe live") puts that site's own records first (site 10);++// - a title match (3) outranks a summary-only match (2), which outranks a body-only one (1);++// - a thin username-titled VOD ranks below a real title+summary inside the same tier.++// The fused order is the final tie-break, so relevance still decides among equals.++async function rankOrder(fused, { match, store }) {++ if (!match || !match.words.length) return fused;++ const hosts = siteHosts(match);++ const rows = new Map();++ for (const hit of fused) rows.set(hit.rid, hit.found !== undefined ? hit.found : await store.byRid(hit.rid));++ return fused++ .filter((hit) => rows.get(hit.rid))++ .map((hit, i) => {++ const found = rows.get(hit.rid);++ return { hit, found, i, tier: docTier(found.doc, match, hosts), thin: isThin(found.doc) };++ })++ .sort((a, b) => b.tier - a.tier || (a.thin === b.thin ? 0 : a.thin ? 1 : -1) || a.i - b.i)++ .map(({ hit, found }) => ({ ...hit, found }));++}++/*** One search as a viewer: the query API, saved-search runs and the HTML page all go through here.* run({ text, query, viewer, limit, cursor, facetKeys, now, since }) → { results, next_cursor, explain, facets? }@@@ -240,6 +264,7 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings}const results = [];++ const seen = new Set();let next = null;let explain;let refNow = now;@@@ -249,6 +274,8 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddingsif (after && after.offset === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');if (after && after.now != null) refNow = after.now;let fused = await hybridCandidates({ match, filters, viewer, now: refNow, vector: state.vector });++ // A plain query's tiers: site first, then title > summary > body, thin VODs last within a tier.++ fused = await rankOrder(fused, { match, store });// Deals: the product the person asked for (its title or category) beats a description-only match.if (filters.owner === 'deals') fused = await dealsOrder(fused, { match, store });let consumed = after ? after.offset : 0;@@@ -260,6 +287,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings// Defence in depth: the vector store is public-only, the keyword SQL is ACL-aware.const found = f.found !== undefined ? f.found : await store.byRid(f.rid);if (!found || !canView(viewer, found.row, found.doc)) continue;++ const key = dedupeKey(found.doc);++ if (seen.has(key)) continue;++ seen.add(key);results.push(projection(found.doc, found.row, f.snip, f.ranks));}if (consumed < fused.length) next = encodeCursor({ k: 'f', o: consumed, i: lastRid, n: refNow, h: hash });@@@ -274,6 +304,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddingsconst found = await store.byRid(h.rid);// Defence in depth: the engine already filtered; re-check against the stored row.if (!found || !canView(viewer, found.row, found.doc)) continue;++ const key = dedupeKey(found.doc);++ if (seen.has(key)) continue;++ seen.add(key);results.push(projection(found.doc, found.row, h.snip, { keyword: position, vector: null }));}next = page.next ? encodeCursor({ k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash }) : null;@@@ -16,10 +16,13 @@* Text is folded before it is indexed or queried (NFKD, combining marks removed), split into runs of letters and* digits and tokenized by the `simple` configuration (lower case, no stemming, no stop words): the index and every* query see the same text. Title, summary and body weigh 10, 4 and 1 (tsvector weights A, B, C), with the freshness-- * boost described below. Every function here is async; put and remove run inside the store's-- * transaction (ambient). Nothing outside this file writes SQL against search_fts.++ * boost described below. The owner and type are not text fields (a word that equals an owner is not a match), and a++ * ranking tier puts a site's own records first, then title matches, then summary-only, then body-only ones++ * (server/ranking.js). Every function here is async; put and remove run inside the store's transaction (ambient).++ * Nothing outside this file writes SQL against search_fts.*/const { EXPOSURE } = require('../document');++const { fold, words, siteOnlyQuery, siteUrlPattern } = require('../ranking');const CONFIG = 'simple';const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: ts_rank weights 1 / 4 / 10@@@ -67,20 +70,6 @@ function freshnessBoost(iso, nowMs, halfLifeDays, weight) {return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);}--/** Diacritics folded, so the index and every query see the same text. */--function fold(s) {-- return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');--}----/**-- * Word tokens: runs of letters and digits, everything else a separator. PostgreSQL's parser would-- * keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first-- * makes "openvibe" find "OpenVibe.Live".-- */--function words(s) {-- return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');--}--function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {const st = {del: db.prepare('DELETE FROM search_fts WHERE rid = ?'),@@@ -130,31 +119,66 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})}/**-- * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the-- * freshness boost at reference time `now` (ms), so lower ranks first and pages of one query-- * (which carry their first page's `now` in the cursor) rank against the same clock.++ * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus the++ * tier (site 10 + title 3 / summary 2 / body 1) and ts_rank times the freshness boost at++ * reference time `now` (ms), so lower ranks first and pages of one query (which carry their++ * first page's `now` in the cursor) rank against the same clock. The owner and type are not in++ * the tsvector: a word that only equals an owner never matches. Rows whose owner and title are++ * the same keep only the best-ranked one.*/function matchedSet({ match, filters, viewer, now = Date.now() }, params) {params.match = tsquery(match);params.r_now = now;params.r_half = freshness.halfLifeDays;params.r_weight = freshness.weight;++ params.site_re = siteUrlPattern(match) || '';const where = filterClause(filters, params);-- const rank = `-(ts_rank('${WEIGHTS}'::float4[], f.tsv, to_tsquery('${CONFIG}', @match), 1)++ const q = `to_tsquery('${CONFIG}', @match)`;++ const base = `ts_rank('${WEIGHTS}'::float4[], f.tsv, ${q}, 1)* CASE WHEN f.fresh_at IS NULL OR @r_weight::float8 = 0 THEN 1-- ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END)`;-- const branches = [`SELECT d.rid AS rid, ${rank} AS rank++ ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END`;++ const title = `ts_rank('{0,0,0,1}'::float4[], f.tsv, ${q}, 0)`;++ const summary = `ts_rank('{0,0,0.4,0}'::float4[], f.tsv, ${q}, 0)`;++ const body = `ts_rank('{0,0.1,0,0}'::float4[], f.tsv, ${q}, 0)`;++ const site = `(@site_re::text <> '' AND d.canonical_url ~* @site_re)`;++ const field = `CASE WHEN ${title} > 0 THEN 3 WHEN ${summary} > 0 THEN 2 WHEN ${body} > 0 THEN 1 ELSE 0 END`;++ const tier = `(CASE WHEN ${site} THEN 10 ELSE 0 END + ${field})`;++ const thin = `(char_length(btrim(f.title)) <= 24 AND char_length(btrim(f.summary)) <= 80)`;++ // Thin VODs sink below documents with a real title and summary inside the same tier; -2 is++ // larger than any base score (ts_rank × freshness < 2), so it decides before relevance does.++ const rank = `-(${tier} * 100 + (${base}) + CASE WHEN ${thin} THEN -2 ELSE 0 END)`;++ const select = `SELECT d.rid AS rid, d.owner AS owner, f.title AS title, ${rank} AS rank`;++ const branches = [`${select}FROM search_fts f JOIN documents d ON d.rid = f.rid-- WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ to_tsquery('${CONFIG}', @match)++ WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ ${q}AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];const acl = aclClause(viewer, params);if (acl) {-- branches.push(`SELECT d.rid AS rid, ${rank} AS rank++ branches.push(`${select}FROM search_fts f JOIN documents d ON d.rid = f.rid-- WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ to_tsquery('${CONFIG}', @match)++ WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ ${q}AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);}-- return branches.join('\nUNION ALL\n');++ // "openvibe live"/"openvibe tools": the site's own records are candidates even when their text++ // does not repeat the site name (the Tools pages never do). They rank in the site tier below the++ // site records that do match the words; the window dedupe folds a document that is in both sets.++ if (params.site_re && siteOnlyQuery(match)) {++ branches.push(`${select}++ FROM search_fts f JOIN documents d ON d.rid = f.rid++ WHERE f.audience = ${EXPOSURE.public_listed} AND d.canonical_url ~* @site_re++ AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`);++ if (acl) {++ branches.push(`${select}++ FROM search_fts f JOIN documents d ON d.rid = f.rid++ WHERE f.audience = ${EXPOSURE.restricted} AND d.canonical_url ~* @site_re++ AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);++ }++ }++ return `SELECT rid, rank FROM (++ SELECT m.rid, m.rank,++ ROW_NUMBER() OVER (PARTITION BY m.owner, lower(btrim(m.title)) ORDER BY m.rank, m.rid) AS rn++ FROM (${branches.join('\nUNION ALL\n')}) m++ ) x WHERE x.rn = 1`;}/** The visible set without a text query (browse). */nnew file mode 100644@@@ -0,0 +1,161 @@++'use strict';++/**++ * Ranking helpers shared by the full-text engine (server/engine/pg.js) and the query path++ * (server/api/query.js): which fields a query matches, whether a document belongs to a site the++ * query names, whether its title is a thin streamer name, and the identity that drops duplicate++ * results from a page.++ *++ * The engine's base score is ts_rank with title A (×10), summary B (×4) and body C (×1), times the++ * freshness boost. That ranks term frequency, but not the things a person means by a plain query:++ *++ * tier = site (10) + field (title 3, summary 2, body 1, none 0)++ *++ * A query naming a site ("openvibe live", "openvibe tools") puts that site's own records first++ * (its home, its help articles, its tool pages), a title match always outranks a summary-only++ * match, and a summary-only match outranks a body-only one. The owner and type are deliberately++ * not text fields: a word that equals a document's owner is not a match (the explicit owner=++ * filter is how an owner narrows a query), and the summary's service boilerplate ("Recorded live++ * stream by … on OpenVibe.Live.") is only summary text, never a title.++ */++++/** Sites whose old name still appears in canonical URLs and help paths. */++const SITE_ALIASES = {++ live: ['openvibe.live', 'openre.stream'],++ openre: ['openre.stream', 'openvibe.live'],++ stream: ['openre.stream', 'openvibe.live'],++ restream: ['openre.stream', 'openvibe.live'],++};++++/** Diacritics folded, so the index and every query see the same text. */++function fold(s) {++ return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');++}++++/** Word tokens: runs of letters and digits, everything else a separator. */++function words(s) {++ return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');++}++++function tokens(s) {++ return fold(s).toLowerCase().match(/[\p{L}\p{N}]+/gu) || [];++}++++function escapeRegex(s) {++ return String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');++}++++/**++ * The site hosts a query names, if it has "openvibe" plus another word: "openvibe live" names++ * openvibe.live and its old name openre.stream; "openvibe tools" names openvibe.tools. The host of++ * an unknown word simply matches nothing. No site without "openvibe" (a plain "live" is text, not++ * the Live site).++ */++function siteHosts(match) {++ const ws = match && Array.isArray(match.words) ? match.words : [];++ if (!ws.includes('openvibe')) return [];++ const hosts = new Set();++ for (const word of ws) {++ if (word === 'openvibe' || word.length < 2) continue;++ for (const host of (SITE_ALIASES[word] || [`openvibe.${word}`])) hosts.add(host);++ }++ return [...hosts].slice(0, 12);++}++++/**++ * The same test as siteHosts(), as a PostgreSQL regex (engine SQL). A document belongs to the++ * site when its host is (a subdomain of) a named host, or when it is a help article under++ * /a/<host>/ for one of them. Null when the query names no site.++ */++function siteUrlPattern(match) {++ const hosts = siteHosts(match);++ if (!hosts.length) return null;++ const alt = hosts.map(escapeRegex).join('|');++ return `^https?://([a-z0-9-]+\\.)*(${alt})(/|$)|^https?://([a-z0-9-]+\\.)*openvibe\\.help/a/(${alt})(/|$)`;++}++++/**++ * Is this a two-word site query ("openvibe live", "openvibe tools")? Then the site's own records are++ * candidates even when the page text never says "openvibe" or the site name: the site membership is++ * the second word, like a filter. A title-only (suggest) query never widens this way.++ */++function siteOnlyQuery(match) {++ const ws = match && Array.isArray(match.words) ? match.words : [];++ return !match.column && ws.length === 2 && ws.includes('openvibe');++}++++function parseUrl(url) {++ try {++ const u = new URL(String(url));++ return { host: u.hostname.toLowerCase(), path: u.pathname };++ } catch {++ return null;++ }++}++++function hostMatches(host, site) {++ return host === site || host.endsWith(`.${site}`);++}++++/** Does this canonical URL belong to one of the sites? Mirrors siteUrlPattern() in JS. */++function urlBelongsToSites(url, hosts) {++ if (!hosts || !hosts.length) return false;++ const parsed = parseUrl(url);++ if (!parsed) return false;++ if (hosts.some((host) => hostMatches(parsed.host, host))) return true;++ if (hostMatches(parsed.host, 'openvibe.help')) {++ const m = /^\/a\/([^/]+)\//.exec(parsed.path);++ if (m && hosts.some((host) => hostMatches(m[1].toLowerCase(), host))) return true;++ }++ return false;++}++++function wordMatches(token, word, prefix) {++ return token === word || (prefix && token.startsWith(word));++}++++/** Does any query word appear in this text? The last word is a prefix when the query says so. */++function textMatches(text, match) {++ if (!match || !Array.isArray(match.words) || !match.words.length) return false;++ const ts = tokens(text);++ return match.words.some((word, i) => ts.some((token) => wordMatches(token, word, Boolean(match.prefix && i === match.words.length - 1))));++}++++/** 3 when the title matches, 2 the summary, 1 the body, 0 no text field (a vector-only hit). */++function fieldTier(doc, match) {++ if (!doc) return 0;++ if (textMatches(doc.title, match)) return 3;++ if (textMatches(doc.summary, match)) return 2;++ if (textMatches(doc.body, match)) return 1;++ return 0;++}++++/** site (10) + the best matching field: a site's own records first, then title > summary > body. */++function docTier(doc, match, hosts = siteHosts(match)) {++ const site = hosts.length && urlBelongsToSites(doc && doc.canonical_url, hosts) ? 10 : 0;++ return site + fieldTier(doc, match);++}++++/**++ * A thin document: a username or a very short title with no real summary ("aaooo.ee", "Omar",++ * "Seattle Maybe 🤔" — a recorded stream whose summary is the site's sentence). It ranks below a++ * document with a real title and summary inside the same tier; the field and site tiers are unchanged.++ */++function isThin(doc) {++ const title = String((doc && doc.title) || '').trim();++ const summary = String((doc && doc.summary) || '').trim();++ if (!title) return true;++ const shortTitle = title.length <= 24;++ const noRealSummary = summary.length === 0 || summary.length <= 80;++ return shortTitle && noRealSummary;++}++++/** The one page identity: same title and owner is one result (keep the best-ranked one). */++function dedupeKey(doc) {++ const title = String((doc && doc.title) || '').trim().replace(/\s+/g, ' ').toLowerCase();++ return `${String((doc && doc.owner) || '').toLowerCase()}\u0000${title}`;++}++++module.exports = {++ SITE_ALIASES, fold, words, tokens, siteHosts, siteOnlyQuery, siteUrlPattern, urlBelongsToSites,++ textMatches, fieldTier, docTier, isThin, dedupeKey,++};@@@ -29,11 +29,12 @@ t('the decay: ×(1+w) today, ×(1+w/2) one half-life ago, → ×1 when old; futuasync function corpus(env = {}) {let clock = NOW;const svc = await boot({ env, now: () => clock });-- // Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed-- // first (lower rid), so without freshness it would come first.-- await svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));-- await svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));-- await svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));++ // Same query term, same length: bm25 ties, so only freshness separates them. Distinct titles keep++ // the page's same-title dedupe from folding them into one. The old one is indexed first (lower++ // rid), so without freshness it would come first.++ await svc.store.apply(doc({ id: 'old', title: 'Aurora report old', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));++ await svc.store.apply(doc({ id: 'mid', title: 'Aurora report mid', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));++ await svc.store.apply(doc({ id: 'new', title: 'Aurora report new', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));return { svc, setClock: (c) => { clock = c; } };}@@@ -315,9 +315,37 @@ t('deals: a title or category match outranks a description-only match (11-01)',assert.deepStrictEqual(r.body.results.find((x) => x.id === 'k2').ranks, { keyword: 1, vector: null });assert.deepStrictEqual(r.body.results.find((x) => x.id === 'bit-blaster').ranks, { keyword: 2, vector: 1 });-- // The order is only the deals owner's: the same two documents unowned keep the fused order.++ // The title tier is not deals-only: across owners, the keyboard's title match now outranks the++ // game's summary-only match too (the deals category rule still covers a category-only match).const all = await request(svc.base, 'GET', '/api/v1/search?q=mechanical%20keyboard&limit=10');-- assert.deepStrictEqual(all.body.results.map((x) => x.id), ['bit-blaster', 'k2'], 'other owners stay on RRF');++ assert.deepStrictEqual(all.body.results.map((x) => x.id), ['k2', 'bit-blaster'], 'title match first outside deals too');++ } finally {++ await svc.stop();++ }++});++++t('ranking: a site query and title tier survive RRF, and a page keeps one title per owner', async () => {++ const vectors = new FakeVectors();++ const embeddings = fakeEmbeddings();++ const svc = await boot({ vectorStore: vectors, embeddingClient: embeddings });++ try {++ await svc.store.apply(doc({ owner: 'help', type: 'article', id: 'live-help', title: 'Is OpenVibe Live on by default?', summary: 'Yes. Every channel streams with OpenVibe.Live on by default.', canonical_url: 'https://openvibe.help/a/openre.stream/is-openvibe-live-on-by-default' }));++ await svc.store.apply(doc({ owner: 'community', type: 'thread', id: 'live-thread', title: 'OpenVibe live thread', summary: 'A discussion about going live.', canonical_url: 'https://openvibe.community/s/roadmap/t/live-thread' }));++ await svc.store.apply(doc({ owner: 'live', type: 'vod', id: 'vod-a', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/1' }));++ await svc.store.apply(doc({ owner: 'live', type: 'vod', id: 'vod-b', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/2' }));++ const rid = await ridMap(svc.db);++ // The vector side likes the thin VODs and the thread; the site record has no vector hit at all.++ vectors.hits = [++ { rid: rid['vod-a'], distance: 0.1 },++ { rid: rid['vod-b'], distance: 0.11 },++ { rid: rid['live-thread'], distance: 0.12 },++ ];++++ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(r.body.explain, { ranking: 'hybrid' });++ assert.deepStrictEqual(r.body.results.map((x) => x.id), ['live-help', 'live-thread', 'vod-a'], 'site first, title before summary, one Omar');++ assert.strictEqual(r.body.results.filter((x) => x.title === 'Omar').length, 1);} finally {await svc.stop();}nnew file mode 100644@@@ -0,0 +1,130 @@++'use strict';++/**++ * Plain-query ranking: a word that only matches an owner is not a match; a title match outranks a++ * summary match, which outranks a body match; a site a query names puts its own records first; a++ * thin username-titled VOD with no real summary loses a score tie; and one page never repeats a++ * title for the same owner. Seeded index, no vectors: this is the keyword order the hybrid page++ * re-applies after RRF (test/hybrid.test.js covers that side).++ */++const assert = require('assert');++const { boot, request, doc, suite } = require('./helpers');++const { isThin, dedupeKey, docTier, siteHosts } = require('../server/ranking');++++const t = suite('ranking');++let svc;++const AT = '2026-11-01T00:00:00Z';++++const put = (overrides) => svc.store.apply(doc({++ published_at: AT, updated_at: AT, summary: '', body: '', ...overrides,++}));++const ids = (r) => r.body.results.map((x) => x.id);++const indexOf = (r, id) => ids(r).indexOf(id);++++t('boot and seed', async () => {++ svc = await boot();++ try {++ // Field order: same query word in title, summary, body, and only in the owner.++ await put({ owner: 'wiki', type: 'page', id: 'aurora-title', title: 'Aurora guide', summary: 'A guide to the lights.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-title' });++ await put({ owner: 'wiki', type: 'page', id: 'aurora-summary', title: 'Weekly notes', summary: 'Aurora was bright this week.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-summary' });++ await put({ owner: 'wiki', type: 'page', id: 'aurora-body', title: 'Picnic plans', summary: 'Sunny afternoon.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-body' });++ await put({ owner: 'aurora', type: 'page', id: 'aurora-owner', title: 'Quiet notes', summary: 'Nothing to see.', body: 'nothing at all', canonical_url: 'https://openvibe.wiki/p/aurora-owner' });++++ // Thin: a username title and no summary. Real: a long summary; the same title/body match.++ await put({ owner: 'live', type: 'vod', id: 'zephyr-thin', title: 'zephyr', summary: '', body: 'zephyr', canonical_url: 'https://openvibe.video/v/9001' });++ await put({ owner: 'live', type: 'vod', id: 'zephyr-real', title: 'zephyr report', summary: 'A full report on the channel, its schedule, its recent streams, and everything that happened this week.', body: 'zephyr', canonical_url: 'https://openvibe.video/v/9002' });++++ // The Live page's own records: help articles (the old openre.stream name too) and a channel.++ await put({ owner: 'help', type: 'article', id: 'help-live-default', title: 'Is OpenVibe Live on by default?', summary: 'Yes. Every channel streams with OpenVibe.Live on by default.', canonical_url: 'https://openvibe.help/a/openre.stream/is-openvibe-live-on-by-default' });++ await put({ owner: 'help', type: 'article', id: 'help-stream-obs', title: 'Can I stream without OBS?', summary: 'Yes. Go live from your browser on OpenVibe.Live: press Go Live and you are live.', canonical_url: 'https://openvibe.help/a/openre.stream/can-i-stream-without-obs' });++ await put({ owner: 'help', type: 'article', id: 'help-go-live', title: 'Can I go live without OBS?', summary: 'Yes. Go live straight from your browser, camera or screen on OpenVibe.Live, with no software to install.', canonical_url: 'https://openvibe.help/a/openvibe.live/can-i-go-live-without-obs' });++ await put({ owner: 'live', type: 'channel', id: 'channel-arman', title: 'Arman (@Arman)', summary: 'Arman streams on OpenVibe.Live.', canonical_url: 'https://openvibe.live/@Arman' });++++ // Thin VODs, two of each title: the page keeps one.++ await put({ owner: 'live', type: 'vod', id: 'vod-omar-a', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9003' });++ await put({ owner: 'live', type: 'vod', id: 'vod-omar-b', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9004' });++ await put({ owner: 'live', type: 'vod', id: 'vod-seattle-a', title: 'Seattle Maybe 🤔', summary: 'Recorded live stream by goosely on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9005' });++ await put({ owner: 'live', type: 'vod', id: 'vod-seattle-b', title: 'Seattle Maybe 🤔', summary: 'Recorded live stream by goosely on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9006' });++ await put({ owner: 'live', type: 'vod', id: 'vod-real', title: 'Allahhuma labbaik (Part 3)', summary: 'Recorded live stream by Arman on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9007' });++ await put({ owner: 'wiki', type: 'page', id: 'omar-summary', title: 'Weekly notes', summary: 'Omar was seen at the market.', body: 'omar', canonical_url: 'https://openvibe.wiki/p/omar-summary' });++++ // The Tools site: a tool page (a subdomain of openvibe.tools), its help article, a community thread.++ await put({ owner: 'tools', type: 'tool', id: 'tool-json', title: 'JSON Formatter', summary: 'Paste messy JSON to pretty-print it. Part of OpenVibe.Tools.', body: 'json formatter', canonical_url: 'https://json.openvibe.tools' });++ await put({ owner: 'tools', type: 'tool', id: 'tool-min', title: 'JSON Minifier', summary: 'Minify JSON in the browser. Part of OpenVibe.Tools.', body: 'json minifier', canonical_url: 'https://json-minifier.openvibe.tools' });++ // The site's own record even when no field repeats "openvibe" or "tools".++ await put({ owner: 'tools', type: 'tool', id: 'tool-silent', title: 'Silent Tool', summary: 'Does one small thing.', body: 'nothing about the name', canonical_url: 'https://silent.openvibe.tools' });++ await put({ owner: 'help', type: 'article', id: 'help-tools', title: 'What kinds of tools are there?', summary: 'OpenVibe.Tools holds generators, converters and small utilities.', canonical_url: 'https://openvibe.help/a/openvibe.tools/what-kinds-of-tools-are-there' });++ await put({ owner: 'community', type: 'thread', id: 'community-tools', title: 'Favourite tools on every device', summary: 'A thread about OpenVibe tools people use every day.', canonical_url: 'https://openvibe.community/s/roadmap/t/favourite-tools' });++ await put({ owner: 'community', type: 'paste', id: 'snippet-json', title: 'Pastebin snippet', summary: 'A pasted file.', body: 'json formatter json formatter', canonical_url: 'https://openvibe.community/p/snippet' });++ } catch (e) {++ await svc.stop();++ throw e;++ }++});++++t('an owner word is not a text match; title beats summary beats body', async () => {++ const r = await request(svc.base, 'GET', '/api/v1/search?q=aurora&limit=10');++ assert.strictEqual(r.status, 200, r.text);++ assert.deepStrictEqual(ids(r), ['aurora-title', 'aurora-summary', 'aurora-body'], 'title > summary > body, owner-only absent');++});++++t('a site query puts that site\'s own records first', async () => {++ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');++ assert.strictEqual(r.status, 200, r.text);++ const first = r.body.results[0];++ assert.ok(['help-live-default', 'help-stream-obs', 'help-go-live', 'channel-arman'].includes(first.id), `first is a Live record: ${first.id}`);++ const site = ['help-live-default', 'help-stream-obs', 'help-go-live', 'channel-arman'];++ const lastSite = Math.max(...site.map((id) => indexOf(r, id)).filter((i) => i >= 0));++ const firstVod = Math.min(...['vod-omar-a', 'vod-seattle-a', 'vod-real'].map((id) => indexOf(r, id)).filter((i) => i >= 0));++ assert.ok(lastSite < firstVod, 'every Live record precedes the first VOD');++++ // A precise title query is still first, even with body noise for both words.++ const json = await request(svc.base, 'GET', '/api/v1/search?q=json%20formatter&limit=10');++ assert.deepStrictEqual(ids(json)[0], 'tool-json');++++ // The Tools site's own records come before a community thread whose title matches "tools".++ const tools = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20tools&limit=10');++ assert.ok(indexOf(tools, 'help-tools') < indexOf(tools, 'community-tools'), 'tools help article first');++ assert.ok(indexOf(tools, 'tool-json') < indexOf(tools, 'community-tools'), 'tool page before the thread');++ assert.ok(indexOf(tools, 'tool-silent') >= 0 && indexOf(tools, 'tool-silent') < indexOf(tools, 'community-tools'), 'a site page without the words is still a site record');++});++++t('a title match outranks a summary-only match (not just a site)', async () => {++ const r = await request(svc.base, 'GET', '/api/v1/search?q=omar&limit=10');++ assert.strictEqual(r.status, 200, r.text);++ // "Omar" is the VOD title; the wiki page only mentions Omar in its summary.++ assert.ok(indexOf(r, 'vod-omar-a') >= 0 && indexOf(r, 'vod-omar-a') < indexOf(r, 'omar-summary'), 'title before summary-only');++++ // And the plain "go live in a browser" question puts the help article first.++ const go = await request(svc.base, 'GET', '/api/v1/search?q=go%20live%20browser&limit=10');++ assert.deepStrictEqual(ids(go)[0], 'help-go-live');++});++++t('a thin username title loses the score tie to a real title and summary', async () => {++ const r = await request(svc.base, 'GET', '/api/v1/search?q=zephyr&limit=10');++ assert.deepStrictEqual(ids(r), ['zephyr-real', 'zephyr-thin'], 'the long summary wins the tie');++ const thin = await svc.store.get('live', 'vod', 'zephyr-thin');++ const real = await svc.store.get('live', 'vod', 'zephyr-real');++ assert.strictEqual(isThin(thin.doc), true);++ assert.strictEqual(isThin(real.doc), false);++});++++t('one page keeps one result per title and owner', async () => {++ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');++ const keys = r.body.results.map(dedupeKey);++ assert.strictEqual(new Set(keys).size, keys.length, 'no repeated title+owner');++ assert.strictEqual(r.body.results.filter((x) => x.title === 'Omar').length, 1);++ assert.strictEqual(r.body.results.filter((x) => x.title === 'Seattle Maybe 🤔').length, 1);++});++++t('helper tiers and dedupe agree with the SQL order', () => {++ const match = { words: ['openvibe', 'live'], prefix: false, column: null };++ const hosts = siteHosts(match);++ assert.deepStrictEqual(hosts.sort(), ['openre.stream', 'openvibe.live']);++ assert.strictEqual(docTier({ title: 'Is OpenVibe Live on?', summary: '', body: '', canonical_url: 'https://openvibe.help/a/openre.stream/x' }, match, hosts), 13);++ assert.strictEqual(docTier({ title: 'Omar', summary: 'Recorded on OpenVibe.Live.', body: '', canonical_url: 'https://openvibe.video/v/1' }, match, hosts), 2);++ assert.strictEqual(dedupeKey({ owner: 'live', title: ' Omar ' }), dedupeKey({ owner: 'LIVE', title: 'omar' }));++});++++t('close', async () => { await svc.stop(); });++++t.run();
@@ -98,21 +98,32 @@ an empty shelf is left out, and without the numbers the sources fall back to the
@@ -98,21 +98,32 @@ an empty shelf is left out, and without the numbers the sources fall back to the
### Ranking
### Ranking
-Full-text results are ordered by full-text relevance (title ×10, summary ×4, body ×1) multiplied by a
-freshness boost:
+Full-text results are ordered by a tier first, then full-text relevance (title ×10, summary ×4, body ×1)
+multiplied by a freshness boost:
```
```
-score = relevance × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)
+tier = site × 10 + field field: title 3 > summary 2 > body 1
+score = tier × 100 + relevance × (1 + W × 2^(−age_days / H))
+ W = SEARCH_FRESHNESS_WEIGHT (default 1)
H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)
H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)
```
```
+A query naming a site ("openvibe live", "openvibe tools") puts that site's own records first: the site is
+the host `openvibe.<word>` (subdomains too) or a help article under `/a/<host>/`, with Live's old
+`openre.stream` name included. A word that matches a document's owner or type is not a text match (the
+`owner=` filter is how an owner narrows a search), a title match always outranks a summary-only match, and
+a summary-only match outranks a body-only one. Within a tier, a thin document (a username or a very short
+title with no real summary: the recorded-stream VODs) ranks below one with a real title and summary, and
+one page keeps one result per title and owner.
+
`age_days` counts from `published_at`, else `updated_at` (an edit does not make an old document
`age_days` counts from `published_at`, else `updated_at` (an edit does not make an old document
new); a future date counts as today and a missing one gets no boost. With the defaults a document
new); a future date counts as today and a missing one gets no boost. With the defaults a document
published today counts ×2.0, 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125 and a year old ×1.0002:
published today counts ×2.0, 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125 and a year old ×1.0002:
relevance still decides, and between comparably relevant documents the newer one ranks first.
relevance still decides, and between comparably relevant documents the newer one ranks first.
`W = 0` turns it off. Suggestions use the same score; browsing without `q` stays newest first. A
`W = 0` turns it off. Suggestions use the same score; browsing without `q` stays newest first. A
-cursor carries its first page's clock, so later pages rank on the same scores
-(`test/freshness.test.js`).
+cursor carries its first page's clock, so later pages rank on the same scores. The hybrid path fuses the
+keyword and vector top 100 by RRF and then applies the same tiers to the fused list
+(`test/freshness.test.js`, `test/ranking.test.js`).
### Saved searches
### Saved searches
@@ -43,9 +43,11 @@ There is no corpus to measure: no product indexes anything today.
@@ -43,9 +43,11 @@ There is no corpus to measure: no product indexes anything today.
- One database: backups, restores and moves are the same as every other service (PostgreSQL,
- One database: backups, restores and moves are the same as every other service (PostgreSQL,
ADR-035).
ADR-035).
-- Ranking is `ts_rank` with title/summary/body weights 10/4/1 times a freshness boost
- ([README "Ranking"](../README.md#ranking)), returned as minus the score so ascending keyset
- cursors work.
+- Ranking is a tier (site 10 + title 3 / summary 2 / body 1) ahead of `ts_rank` with
+ title/summary/body weights 10/4/1 times a freshness boost ([README "Ranking"](../README.md#ranking)),
+ returned as minus the score so ascending keyset cursors work. The owner and type are not text
+ fields, thin username-titled VODs sink inside a tier, and one page keeps one result per title and
+ owner.
- Text is folded (diacritics removed) and split into runs of letters and digits before the
- Text is folded (diacritics removed) and split into runs of letters and digits before the
`simple` configuration sees it, so tokens are predictable across languages (Japanese/Chinese get
`simple` configuration sees it, so tokens are predictable across languages (Japanese/Chinese get
no word segmentation; a later engine or ICU tokenizer fixes that) and a query and the index
no word segmentation; a later engine or ICU tokenizer fixes that) and a query and the index
@@ -29,6 +29,7 @@ const { AuthError } = require('../auth');
@@ -29,6 +29,7 @@ const { AuthError } = require('../auth');
const { OWNER_RE, TYPE_RE, ID_RE } = require('./documents');
const { OWNER_RE, TYPE_RE, ID_RE } = require('./documents');
const { fuseRrf } = require('../fabric/rrf');
const { fuseRrf } = require('../fabric/rrf');
const { createQueryEmbeddingCache } = require('../fabric/embed');
const { createQueryEmbeddingCache } = require('../fabric/embed');
+const { docTier, isThin, dedupeKey, siteHosts } = require('../ranking');
const FACET_KEY_RE = /^[a-z][a-z0-9_]{0,39}$/;
const FACET_KEY_RE = /^[a-z][a-z0-9_]{0,39}$/;
const LANG_RE = /^[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*$/;
const LANG_RE = /^[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*$/;
@@ -162,6 +163,29 @@ async function dealsOrder(fused, { match, store }) {
@@ -162,6 +163,29 @@ async function dealsOrder(fused, { match, store }) {
.map(({ hit }) => ({ ...hit, found: rows.get(hit.rid) }));
.map(({ hit }) => ({ ...hit, found: rows.get(hit.rid) }));
}
}
+// ── A plain query's order (11-01) ──────────────────────────────────────────────────────────────
+// The keyword engine already ranks with the same tiers (server/engine/pg.js), but a hybrid page is
+// the RRF order of two top-100 lists, and a strong vector hit could still lift a summary-only VOD
+// above a help article whose title says what was asked. Re-apply the tiers to the fused list, so:
+// - a query naming a site ("openvibe live") puts that site's own records first (site 10);
+// - a title match (3) outranks a summary-only match (2), which outranks a body-only one (1);
+// - a thin username-titled VOD ranks below a real title+summary inside the same tier.
+// The fused order is the final tie-break, so relevance still decides among equals.
+async function rankOrder(fused, { match, store }) {
+ if (!match || !match.words.length) return fused;
+ const hosts = siteHosts(match);
+ const rows = new Map();
+ for (const hit of fused) rows.set(hit.rid, hit.found !== undefined ? hit.found : await store.byRid(hit.rid));
+ return fused
+ .filter((hit) => rows.get(hit.rid))
+ .map((hit, i) => {
+ const found = rows.get(hit.rid);
+ return { hit, found, i, tier: docTier(found.doc, match, hosts), thin: isThin(found.doc) };
+ })
+ .sort((a, b) => b.tier - a.tier || (a.thin === b.thin ? 0 : a.thin ? 1 : -1) || a.i - b.i)
+ .map(({ hit, found }) => ({ ...hit, found }));
+}
+
/**
/**
* One search as a viewer: the query API, saved-search runs and the HTML page all go through here.
* One search as a viewer: the query API, saved-search runs and the HTML page all go through here.
* run({ text, query, viewer, limit, cursor, facetKeys, now, since }) → { results, next_cursor, explain, facets? }
* run({ text, query, viewer, limit, cursor, facetKeys, now, since }) → { results, next_cursor, explain, facets? }
@@ -240,6 +264,7 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
@@ -240,6 +264,7 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
}
}
const results = [];
const results = [];
+ const seen = new Set();
let next = null;
let next = null;
let explain;
let explain;
let refNow = now;
let refNow = now;
@@ -249,6 +274,8 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
@@ -249,6 +274,8 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
if (after && after.offset === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.offset === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.now != null) refNow = after.now;
if (after && after.now != null) refNow = after.now;
let fused = await hybridCandidates({ match, filters, viewer, now: refNow, vector: state.vector });
let fused = await hybridCandidates({ match, filters, viewer, now: refNow, vector: state.vector });
+ // A plain query's tiers: site first, then title > summary > body, thin VODs last within a tier.
+ fused = await rankOrder(fused, { match, store });
// Deals: the product the person asked for (its title or category) beats a description-only match.
// Deals: the product the person asked for (its title or category) beats a description-only match.
if (filters.owner === 'deals') fused = await dealsOrder(fused, { match, store });
if (filters.owner === 'deals') fused = await dealsOrder(fused, { match, store });
let consumed = after ? after.offset : 0;
let consumed = after ? after.offset : 0;
@@ -260,6 +287,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
@@ -260,6 +287,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
// Defence in depth: the vector store is public-only, the keyword SQL is ACL-aware.
// Defence in depth: the vector store is public-only, the keyword SQL is ACL-aware.
const found = f.found !== undefined ? f.found : await store.byRid(f.rid);
const found = f.found !== undefined ? f.found : await store.byRid(f.rid);
if (!found || !canView(viewer, found.row, found.doc)) continue;
if (!found || !canView(viewer, found.row, found.doc)) continue;
+ const key = dedupeKey(found.doc);
+ if (seen.has(key)) continue;
+ seen.add(key);
results.push(projection(found.doc, found.row, f.snip, f.ranks));
results.push(projection(found.doc, found.row, f.snip, f.ranks));
}
}
if (consumed < fused.length) next = encodeCursor({ k: 'f', o: consumed, i: lastRid, n: refNow, h: hash });
if (consumed < fused.length) next = encodeCursor({ k: 'f', o: consumed, i: lastRid, n: refNow, h: hash });
@@ -274,6 +304,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
@@ -274,6 +304,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
const found = await store.byRid(h.rid);
const found = await store.byRid(h.rid);
// Defence in depth: the engine already filtered; re-check against the stored row.
// Defence in depth: the engine already filtered; re-check against the stored row.
if (!found || !canView(viewer, found.row, found.doc)) continue;
if (!found || !canView(viewer, found.row, found.doc)) continue;
+ const key = dedupeKey(found.doc);
+ if (seen.has(key)) continue;
+ seen.add(key);
results.push(projection(found.doc, found.row, h.snip, { keyword: position, vector: null }));
results.push(projection(found.doc, found.row, h.snip, { keyword: position, vector: null }));
}
}
next = page.next ? encodeCursor({ k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash }) : null;
next = page.next ? encodeCursor({ k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash }) : null;
@@ -16,10 +16,13 @@
@@ -16,10 +16,13 @@
* Text is folded before it is indexed or queried (NFKD, combining marks removed), split into runs of letters and
* Text is folded before it is indexed or queried (NFKD, combining marks removed), split into runs of letters and
* digits and tokenized by the `simple` configuration (lower case, no stemming, no stop words): the index and every
* digits and tokenized by the `simple` configuration (lower case, no stemming, no stop words): the index and every
* query see the same text. Title, summary and body weigh 10, 4 and 1 (tsvector weights A, B, C), with the freshness
* query see the same text. Title, summary and body weigh 10, 4 and 1 (tsvector weights A, B, C), with the freshness
- * boost described below. Every function here is async; put and remove run inside the store's
- * transaction (ambient). Nothing outside this file writes SQL against search_fts.
+ * boost described below. The owner and type are not text fields (a word that equals an owner is not a match), and a
+ * ranking tier puts a site's own records first, then title matches, then summary-only, then body-only ones
+ * (server/ranking.js). Every function here is async; put and remove run inside the store's transaction (ambient).
+ * Nothing outside this file writes SQL against search_fts.
*/
*/
const { EXPOSURE } = require('../document');
const { EXPOSURE } = require('../document');
+const { fold, words, siteOnlyQuery, siteUrlPattern } = require('../ranking');
const CONFIG = 'simple';
const CONFIG = 'simple';
const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: ts_rank weights 1 / 4 / 10
const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: ts_rank weights 1 / 4 / 10
@@ -67,20 +70,6 @@ function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
@@ -67,20 +70,6 @@ function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
}
}
-/** Diacritics folded, so the index and every query see the same text. */
-function fold(s) {
- return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');
-}
-
-/**
- * Word tokens: runs of letters and digits, everything else a separator. PostgreSQL's parser would
- * keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first
- * makes "openvibe" find "OpenVibe.Live".
- */
-function words(s) {
- return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');
-}
-
function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {
function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {
const st = {
const st = {
del: db.prepare('DELETE FROM search_fts WHERE rid = ?'),
del: db.prepare('DELETE FROM search_fts WHERE rid = ?'),
@@ -130,31 +119,66 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
@@ -130,31 +119,66 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
}
}
/**
/**
- * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the
- * freshness boost at reference time `now` (ms), so lower ranks first and pages of one query
- * (which carry their first page's `now` in the cursor) rank against the same clock.
+ * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus the
+ * tier (site 10 + title 3 / summary 2 / body 1) and ts_rank times the freshness boost at
+ * reference time `now` (ms), so lower ranks first and pages of one query (which carry their
+ * first page's `now` in the cursor) rank against the same clock. The owner and type are not in
+ * the tsvector: a word that only equals an owner never matches. Rows whose owner and title are
+ * the same keep only the best-ranked one.
*/
*/
function matchedSet({ match, filters, viewer, now = Date.now() }, params) {
function matchedSet({ match, filters, viewer, now = Date.now() }, params) {
params.match = tsquery(match);
params.match = tsquery(match);
params.r_now = now;
params.r_now = now;
params.r_half = freshness.halfLifeDays;
params.r_half = freshness.halfLifeDays;
params.r_weight = freshness.weight;
params.r_weight = freshness.weight;
+ params.site_re = siteUrlPattern(match) || '';
const where = filterClause(filters, params);
const where = filterClause(filters, params);
- const rank = `-(ts_rank('${WEIGHTS}'::float4[], f.tsv, to_tsquery('${CONFIG}', @match), 1)
+ const q = `to_tsquery('${CONFIG}', @match)`;
+ const base = `ts_rank('${WEIGHTS}'::float4[], f.tsv, ${q}, 1)
* CASE WHEN f.fresh_at IS NULL OR @r_weight::float8 = 0 THEN 1
* CASE WHEN f.fresh_at IS NULL OR @r_weight::float8 = 0 THEN 1
- ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END)`;
- const branches = [`SELECT d.rid AS rid, ${rank} AS rank
+ ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END`;
+ const title = `ts_rank('{0,0,0,1}'::float4[], f.tsv, ${q}, 0)`;
+ const summary = `ts_rank('{0,0,0.4,0}'::float4[], f.tsv, ${q}, 0)`;
+ const body = `ts_rank('{0,0.1,0,0}'::float4[], f.tsv, ${q}, 0)`;
+ const site = `(@site_re::text <> '' AND d.canonical_url ~* @site_re)`;
+ const field = `CASE WHEN ${title} > 0 THEN 3 WHEN ${summary} > 0 THEN 2 WHEN ${body} > 0 THEN 1 ELSE 0 END`;
+ const tier = `(CASE WHEN ${site} THEN 10 ELSE 0 END + ${field})`;
+ const thin = `(char_length(btrim(f.title)) <= 24 AND char_length(btrim(f.summary)) <= 80)`;
+ // Thin VODs sink below documents with a real title and summary inside the same tier; -2 is
+ // larger than any base score (ts_rank × freshness < 2), so it decides before relevance does.
+ const rank = `-(${tier} * 100 + (${base}) + CASE WHEN ${thin} THEN -2 ELSE 0 END)`;
+ const select = `SELECT d.rid AS rid, d.owner AS owner, f.title AS title, ${rank} AS rank`;
+ const branches = [`${select}
FROM search_fts f JOIN documents d ON d.rid = f.rid
FROM search_fts f JOIN documents d ON d.rid = f.rid
- WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ to_tsquery('${CONFIG}', @match)
+ WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ ${q}
AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];
AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];
const acl = aclClause(viewer, params);
const acl = aclClause(viewer, params);
if (acl) {
if (acl) {
- branches.push(`SELECT d.rid AS rid, ${rank} AS rank
+ branches.push(`${select}
FROM search_fts f JOIN documents d ON d.rid = f.rid
FROM search_fts f JOIN documents d ON d.rid = f.rid
- WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ to_tsquery('${CONFIG}', @match)
+ WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ ${q}
AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
}
}
- return branches.join('\nUNION ALL\n');
+ // "openvibe live"/"openvibe tools": the site's own records are candidates even when their text
+ // does not repeat the site name (the Tools pages never do). They rank in the site tier below the
+ // site records that do match the words; the window dedupe folds a document that is in both sets.
+ if (params.site_re && siteOnlyQuery(match)) {
+ branches.push(`${select}
+ FROM search_fts f JOIN documents d ON d.rid = f.rid
+ WHERE f.audience = ${EXPOSURE.public_listed} AND d.canonical_url ~* @site_re
+ AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`);
+ if (acl) {
+ branches.push(`${select}
+ FROM search_fts f JOIN documents d ON d.rid = f.rid
+ WHERE f.audience = ${EXPOSURE.restricted} AND d.canonical_url ~* @site_re
+ AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
+ }
+ }
+ return `SELECT rid, rank FROM (
+ SELECT m.rid, m.rank,
+ ROW_NUMBER() OVER (PARTITION BY m.owner, lower(btrim(m.title)) ORDER BY m.rank, m.rid) AS rn
+ FROM (${branches.join('\nUNION ALL\n')}) m
+ ) x WHERE x.rn = 1`;
}
}
/** The visible set without a text query (browse). */
/** The visible set without a text query (browse). */
new file mode 100644
new file mode 100644
@@ -0,0 +1,161 @@
@@ -0,0 +1,161 @@
+'use strict';
+/**
+ * Ranking helpers shared by the full-text engine (server/engine/pg.js) and the query path
+ * (server/api/query.js): which fields a query matches, whether a document belongs to a site the
+ * query names, whether its title is a thin streamer name, and the identity that drops duplicate
+ * results from a page.
+ *
+ * The engine's base score is ts_rank with title A (×10), summary B (×4) and body C (×1), times the
+ * freshness boost. That ranks term frequency, but not the things a person means by a plain query:
+ *
+ * tier = site (10) + field (title 3, summary 2, body 1, none 0)
+ *
+ * A query naming a site ("openvibe live", "openvibe tools") puts that site's own records first
+ * (its home, its help articles, its tool pages), a title match always outranks a summary-only
+ * match, and a summary-only match outranks a body-only one. The owner and type are deliberately
+ * not text fields: a word that equals a document's owner is not a match (the explicit owner=
+ * filter is how an owner narrows a query), and the summary's service boilerplate ("Recorded live
+ * stream by … on OpenVibe.Live.") is only summary text, never a title.
+ */
+
+/** Sites whose old name still appears in canonical URLs and help paths. */
+const SITE_ALIASES = {
+ live: ['openvibe.live', 'openre.stream'],
+ openre: ['openre.stream', 'openvibe.live'],
+ stream: ['openre.stream', 'openvibe.live'],
+ restream: ['openre.stream', 'openvibe.live'],
+};
+
+/** Diacritics folded, so the index and every query see the same text. */
+function fold(s) {
+ return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');
+}
+
+/** Word tokens: runs of letters and digits, everything else a separator. */
+function words(s) {
+ return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');
+}
+
+function tokens(s) {
+ return fold(s).toLowerCase().match(/[\p{L}\p{N}]+/gu) || [];
+}
+
+function escapeRegex(s) {
+ return String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
+
+/**
+ * The site hosts a query names, if it has "openvibe" plus another word: "openvibe live" names
+ * openvibe.live and its old name openre.stream; "openvibe tools" names openvibe.tools. The host of
+ * an unknown word simply matches nothing. No site without "openvibe" (a plain "live" is text, not
+ * the Live site).
+ */
+function siteHosts(match) {
+ const ws = match && Array.isArray(match.words) ? match.words : [];
+ if (!ws.includes('openvibe')) return [];
+ const hosts = new Set();
+ for (const word of ws) {
+ if (word === 'openvibe' || word.length < 2) continue;
+ for (const host of (SITE_ALIASES[word] || [`openvibe.${word}`])) hosts.add(host);
+ }
+ return [...hosts].slice(0, 12);
+}
+
+/**
+ * The same test as siteHosts(), as a PostgreSQL regex (engine SQL). A document belongs to the
+ * site when its host is (a subdomain of) a named host, or when it is a help article under
+ * /a/<host>/ for one of them. Null when the query names no site.
+ */
+function siteUrlPattern(match) {
+ const hosts = siteHosts(match);
+ if (!hosts.length) return null;
+ const alt = hosts.map(escapeRegex).join('|');
+ return `^https?://([a-z0-9-]+\\.)*(${alt})(/|$)|^https?://([a-z0-9-]+\\.)*openvibe\\.help/a/(${alt})(/|$)`;
+}
+
+/**
+ * Is this a two-word site query ("openvibe live", "openvibe tools")? Then the site's own records are
+ * candidates even when the page text never says "openvibe" or the site name: the site membership is
+ * the second word, like a filter. A title-only (suggest) query never widens this way.
+ */
+function siteOnlyQuery(match) {
+ const ws = match && Array.isArray(match.words) ? match.words : [];
+ return !match.column && ws.length === 2 && ws.includes('openvibe');
+}
+
+function parseUrl(url) {
+ try {
+ const u = new URL(String(url));
+ return { host: u.hostname.toLowerCase(), path: u.pathname };
+ } catch {
+ return null;
+ }
+}
+
+function hostMatches(host, site) {
+ return host === site || host.endsWith(`.${site}`);
+}
+
+/** Does this canonical URL belong to one of the sites? Mirrors siteUrlPattern() in JS. */
+function urlBelongsToSites(url, hosts) {
+ if (!hosts || !hosts.length) return false;
+ const parsed = parseUrl(url);
+ if (!parsed) return false;
+ if (hosts.some((host) => hostMatches(parsed.host, host))) return true;
+ if (hostMatches(parsed.host, 'openvibe.help')) {
+ const m = /^\/a\/([^/]+)\//.exec(parsed.path);
+ if (m && hosts.some((host) => hostMatches(m[1].toLowerCase(), host))) return true;
+ }
+ return false;
+}
+
+function wordMatches(token, word, prefix) {
+ return token === word || (prefix && token.startsWith(word));
+}
+
+/** Does any query word appear in this text? The last word is a prefix when the query says so. */
+function textMatches(text, match) {
+ if (!match || !Array.isArray(match.words) || !match.words.length) return false;
+ const ts = tokens(text);
+ return match.words.some((word, i) => ts.some((token) => wordMatches(token, word, Boolean(match.prefix && i === match.words.length - 1))));
+}
+
+/** 3 when the title matches, 2 the summary, 1 the body, 0 no text field (a vector-only hit). */
+function fieldTier(doc, match) {
+ if (!doc) return 0;
+ if (textMatches(doc.title, match)) return 3;
+ if (textMatches(doc.summary, match)) return 2;
+ if (textMatches(doc.body, match)) return 1;
+ return 0;
+}
+
+/** site (10) + the best matching field: a site's own records first, then title > summary > body. */
+function docTier(doc, match, hosts = siteHosts(match)) {
+ const site = hosts.length && urlBelongsToSites(doc && doc.canonical_url, hosts) ? 10 : 0;
+ return site + fieldTier(doc, match);
+}
+
+/**
+ * A thin document: a username or a very short title with no real summary ("aaooo.ee", "Omar",
+ * "Seattle Maybe 🤔" — a recorded stream whose summary is the site's sentence). It ranks below a
+ * document with a real title and summary inside the same tier; the field and site tiers are unchanged.
+ */
+function isThin(doc) {
+ const title = String((doc && doc.title) || '').trim();
+ const summary = String((doc && doc.summary) || '').trim();
+ if (!title) return true;
+ const shortTitle = title.length <= 24;
+ const noRealSummary = summary.length === 0 || summary.length <= 80;
+ return shortTitle && noRealSummary;
+}
+
+/** The one page identity: same title and owner is one result (keep the best-ranked one). */
+function dedupeKey(doc) {
+ const title = String((doc && doc.title) || '').trim().replace(/\s+/g, ' ').toLowerCase();
+ return `${String((doc && doc.owner) || '').toLowerCase()}\u0000${title}`;
+}
+
+module.exports = {
+ SITE_ALIASES, fold, words, tokens, siteHosts, siteOnlyQuery, siteUrlPattern, urlBelongsToSites,
+ textMatches, fieldTier, docTier, isThin, dedupeKey,
+};
@@ -29,11 +29,12 @@ t('the decay: ×(1+w) today, ×(1+w/2) one half-life ago, → ×1 when old; futu
@@ -29,11 +29,12 @@ t('the decay: ×(1+w) today, ×(1+w/2) one half-life ago, → ×1 when old; futu
async function corpus(env = {}) {
async function corpus(env = {}) {
let clock = NOW;
let clock = NOW;
const svc = await boot({ env, now: () => clock });
const svc = await boot({ env, now: () => clock });
- // Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed
- // first (lower rid), so without freshness it would come first.
- await svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));
- await svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));
- await svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));
+ // Same query term, same length: bm25 ties, so only freshness separates them. Distinct titles keep
+ // the page's same-title dedupe from folding them into one. The old one is indexed first (lower
+ // rid), so without freshness it would come first.
+ await svc.store.apply(doc({ id: 'old', title: 'Aurora report old', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));
+ await svc.store.apply(doc({ id: 'mid', title: 'Aurora report mid', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));
+ await svc.store.apply(doc({ id: 'new', title: 'Aurora report new', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));
return { svc, setClock: (c) => { clock = c; } };
return { svc, setClock: (c) => { clock = c; } };
}
}
@@ -315,9 +315,37 @@ t('deals: a title or category match outranks a description-only match (11-01)',
@@ -315,9 +315,37 @@ t('deals: a title or category match outranks a description-only match (11-01)',
assert.deepStrictEqual(r.body.results.find((x) => x.id === 'k2').ranks, { keyword: 1, vector: null });
assert.deepStrictEqual(r.body.results.find((x) => x.id === 'k2').ranks, { keyword: 1, vector: null });
assert.deepStrictEqual(r.body.results.find((x) => x.id === 'bit-blaster').ranks, { keyword: 2, vector: 1 });
assert.deepStrictEqual(r.body.results.find((x) => x.id === 'bit-blaster').ranks, { keyword: 2, vector: 1 });
- // The order is only the deals owner's: the same two documents unowned keep the fused order.
+ // The title tier is not deals-only: across owners, the keyboard's title match now outranks the
+ // game's summary-only match too (the deals category rule still covers a category-only match).
const all = await request(svc.base, 'GET', '/api/v1/search?q=mechanical%20keyboard&limit=10');
const all = await request(svc.base, 'GET', '/api/v1/search?q=mechanical%20keyboard&limit=10');
- assert.deepStrictEqual(all.body.results.map((x) => x.id), ['bit-blaster', 'k2'], 'other owners stay on RRF');
+ assert.deepStrictEqual(all.body.results.map((x) => x.id), ['k2', 'bit-blaster'], 'title match first outside deals too');
+ } finally {
+ await svc.stop();
+ }
+});
+
+t('ranking: a site query and title tier survive RRF, and a page keeps one title per owner', async () => {
+ const vectors = new FakeVectors();
+ const embeddings = fakeEmbeddings();
+ const svc = await boot({ vectorStore: vectors, embeddingClient: embeddings });
+ try {
+ await svc.store.apply(doc({ owner: 'help', type: 'article', id: 'live-help', title: 'Is OpenVibe Live on by default?', summary: 'Yes. Every channel streams with OpenVibe.Live on by default.', canonical_url: 'https://openvibe.help/a/openre.stream/is-openvibe-live-on-by-default' }));
+ await svc.store.apply(doc({ owner: 'community', type: 'thread', id: 'live-thread', title: 'OpenVibe live thread', summary: 'A discussion about going live.', canonical_url: 'https://openvibe.community/s/roadmap/t/live-thread' }));
+ await svc.store.apply(doc({ owner: 'live', type: 'vod', id: 'vod-a', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/1' }));
+ await svc.store.apply(doc({ owner: 'live', type: 'vod', id: 'vod-b', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/2' }));
+ const rid = await ridMap(svc.db);
+ // The vector side likes the thin VODs and the thread; the site record has no vector hit at all.
+ vectors.hits = [
+ { rid: rid['vod-a'], distance: 0.1 },
+ { rid: rid['vod-b'], distance: 0.11 },
+ { rid: rid['live-thread'], distance: 0.12 },
+ ];
+
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(r.body.explain, { ranking: 'hybrid' });
+ assert.deepStrictEqual(r.body.results.map((x) => x.id), ['live-help', 'live-thread', 'vod-a'], 'site first, title before summary, one Omar');
+ assert.strictEqual(r.body.results.filter((x) => x.title === 'Omar').length, 1);
} finally {
} finally {
await svc.stop();
await svc.stop();
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,130 @@
@@ -0,0 +1,130 @@
+'use strict';
+/**
+ * Plain-query ranking: a word that only matches an owner is not a match; a title match outranks a
+ * summary match, which outranks a body match; a site a query names puts its own records first; a
+ * thin username-titled VOD with no real summary loses a score tie; and one page never repeats a
+ * title for the same owner. Seeded index, no vectors: this is the keyword order the hybrid page
+ * re-applies after RRF (test/hybrid.test.js covers that side).
+ */
+const assert = require('assert');
+const { boot, request, doc, suite } = require('./helpers');
+const { isThin, dedupeKey, docTier, siteHosts } = require('../server/ranking');
+
+const t = suite('ranking');
+let svc;
+const AT = '2026-11-01T00:00:00Z';
+
+const put = (overrides) => svc.store.apply(doc({
+ published_at: AT, updated_at: AT, summary: '', body: '', ...overrides,
+}));
+const ids = (r) => r.body.results.map((x) => x.id);
+const indexOf = (r, id) => ids(r).indexOf(id);
+
+t('boot and seed', async () => {
+ svc = await boot();
+ try {
+ // Field order: same query word in title, summary, body, and only in the owner.
+ await put({ owner: 'wiki', type: 'page', id: 'aurora-title', title: 'Aurora guide', summary: 'A guide to the lights.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-title' });
+ await put({ owner: 'wiki', type: 'page', id: 'aurora-summary', title: 'Weekly notes', summary: 'Aurora was bright this week.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-summary' });
+ await put({ owner: 'wiki', type: 'page', id: 'aurora-body', title: 'Picnic plans', summary: 'Sunny afternoon.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-body' });
+ await put({ owner: 'aurora', type: 'page', id: 'aurora-owner', title: 'Quiet notes', summary: 'Nothing to see.', body: 'nothing at all', canonical_url: 'https://openvibe.wiki/p/aurora-owner' });
+
+ // Thin: a username title and no summary. Real: a long summary; the same title/body match.
+ await put({ owner: 'live', type: 'vod', id: 'zephyr-thin', title: 'zephyr', summary: '', body: 'zephyr', canonical_url: 'https://openvibe.video/v/9001' });
+ await put({ owner: 'live', type: 'vod', id: 'zephyr-real', title: 'zephyr report', summary: 'A full report on the channel, its schedule, its recent streams, and everything that happened this week.', body: 'zephyr', canonical_url: 'https://openvibe.video/v/9002' });
+
+ // The Live page's own records: help articles (the old openre.stream name too) and a channel.
+ await put({ owner: 'help', type: 'article', id: 'help-live-default', title: 'Is OpenVibe Live on by default?', summary: 'Yes. Every channel streams with OpenVibe.Live on by default.', canonical_url: 'https://openvibe.help/a/openre.stream/is-openvibe-live-on-by-default' });
+ await put({ owner: 'help', type: 'article', id: 'help-stream-obs', title: 'Can I stream without OBS?', summary: 'Yes. Go live from your browser on OpenVibe.Live: press Go Live and you are live.', canonical_url: 'https://openvibe.help/a/openre.stream/can-i-stream-without-obs' });
+ await put({ owner: 'help', type: 'article', id: 'help-go-live', title: 'Can I go live without OBS?', summary: 'Yes. Go live straight from your browser, camera or screen on OpenVibe.Live, with no software to install.', canonical_url: 'https://openvibe.help/a/openvibe.live/can-i-go-live-without-obs' });
+ await put({ owner: 'live', type: 'channel', id: 'channel-arman', title: 'Arman (@Arman)', summary: 'Arman streams on OpenVibe.Live.', canonical_url: 'https://openvibe.live/@Arman' });
+
+ // Thin VODs, two of each title: the page keeps one.
+ await put({ owner: 'live', type: 'vod', id: 'vod-omar-a', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9003' });
+ await put({ owner: 'live', type: 'vod', id: 'vod-omar-b', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9004' });
+ await put({ owner: 'live', type: 'vod', id: 'vod-seattle-a', title: 'Seattle Maybe 🤔', summary: 'Recorded live stream by goosely on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9005' });
+ await put({ owner: 'live', type: 'vod', id: 'vod-seattle-b', title: 'Seattle Maybe 🤔', summary: 'Recorded live stream by goosely on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9006' });
+ await put({ owner: 'live', type: 'vod', id: 'vod-real', title: 'Allahhuma labbaik (Part 3)', summary: 'Recorded live stream by Arman on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9007' });
+ await put({ owner: 'wiki', type: 'page', id: 'omar-summary', title: 'Weekly notes', summary: 'Omar was seen at the market.', body: 'omar', canonical_url: 'https://openvibe.wiki/p/omar-summary' });
+
+ // The Tools site: a tool page (a subdomain of openvibe.tools), its help article, a community thread.
+ await put({ owner: 'tools', type: 'tool', id: 'tool-json', title: 'JSON Formatter', summary: 'Paste messy JSON to pretty-print it. Part of OpenVibe.Tools.', body: 'json formatter', canonical_url: 'https://json.openvibe.tools' });
+ await put({ owner: 'tools', type: 'tool', id: 'tool-min', title: 'JSON Minifier', summary: 'Minify JSON in the browser. Part of OpenVibe.Tools.', body: 'json minifier', canonical_url: 'https://json-minifier.openvibe.tools' });
+ // The site's own record even when no field repeats "openvibe" or "tools".
+ await put({ owner: 'tools', type: 'tool', id: 'tool-silent', title: 'Silent Tool', summary: 'Does one small thing.', body: 'nothing about the name', canonical_url: 'https://silent.openvibe.tools' });
+ await put({ owner: 'help', type: 'article', id: 'help-tools', title: 'What kinds of tools are there?', summary: 'OpenVibe.Tools holds generators, converters and small utilities.', canonical_url: 'https://openvibe.help/a/openvibe.tools/what-kinds-of-tools-are-there' });
+ await put({ owner: 'community', type: 'thread', id: 'community-tools', title: 'Favourite tools on every device', summary: 'A thread about OpenVibe tools people use every day.', canonical_url: 'https://openvibe.community/s/roadmap/t/favourite-tools' });
+ await put({ owner: 'community', type: 'paste', id: 'snippet-json', title: 'Pastebin snippet', summary: 'A pasted file.', body: 'json formatter json formatter', canonical_url: 'https://openvibe.community/p/snippet' });
+ } catch (e) {
+ await svc.stop();
+ throw e;
+ }
+});
+
+t('an owner word is not a text match; title beats summary beats body', async () => {
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=aurora&limit=10');
+ assert.strictEqual(r.status, 200, r.text);
+ assert.deepStrictEqual(ids(r), ['aurora-title', 'aurora-summary', 'aurora-body'], 'title > summary > body, owner-only absent');
+});
+
+t('a site query puts that site\'s own records first', async () => {
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');
+ assert.strictEqual(r.status, 200, r.text);
+ const first = r.body.results[0];
+ assert.ok(['help-live-default', 'help-stream-obs', 'help-go-live', 'channel-arman'].includes(first.id), `first is a Live record: ${first.id}`);
+ const site = ['help-live-default', 'help-stream-obs', 'help-go-live', 'channel-arman'];
+ const lastSite = Math.max(...site.map((id) => indexOf(r, id)).filter((i) => i >= 0));
+ const firstVod = Math.min(...['vod-omar-a', 'vod-seattle-a', 'vod-real'].map((id) => indexOf(r, id)).filter((i) => i >= 0));
+ assert.ok(lastSite < firstVod, 'every Live record precedes the first VOD');
+
+ // A precise title query is still first, even with body noise for both words.
+ const json = await request(svc.base, 'GET', '/api/v1/search?q=json%20formatter&limit=10');
+ assert.deepStrictEqual(ids(json)[0], 'tool-json');
+
+ // The Tools site's own records come before a community thread whose title matches "tools".
+ const tools = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20tools&limit=10');
+ assert.ok(indexOf(tools, 'help-tools') < indexOf(tools, 'community-tools'), 'tools help article first');
+ assert.ok(indexOf(tools, 'tool-json') < indexOf(tools, 'community-tools'), 'tool page before the thread');
+ assert.ok(indexOf(tools, 'tool-silent') >= 0 && indexOf(tools, 'tool-silent') < indexOf(tools, 'community-tools'), 'a site page without the words is still a site record');
+});
+
+t('a title match outranks a summary-only match (not just a site)', async () => {
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=omar&limit=10');
+ assert.strictEqual(r.status, 200, r.text);
+ // "Omar" is the VOD title; the wiki page only mentions Omar in its summary.
+ assert.ok(indexOf(r, 'vod-omar-a') >= 0 && indexOf(r, 'vod-omar-a') < indexOf(r, 'omar-summary'), 'title before summary-only');
+
+ // And the plain "go live in a browser" question puts the help article first.
+ const go = await request(svc.base, 'GET', '/api/v1/search?q=go%20live%20browser&limit=10');
+ assert.deepStrictEqual(ids(go)[0], 'help-go-live');
+});
+
+t('a thin username title loses the score tie to a real title and summary', async () => {
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=zephyr&limit=10');
+ assert.deepStrictEqual(ids(r), ['zephyr-real', 'zephyr-thin'], 'the long summary wins the tie');
+ const thin = await svc.store.get('live', 'vod', 'zephyr-thin');
+ const real = await svc.store.get('live', 'vod', 'zephyr-real');
+ assert.strictEqual(isThin(thin.doc), true);
+ assert.strictEqual(isThin(real.doc), false);
+});
+
+t('one page keeps one result per title and owner', async () => {
+ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');
+ const keys = r.body.results.map(dedupeKey);
+ assert.strictEqual(new Set(keys).size, keys.length, 'no repeated title+owner');
+ assert.strictEqual(r.body.results.filter((x) => x.title === 'Omar').length, 1);
+ assert.strictEqual(r.body.results.filter((x) => x.title === 'Seattle Maybe 🤔').length, 1);
+});
+
+t('helper tiers and dedupe agree with the SQL order', () => {
+ const match = { words: ['openvibe', 'live'], prefix: false, column: null };
+ const hosts = siteHosts(match);
+ assert.deepStrictEqual(hosts.sort(), ['openre.stream', 'openvibe.live']);
+ assert.strictEqual(docTier({ title: 'Is OpenVibe Live on?', summary: '', body: '', canonical_url: 'https://openvibe.help/a/openre.stream/x' }, match, hosts), 13);
+ assert.strictEqual(docTier({ title: 'Omar', summary: 'Recorded on OpenVibe.Live.', body: '', canonical_url: 'https://openvibe.video/v/1' }, match, hosts), 2);
+ assert.strictEqual(dedupeKey({ owner: 'live', title: ' Omar ' }), dedupeKey({ owner: 'LIVE', title: 'omar' }));
+});
+
+t('close', async () => { await svc.stop(); });
+
+t.run();