openvibers/openvibe.search Commits

Search ranks titles above owners and drops repeats (#176)

OP OpenVibers committed 12 days ago · bb16b6b300 · 8 files, +431 −41

Plain queries were filling with thin Live VODs: their service boilerplate summary matched the site name and a short title inflated ts_rank, while a site query never treated the site's own pages as the site. Ranking now puts a site's records first (site 10 + title 3 / summary 2 / body 1), treats owner/type as filters rather than text, sinks thin username-titled docs inside a tier, and keeps one result per title and owner. The hybrid path re-applies the same tiers after RRF.

Changes

ddiff --git a/README.md b/README.md
iindex 8b96192..8f0e015 100644
---- a/README.md
++++ b/README.md
@@@ -98,21 +98,32 @@ an empty shelf is left out, and without the numbers the sources fall back to the
### Ranking
--Full-text results are ordered by full-text relevance (title ×10, summary ×4, body ×1) multiplied by a
--freshness boost:
++Full-text results are ordered by a tier first, then full-text relevance (title ×10, summary ×4, body ×1)
++multiplied by a freshness boost:
```
--score = relevance × (1 + W × 2^(−age_days / H)) W = SEARCH_FRESHNESS_WEIGHT (default 1)
++tier = site × 10 + field field: title 3 > summary 2 > body 1
++score = tier × 100 + relevance × (1 + W × 2^(−age_days / H))
++ W = SEARCH_FRESHNESS_WEIGHT (default 1)
H = SEARCH_FRESHNESS_HALF_LIFE_DAYS (default 30)
```
++A query naming a site ("openvibe live", "openvibe tools") puts that site's own records first: the site is
++the host `openvibe.<word>` (subdomains too) or a help article under `/a/<host>/`, with Live's old
++`openre.stream` name included. A word that matches a document's owner or type is not a text match (the
++`owner=` filter is how an owner narrows a search), a title match always outranks a summary-only match, and
++a summary-only match outranks a body-only one. Within a tier, a thin document (a username or a very short
++title with no real summary: the recorded-stream VODs) ranks below one with a real title and summary, and
++one page keeps one result per title and owner.
++
`age_days` counts from `published_at`, else `updated_at` (an edit does not make an old document
new); a future date counts as today and a missing one gets no boost. With the defaults a document
published today counts ×2.0, 30 days old ×1.5, 60 days ×1.25, 90 days ×1.125 and a year old ×1.0002:
relevance still decides, and between comparably relevant documents the newer one ranks first.
`W = 0` turns it off. Suggestions use the same score; browsing without `q` stays newest first. A
--cursor carries its first page's clock, so later pages rank on the same scores
--(`test/freshness.test.js`).
++cursor carries its first page's clock, so later pages rank on the same scores. The hybrid path fuses the
++keyword and vector top 100 by RRF and then applies the same tiers to the fused list
++(`test/freshness.test.js`, `test/ranking.test.js`).
### Saved searches
ddiff --git a/docs/adr-engine.md b/docs/adr-engine.md
iindex 2399cf7..ad0b822 100644
---- a/docs/adr-engine.md
++++ b/docs/adr-engine.md
@@@ -43,9 +43,11 @@ There is no corpus to measure: no product indexes anything today.
- One database: backups, restores and moves are the same as every other service (PostgreSQL,
ADR-035).
--- Ranking is `ts_rank` with title/summary/body weights 10/4/1 times a freshness boost
-- ([README "Ranking"](../README.md#ranking)), returned as minus the score so ascending keyset
-- cursors work.
++- Ranking is a tier (site 10 + title 3 / summary 2 / body 1) ahead of `ts_rank` with
++ title/summary/body weights 10/4/1 times a freshness boost ([README "Ranking"](../README.md#ranking)),
++ returned as minus the score so ascending keyset cursors work. The owner and type are not text
++ fields, thin username-titled VODs sink inside a tier, and one page keeps one result per title and
++ owner.
- Text is folded (diacritics removed) and split into runs of letters and digits before the
`simple` configuration sees it, so tokens are predictable across languages (Japanese/Chinese get
no word segmentation; a later engine or ICU tokenizer fixes that) and a query and the index
ddiff --git a/server/api/query.js b/server/api/query.js
iindex 617ecef..92bb5a1 100644
---- a/server/api/query.js
++++ b/server/api/query.js
@@@ -29,6 +29,7 @@ const { AuthError } = require('../auth');
const { OWNER_RE, TYPE_RE, ID_RE } = require('./documents');
const { fuseRrf } = require('../fabric/rrf');
const { createQueryEmbeddingCache } = require('../fabric/embed');
++const { docTier, isThin, dedupeKey, siteHosts } = require('../ranking');
const FACET_KEY_RE = /^[a-z][a-z0-9_]{0,39}$/;
const LANG_RE = /^[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*$/;
@@@ -162,6 +163,29 @@ async function dealsOrder(fused, { match, store }) {
.map(({ hit }) => ({ ...hit, found: rows.get(hit.rid) }));
}
++// ── A plain query's order (11-01) ──────────────────────────────────────────────────────────────
++// The keyword engine already ranks with the same tiers (server/engine/pg.js), but a hybrid page is
++// the RRF order of two top-100 lists, and a strong vector hit could still lift a summary-only VOD
++// above a help article whose title says what was asked. Re-apply the tiers to the fused list, so:
++// - a query naming a site ("openvibe live") puts that site's own records first (site 10);
++// - a title match (3) outranks a summary-only match (2), which outranks a body-only one (1);
++// - a thin username-titled VOD ranks below a real title+summary inside the same tier.
++// The fused order is the final tie-break, so relevance still decides among equals.
++async function rankOrder(fused, { match, store }) {
++ if (!match || !match.words.length) return fused;
++ const hosts = siteHosts(match);
++ const rows = new Map();
++ for (const hit of fused) rows.set(hit.rid, hit.found !== undefined ? hit.found : await store.byRid(hit.rid));
++ return fused
++ .filter((hit) => rows.get(hit.rid))
++ .map((hit, i) => {
++ const found = rows.get(hit.rid);
++ return { hit, found, i, tier: docTier(found.doc, match, hosts), thin: isThin(found.doc) };
++ })
++ .sort((a, b) => b.tier - a.tier || (a.thin === b.thin ? 0 : a.thin ? 1 : -1) || a.i - b.i)
++ .map(({ hit, found }) => ({ ...hit, found }));
++}
++
/**
* One search as a viewer: the query API, saved-search runs and the HTML page all go through here.
* run({ text, query, viewer, limit, cursor, facetKeys, now, since }) → { results, next_cursor, explain, facets? }
@@@ -240,6 +264,7 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
}
const results = [];
++ const seen = new Set();
let next = null;
let explain;
let refNow = now;
@@@ -249,6 +274,8 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
if (after && after.offset === undefined) throw new QueryError('search.bad_cursor', 'cursor does not belong to this query');
if (after && after.now != null) refNow = after.now;
let fused = await hybridCandidates({ match, filters, viewer, now: refNow, vector: state.vector });
++ // A plain query's tiers: site first, then title > summary > body, thin VODs last within a tier.
++ fused = await rankOrder(fused, { match, store });
// Deals: the product the person asked for (its title or category) beats a description-only match.
if (filters.owner === 'deals') fused = await dealsOrder(fused, { match, store });
let consumed = after ? after.offset : 0;
@@@ -260,6 +287,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
// Defence in depth: the vector store is public-only, the keyword SQL is ACL-aware.
const found = f.found !== undefined ? f.found : await store.byRid(f.rid);
if (!found || !canView(viewer, found.row, found.doc)) continue;
++ const key = dedupeKey(found.doc);
++ if (seen.has(key)) continue;
++ seen.add(key);
results.push(projection(found.doc, found.row, f.snip, f.ranks));
}
if (consumed < fused.length) next = encodeCursor({ k: 'f', o: consumed, i: lastRid, n: refNow, h: hash });
@@@ -274,6 +304,9 @@ function createSearcher({ config, store, engine, vectorStore = null, embeddings
const found = await store.byRid(h.rid);
// Defence in depth: the engine already filtered; re-check against the stored row.
if (!found || !canView(viewer, found.row, found.doc)) continue;
++ const key = dedupeKey(found.doc);
++ if (seen.has(key)) continue;
++ seen.add(key);
results.push(projection(found.doc, found.row, h.snip, { keyword: position, vector: null }));
}
next = page.next ? encodeCursor({ k: 'r', r: page.next.rank, i: page.next.rid, n: refNow, h: hash }) : null;
ddiff --git a/server/engine/pg.js b/server/engine/pg.js
iindex 27bc6f0..0ec27b0 100644
---- a/server/engine/pg.js
++++ b/server/engine/pg.js
@@@ -16,10 +16,13 @@
* Text is folded before it is indexed or queried (NFKD, combining marks removed), split into runs of letters and
* digits and tokenized by the `simple` configuration (lower case, no stemming, no stop words): the index and every
* query see the same text. Title, summary and body weigh 10, 4 and 1 (tsvector weights A, B, C), with the freshness
-- * boost described below. Every function here is async; put and remove run inside the store's
-- * transaction (ambient). Nothing outside this file writes SQL against search_fts.
++ * boost described below. The owner and type are not text fields (a word that equals an owner is not a match), and a
++ * ranking tier puts a site's own records first, then title matches, then summary-only, then body-only ones
++ * (server/ranking.js). Every function here is async; put and remove run inside the store's transaction (ambient).
++ * Nothing outside this file writes SQL against search_fts.
*/
const { EXPOSURE } = require('../document');
++const { fold, words, siteOnlyQuery, siteUrlPattern } = require('../ranking');
const CONFIG = 'simple';
const WEIGHTS = '{0.1, 0.1, 0.4, 1.0}'; // D, C body, B summary, A title: ts_rank weights 1 / 4 / 10
@@@ -67,20 +70,6 @@ function freshnessBoost(iso, nowMs, halfLifeDays, weight) {
return 1 + weight * Math.pow(2, -ageDays / halfLifeDays);
}
--/** Diacritics folded, so the index and every query see the same text. */
--function fold(s) {
-- return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');
--}
--
--/**
-- * Word tokens: runs of letters and digits, everything else a separator. PostgreSQL's parser would
-- * keep "OpenVibe.Live", "a_b" or "3.5" as single tokens (host, word, number); separating them first
-- * makes "openvibe" find "OpenVibe.Live".
-- */
--function words(s) {
-- return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');
--}
--
function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {}) {
const st = {
del: db.prepare('DELETE FROM search_fts WHERE rid = ?'),
@@@ -130,31 +119,66 @@ function createEngine(db, { freshness = { weight: 1, halfLifeDays: 30 } } = {})
}
/**
-- * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus ts_rank times the
-- * freshness boost at reference time `now` (ms), so lower ranks first and pages of one query
-- * (which carry their first page's `now` in the cursor) rank against the same clock.
++ * The visible, matching set as (rid, rank) rows. match = toMatch()'s query. rank is minus the
++ * tier (site 10 + title 3 / summary 2 / body 1) and ts_rank times the freshness boost at
++ * reference time `now` (ms), so lower ranks first and pages of one query (which carry their
++ * first page's `now` in the cursor) rank against the same clock. The owner and type are not in
++ * the tsvector: a word that only equals an owner never matches. Rows whose owner and title are
++ * the same keep only the best-ranked one.
*/
function matchedSet({ match, filters, viewer, now = Date.now() }, params) {
params.match = tsquery(match);
params.r_now = now;
params.r_half = freshness.halfLifeDays;
params.r_weight = freshness.weight;
++ params.site_re = siteUrlPattern(match) || '';
const where = filterClause(filters, params);
-- const rank = `-(ts_rank('${WEIGHTS}'::float4[], f.tsv, to_tsquery('${CONFIG}', @match), 1)
++ const q = `to_tsquery('${CONFIG}', @match)`;
++ const base = `ts_rank('${WEIGHTS}'::float4[], f.tsv, ${q}, 1)
* CASE WHEN f.fresh_at IS NULL OR @r_weight::float8 = 0 THEN 1
-- ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END)`;
-- const branches = [`SELECT d.rid AS rid, ${rank} AS rank
++ ELSE 1 + @r_weight::float8 * power(2::float8, -(GREATEST(0, @r_now::bigint - f.fresh_at)::float8 / ${DAY_MS}) / @r_half::float8) END`;
++ const title = `ts_rank('{0,0,0,1}'::float4[], f.tsv, ${q}, 0)`;
++ const summary = `ts_rank('{0,0,0.4,0}'::float4[], f.tsv, ${q}, 0)`;
++ const body = `ts_rank('{0,0.1,0,0}'::float4[], f.tsv, ${q}, 0)`;
++ const site = `(@site_re::text <> '' AND d.canonical_url ~* @site_re)`;
++ const field = `CASE WHEN ${title} > 0 THEN 3 WHEN ${summary} > 0 THEN 2 WHEN ${body} > 0 THEN 1 ELSE 0 END`;
++ const tier = `(CASE WHEN ${site} THEN 10 ELSE 0 END + ${field})`;
++ const thin = `(char_length(btrim(f.title)) <= 24 AND char_length(btrim(f.summary)) <= 80)`;
++ // Thin VODs sink below documents with a real title and summary inside the same tier; -2 is
++ // larger than any base score (ts_rank × freshness < 2), so it decides before relevance does.
++ const rank = `-(${tier} * 100 + (${base}) + CASE WHEN ${thin} THEN -2 ELSE 0 END)`;
++ const select = `SELECT d.rid AS rid, d.owner AS owner, f.title AS title, ${rank} AS rank`;
++ const branches = [`${select}
FROM search_fts f JOIN documents d ON d.rid = f.rid
-- WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ to_tsquery('${CONFIG}', @match)
++ WHERE f.audience = ${EXPOSURE.public_listed} AND f.tsv @@ ${q}
AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`];
const acl = aclClause(viewer, params);
if (acl) {
-- branches.push(`SELECT d.rid AS rid, ${rank} AS rank
++ branches.push(`${select}
FROM search_fts f JOIN documents d ON d.rid = f.rid
-- WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ to_tsquery('${CONFIG}', @match)
++ WHERE f.audience = ${EXPOSURE.restricted} AND f.tsv @@ ${q}
AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
}
-- return branches.join('\nUNION ALL\n');
++ // "openvibe live"/"openvibe tools": the site's own records are candidates even when their text
++ // does not repeat the site name (the Tools pages never do). They rank in the site tier below the
++ // site records that do match the words; the window dedupe folds a document that is in both sets.
++ if (params.site_re && siteOnlyQuery(match)) {
++ branches.push(`${select}
++ FROM search_fts f JOIN documents d ON d.rid = f.rid
++ WHERE f.audience = ${EXPOSURE.public_listed} AND d.canonical_url ~* @site_re
++ AND d.exposure = ${EXPOSURE.public_listed} AND d.deleted = 0${where}`);
++ if (acl) {
++ branches.push(`${select}
++ FROM search_fts f JOIN documents d ON d.rid = f.rid
++ WHERE f.audience = ${EXPOSURE.restricted} AND d.canonical_url ~* @site_re
++ AND d.exposure = ${EXPOSURE.restricted} AND d.deleted = 0${where} AND ${acl}`);
++ }
++ }
++ return `SELECT rid, rank FROM (
++ SELECT m.rid, m.rank,
++ ROW_NUMBER() OVER (PARTITION BY m.owner, lower(btrim(m.title)) ORDER BY m.rank, m.rid) AS rn
++ FROM (${branches.join('\nUNION ALL\n')}) m
++ ) x WHERE x.rn = 1`;
}
/** The visible set without a text query (browse). */
ddiff --git a/server/ranking.js b/server/ranking.js
nnew file mode 100644
iindex 0000000..d4efca4
---- /dev/null
++++ b/server/ranking.js
@@@ -0,0 +1,161 @@
++'use strict';
++/**
++ * Ranking helpers shared by the full-text engine (server/engine/pg.js) and the query path
++ * (server/api/query.js): which fields a query matches, whether a document belongs to a site the
++ * query names, whether its title is a thin streamer name, and the identity that drops duplicate
++ * results from a page.
++ *
++ * The engine's base score is ts_rank with title A (×10), summary B (×4) and body C (×1), times the
++ * freshness boost. That ranks term frequency, but not the things a person means by a plain query:
++ *
++ * tier = site (10) + field (title 3, summary 2, body 1, none 0)
++ *
++ * A query naming a site ("openvibe live", "openvibe tools") puts that site's own records first
++ * (its home, its help articles, its tool pages), a title match always outranks a summary-only
++ * match, and a summary-only match outranks a body-only one. The owner and type are deliberately
++ * not text fields: a word that equals a document's owner is not a match (the explicit owner=
++ * filter is how an owner narrows a query), and the summary's service boilerplate ("Recorded live
++ * stream by … on OpenVibe.Live.") is only summary text, never a title.
++ */
++
++/** Sites whose old name still appears in canonical URLs and help paths. */
++const SITE_ALIASES = {
++ live: ['openvibe.live', 'openre.stream'],
++ openre: ['openre.stream', 'openvibe.live'],
++ stream: ['openre.stream', 'openvibe.live'],
++ restream: ['openre.stream', 'openvibe.live'],
++};
++
++/** Diacritics folded, so the index and every query see the same text. */
++function fold(s) {
++ return String(s || '').normalize('NFKD').replace(/\p{M}+/gu, '').normalize('NFC');
++}
++
++/** Word tokens: runs of letters and digits, everything else a separator. */
++function words(s) {
++ return fold(s).replace(/[^\p{L}\p{N}]+/gu, ' ');
++}
++
++function tokens(s) {
++ return fold(s).toLowerCase().match(/[\p{L}\p{N}]+/gu) || [];
++}
++
++function escapeRegex(s) {
++ return String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
++}
++
++/**
++ * The site hosts a query names, if it has "openvibe" plus another word: "openvibe live" names
++ * openvibe.live and its old name openre.stream; "openvibe tools" names openvibe.tools. The host of
++ * an unknown word simply matches nothing. No site without "openvibe" (a plain "live" is text, not
++ * the Live site).
++ */
++function siteHosts(match) {
++ const ws = match && Array.isArray(match.words) ? match.words : [];
++ if (!ws.includes('openvibe')) return [];
++ const hosts = new Set();
++ for (const word of ws) {
++ if (word === 'openvibe' || word.length < 2) continue;
++ for (const host of (SITE_ALIASES[word] || [`openvibe.${word}`])) hosts.add(host);
++ }
++ return [...hosts].slice(0, 12);
++}
++
++/**
++ * The same test as siteHosts(), as a PostgreSQL regex (engine SQL). A document belongs to the
++ * site when its host is (a subdomain of) a named host, or when it is a help article under
++ * /a/<host>/ for one of them. Null when the query names no site.
++ */
++function siteUrlPattern(match) {
++ const hosts = siteHosts(match);
++ if (!hosts.length) return null;
++ const alt = hosts.map(escapeRegex).join('|');
++ return `^https?://([a-z0-9-]+\\.)*(${alt})(/|$)|^https?://([a-z0-9-]+\\.)*openvibe\\.help/a/(${alt})(/|$)`;
++}
++
++/**
++ * Is this a two-word site query ("openvibe live", "openvibe tools")? Then the site's own records are
++ * candidates even when the page text never says "openvibe" or the site name: the site membership is
++ * the second word, like a filter. A title-only (suggest) query never widens this way.
++ */
++function siteOnlyQuery(match) {
++ const ws = match && Array.isArray(match.words) ? match.words : [];
++ return !match.column && ws.length === 2 && ws.includes('openvibe');
++}
++
++function parseUrl(url) {
++ try {
++ const u = new URL(String(url));
++ return { host: u.hostname.toLowerCase(), path: u.pathname };
++ } catch {
++ return null;
++ }
++}
++
++function hostMatches(host, site) {
++ return host === site || host.endsWith(`.${site}`);
++}
++
++/** Does this canonical URL belong to one of the sites? Mirrors siteUrlPattern() in JS. */
++function urlBelongsToSites(url, hosts) {
++ if (!hosts || !hosts.length) return false;
++ const parsed = parseUrl(url);
++ if (!parsed) return false;
++ if (hosts.some((host) => hostMatches(parsed.host, host))) return true;
++ if (hostMatches(parsed.host, 'openvibe.help')) {
++ const m = /^\/a\/([^/]+)\//.exec(parsed.path);
++ if (m && hosts.some((host) => hostMatches(m[1].toLowerCase(), host))) return true;
++ }
++ return false;
++}
++
++function wordMatches(token, word, prefix) {
++ return token === word || (prefix && token.startsWith(word));
++}
++
++/** Does any query word appear in this text? The last word is a prefix when the query says so. */
++function textMatches(text, match) {
++ if (!match || !Array.isArray(match.words) || !match.words.length) return false;
++ const ts = tokens(text);
++ return match.words.some((word, i) => ts.some((token) => wordMatches(token, word, Boolean(match.prefix && i === match.words.length - 1))));
++}
++
++/** 3 when the title matches, 2 the summary, 1 the body, 0 no text field (a vector-only hit). */
++function fieldTier(doc, match) {
++ if (!doc) return 0;
++ if (textMatches(doc.title, match)) return 3;
++ if (textMatches(doc.summary, match)) return 2;
++ if (textMatches(doc.body, match)) return 1;
++ return 0;
++}
++
++/** site (10) + the best matching field: a site's own records first, then title > summary > body. */
++function docTier(doc, match, hosts = siteHosts(match)) {
++ const site = hosts.length && urlBelongsToSites(doc && doc.canonical_url, hosts) ? 10 : 0;
++ return site + fieldTier(doc, match);
++}
++
++/**
++ * A thin document: a username or a very short title with no real summary ("aaooo.ee", "Omar",
++ * "Seattle Maybe 🤔" — a recorded stream whose summary is the site's sentence). It ranks below a
++ * document with a real title and summary inside the same tier; the field and site tiers are unchanged.
++ */
++function isThin(doc) {
++ const title = String((doc && doc.title) || '').trim();
++ const summary = String((doc && doc.summary) || '').trim();
++ if (!title) return true;
++ const shortTitle = title.length <= 24;
++ const noRealSummary = summary.length === 0 || summary.length <= 80;
++ return shortTitle && noRealSummary;
++}
++
++/** The one page identity: same title and owner is one result (keep the best-ranked one). */
++function dedupeKey(doc) {
++ const title = String((doc && doc.title) || '').trim().replace(/\s+/g, ' ').toLowerCase();
++ return `${String((doc && doc.owner) || '').toLowerCase()}\u0000${title}`;
++}
++
++module.exports = {
++ SITE_ALIASES, fold, words, tokens, siteHosts, siteOnlyQuery, siteUrlPattern, urlBelongsToSites,
++ textMatches, fieldTier, docTier, isThin, dedupeKey,
++};
ddiff --git a/test/freshness.test.js b/test/freshness.test.js
iindex f3afe5a..bcc5d10 100644
---- a/test/freshness.test.js
++++ b/test/freshness.test.js
@@@ -29,11 +29,12 @@ t('the decay: ×(1+w) today, ×(1+w/2) one half-life ago, → ×1 when old; futu
async function corpus(env = {}) {
let clock = NOW;
const svc = await boot({ env, now: () => clock });
-- // Same text, same length: bm25 ties, so only freshness separates them. The old one is indexed
-- // first (lower rid), so without freshness it would come first.
-- await svc.store.apply(doc({ id: 'old', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));
-- await svc.store.apply(doc({ id: 'mid', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));
-- await svc.store.apply(doc({ id: 'new', title: 'Aurora report', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));
++ // Same query term, same length: bm25 ties, so only freshness separates them. Distinct titles keep
++ // the page's same-title dedupe from folding them into one. The old one is indexed first (lower
++ // rid), so without freshness it would come first.
++ await svc.store.apply(doc({ id: 'old', title: 'Aurora report old', body: 'aurora seen tonight', published_at: daysAgo(400), updated_at: daysAgo(400) }));
++ await svc.store.apply(doc({ id: 'mid', title: 'Aurora report mid', body: 'aurora seen tonight', published_at: daysAgo(20), updated_at: daysAgo(20) }));
++ await svc.store.apply(doc({ id: 'new', title: 'Aurora report new', body: 'aurora seen tonight', published_at: daysAgo(1), updated_at: daysAgo(1) }));
return { svc, setClock: (c) => { clock = c; } };
}
ddiff --git a/test/hybrid.test.js b/test/hybrid.test.js
iindex 0c3c6e1..7711f52 100644
---- a/test/hybrid.test.js
++++ b/test/hybrid.test.js
@@@ -315,9 +315,37 @@ t('deals: a title or category match outranks a description-only match (11-01)',
assert.deepStrictEqual(r.body.results.find((x) => x.id === 'k2').ranks, { keyword: 1, vector: null });
assert.deepStrictEqual(r.body.results.find((x) => x.id === 'bit-blaster').ranks, { keyword: 2, vector: 1 });
-- // The order is only the deals owner's: the same two documents unowned keep the fused order.
++ // The title tier is not deals-only: across owners, the keyboard's title match now outranks the
++ // game's summary-only match too (the deals category rule still covers a category-only match).
const all = await request(svc.base, 'GET', '/api/v1/search?q=mechanical%20keyboard&limit=10');
-- assert.deepStrictEqual(all.body.results.map((x) => x.id), ['bit-blaster', 'k2'], 'other owners stay on RRF');
++ assert.deepStrictEqual(all.body.results.map((x) => x.id), ['k2', 'bit-blaster'], 'title match first outside deals too');
++ } finally {
++ await svc.stop();
++ }
++});
++
++t('ranking: a site query and title tier survive RRF, and a page keeps one title per owner', async () => {
++ const vectors = new FakeVectors();
++ const embeddings = fakeEmbeddings();
++ const svc = await boot({ vectorStore: vectors, embeddingClient: embeddings });
++ try {
++ await svc.store.apply(doc({ owner: 'help', type: 'article', id: 'live-help', title: 'Is OpenVibe Live on by default?', summary: 'Yes. Every channel streams with OpenVibe.Live on by default.', canonical_url: 'https://openvibe.help/a/openre.stream/is-openvibe-live-on-by-default' }));
++ await svc.store.apply(doc({ owner: 'community', type: 'thread', id: 'live-thread', title: 'OpenVibe live thread', summary: 'A discussion about going live.', canonical_url: 'https://openvibe.community/s/roadmap/t/live-thread' }));
++ await svc.store.apply(doc({ owner: 'live', type: 'vod', id: 'vod-a', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/1' }));
++ await svc.store.apply(doc({ owner: 'live', type: 'vod', id: 'vod-b', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/2' }));
++ const rid = await ridMap(svc.db);
++ // The vector side likes the thin VODs and the thread; the site record has no vector hit at all.
++ vectors.hits = [
++ { rid: rid['vod-a'], distance: 0.1 },
++ { rid: rid['vod-b'], distance: 0.11 },
++ { rid: rid['live-thread'], distance: 0.12 },
++ ];
++
++ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');
++ assert.strictEqual(r.status, 200, r.text);
++ assert.deepStrictEqual(r.body.explain, { ranking: 'hybrid' });
++ assert.deepStrictEqual(r.body.results.map((x) => x.id), ['live-help', 'live-thread', 'vod-a'], 'site first, title before summary, one Omar');
++ assert.strictEqual(r.body.results.filter((x) => x.title === 'Omar').length, 1);
} finally {
await svc.stop();
}
ddiff --git a/test/ranking.test.js b/test/ranking.test.js
nnew file mode 100644
iindex 0000000..e48c698
---- /dev/null
++++ b/test/ranking.test.js
@@@ -0,0 +1,130 @@
++'use strict';
++/**
++ * Plain-query ranking: a word that only matches an owner is not a match; a title match outranks a
++ * summary match, which outranks a body match; a site a query names puts its own records first; a
++ * thin username-titled VOD with no real summary loses a score tie; and one page never repeats a
++ * title for the same owner. Seeded index, no vectors: this is the keyword order the hybrid page
++ * re-applies after RRF (test/hybrid.test.js covers that side).
++ */
++const assert = require('assert');
++const { boot, request, doc, suite } = require('./helpers');
++const { isThin, dedupeKey, docTier, siteHosts } = require('../server/ranking');
++
++const t = suite('ranking');
++let svc;
++const AT = '2026-11-01T00:00:00Z';
++
++const put = (overrides) => svc.store.apply(doc({
++ published_at: AT, updated_at: AT, summary: '', body: '', ...overrides,
++}));
++const ids = (r) => r.body.results.map((x) => x.id);
++const indexOf = (r, id) => ids(r).indexOf(id);
++
++t('boot and seed', async () => {
++ svc = await boot();
++ try {
++ // Field order: same query word in title, summary, body, and only in the owner.
++ await put({ owner: 'wiki', type: 'page', id: 'aurora-title', title: 'Aurora guide', summary: 'A guide to the lights.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-title' });
++ await put({ owner: 'wiki', type: 'page', id: 'aurora-summary', title: 'Weekly notes', summary: 'Aurora was bright this week.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-summary' });
++ await put({ owner: 'wiki', type: 'page', id: 'aurora-body', title: 'Picnic plans', summary: 'Sunny afternoon.', body: 'aurora', canonical_url: 'https://openvibe.wiki/p/aurora-body' });
++ await put({ owner: 'aurora', type: 'page', id: 'aurora-owner', title: 'Quiet notes', summary: 'Nothing to see.', body: 'nothing at all', canonical_url: 'https://openvibe.wiki/p/aurora-owner' });
++
++ // Thin: a username title and no summary. Real: a long summary; the same title/body match.
++ await put({ owner: 'live', type: 'vod', id: 'zephyr-thin', title: 'zephyr', summary: '', body: 'zephyr', canonical_url: 'https://openvibe.video/v/9001' });
++ await put({ owner: 'live', type: 'vod', id: 'zephyr-real', title: 'zephyr report', summary: 'A full report on the channel, its schedule, its recent streams, and everything that happened this week.', body: 'zephyr', canonical_url: 'https://openvibe.video/v/9002' });
++
++ // The Live page's own records: help articles (the old openre.stream name too) and a channel.
++ await put({ owner: 'help', type: 'article', id: 'help-live-default', title: 'Is OpenVibe Live on by default?', summary: 'Yes. Every channel streams with OpenVibe.Live on by default.', canonical_url: 'https://openvibe.help/a/openre.stream/is-openvibe-live-on-by-default' });
++ await put({ owner: 'help', type: 'article', id: 'help-stream-obs', title: 'Can I stream without OBS?', summary: 'Yes. Go live from your browser on OpenVibe.Live: press Go Live and you are live.', canonical_url: 'https://openvibe.help/a/openre.stream/can-i-stream-without-obs' });
++ await put({ owner: 'help', type: 'article', id: 'help-go-live', title: 'Can I go live without OBS?', summary: 'Yes. Go live straight from your browser, camera or screen on OpenVibe.Live, with no software to install.', canonical_url: 'https://openvibe.help/a/openvibe.live/can-i-go-live-without-obs' });
++ await put({ owner: 'live', type: 'channel', id: 'channel-arman', title: 'Arman (@Arman)', summary: 'Arman streams on OpenVibe.Live.', canonical_url: 'https://openvibe.live/@Arman' });
++
++ // Thin VODs, two of each title: the page keeps one.
++ await put({ owner: 'live', type: 'vod', id: 'vod-omar-a', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9003' });
++ await put({ owner: 'live', type: 'vod', id: 'vod-omar-b', title: 'Omar', summary: 'Recorded live stream by omar on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9004' });
++ await put({ owner: 'live', type: 'vod', id: 'vod-seattle-a', title: 'Seattle Maybe 🤔', summary: 'Recorded live stream by goosely on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9005' });
++ await put({ owner: 'live', type: 'vod', id: 'vod-seattle-b', title: 'Seattle Maybe 🤔', summary: 'Recorded live stream by goosely on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9006' });
++ await put({ owner: 'live', type: 'vod', id: 'vod-real', title: 'Allahhuma labbaik (Part 3)', summary: 'Recorded live stream by Arman on OpenVibe.Live.', canonical_url: 'https://openvibe.video/v/9007' });
++ await put({ owner: 'wiki', type: 'page', id: 'omar-summary', title: 'Weekly notes', summary: 'Omar was seen at the market.', body: 'omar', canonical_url: 'https://openvibe.wiki/p/omar-summary' });
++
++ // The Tools site: a tool page (a subdomain of openvibe.tools), its help article, a community thread.
++ await put({ owner: 'tools', type: 'tool', id: 'tool-json', title: 'JSON Formatter', summary: 'Paste messy JSON to pretty-print it. Part of OpenVibe.Tools.', body: 'json formatter', canonical_url: 'https://json.openvibe.tools' });
++ await put({ owner: 'tools', type: 'tool', id: 'tool-min', title: 'JSON Minifier', summary: 'Minify JSON in the browser. Part of OpenVibe.Tools.', body: 'json minifier', canonical_url: 'https://json-minifier.openvibe.tools' });
++ // The site's own record even when no field repeats "openvibe" or "tools".
++ await put({ owner: 'tools', type: 'tool', id: 'tool-silent', title: 'Silent Tool', summary: 'Does one small thing.', body: 'nothing about the name', canonical_url: 'https://silent.openvibe.tools' });
++ await put({ owner: 'help', type: 'article', id: 'help-tools', title: 'What kinds of tools are there?', summary: 'OpenVibe.Tools holds generators, converters and small utilities.', canonical_url: 'https://openvibe.help/a/openvibe.tools/what-kinds-of-tools-are-there' });
++ await put({ owner: 'community', type: 'thread', id: 'community-tools', title: 'Favourite tools on every device', summary: 'A thread about OpenVibe tools people use every day.', canonical_url: 'https://openvibe.community/s/roadmap/t/favourite-tools' });
++ await put({ owner: 'community', type: 'paste', id: 'snippet-json', title: 'Pastebin snippet', summary: 'A pasted file.', body: 'json formatter json formatter', canonical_url: 'https://openvibe.community/p/snippet' });
++ } catch (e) {
++ await svc.stop();
++ throw e;
++ }
++});
++
++t('an owner word is not a text match; title beats summary beats body', async () => {
++ const r = await request(svc.base, 'GET', '/api/v1/search?q=aurora&limit=10');
++ assert.strictEqual(r.status, 200, r.text);
++ assert.deepStrictEqual(ids(r), ['aurora-title', 'aurora-summary', 'aurora-body'], 'title > summary > body, owner-only absent');
++});
++
++t('a site query puts that site\'s own records first', async () => {
++ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');
++ assert.strictEqual(r.status, 200, r.text);
++ const first = r.body.results[0];
++ assert.ok(['help-live-default', 'help-stream-obs', 'help-go-live', 'channel-arman'].includes(first.id), `first is a Live record: ${first.id}`);
++ const site = ['help-live-default', 'help-stream-obs', 'help-go-live', 'channel-arman'];
++ const lastSite = Math.max(...site.map((id) => indexOf(r, id)).filter((i) => i >= 0));
++ const firstVod = Math.min(...['vod-omar-a', 'vod-seattle-a', 'vod-real'].map((id) => indexOf(r, id)).filter((i) => i >= 0));
++ assert.ok(lastSite < firstVod, 'every Live record precedes the first VOD');
++
++ // A precise title query is still first, even with body noise for both words.
++ const json = await request(svc.base, 'GET', '/api/v1/search?q=json%20formatter&limit=10');
++ assert.deepStrictEqual(ids(json)[0], 'tool-json');
++
++ // The Tools site's own records come before a community thread whose title matches "tools".
++ const tools = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20tools&limit=10');
++ assert.ok(indexOf(tools, 'help-tools') < indexOf(tools, 'community-tools'), 'tools help article first');
++ assert.ok(indexOf(tools, 'tool-json') < indexOf(tools, 'community-tools'), 'tool page before the thread');
++ assert.ok(indexOf(tools, 'tool-silent') >= 0 && indexOf(tools, 'tool-silent') < indexOf(tools, 'community-tools'), 'a site page without the words is still a site record');
++});
++
++t('a title match outranks a summary-only match (not just a site)', async () => {
++ const r = await request(svc.base, 'GET', '/api/v1/search?q=omar&limit=10');
++ assert.strictEqual(r.status, 200, r.text);
++ // "Omar" is the VOD title; the wiki page only mentions Omar in its summary.
++ assert.ok(indexOf(r, 'vod-omar-a') >= 0 && indexOf(r, 'vod-omar-a') < indexOf(r, 'omar-summary'), 'title before summary-only');
++
++ // And the plain "go live in a browser" question puts the help article first.
++ const go = await request(svc.base, 'GET', '/api/v1/search?q=go%20live%20browser&limit=10');
++ assert.deepStrictEqual(ids(go)[0], 'help-go-live');
++});
++
++t('a thin username title loses the score tie to a real title and summary', async () => {
++ const r = await request(svc.base, 'GET', '/api/v1/search?q=zephyr&limit=10');
++ assert.deepStrictEqual(ids(r), ['zephyr-real', 'zephyr-thin'], 'the long summary wins the tie');
++ const thin = await svc.store.get('live', 'vod', 'zephyr-thin');
++ const real = await svc.store.get('live', 'vod', 'zephyr-real');
++ assert.strictEqual(isThin(thin.doc), true);
++ assert.strictEqual(isThin(real.doc), false);
++});
++
++t('one page keeps one result per title and owner', async () => {
++ const r = await request(svc.base, 'GET', '/api/v1/search?q=openvibe%20live&limit=10');
++ const keys = r.body.results.map(dedupeKey);
++ assert.strictEqual(new Set(keys).size, keys.length, 'no repeated title+owner');
++ assert.strictEqual(r.body.results.filter((x) => x.title === 'Omar').length, 1);
++ assert.strictEqual(r.body.results.filter((x) => x.title === 'Seattle Maybe 🤔').length, 1);
++});
++
++t('helper tiers and dedupe agree with the SQL order', () => {
++ const match = { words: ['openvibe', 'live'], prefix: false, column: null };
++ const hosts = siteHosts(match);
++ assert.deepStrictEqual(hosts.sort(), ['openre.stream', 'openvibe.live']);
++ assert.strictEqual(docTier({ title: 'Is OpenVibe Live on?', summary: '', body: '', canonical_url: 'https://openvibe.help/a/openre.stream/x' }, match, hosts), 13);
++ assert.strictEqual(docTier({ title: 'Omar', summary: 'Recorded on OpenVibe.Live.', body: '', canonical_url: 'https://openvibe.video/v/1' }, match, hosts), 2);
++ assert.strictEqual(dedupeKey({ owner: 'live', title: ' Omar ' }), dedupeKey({ owner: 'LIVE', title: 'omar' }));
++});
++
++t('close', async () => { await svc.stop(); });
++
++t.run();