openvibers/openvibe.meme Commits

Meme entries reach OpenVibe.Search (search-relay) (#65)

OP OpenVibers committed 11 days ago · 44d103a68b · 11 files, +133 −13

Every public entry has queued a search.index-document@1 in meme_outbox since launch, and nothing
sent them: 118 rows, none delivered. openvibe-publishing/search-relay (v1.5.0) now sends them
with Meme's search.document.write token, a few seconds after a change and every minute; off
without MEME_SEARCH_URL, never under MEME_DRILL, stopped on shutdown.

Changes

ddiff --git a/.env.example b/.env.example
iindex 75627e6..7695b9d 100644
---- a/.env.example
++++ b/.env.example
@@@ -34,6 +34,8 @@ OV_OAUTH_CLIENT_SECRET=
  MEME_EVENTS_SECRET=
  MEME_EVENTS_URL=
  #MEME_EVENTS_ENDPOINT=http://127.0.0.1:5120/internal/events
++# OpenVibe.Search's owner API: the queued Search documents (meme_outbox) are delivered there. Unset: they wait.
++#MEME_SEARCH_URL=http://127.0.0.1:4710
  #MEME_EVENTS_INTERVAL_MS=5000
  
  # The public APIs the importer and the citation re-verifier read. No service grant is used for them.
ddiff --git a/README.md b/README.md
iindex eb293f3..aa5e98a 100644
---- a/README.md
++++ b/README.md
@@@ -100,8 +100,9 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta
    The one service grant Meme holds is OpenVibe.AI's `ai.run.create`, used by the judge and the encyclopedia.
  - **Identity, roles and accounts** — OpenVibe.Network. Meme verifies session tokens against its JWKS; a staff
    route asks for `global_mod` or above.
--- **Search** — OpenVibe.Search. Meme produces `search.index-document@1` documents in its own outbox; the Search
--  document writer is not wired yet because `meme` has no `search.document.write` grant.
++- **Search** — OpenVibe.Search. Meme produces `search.index-document@1` documents in its own outbox, and
++  `openvibe-publishing/search-relay` delivers them to Search's owner API with Meme's `search.document.write` grant
++  (`MEME_SEARCH_URL`; unset, they wait).
  - **Events, the host and the database** — OpenVibe.Events, OpenVibe.Host.
  
  ## Depends on
@@@ -304,8 +305,8 @@ A transient 5xx is recorded and changes nothing: a hiccup must not erase a real
  verbatim quotes as plain text), facets `kind`/`status`/`source`, authorship `ai_assisted`, and one provenance
  reference per live citation. Publishing or editing queues `meme.index_document.upserted`; hiding, deleting or
  losing the origin queues `meme.index_document.deleted` (a tombstone wins over an older revision). Rows live in
--`meme_outbox` in the same transaction as the entry change and wait for the `search.document.write` grant; the relay
--is deliberately not part of v1. The `search-feed` recipe (Work, Help, Rent, Quest) is followed: a document from one
++`meme_outbox` in the same transaction as the entry change; `openvibe-publishing/search-relay` (v1.5.0) sends them to
++Search's owner API a few seconds after the change and every minute, oldest first, superseding a document's older rows. The `search-feed` recipe (Work, Help, Rent, Quest) is followed: a document from one
  row, a sequencer revision in the row, a tombstone when the page is no longer public.
  
  ## Pages
ddiff --git a/STATUS.json b/STATUS.json
iindex 4fd3d60..f59b706 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -40,13 +40,12 @@
      "Search Fabric documents (owner meme, type entry, /m/<slug>) with scope/labels facets and the origin as provenance, in meme_outbox, with a tombstone when an entry is hidden",
      "The caption generator: a template, 1–4 lines × 120 characters, a blocked-word list, server-rendered HTML preview, text lines stored in jsonb",
      "The API: public cursor-paged entries, an entry with citations and timeline, a zero-filled use series, trending, the week, templates, captions, reports, and staff publish/hide/patch",
--    "Search Fabric documents (owner meme, type entry, /m/<slug>) in meme_outbox, with a tombstone when an entry is hidden",
++    "Search Fabric documents (owner meme, type entry, /m/<slug>) in meme_outbox, with a tombstone when an entry is hidden, delivered to Search's owner API by openvibe-publishing/search-relay",
      "Sign-in with OpenVibe.Network (PKCE S256, meme_at/meme_rt/meme_oauth), staff role global_mod+",
      "Account export and deletion over votes, captions and reports",
      "Server-rendered pages with stable mm- ids/classes, cache headers, discovery artifacts and deploy files"
    ],
    "notYet": [
--    "The Search document relay: documents wait in meme_outbox until Network grants meme search.document.write",
      "A per-message Chat permalink API; a chat citation stores the message id and the public history is re-read to verify it",
      "Automatic status transitions by the proposed rising/peaked/classic/dead rules; the importer sets a first status and staff can change it",
      "Real template artwork: v1 ships three data-free local SVG placeholders; a real template is an openvibe.media object",
ddiff --git a/package-lock.json b/package-lock.json
iindex 2a4d01a..b73e6d0 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -17,6 +17,7 @@
          "iovalkey": "^0.4.0",
          "opentype.js": "^1.3.4",
          "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.234.0",
++        "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
          "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
          "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
          "pg": "^8.23.0",
@@@ -1646,6 +1647,24 @@
          "node": ">=20"
        }
      },
++    "node_modules/openvibe-publishing": {
++      "version": "1.5.0",
++      "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
++      "integrity": "sha512-8cKVhJvpbDgWuGR+Dh1mpgqfWPJoC7EIcw7eTRc5/2ueDc8caCA7XKdcpAHNZYMdukIAL8o7FvxE8QV5/1uSLw==",
++      "license": "MIT",
++      "engines": {
++        "node": ">=20"
++      },
++      "peerDependencies": {
++        "openvibe-sdk": ">=0.15.0",
++        "openvibe-shared": ">=1.5.0"
++      },
++      "peerDependenciesMeta": {
++        "openvibe-shared": {
++          "optional": true
++        }
++      }
++    },
      "node_modules/openvibe-sdk": {
        "version": "0.81.0",
        "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
ddiff --git a/package.json b/package.json
iindex 20b7eed..88a4c08 100644
---- a/package.json
++++ b/package.json
@@@ -28,10 +28,11 @@
      "iovalkey": "^0.4.0",
      "opentype.js": "^1.3.4",
      "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.234.0",
++    "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
      "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
      "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
      "pg": "^8.23.0",
--        "sharp": "^0.33.5"
++    "sharp": "^0.33.5"
    },
    "engines": {
      "node": ">=22.0.0"
ddiff --git a/server/app.js b/server/app.js
iindex 2322a43..bcd31d7 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -35,6 +35,7 @@ const { raw } = require('./render/html');
  const accountDataLib = require('./identity/account-data');
  const { createStore } = require('./store');
  const { createSearchIndex } = require('./search');
++const { createSearchRelay } = require('openvibe-publishing/search-relay');
  const { createUpstreams } = require('./upstreams');
  const { createImporter } = require('./importer');
  const { createReverifier } = require('./reverify');
@@@ -75,7 +76,16 @@ async function createApp(opts = {}) {
      const sso = createSso({ config, keys, fetchImpl, now: s.now, log });
      const principal = createPrincipal({ config, keys });
  
--    const search = createSearchIndex({ config, s });
++    // The queued Search documents (meme_outbox) go to OpenVibe.Search's owner API with this service's token; off
++    // without MEME_SEARCH_URL or the client secret. Started by server/index.js, never under MEME_DRILL.
++    const searchRelay = createSearchRelay({
++        db: s.db, table: 'meme_outbox', owner: 'meme', searchUrl: config.search.url, fetchImpl, now: s.now, log,
++        auth: config.search.url && config.oauth.clientSecret ? contracts.serviceAuth.createTokenClient({
++            tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,
++            audience: 'openvibe.search', scope: 'search.document.write', fetchImpl,
++        }) : null,
++    });
++    const search = createSearchIndex({ config, s, onQueued: () => searchRelay.kick() });
      const entries = createStore(s, { search });
      const upstreams = createUpstreams({ config, fetchImpl, log });
      const importer = createImporter({ config, s, entries, upstreams, fetchImpl, log });
@@@ -108,7 +118,7 @@ async function createApp(opts = {}) {
          : async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Meme cannot answer account events'); });
  
      const ctx = {
--        config, s, keys, sso, principal, search, entries, upstreams, importer, reverifier, accountData, accountSend, log,
++        config, s, keys, sso, principal, search, searchRelay, entries, upstreams, importer, reverifier, accountData, accountSend, log,
          ai: aiRunner, judgeAi: ai, context, judge, judgeJob,
          media, trends, reddit, sourceRegistration,
          redditReader, analyzer,
ddiff --git a/server/config.js b/server/config.js
iindex d562011..237c5f9 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -61,6 +61,10 @@ function load(env = process.env) {
              scope: env.MEME_EVENTS_SCOPE || 'events.event.publish',
          },
  
++        // OpenVibe.Search's owner API: the documents queued in meme_outbox are delivered there (server/app.js,
++        // openvibe-publishing/search-relay) with this service's token (search.document.write). Unset: they wait.
++        search: { url: trim(env.MEME_SEARCH_URL || '') },
++
          // The public APIs the importer reads. No service grant is used in v1: these are the anonymous reads a
          // person can make too (OpenVibe.Live's Rundown and transcripts, OpenVibe.Chat's history and timeline).
          upstreams: {
ddiff --git a/server/index.js b/server/index.js
iindex 1bd947a..c293603 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -23,6 +23,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
              () => ctx.reverifier.stop(),
              () => ctx.judgeJob.stop(),
              () => ctx.reddit.stop(),
++            () => ctx.searchRelay.stop(),
              async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
              () => ctx.s.close(),
              ...extra,
@@@ -54,6 +55,8 @@ async function start() {
      ctx.reverifier.start();
      ctx.judgeJob.start();
      ctx.reddit.start();
++    // Queued Search documents → OpenVibe.Search (a tick a few seconds after boot, then every minute).
++    if (!config.drill) ctx.searchRelay.start();
      // The Reddit source is created disabled in Sources, once, when the owner asks (MEME_SOURCES_REGISTER=1).
      if (!config.drill) ctx.sourceRegistration.ensure().catch(() => {});
  
ddiff --git a/server/search.js b/server/search.js
iindex 2e673a1..1c80679 100644
---- a/server/search.js
++++ b/server/search.js
@@@ -7,8 +7,9 @@
   * summary, the verbatim quotes as plain-text body, facets `kind`/`status`/`source`, authorship `ai_assisted` and
   * one provenance reference per citation — the same shape Work, Help and the other record products send.
   *
-- * No service grant is needed to queue the rows: they wait in the outbox until OpenVibe.Search's document writer
-- * is configured for `meme` (research §6.2). The relay is deliberately not part of v1.
++ * No service grant is needed to queue the rows. openvibe-publishing/search-relay (server/app.js) delivers them to
++ * Search's owner API with this service's token (search.document.write), a few seconds after each change (onQueued)
++ * and every minute; without MEME_SEARCH_URL they wait.
   */
  const { ids } = require('openvibe-contracts');
  
@@@ -73,7 +74,7 @@ function documentOf(entry, citations = [], { baseUrl = 'https://openvibe.meme',
      };
  }
  
--function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {
++function createSearchIndex({ config, s, baseUrl = config.baseUrl, onQueued = null }) {
      const nowIso = () => new Date(s.now()).toISOString();
  
      async function enqueue(topic, payload, t = null) {
@@@ -84,6 +85,7 @@ function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {
               VALUES ($1, $2, $3::jsonb, $4::timestamptz, $4::timestamptz)`,
              [eventId, topic, JSON.stringify(payload), nowIso()],
          );
++        if (onQueued) onQueued();
          return eventId;
      }
  
ddiff --git a/server/views/index.js b/server/views/index.js
iindex 73448e2..9e9bba1 100644
---- a/server/views/index.js
++++ b/server/views/index.js
@@@ -254,7 +254,7 @@ ${ep('POST', '/api/v1/entries/:slug/hide', 'Remove it from public lists and tomb
  ${ep('PATCH', '/api/v1/entries/:slug', '<code>{ title, kind, status }</code>: change an entry; status history is kept.')}
  ${ep('GET', '/api/v1/reports', 'The open takedown/correction reports.')}
  </dl></section>
--<section id="mm-search"><h2>Search</h2><p class="mm-fine">Each public entry is one <code>search.index-document@1</code> (owner <code>meme</code>, type <code>entry</code>, canonical page <code>/m/&lt;slug&gt;</code>) queued in <code>meme_outbox</code>; hiding or deleting an entry queues a tombstone. The importer reads only public APIs and never publishes: a candidate stays hidden until a staff person publishes it.</p></section>
++<section id="mm-search"><h2>Search</h2><p class="mm-fine">Each public entry is one <code>search.index-document@1</code> (owner <code>meme</code>, type <code>entry</code>, canonical page <code>/m/&lt;slug&gt;</code>) queued in <code>meme_outbox</code> and delivered to OpenVibe.Search within a minute; hiding or deleting an entry queues a tombstone. The importer reads only public APIs and never publishes: a candidate stays hidden until a staff person publishes it.</p></section>
  </div>`;
  }
  
ddiff --git a/test/search-relay.test.js b/test/search-relay.test.js
nnew file mode 100644
iindex 0000000..9dbaf3f
---- /dev/null
++++ b/test/search-relay.test.js
@@@ -0,0 +1,79 @@
++'use strict';
++/**
++ * The queued Search documents reach OpenVibe.Search (server/app.js, openvibe-publishing/search-relay): with
++ * MEME_SEARCH_URL set, publishing an entry PUTs its document to Search's owner API with a service token for
++ * openvibe.search, and hiding it DELETEs it at the next revision; a delivered row leaves the queue. Without the URL
++ * the relay is off and the rows wait.
++ */
++const assert = require('assert');
++const http = require('http');
++const { boot, check, done } = require('./helpers/boot');
++
++function fakeSearch() {
++    const calls = [];
++    const server = http.createServer((req, res) => {
++        let body = '';
++        req.on('data', (c) => { body += c; });
++        req.on('end', () => {
++            calls.push({ method: req.method, url: req.url, auth: req.headers.authorization || '', body: body ? JSON.parse(body) : null });
++            res.writeHead(200, { 'content-type': 'application/json' });
++            res.end(JSON.stringify({ outcome: 'applied' }));
++        });
++    });
++    return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ url: `http://127.0.0.1:${server.address().port}`, calls, close: () => new Promise((r) => server.close(r)) })));
++}
++
++async function makeEntry(t, slug) {
++    return t.ctx.entries.createCandidate({
++        slug, title: 'Robot is offline', kind: 'catchphrase', summary: 'The robot is offline, again.', status: 'rising',
++        first_seen_at: '2026-10-20T10:00:00Z', last_seen_at: new Date().toISOString(), visibility: 'hidden',
++        citations: [{ role: 'origin', source_service: 'chat', source_type: 'chat_message', source_id: '1001', chat_message_id: 1001, quote_text: 'the robot is offline again', url: `${t.upstreams.url}/`, observed_at: '2026-10-20T10:00:00Z' }],
++    });
++}
++
++(async () => {
++    const search = await fakeSearch();
++    const t = await boot({ env: { MEME_SEARCH_URL: search.url } });
++    try {
++        const mod = t.network.addUser('mod', { role: 'global_mod' });
++        await makeEntry(t, 'relay-proof');
++        const e = await t.ctx.s.db.maybe("SELECT id FROM meme_entries WHERE slug = 'relay-proof'");
++
++        await check('publishing an entry PUTs its document to Search with a service token, and the row leaves the queue', async () => {
++            assert.strictEqual(t.ctx.searchRelay.enabled, true);
++            assert.strictEqual((await t.get('/api/v1/entries/relay-proof/publish', { as: mod, method: 'POST' })).status, 200);
++            const out = await t.ctx.searchRelay.drain();
++            assert.ok(out.sent >= 1, JSON.stringify(out));
++            const put = search.calls.find((c) => c.method === 'PUT');
++            assert.ok(put, 'a PUT');
++            assert.strictEqual(put.url, `/api/v1/documents/meme/entry/${e.id}`);
++            assert.match(put.auth, /^Bearer \S+/);
++            assert.strictEqual(put.body.canonical_url, 'https://openvibe.meme/m/relay-proof');
++            assert.strictEqual(put.body.title, 'Robot is offline');
++            assert.strictEqual((await t.ctx.searchRelay.status()).pending, 0);
++        });
++
++        await check('hiding it DELETEs the document at the next revision', async () => {
++            const before = search.calls.length;
++            assert.strictEqual((await t.get('/api/v1/entries/relay-proof/hide', { as: mod, method: 'POST' })).status, 200);
++            await t.ctx.searchRelay.drain();
++            const del = search.calls.slice(before).find((c) => c.method === 'DELETE');
++            assert.ok(del, 'a DELETE');
++            assert.match(del.url, new RegExp(`^/api/v1/documents/meme/entry/${e.id}\\?revision=\\d+$`));
++        });
++    } finally {
++        await t.close();
++        await search.close();
++    }
++
++    const off = await boot();
++    try {
++        await check('without MEME_SEARCH_URL the relay is off', async () => {
++            assert.strictEqual(off.ctx.searchRelay.enabled, false);
++            assert.strictEqual(await off.ctx.searchRelay.drain(), null);
++        });
++    } finally {
++        await off.close();
++        done();
++    }
++})();