openvibers/openvibe.fun Commits

Fun quests and plans reach OpenVibe.Search (search-relay) (#59)

OP OpenVibers committed 11 days ago · 91bb743630 · 11 files, +113 −12

fun_outbox queued a search.index-document@1 per public quest and plan and nothing sent them.
openvibe-publishing/search-relay (v1.5.0) now delivers them with Fun's search.document.write
token, a few seconds after a change and every minute; off without FUN_SEARCH_URL, never under
FUN_DRILL, stopped on shutdown.

Changes

ddiff --git a/.env.example b/.env.example
iindex b35e487..7f7a0dd 100644
---- a/.env.example
++++ b/.env.example
@@@ -35,6 +35,8 @@ OV_OAUTH_CLIENT_SECRET=
  # needs the events.subscription.manage capability for svc:fun.
  FUN_EVENTS_SECRET=
  FUN_EVENTS_URL=
++# OpenVibe.Search's owner API: the queued Search documents (fun_outbox) are delivered there. Unset: they wait.
++#FUN_SEARCH_URL=http://127.0.0.1:4710
  #FUN_EVENTS_ENDPOINT=http://127.0.0.1:5140/internal/events
  #FUN_EVENTS_INTERVAL_MS=5000
  
ddiff --git a/README.md b/README.md
iindex d941843..4c8ffec 100644
---- a/README.md
++++ b/README.md
@@@ -50,8 +50,9 @@ external write is a plan added to *your own* Today.
  - **Identity and roles** — OpenVibe.Network. Fun is OAuth client `fun` (PKCE S256) and verifies session tokens
    against the Network JWKS; staff means Network role `global_mod` or above.
  - **OpenStreetMap and Ticketmaster** — external public sources, cached and credited; never a paid provider.
--- **Search** — OpenVibe.Search receives `search.index-document@1` documents from `fun_outbox` when its writer is
--  configured; the relay needs `search.document.write` and is not in v1.
++- **Search** — OpenVibe.Search receives `search.index-document@1` documents from `fun_outbox`:
++  `openvibe-publishing/search-relay` delivers them to its owner API with Fun's `search.document.write` grant
++  (`FUN_SEARCH_URL`; unset, they wait).
  - **Events, the host and the database** — OpenVibe.Events, OpenVibe.Host.
  
  ## Capabilities and grants
@@@ -70,8 +71,8 @@ cookie or a Network user token; a signed-in write from the cookie must come from
    to credit OpenCoins through OpenVibe.Network after a quest is verified. Off by default; the badge is still paid.
  
  **Not needed in v1:** no Live, Chat, Today, Quest, Media or AI service token. The Today write uses the signed-in
--person's own Network token, and the guide reads public OSM/Ticketmaster endpoints. A future Search relay needs
--`search.document.write`; a future streamer-ownership proof could use a Live capability, but v1 checks the Network
++person's own Network token, and the guide reads public OSM/Ticketmaster endpoints. The Search relay uses
++`search.document.write` (held); a future streamer-ownership proof could use a Live capability, but v1 checks the Network
  username against the public Live channel.
  
  ## API
ddiff --git a/STATUS.json b/STATUS.json
iindex 03cb5ff..3fbe4c6 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -27,7 +27,7 @@
      "A deterministic local guide from Nominatim/Overpass with a contact User-Agent, a per-host pacer, a seven-day cache, stale fallback and ODbL attribution",
      "Ticketmaster Discovery events only when TICKETMASTER_API_KEY is set; the key is never cached, returned or logged",
      "Group plans with options, one vote per person, a tie refused, an owner/staff decision and a winner added to the viewer's own Today via the person's token",
--    "Search Fabric documents (owner fun; types quest and plan) in fun_outbox, with valid tombstones on hide/cancel",
++    "Search Fabric documents (owner fun; types quest and plan) in fun_outbox, with valid tombstones on hide/cancel, delivered to Search's owner API by openvibe-publishing/search-relay",
      "Sign-in with OpenVibe.Network (PKCE S256, fun_at/fun_rt/fun_oauth), staff role global_mod+",
      "Account export and deletion: personal rows erased; public quests/boards and shared plans anonymized, never silently kept with a name",
      "Server-rendered pages with stable fu- ids/classes, a no-JS OBS overlay, discovery artifacts and deploy files"
@@@ -36,7 +36,6 @@
      "An AI chat or model-generated guide: v1 is deterministic and adds no AI capability",
      "Ticketmaster events without a key: v1 is OSM-only by default",
      "OpenCoins without the FUN_COINS=on setting and the identity.subject.resolve + network.coins.credit grants: the badge is still paid",
--    "The Search document relay: documents wait in fun_outbox until Network grants fun search.document.write",
      "A service-token proof of Live channel ownership: v1 checks the public channel and matching Network username",
      "Fan-out of a decided group plan to every member's Today: each viewer adds it to their own with their own token",
      "Parsing OSM opening_hours into open-now; the guide shows the raw OSM value",
ddiff --git a/package-lock.json b/package-lock.json
iindex dc9b1e3..282be5a 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -16,6 +16,7 @@
          "helmet": "^8.1.0",
          "iovalkey": "^0.4.0",
          "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.236.0",
++        "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
          "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
          "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
          "pg": "^8.23.0"
@@@ -1165,6 +1166,24 @@
          "node": ">=20"
        }
      },
++    "node_modules/openvibe-publishing": {
++      "version": "1.5.0",
++      "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
++      "integrity": "sha512-8cKVhJvpbDgWuGR+Dh1mpgqfWPJoC7EIcw7eTRc5/2ueDc8caCA7XKdcpAHNZYMdukIAL8o7FvxE8QV5/1uSLw==",
++      "license": "MIT",
++      "engines": {
++        "node": ">=20"
++      },
++      "peerDependencies": {
++        "openvibe-sdk": ">=0.15.0",
++        "openvibe-shared": ">=1.5.0"
++      },
++      "peerDependenciesMeta": {
++        "openvibe-shared": {
++          "optional": true
++        }
++      }
++    },
      "node_modules/openvibe-sdk": {
        "version": "0.81.0",
        "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
ddiff --git a/package.json b/package.json
iindex c09dda1..46609fa 100644
---- a/package.json
++++ b/package.json
@@@ -23,6 +23,7 @@
      "helmet": "^8.1.0",
      "iovalkey": "^0.4.0",
      "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.236.0",
++    "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
      "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
      "openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
      "pg": "^8.23.0"
ddiff --git a/server/app.js b/server/app.js
iindex b2a43ea..e0c4f8a 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -35,6 +35,7 @@ const { raw } = require('./render/html');
  const accountDataLib = require('./identity/account-data');
  const { createStore } = require('./store');
  const { createSearchIndex } = require('./search');
++const { createSearchRelay } = require('openvibe-publishing/search-relay');
  const { createUpstreams } = require('./upstreams');
  const { createGuide } = require('./guide');
  const { createUpstreams: createGuideUpstreams } = require('./guide/upstream');
@@@ -61,7 +62,16 @@ async function createApp(opts = {}) {
      const sso = createSso({ config, keys, fetchImpl, now: s.now, log });
      const principal = createPrincipal({ config, keys });
  
--    const search = createSearchIndex({ config, s });
++    // The queued Search documents (fun_outbox) go to OpenVibe.Search's owner API with this service's token; off
++    // without FUN_SEARCH_URL or the client secret. Started by server/index.js, never under FUN_DRILL.
++    const searchRelay = createSearchRelay({
++        db: s.db, table: 'fun_outbox', owner: 'fun', searchUrl: config.search.url, fetchImpl, now: s.now, log,
++        auth: config.search.url && config.oauth.clientSecret ? contracts.serviceAuth.createTokenClient({
++            tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,
++            audience: 'openvibe.search', scope: 'search.document.write', fetchImpl,
++        }) : null,
++    });
++    const search = createSearchIndex({ config, s, onQueued: () => searchRelay.kick() });
      const store = createStore(s, { search, config });
      const upstreams = createUpstreams({ config, fetchImpl, now: s.now, log });
      const guideUpstream = createGuideUpstreams({ config, s, fetchImpl, now: s.now, log });
@@@ -77,7 +87,7 @@ async function createApp(opts = {}) {
          : async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Fun cannot answer account events'); });
  
      const ctx = {
--        config, s, keys, sso, principal, search, store, upstreams, guide, guideUpstream, coins, chatSync,
++        config, s, keys, sso, principal, search, searchRelay, store, upstreams, guide, guideUpstream, coins, chatSync,
          accountData, accountSend, log,
      };
  
ddiff --git a/server/config.js b/server/config.js
iindex 6310f27..99261ef 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -62,6 +62,10 @@ function load(env = process.env) {
              scope: env.FUN_EVENTS_SCOPE || 'events.subscription.manage',
          },
  
++        // OpenVibe.Search's owner API: the documents queued in fun_outbox are delivered there (server/app.js,
++        // openvibe-publishing/search-relay) with this service's token (search.document.write). Unset: they wait.
++        search: { url: trim(env.FUN_SEARCH_URL || '') },
++
          // The public APIs Fun reads: Live's channel status, Chat's public history (chat votes are read, never
          // posted), and Today's public event API (called with the viewer's own Network token, never a service one).
          upstreams: {
ddiff --git a/server/index.js b/server/index.js
iindex cf25c79..d682ec9 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -21,6 +21,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
              () => ctx.keys.client.stop(),
              () => ctx.chatSync.stop(),
              () => ctx.coins.stop(),
++            () => ctx.searchRelay.stop(),
              async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
              () => ctx.s.close(),
              ...extra,
@@@ -48,6 +49,8 @@ async function start() {
      // Jobs. Chat votes are read-only; both refuse to start under FUN_DRILL, where tests call runOnce/retryPending.
      ctx.chatSync.start();
      ctx.coins.start();
++    // Queued Search documents → OpenVibe.Search (a tick a few seconds after boot, then every minute).
++    if (!config.drill) ctx.searchRelay.start();
  
      // The two account topics at OpenVibe.Events: idempotent; off without the URL, the delivery secret and the client
      // secret. The consumer itself is mounted in server/app.js.
ddiff --git a/server/search.js b/server/search.js
iindex 118caeb..1375f17 100644
---- a/server/search.js
++++ b/server/search.js
@@@ -4,8 +4,9 @@
   * OpenVibe.Fun's Search Fabric feed. A public IRL quest is one `search.index-document@1` (owner `fun`, type `quest`,
   * page `/q/<slug>`); a public group plan is another (type `plan`, page `/plans/<slug>`). They are queued in
   * `fun_outbox` in the same transaction as the change; hiding or cancelling one queues a tombstone, which wins over
-- * any older revision. The relay is deliberately not part of v1: rows wait until OpenVibe.Search's document writer
-- * is configured for `fun` (it needs the search.document.write grant).
++ * any older revision. openvibe-publishing/search-relay (server/app.js) delivers them to Search's owner API with
++ * this service's token (search.document.write), a few seconds after each change and every minute; without
++ * FUN_SEARCH_URL they wait.
   *
   * A quest/plan body is the public text the person wrote plus the public facts around it. No voter, subject, chat
   * author or token is ever part of a document.
@@@ -107,7 +108,7 @@ function planDocument(p, options = [], { baseUrl = 'https://openvibe.fun' } = {}
      };
  }
  
--function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {
++function createSearchIndex({ config, s, baseUrl = config.baseUrl, onQueued = null }) {
      const nowIso = () => new Date(s.now()).toISOString();
      const base = String(baseUrl).replace(/\/+$/, '');
  
@@@ -119,6 +120,7 @@ function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {
               VALUES ($1, $2, $3::jsonb, $4::timestamptz, $4::timestamptz)`,
              [eventId, topic, JSON.stringify(payload), nowIso()],
          );
++        if (onQueued) onQueued();
          return eventId;
      }
  
ddiff --git a/server/views/index.js b/server/views/index.js
iindex 3a53ab3..f5a1a7b 100644
---- a/server/views/index.js
++++ b/server/views/index.js
@@@ -398,7 +398,7 @@ ${ep('POST', '/api/v1/plans/:slug/options|vote|decide', 'Add an option, cast one
  ${ep('POST', '/api/v1/plans/:slug/today', 'Adds the winner to the caller\'s own OpenVibe.Today with their token and an idempotency key.')}
  ${ep('POST', '/api/v1/plans/:slug/hide', 'Staff.')}
  </dl></section>
--<section id="fu-search"><h2>Search</h2><p class="fu-fine">Each public quest is one <code>search.index-document@1</code> (owner <code>fun</code>, type <code>quest</code>, <code>/q/&lt;slug&gt;</code>); each public plan is type <code>plan</code> (<code>/plans/&lt;slug&gt;</code>). They wait in <code>fun_outbox</code>; hiding or cancelling queues a tombstone. The relay needs the <code>search.document.write</code> grant and is not in v1.</p></section>
++<section id="fu-search"><h2>Search</h2><p class="fu-fine">Each public quest is one <code>search.index-document@1</code> (owner <code>fun</code>, type <code>quest</code>, <code>/q/&lt;slug&gt;</code>); each public plan is type <code>plan</code> (<code>/plans/&lt;slug&gt;</code>). They are queued in <code>fun_outbox</code> and delivered to OpenVibe.Search within a minute; hiding or cancelling queues a tombstone.</p></section>
  </div>`;
  }
  
ddiff --git a/test/search-relay.test.js b/test/search-relay.test.js
nnew file mode 100644
iindex 0000000..dab4ef2
---- /dev/null
++++ b/test/search-relay.test.js
@@@ -0,0 +1,60 @@
++'use strict';
++/**
++ * The queued Search documents reach OpenVibe.Search (server/app.js, openvibe-publishing/search-relay): with
++ * FUN_SEARCH_URL set, accepting a quest PUTs its document to Search's owner API with a service token for
++ * openvibe.search, and the row leaves the queue. Without the URL the relay is off and the rows wait.
++ */
++const assert = require('assert');
++const http = require('http');
++const { boot, check, done } = require('./helpers/boot');
++
++function fakeSearch() {
++    const calls = [];
++    const server = http.createServer((req, res) => {
++        let body = '';
++        req.on('data', (c) => { body += c; });
++        req.on('end', () => {
++            calls.push({ method: req.method, url: req.url, auth: req.headers.authorization || '', body: body ? JSON.parse(body) : null });
++            res.writeHead(200, { 'content-type': 'application/json' });
++            res.end(JSON.stringify({ outcome: 'applied' }));
++        });
++    });
++    return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ url: `http://127.0.0.1:${server.address().port}`, calls, close: () => new Promise((r) => server.close(r)) })));
++}
++
++(async () => {
++    const search = await fakeSearch();
++    const t = await boot({ env: { FUN_SEARCH_URL: search.url } });
++    try {
++        const ada = t.network.addUser('ada');
++        const kim = t.network.addUser('kim');
++
++        await check('an accepted quest is PUT to Search with a service token, and the row leaves the queue', async () => {
++            assert.strictEqual(t.ctx.searchRelay.enabled, true);
++            const b = await t.get('/api/v1/quests/boards', { as: ada, json: { live_username: 'ada', title: 'Relay board' } });
++            const quest = (await t.get(`/api/v1/quests/boards/${b.json().id}/suggest`, { as: kim, json: { title: 'Find the oldest tree', brief: 'Walk to the park.' } })).json();
++            assert.strictEqual((await t.get(`/api/v1/quests/${quest.slug}/accept`, { as: ada, json: {} })).status, 200);
++            const out = await t.ctx.searchRelay.drain();
++            assert.ok(out.sent >= 1, JSON.stringify(out));
++            const put = search.calls.find((c) => c.method === 'PUT' && c.url === `/api/v1/documents/fun/quest/${quest.id}`);
++            assert.ok(put, JSON.stringify(search.calls.map((c) => `${c.method} ${c.url}`)));
++            assert.match(put.auth, /^Bearer \S+/);
++            assert.strictEqual(put.body.canonical_url, `https://openvibe.fun/q/${quest.slug}`);
++            assert.strictEqual((await t.ctx.searchRelay.status()).pending, 0);
++        });
++    } finally {
++        await t.close();
++        await search.close();
++    }
++
++    const off = await boot();
++    try {
++        await check('without FUN_SEARCH_URL the relay is off', async () => {
++            assert.strictEqual(off.ctx.searchRelay.enabled, false);
++            assert.strictEqual(await off.ctx.searchRelay.drain(), null);
++        });
++    } finally {
++        await off.close();
++    }
++    done();
++})();