openvibers/openvibe.exchange Commits
Exchange listings reach OpenVibe.Search (search-relay) (#59)
Changes
@@@ -47,6 +47,8 @@ OV_OAUTH_CLIENT_SECRET=# (comma-separated for rotation, 32+ characters each; unset makes the route answer 503).EXCHANGE_EVENTS_SECRET=EXCHANGE_EVENTS_URL=++# OpenVibe.Search's owner API: the queued Search documents (exchange_outbox) are delivered there. Unset: they wait.++#EXCHANGE_SEARCH_URL=http://127.0.0.1:4710#EXCHANGE_EVENTS_ENDPOINT=http://127.0.0.1:5150/internal/events#EXCHANGE_EVENTS_INTERVAL_MS=5000@@@ -60,8 +60,9 @@ v1 cannot represent one.asks for `global_mod` or above. A passport is enforced by Network grants in the plan; v1 cannot create one (see"Grants it will need").- **The sandbox runtime** — OpenVibe.Run. v1's sandbox is a dry run; a real execution needs a Run capability.--- **Search** — OpenVibe.Search. Exchange produces `search.index-document@1` documents in its own outbox; the relay-- is not wired because `exchange` has no `search.document.write` grant yet.++- **Search** — OpenVibe.Search. Exchange produces `search.index-document@1` documents in its own outbox, and++ `openvibe-publishing/search-relay` delivers them to Search's owner API with the `search.document.write` grant++ (`EXCHANGE_SEARCH_URL`; unset, they wait).- **Billing, transfers, cashouts** — OpenVibe.Billing. Not used at all in v1.- **The listings engine and local safety rules** — OpenVibe.Rent. Exchange borrows the shape of the rules (no streetaddresses, a fixed prohibited list, staff hide) but has its own table; it does not call Rent.@@@ -149,8 +150,8 @@ This is deliberately honest: v1 does not run a listing's code. A real sandbox neEach public listing is one `search.index-document@1` (owner `exchange`, type `listing`, canonical page`/listings/<slug>`) queued in `exchange_outbox` in the same transaction as the listing change. A draft writes--nothing; closing or hiding writes a tombstone. The outbox waits for `search.document.write`; the relay is not part of--v1.++nothing; closing or hiding writes a tombstone. `openvibe-publishing/search-relay` (v1.5.0) sends the rows to Search's owner++API a few seconds after the change and every minute, oldest first, superseding a listing's older rows.## Deploy@@@ -179,7 +180,7 @@ services, requested from the main session (this service registers no capability| `network.grants.write` | OpenVibe.Network | create the developer-app grant from the passport snapshot at install | **proposed**; not in contracts v0.229.0 yet || `network.grants.revoke` | OpenVibe.Network | revoke that grant everywhere at one-tap revoke | **proposed**; not in contracts v0.229.0 yet || `run.sandbox.execute` | OpenVibe.Run | execute a listing's entry point in a real sandbox and run its own tests | **proposed**; Run has no sandbox today. Existing `run.job.submit` could back it if Run adds a sandbox job class |--| `search.document.write` | OpenVibe.Search | relay the `exchange_outbox` documents | existing capability, not granted |++| `search.document.write` | OpenVibe.Search | relay the `exchange_outbox` documents | granted || `network.account.export.contribute` | OpenVibe.Network | push a person's listings/installs/trials/offers export | existing capability, not granted || `network.account.deletion.confirm` | OpenVibe.Network | confirm the erasure and counts | existing capability, not granted || `events.subscription.manage` | OpenVibe.Events | create the two account subscriptions | existing scope, client credentials |@@@ -26,7 +26,7 @@"Free claims and swap offers with a required give on swaps; the owner accepts/declines, the sender withdraws; accepting closes the listing and tombstones it","A fixed prohibited-items and no-money filter on every listing and offer","The API: public cursor-paged listings, detail, passport, tests, owner drafts, sandbox, install, installs, offers and staff hide",-- "Search Fabric documents (owner exchange, type listing, /listings/<slug>) in exchange_outbox, with a tombstone when closed or hidden",++ "Search Fabric documents (owner exchange, type listing, /listings/<slug>) in exchange_outbox, with a tombstone when closed or hidden, delivered to Search's owner API by openvibe-publishing/search-relay","Sign-in with OpenVibe.Network (PKCE S256, exchange_at/exchange_rt/exchange_oauth), staff role global_mod+","Account export and deletion over listings, installs, sandbox trials and offers; a deleted listing is anonymised, closed and tombstoned","Server-rendered pages with stable ex- ids/classes, cache headers, discovery artifacts, deploy files and the first-proof seed"@@@ -34,7 +34,6 @@"notYet": ["The Network grant write/revoke: v1 keeps the install as an Exchange record; network.grants.write and network.grants.revoke are proposed and not in contracts v0.229.0","A real sandbox: v1 is a manifest/passport dry run; executing a listing needs OpenVibe.Run and a run.sandbox.execute capability",-- "The Search document relay: documents wait in exchange_outbox until Network grants exchange search.document.write","Paid listings through Billing (v2): no intent, transfer, cashout, entitlement or usage call exists in v1","Reviews and mutual reviews; the listing page shows no reviews yet","A user report form and moderation queue beyond the fixed filter and staff hide",@@@ -16,6 +16,7 @@"helmet": "^8.1.0","iovalkey": "^0.4.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",++ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0","pg": "^8.23.0"@@@ -1165,6 +1166,24 @@"node": ">=20"}},++ "node_modules/openvibe-publishing": {++ "version": "1.5.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",++ "integrity": "sha512-8cKVhJvpbDgWuGR+Dh1mpgqfWPJoC7EIcw7eTRc5/2ueDc8caCA7XKdcpAHNZYMdukIAL8o7FvxE8QV5/1uSLw==",++ "license": "MIT",++ "engines": {++ "node": ">=20"++ },++ "peerDependencies": {++ "openvibe-sdk": ">=0.15.0",++ "openvibe-shared": ">=1.5.0"++ },++ "peerDependenciesMeta": {++ "openvibe-shared": {++ "optional": true++ }++ }++ },"node_modules/openvibe-sdk": {"version": "0.81.0","resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",@@@ -25,6 +25,7 @@"helmet": "^8.1.0","iovalkey": "^0.4.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",++ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0","pg": "^8.23.0"@@@ -37,6 +37,7 @@ const { raw } = require('./render/html');const accountDataLib = require('./identity/account-data');const { createStore } = require('./store');const { createSearchIndex } = require('./search');++const { createSearchRelay } = require('openvibe-publishing/search-relay');const { createFirstParty } = require('./jobs/first-party');const { createNetworkSender } = require('openvibe-sdk/account-data');@@@ -59,7 +60,16 @@ async function createApp(opts = {}) {const sso = createSso({ config, keys, fetchImpl, now: s.now, log });const principal = createPrincipal({ config, keys });-- const search = createSearchIndex({ config, s });++ // The queued Search documents (exchange_outbox) go to OpenVibe.Search's owner API with this service's token;++ // off without EXCHANGE_SEARCH_URL or the client secret. Started by server/index.js, never under EXCHANGE_DRILL.++ const searchRelay = createSearchRelay({++ db: s.db, table: 'exchange_outbox', owner: 'exchange', searchUrl: config.search.url, fetchImpl, now: s.now, log,++ auth: config.search.url && config.oauth.clientSecret ? contracts.serviceAuth.createTokenClient({++ tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,++ audience: 'openvibe.search', scope: 'search.document.write', fetchImpl,++ }) : null,++ });++ const search = createSearchIndex({ config, s, onQueued: () => searchRelay.kick() });const entries = createStore(s, { search, config });const firstParty = createFirstParty({ config, s, entries, fetchImpl, log, now: s.now });@@@ -70,7 +80,7 @@ async function createApp(opts = {}) {? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl }): async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Exchange cannot answer account events'); });-- const ctx = { config, s, keys, sso, principal, search, entries, firstParty, accountData, accountSend, log };++ const ctx = { config, s, keys, sso, principal, search, searchRelay, entries, firstParty, accountData, accountSend, log };const app = express();app.disable('x-powered-by');@@@ -61,6 +61,10 @@ function load(env = process.env) {scope: env.EXCHANGE_EVENTS_SCOPE || 'events.event.publish',},++ // OpenVibe.Search's owner API: the documents queued in exchange_outbox are delivered there (server/app.js,++ // openvibe-publishing/search-relay) with this service's token (search.document.write). Unset: they wait.++ search: { url: trim(env.EXCHANGE_SEARCH_URL || '') },++// The sandbox: a manifest dry run, not a code execution. A passed trial is good for this many minutes and// is required before an install; a person gets at most this many trials a day.sandbox: {@@@ -60,7 +60,7 @@ function createDiscoveryRoutes(ctx) {res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({name: SITE_NAME,summary: 'OpenVibe.Exchange is the OpenVibe marketplace for what people make: agents, MCP tools, templates (Website, Report, Space), design themes and kits, game assets and creator services. Every public listing shows a permission passport (exactly what it can read, write, spend or contact), its licence, the conformance tests for that version, and a sandbox. A passed sandbox trial is required before an install; a revoke removes the installation in one tap. v1 is free and swap only: there is no money and a passport may not declare a spend.',-- details: 'GET /api/v1/listings lists public listings; GET /api/v1/listings/:slug adds the passport, manifest, licence, tests and counters; GET /api/v1/listings/:slug/passport is the exact passport an installer accepts. The sandbox is a manifest dry run in v1: Exchange does not execute a listing\'s code. Every public listing is one search.index-document@1 (owner exchange, type listing) queued in exchange_outbox.',++ details: 'GET /api/v1/listings lists public listings; GET /api/v1/listings/:slug adds the passport, manifest, licence, tests and counters; GET /api/v1/listings/:slug/passport is the exact passport an installer accepts. The sandbox is a manifest dry run in v1: Exchange does not execute a listing\'s code. Every public listing is one search.index-document@1 (owner exchange, type listing) queued in exchange_outbox and delivered to OpenVibe.Search within a minute.',sections: [{ title: 'Start here', links: [{ title: 'OpenVibe.Exchange', url: abs('/'), note: 'free and swap listings' },@@@ -20,6 +20,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }() => { for (const t of timers) clearInterval(t); },() => ctx.keys.client.stop(),() => ctx.firstParty.stop(),++ () => ctx.searchRelay.stop(),async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },() => ctx.s.close(),...extra,@@@ -46,6 +47,8 @@ async function start() {ctx.telemetryTick.unref();ctx.keys.client.start();ctx.firstParty.start();++ // Queued Search documents → OpenVibe.Search (a tick a few seconds after boot, then every minute).++ if (!config.drill) ctx.searchRelay.start();// The two account topics (export_requested, deleted) at OpenVibe.Events: idempotent; off without the URL, the// delivery secret and the client secret. The consumer itself is mounted in server/app.js.@@@ -7,8 +7,9 @@* revision. The document carries the maker's copy, the passport's declared permissions and the licence, so Search* can filter free/swap and kinds without reading the listing page.*-- * No service grant is needed to queue the rows: they wait in the outbox until OpenVibe.Search's document writer is-- * configured for `exchange` (README, "Grants it will need"). The relay is deliberately not part of v1.++ * No service grant is needed to queue the rows. openvibe-publishing/search-relay (server/app.js) delivers them to++ * Search's owner API with this service's token (search.document.write), a few seconds after each change++ * (onQueued) and every minute; without EXCHANGE_SEARCH_URL they wait.*/const { ids } = require('openvibe-contracts');@@@ -67,7 +68,7 @@ function documentOf(listing, { baseUrl = 'https://openvibe.exchange' } = {}) {};}--function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {++function createSearchIndex({ config, s, baseUrl = config.baseUrl, onQueued = null }) {const nowIso = () => new Date(s.now()).toISOString();async function enqueue(topic, payload, t = null) {@@@ -78,6 +79,7 @@ function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {VALUES ($1, $2, $3::jsonb, $4::timestamptz, $4::timestamptz)`,[eventId, topic, JSON.stringify(payload), nowIso()],);++ if (onQueued) onQueued();return eventId;}@@@ -439,7 +439,7 @@ ${ep('GET', '/api/v1/me/listings', 'Every listing of yours, all statuses.')}${ep('POST', '/api/v1/listings/:slug/hide', 'Hide a listing and tombstone its Search document.')}${ep('GET', '/api/v1/listings/:slug/offers', 'The open claims/offers on a listing.')}</dl></section>--<section id="ex-search"><h2>Search</h2><p class="ex-fine">Each public listing is one <code>search.index-document@1</code> (owner <code>exchange</code>, type <code>listing</code>, canonical page <code>/listings/<slug></code>) queued in <code>exchange_outbox</code>; closing or hiding it queues a tombstone. The outbox waits for Network's <code>search.document.write</code> grant; the relay is not part of v1.</p></section>++<section id="ex-search"><h2>Search</h2><p class="ex-fine">Each public listing is one <code>search.index-document@1</code> (owner <code>exchange</code>, type <code>listing</code>, canonical page <code>/listings/<slug></code>) queued in <code>exchange_outbox</code> and delivered to OpenVibe.Search within a minute; closing or hiding it queues a tombstone.</p></section><section id="ex-grants"><h2>Grants v1 does not have yet</h2><p class="ex-fine">The plan enforces a passport with Network grants and executes a listing in a Run sandbox. v1 does neither: an install is an Exchange record and the sandbox is a dry run. The service will ask for <code>network.grants.write</code> and <code>network.grants.revoke</code> before an install/revoke can reach Network, <code>run.sandbox.execute</code> before a trial can run code, and <code>search.document.write</code> before the outbox can reach Search. Account export and deletion need <code>network.account.export.contribute</code> and <code>network.account.deletion.confirm</code>; the two Events subscriptions need <code>events.subscription.manage</code>.</p></section></div>`;}nnew file mode 100644@@@ -0,0 +1,70 @@++'use strict';++/**++ * The queued Search documents reach OpenVibe.Search (server/app.js, openvibe-publishing/search-relay): with++ * EXCHANGE_SEARCH_URL set, publishing a listing PUTs its document to Search's owner API with a service token for++ * openvibe.search, and closing it DELETEs it at the listing's new revision; a delivered row leaves the queue. Without++ * the URL the relay is off and the rows wait.++ */++const assert = require('assert');++const http = require('http');++const { boot, check, done } = require('./helpers/boot');++const fixtures = require('./helpers/fixtures');++++function fakeSearch() {++ const calls = [];++ const server = http.createServer((req, res) => {++ let body = '';++ req.on('data', (c) => { body += c; });++ req.on('end', () => {++ calls.push({ method: req.method, url: req.url, auth: req.headers.authorization || '', body: body ? JSON.parse(body) : null });++ res.writeHead(200, { 'content-type': 'application/json' });++ res.end(JSON.stringify({ outcome: 'applied' }));++ });++ });++ return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ url: `http://127.0.0.1:${server.address().port}`, calls, close: () => new Promise((r) => server.close(r)) })));++}++++(async () => {++ const search = await fakeSearch();++ const t = await boot({ env: { EXCHANGE_SEARCH_URL: search.url } });++ try {++ const alice = t.network.addUser('alice');++ const l = await fixtures.listed(t, alice, { title: 'A relayed template' });++++ await check('a listed listing is PUT to Search with a service token, and the row leaves the queue', async () => {++ assert.strictEqual(t.ctx.searchRelay.enabled, true);++ const out = await t.ctx.searchRelay.drain();++ assert.ok(out.sent >= 1, JSON.stringify(out));++ const put = search.calls.find((c) => c.method === 'PUT' && c.url === `/api/v1/documents/exchange/listing/${l.id}`);++ assert.ok(put, `a PUT for ${l.id}: ${JSON.stringify(search.calls.map((c) => `${c.method} ${c.url}`))}`);++ assert.match(put.auth, /^Bearer \S+/);++ assert.strictEqual(put.body.canonical_url, `https://openvibe.exchange/listings/${l.slug}`);++ assert.strictEqual(put.body.title, 'A relayed template');++ assert.strictEqual((await t.ctx.searchRelay.status()).pending, 0);++ });++++ await check('closing it DELETEs the document at the listing\'s new revision', async () => {++ const before = search.calls.length;++ const r = await t.get(`/api/v1/listings/${l.slug}/close`, { as: alice, method: 'POST', json: { reason: 'withdrawn' } });++ assert.strictEqual(r.status, 200);++ await t.ctx.searchRelay.drain();++ const del = search.calls.slice(before).find((c) => c.method === 'DELETE');++ assert.ok(del, 'a DELETE');++ assert.strictEqual(del.url, `/api/v1/documents/exchange/listing/${l.id}?revision=${r.json().revision}`);++ });++ } finally {++ await t.close();++ await search.close();++ }++++ const off = await boot();++ try {++ await check('without EXCHANGE_SEARCH_URL the relay is off', async () => {++ assert.strictEqual(off.ctx.searchRelay.enabled, false);++ assert.strictEqual(await off.ctx.searchRelay.drain(), null);++ });++ } finally {++ await off.close();++ }++ done();++})();
@@ -47,6 +47,8 @@ OV_OAUTH_CLIENT_SECRET=
@@ -47,6 +47,8 @@ OV_OAUTH_CLIENT_SECRET=
# (comma-separated for rotation, 32+ characters each; unset makes the route answer 503).
# (comma-separated for rotation, 32+ characters each; unset makes the route answer 503).
EXCHANGE_EVENTS_SECRET=
EXCHANGE_EVENTS_SECRET=
EXCHANGE_EVENTS_URL=
EXCHANGE_EVENTS_URL=
+# OpenVibe.Search's owner API: the queued Search documents (exchange_outbox) are delivered there. Unset: they wait.
+#EXCHANGE_SEARCH_URL=http://127.0.0.1:4710
#EXCHANGE_EVENTS_ENDPOINT=http://127.0.0.1:5150/internal/events
#EXCHANGE_EVENTS_ENDPOINT=http://127.0.0.1:5150/internal/events
#EXCHANGE_EVENTS_INTERVAL_MS=5000
#EXCHANGE_EVENTS_INTERVAL_MS=5000
@@ -60,8 +60,9 @@ v1 cannot represent one.
@@ -60,8 +60,9 @@ v1 cannot represent one.
asks for `global_mod` or above. A passport is enforced by Network grants in the plan; v1 cannot create one (see
asks for `global_mod` or above. A passport is enforced by Network grants in the plan; v1 cannot create one (see
"Grants it will need").
"Grants it will need").
- **The sandbox runtime** — OpenVibe.Run. v1's sandbox is a dry run; a real execution needs a Run capability.
- **The sandbox runtime** — OpenVibe.Run. v1's sandbox is a dry run; a real execution needs a Run capability.
-- **Search** — OpenVibe.Search. Exchange produces `search.index-document@1` documents in its own outbox; the relay
- is not wired because `exchange` has no `search.document.write` grant yet.
+- **Search** — OpenVibe.Search. Exchange produces `search.index-document@1` documents in its own outbox, and
+ `openvibe-publishing/search-relay` delivers them to Search's owner API with the `search.document.write` grant
+ (`EXCHANGE_SEARCH_URL`; unset, they wait).
- **Billing, transfers, cashouts** — OpenVibe.Billing. Not used at all in v1.
- **Billing, transfers, cashouts** — OpenVibe.Billing. Not used at all in v1.
- **The listings engine and local safety rules** — OpenVibe.Rent. Exchange borrows the shape of the rules (no street
- **The listings engine and local safety rules** — OpenVibe.Rent. Exchange borrows the shape of the rules (no street
addresses, a fixed prohibited list, staff hide) but has its own table; it does not call Rent.
addresses, a fixed prohibited list, staff hide) but has its own table; it does not call Rent.
@@ -149,8 +150,8 @@ This is deliberately honest: v1 does not run a listing's code. A real sandbox ne
@@ -149,8 +150,8 @@ This is deliberately honest: v1 does not run a listing's code. A real sandbox ne
Each public listing is one `search.index-document@1` (owner `exchange`, type `listing`, canonical page
Each public listing is one `search.index-document@1` (owner `exchange`, type `listing`, canonical page
`/listings/<slug>`) queued in `exchange_outbox` in the same transaction as the listing change. A draft writes
`/listings/<slug>`) queued in `exchange_outbox` in the same transaction as the listing change. A draft writes
-nothing; closing or hiding writes a tombstone. The outbox waits for `search.document.write`; the relay is not part of
-v1.
+nothing; closing or hiding writes a tombstone. `openvibe-publishing/search-relay` (v1.5.0) sends the rows to Search's owner
+API a few seconds after the change and every minute, oldest first, superseding a listing's older rows.
## Deploy
## Deploy
@@ -179,7 +180,7 @@ services, requested from the main session (this service registers no capability
@@ -179,7 +180,7 @@ services, requested from the main session (this service registers no capability
| `network.grants.write` | OpenVibe.Network | create the developer-app grant from the passport snapshot at install | **proposed**; not in contracts v0.229.0 yet |
| `network.grants.write` | OpenVibe.Network | create the developer-app grant from the passport snapshot at install | **proposed**; not in contracts v0.229.0 yet |
| `network.grants.revoke` | OpenVibe.Network | revoke that grant everywhere at one-tap revoke | **proposed**; not in contracts v0.229.0 yet |
| `network.grants.revoke` | OpenVibe.Network | revoke that grant everywhere at one-tap revoke | **proposed**; not in contracts v0.229.0 yet |
| `run.sandbox.execute` | OpenVibe.Run | execute a listing's entry point in a real sandbox and run its own tests | **proposed**; Run has no sandbox today. Existing `run.job.submit` could back it if Run adds a sandbox job class |
| `run.sandbox.execute` | OpenVibe.Run | execute a listing's entry point in a real sandbox and run its own tests | **proposed**; Run has no sandbox today. Existing `run.job.submit` could back it if Run adds a sandbox job class |
-| `search.document.write` | OpenVibe.Search | relay the `exchange_outbox` documents | existing capability, not granted |
+| `search.document.write` | OpenVibe.Search | relay the `exchange_outbox` documents | granted |
| `network.account.export.contribute` | OpenVibe.Network | push a person's listings/installs/trials/offers export | existing capability, not granted |
| `network.account.export.contribute` | OpenVibe.Network | push a person's listings/installs/trials/offers export | existing capability, not granted |
| `network.account.deletion.confirm` | OpenVibe.Network | confirm the erasure and counts | existing capability, not granted |
| `network.account.deletion.confirm` | OpenVibe.Network | confirm the erasure and counts | existing capability, not granted |
| `events.subscription.manage` | OpenVibe.Events | create the two account subscriptions | existing scope, client credentials |
| `events.subscription.manage` | OpenVibe.Events | create the two account subscriptions | existing scope, client credentials |
@@ -26,7 +26,7 @@
@@ -26,7 +26,7 @@
"Free claims and swap offers with a required give on swaps; the owner accepts/declines, the sender withdraws; accepting closes the listing and tombstones it",
"Free claims and swap offers with a required give on swaps; the owner accepts/declines, the sender withdraws; accepting closes the listing and tombstones it",
"A fixed prohibited-items and no-money filter on every listing and offer",
"A fixed prohibited-items and no-money filter on every listing and offer",
"The API: public cursor-paged listings, detail, passport, tests, owner drafts, sandbox, install, installs, offers and staff hide",
"The API: public cursor-paged listings, detail, passport, tests, owner drafts, sandbox, install, installs, offers and staff hide",
- "Search Fabric documents (owner exchange, type listing, /listings/<slug>) in exchange_outbox, with a tombstone when closed or hidden",
+ "Search Fabric documents (owner exchange, type listing, /listings/<slug>) in exchange_outbox, with a tombstone when closed or hidden, delivered to Search's owner API by openvibe-publishing/search-relay",
"Sign-in with OpenVibe.Network (PKCE S256, exchange_at/exchange_rt/exchange_oauth), staff role global_mod+",
"Sign-in with OpenVibe.Network (PKCE S256, exchange_at/exchange_rt/exchange_oauth), staff role global_mod+",
"Account export and deletion over listings, installs, sandbox trials and offers; a deleted listing is anonymised, closed and tombstoned",
"Account export and deletion over listings, installs, sandbox trials and offers; a deleted listing is anonymised, closed and tombstoned",
"Server-rendered pages with stable ex- ids/classes, cache headers, discovery artifacts, deploy files and the first-proof seed"
"Server-rendered pages with stable ex- ids/classes, cache headers, discovery artifacts, deploy files and the first-proof seed"
@@ -34,7 +34,6 @@
@@ -34,7 +34,6 @@
"notYet": [
"notYet": [
"The Network grant write/revoke: v1 keeps the install as an Exchange record; network.grants.write and network.grants.revoke are proposed and not in contracts v0.229.0",
"The Network grant write/revoke: v1 keeps the install as an Exchange record; network.grants.write and network.grants.revoke are proposed and not in contracts v0.229.0",
"A real sandbox: v1 is a manifest/passport dry run; executing a listing needs OpenVibe.Run and a run.sandbox.execute capability",
"A real sandbox: v1 is a manifest/passport dry run; executing a listing needs OpenVibe.Run and a run.sandbox.execute capability",
- "The Search document relay: documents wait in exchange_outbox until Network grants exchange search.document.write",
"Paid listings through Billing (v2): no intent, transfer, cashout, entitlement or usage call exists in v1",
"Paid listings through Billing (v2): no intent, transfer, cashout, entitlement or usage call exists in v1",
"Reviews and mutual reviews; the listing page shows no reviews yet",
"Reviews and mutual reviews; the listing page shows no reviews yet",
"A user report form and moderation queue beyond the fixed filter and staff hide",
"A user report form and moderation queue beyond the fixed filter and staff hide",
@@ -16,6 +16,7 @@
@@ -16,6 +16,7 @@
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",
+ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -1165,6 +1166,24 @@
@@ -1165,6 +1166,24 @@
"node": ">=20"
"node": ">=20"
}
}
},
},
+ "node_modules/openvibe-publishing": {
+ "version": "1.5.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
+ "integrity": "sha512-8cKVhJvpbDgWuGR+Dh1mpgqfWPJoC7EIcw7eTRc5/2ueDc8caCA7XKdcpAHNZYMdukIAL8o7FvxE8QV5/1uSLw==",
+ "license": "MIT",
+ "engines": {
+ "node": ">=20"
+ },
+ "peerDependencies": {
+ "openvibe-sdk": ">=0.15.0",
+ "openvibe-shared": ">=1.5.0"
+ },
+ "peerDependenciesMeta": {
+ "openvibe-shared": {
+ "optional": true
+ }
+ }
+ },
"node_modules/openvibe-sdk": {
"node_modules/openvibe-sdk": {
"version": "0.81.0",
"version": "0.81.0",
"resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
"resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
@@ -25,6 +25,7 @@
@@ -25,6 +25,7 @@
"helmet": "^8.1.0",
"helmet": "^8.1.0",
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.235.0",
+ "openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.5.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.81.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.103.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -37,6 +37,7 @@ const { raw } = require('./render/html');
@@ -37,6 +37,7 @@ const { raw } = require('./render/html');
const accountDataLib = require('./identity/account-data');
const accountDataLib = require('./identity/account-data');
const { createStore } = require('./store');
const { createStore } = require('./store');
const { createSearchIndex } = require('./search');
const { createSearchIndex } = require('./search');
+const { createSearchRelay } = require('openvibe-publishing/search-relay');
const { createFirstParty } = require('./jobs/first-party');
const { createFirstParty } = require('./jobs/first-party');
const { createNetworkSender } = require('openvibe-sdk/account-data');
const { createNetworkSender } = require('openvibe-sdk/account-data');
@@ -59,7 +60,16 @@ async function createApp(opts = {}) {
@@ -59,7 +60,16 @@ async function createApp(opts = {}) {
const sso = createSso({ config, keys, fetchImpl, now: s.now, log });
const sso = createSso({ config, keys, fetchImpl, now: s.now, log });
const principal = createPrincipal({ config, keys });
const principal = createPrincipal({ config, keys });
- const search = createSearchIndex({ config, s });
+ // The queued Search documents (exchange_outbox) go to OpenVibe.Search's owner API with this service's token;
+ // off without EXCHANGE_SEARCH_URL or the client secret. Started by server/index.js, never under EXCHANGE_DRILL.
+ const searchRelay = createSearchRelay({
+ db: s.db, table: 'exchange_outbox', owner: 'exchange', searchUrl: config.search.url, fetchImpl, now: s.now, log,
+ auth: config.search.url && config.oauth.clientSecret ? contracts.serviceAuth.createTokenClient({
+ tokenUrl: `${config.networkInternalUrl}/oauth/token`, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,
+ audience: 'openvibe.search', scope: 'search.document.write', fetchImpl,
+ }) : null,
+ });
+ const search = createSearchIndex({ config, s, onQueued: () => searchRelay.kick() });
const entries = createStore(s, { search, config });
const entries = createStore(s, { search, config });
const firstParty = createFirstParty({ config, s, entries, fetchImpl, log, now: s.now });
const firstParty = createFirstParty({ config, s, entries, fetchImpl, log, now: s.now });
@@ -70,7 +80,7 @@ async function createApp(opts = {}) {
@@ -70,7 +80,7 @@ async function createApp(opts = {}) {
? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })
? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Exchange cannot answer account events'); });
: async () => { throw new Error('OV_OAUTH_CLIENT_SECRET is not set: Exchange cannot answer account events'); });
- const ctx = { config, s, keys, sso, principal, search, entries, firstParty, accountData, accountSend, log };
+ const ctx = { config, s, keys, sso, principal, search, searchRelay, entries, firstParty, accountData, accountSend, log };
const app = express();
const app = express();
app.disable('x-powered-by');
app.disable('x-powered-by');
@@ -61,6 +61,10 @@ function load(env = process.env) {
@@ -61,6 +61,10 @@ function load(env = process.env) {
scope: env.EXCHANGE_EVENTS_SCOPE || 'events.event.publish',
scope: env.EXCHANGE_EVENTS_SCOPE || 'events.event.publish',
},
},
+ // OpenVibe.Search's owner API: the documents queued in exchange_outbox are delivered there (server/app.js,
+ // openvibe-publishing/search-relay) with this service's token (search.document.write). Unset: they wait.
+ search: { url: trim(env.EXCHANGE_SEARCH_URL || '') },
+
// The sandbox: a manifest dry run, not a code execution. A passed trial is good for this many minutes and
// The sandbox: a manifest dry run, not a code execution. A passed trial is good for this many minutes and
// is required before an install; a person gets at most this many trials a day.
// is required before an install; a person gets at most this many trials a day.
sandbox: {
sandbox: {
@@ -60,7 +60,7 @@ function createDiscoveryRoutes(ctx) {
@@ -60,7 +60,7 @@ function createDiscoveryRoutes(ctx) {
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({
name: SITE_NAME,
name: SITE_NAME,
summary: 'OpenVibe.Exchange is the OpenVibe marketplace for what people make: agents, MCP tools, templates (Website, Report, Space), design themes and kits, game assets and creator services. Every public listing shows a permission passport (exactly what it can read, write, spend or contact), its licence, the conformance tests for that version, and a sandbox. A passed sandbox trial is required before an install; a revoke removes the installation in one tap. v1 is free and swap only: there is no money and a passport may not declare a spend.',
summary: 'OpenVibe.Exchange is the OpenVibe marketplace for what people make: agents, MCP tools, templates (Website, Report, Space), design themes and kits, game assets and creator services. Every public listing shows a permission passport (exactly what it can read, write, spend or contact), its licence, the conformance tests for that version, and a sandbox. A passed sandbox trial is required before an install; a revoke removes the installation in one tap. v1 is free and swap only: there is no money and a passport may not declare a spend.',
- details: 'GET /api/v1/listings lists public listings; GET /api/v1/listings/:slug adds the passport, manifest, licence, tests and counters; GET /api/v1/listings/:slug/passport is the exact passport an installer accepts. The sandbox is a manifest dry run in v1: Exchange does not execute a listing\'s code. Every public listing is one search.index-document@1 (owner exchange, type listing) queued in exchange_outbox.',
+ details: 'GET /api/v1/listings lists public listings; GET /api/v1/listings/:slug adds the passport, manifest, licence, tests and counters; GET /api/v1/listings/:slug/passport is the exact passport an installer accepts. The sandbox is a manifest dry run in v1: Exchange does not execute a listing\'s code. Every public listing is one search.index-document@1 (owner exchange, type listing) queued in exchange_outbox and delivered to OpenVibe.Search within a minute.',
sections: [
sections: [
{ title: 'Start here', links: [
{ title: 'Start here', links: [
{ title: 'OpenVibe.Exchange', url: abs('/'), note: 'free and swap listings' },
{ title: 'OpenVibe.Exchange', url: abs('/'), note: 'free and swap listings' },
@@ -20,6 +20,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
@@ -20,6 +20,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
() => { for (const t of timers) clearInterval(t); },
() => { for (const t of timers) clearInterval(t); },
() => ctx.keys.client.stop(),
() => ctx.keys.client.stop(),
() => ctx.firstParty.stop(),
() => ctx.firstParty.stop(),
+ () => ctx.searchRelay.stop(),
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
() => ctx.s.close(),
() => ctx.s.close(),
...extra,
...extra,
@@ -46,6 +47,8 @@ async function start() {
@@ -46,6 +47,8 @@ async function start() {
ctx.telemetryTick.unref();
ctx.telemetryTick.unref();
ctx.keys.client.start();
ctx.keys.client.start();
ctx.firstParty.start();
ctx.firstParty.start();
+ // Queued Search documents → OpenVibe.Search (a tick a few seconds after boot, then every minute).
+ if (!config.drill) ctx.searchRelay.start();
// The two account topics (export_requested, deleted) at OpenVibe.Events: idempotent; off without the URL, the
// The two account topics (export_requested, deleted) at OpenVibe.Events: idempotent; off without the URL, the
// delivery secret and the client secret. The consumer itself is mounted in server/app.js.
// delivery secret and the client secret. The consumer itself is mounted in server/app.js.
@@ -7,8 +7,9 @@
@@ -7,8 +7,9 @@
* revision. The document carries the maker's copy, the passport's declared permissions and the licence, so Search
* revision. The document carries the maker's copy, the passport's declared permissions and the licence, so Search
* can filter free/swap and kinds without reading the listing page.
* can filter free/swap and kinds without reading the listing page.
*
*
- * No service grant is needed to queue the rows: they wait in the outbox until OpenVibe.Search's document writer is
- * configured for `exchange` (README, "Grants it will need"). The relay is deliberately not part of v1.
+ * No service grant is needed to queue the rows. openvibe-publishing/search-relay (server/app.js) delivers them to
+ * Search's owner API with this service's token (search.document.write), a few seconds after each change
+ * (onQueued) and every minute; without EXCHANGE_SEARCH_URL they wait.
*/
*/
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
@@ -67,7 +68,7 @@ function documentOf(listing, { baseUrl = 'https://openvibe.exchange' } = {}) {
@@ -67,7 +68,7 @@ function documentOf(listing, { baseUrl = 'https://openvibe.exchange' } = {}) {
};
};
}
}
-function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {
+function createSearchIndex({ config, s, baseUrl = config.baseUrl, onQueued = null }) {
const nowIso = () => new Date(s.now()).toISOString();
const nowIso = () => new Date(s.now()).toISOString();
async function enqueue(topic, payload, t = null) {
async function enqueue(topic, payload, t = null) {
@@ -78,6 +79,7 @@ function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {
@@ -78,6 +79,7 @@ function createSearchIndex({ config, s, baseUrl = config.baseUrl }) {
VALUES ($1, $2, $3::jsonb, $4::timestamptz, $4::timestamptz)`,
VALUES ($1, $2, $3::jsonb, $4::timestamptz, $4::timestamptz)`,
[eventId, topic, JSON.stringify(payload), nowIso()],
[eventId, topic, JSON.stringify(payload), nowIso()],
);
);
+ if (onQueued) onQueued();
return eventId;
return eventId;
}
}
@@ -439,7 +439,7 @@ ${ep('GET', '/api/v1/me/listings', 'Every listing of yours, all statuses.')}
@@ -439,7 +439,7 @@ ${ep('GET', '/api/v1/me/listings', 'Every listing of yours, all statuses.')}
${ep('POST', '/api/v1/listings/:slug/hide', 'Hide a listing and tombstone its Search document.')}
${ep('POST', '/api/v1/listings/:slug/hide', 'Hide a listing and tombstone its Search document.')}
${ep('GET', '/api/v1/listings/:slug/offers', 'The open claims/offers on a listing.')}
${ep('GET', '/api/v1/listings/:slug/offers', 'The open claims/offers on a listing.')}
</dl></section>
</dl></section>
-<section id="ex-search"><h2>Search</h2><p class="ex-fine">Each public listing is one <code>search.index-document@1</code> (owner <code>exchange</code>, type <code>listing</code>, canonical page <code>/listings/<slug></code>) queued in <code>exchange_outbox</code>; closing or hiding it queues a tombstone. The outbox waits for Network's <code>search.document.write</code> grant; the relay is not part of v1.</p></section>
+<section id="ex-search"><h2>Search</h2><p class="ex-fine">Each public listing is one <code>search.index-document@1</code> (owner <code>exchange</code>, type <code>listing</code>, canonical page <code>/listings/<slug></code>) queued in <code>exchange_outbox</code> and delivered to OpenVibe.Search within a minute; closing or hiding it queues a tombstone.</p></section>
<section id="ex-grants"><h2>Grants v1 does not have yet</h2><p class="ex-fine">The plan enforces a passport with Network grants and executes a listing in a Run sandbox. v1 does neither: an install is an Exchange record and the sandbox is a dry run. The service will ask for <code>network.grants.write</code> and <code>network.grants.revoke</code> before an install/revoke can reach Network, <code>run.sandbox.execute</code> before a trial can run code, and <code>search.document.write</code> before the outbox can reach Search. Account export and deletion need <code>network.account.export.contribute</code> and <code>network.account.deletion.confirm</code>; the two Events subscriptions need <code>events.subscription.manage</code>.</p></section>
<section id="ex-grants"><h2>Grants v1 does not have yet</h2><p class="ex-fine">The plan enforces a passport with Network grants and executes a listing in a Run sandbox. v1 does neither: an install is an Exchange record and the sandbox is a dry run. The service will ask for <code>network.grants.write</code> and <code>network.grants.revoke</code> before an install/revoke can reach Network, <code>run.sandbox.execute</code> before a trial can run code, and <code>search.document.write</code> before the outbox can reach Search. Account export and deletion need <code>network.account.export.contribute</code> and <code>network.account.deletion.confirm</code>; the two Events subscriptions need <code>events.subscription.manage</code>.</p></section>
</div>`;
</div>`;
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,70 @@
@@ -0,0 +1,70 @@
+'use strict';
+/**
+ * The queued Search documents reach OpenVibe.Search (server/app.js, openvibe-publishing/search-relay): with
+ * EXCHANGE_SEARCH_URL set, publishing a listing PUTs its document to Search's owner API with a service token for
+ * openvibe.search, and closing it DELETEs it at the listing's new revision; a delivered row leaves the queue. Without
+ * the URL the relay is off and the rows wait.
+ */
+const assert = require('assert');
+const http = require('http');
+const { boot, check, done } = require('./helpers/boot');
+const fixtures = require('./helpers/fixtures');
+
+function fakeSearch() {
+ const calls = [];
+ const server = http.createServer((req, res) => {
+ let body = '';
+ req.on('data', (c) => { body += c; });
+ req.on('end', () => {
+ calls.push({ method: req.method, url: req.url, auth: req.headers.authorization || '', body: body ? JSON.parse(body) : null });
+ res.writeHead(200, { 'content-type': 'application/json' });
+ res.end(JSON.stringify({ outcome: 'applied' }));
+ });
+ });
+ return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ url: `http://127.0.0.1:${server.address().port}`, calls, close: () => new Promise((r) => server.close(r)) })));
+}
+
+(async () => {
+ const search = await fakeSearch();
+ const t = await boot({ env: { EXCHANGE_SEARCH_URL: search.url } });
+ try {
+ const alice = t.network.addUser('alice');
+ const l = await fixtures.listed(t, alice, { title: 'A relayed template' });
+
+ await check('a listed listing is PUT to Search with a service token, and the row leaves the queue', async () => {
+ assert.strictEqual(t.ctx.searchRelay.enabled, true);
+ const out = await t.ctx.searchRelay.drain();
+ assert.ok(out.sent >= 1, JSON.stringify(out));
+ const put = search.calls.find((c) => c.method === 'PUT' && c.url === `/api/v1/documents/exchange/listing/${l.id}`);
+ assert.ok(put, `a PUT for ${l.id}: ${JSON.stringify(search.calls.map((c) => `${c.method} ${c.url}`))}`);
+ assert.match(put.auth, /^Bearer \S+/);
+ assert.strictEqual(put.body.canonical_url, `https://openvibe.exchange/listings/${l.slug}`);
+ assert.strictEqual(put.body.title, 'A relayed template');
+ assert.strictEqual((await t.ctx.searchRelay.status()).pending, 0);
+ });
+
+ await check('closing it DELETEs the document at the listing\'s new revision', async () => {
+ const before = search.calls.length;
+ const r = await t.get(`/api/v1/listings/${l.slug}/close`, { as: alice, method: 'POST', json: { reason: 'withdrawn' } });
+ assert.strictEqual(r.status, 200);
+ await t.ctx.searchRelay.drain();
+ const del = search.calls.slice(before).find((c) => c.method === 'DELETE');
+ assert.ok(del, 'a DELETE');
+ assert.strictEqual(del.url, `/api/v1/documents/exchange/listing/${l.id}?revision=${r.json().revision}`);
+ });
+ } finally {
+ await t.close();
+ await search.close();
+ }
+
+ const off = await boot();
+ try {
+ await check('without EXCHANGE_SEARCH_URL the relay is off', async () => {
+ assert.strictEqual(off.ctx.searchRelay.enabled, false);
+ assert.strictEqual(await off.ctx.searchRelay.drain(), null);
+ });
+ } finally {
+ await off.close();
+ }
+ done();
+})();