openvibers/openvibe.tools Commits
Tools job events: the event_outbox table is an owner migration, never runtime DDL (plan T8 review): outboxFromEnv no longer calls createPgOutbox.ensureSchema through the runtime role (DATABASE_URL may have no DDL rights); outbox.ready checks the table and names migrations/0004_event_outbox.sql when it is missing. job-events.test.js: no runtime DDL, the missing-table report. The migration file itself follows (its DDL mirrors the installed openvibe-sdk).
Changes
@@@ -630,8 +630,10 @@ when the gateway's `/api/ready` or `/release.json` does not come up with every uAll eight apps move from `better-sqlite3` to **one `tools` PostgreSQL database** and to **Valkey** for theguard's throttle state. The schema is in `migrations/0001_tools.sql` (job store, `guard_abuse`) and--`migrations/0002_analytics.sql` (request analytics, the DDL of `openvibe-shared/analytics/pg.js`); the event--outbox and the revocation store create their own tables at boot. `scripts/migrate-to-postgres.js` imports++`migrations/0002_analytics.sql` (request analytics, the DDL of `openvibe-shared/analytics/pg.js`),++`migrations/0003_token_revocations.sql` and `migrations/0004_event_outbox.sql` (the tools.job.* outbox,++`openvibe-sdk` createPgOutbox's table). All of them run as the owner (`DATABASE_DIRECT_URL`); nothing creates a++table at runtime, so the runtime role needs no DDL rights. `scripts/migrate-to-postgres.js` importsevery `apps/*/data/{guard,jobs,analytics}.db` into that one database (each file's rows tagged with its app):run it with `--pglite` for an in-memory rehearsal, `--dry-run --url <scratch>` for a scratch database, orwith `DATABASE_DIRECT_URL` set for the cutover; it prints a per-table count and checksum report.@@@ -18,7 +18,8 @@// output data or a browser session: a session-owned job has owner null.//// Inert unless EVENTS_URL is set (and OV_OAUTH_CLIENT_SECRET, for the tools service token with--// events.event.publish on openvibe.events): no outbox table, nothing written, nothing relayed.++// events.event.publish on openvibe.events): nothing written, nothing relayed. The event_outbox table is++// migrations/0004_event_outbox.sql (run as the owner), never created at runtime.// EVENTS_PUBLISH=off turns it off with EVENTS_URL set.// ═══════════════════════════════════════════════════════════════@@@ -165,8 +166,10 @@ function createJobEvents({ contracts, service, outbox = null, referenceCount = a* OV_NETWORK_INTERNAL_URL token endpoint host (default http://127.0.0.1:4000)* EVENTS_RELAY_INTERVAL_MS relay poll (default 2000)*-- * The outbox table is created at boot (createPgOutbox.ensureSchema); outbox.ready is that promise, for a-- * test (or a caller) that must know the table exists.++ * The event_outbox table is an owner migration (migrations/0004_event_outbox.sql, DATABASE_DIRECT_URL):++ * the runtime role (DATABASE_URL) may have no DDL rights, so createPgOutbox.ensureSchema is never called++ * here. outbox.ready checks the table is there (after the app's migrations) and rejects, naming the++ * migration, when it is not; index.js logs that.** @param {object} o* @param {object} o.db openvibe-sdk/db handle — the outbox table lives beside tool_jobs@@@ -203,7 +206,9 @@ function outboxFromEnv({ db, sdk, env = process.env, fetch: fetchImpl, log = con},});outbox.url = url;-- outbox.ready = outbox.ensureSchema();++ outbox.ready = Promise.resolve(db.maybe("SELECT to_regclass('event_outbox') IS NOT NULL AS ok")).then((r) => {++ if (!r || !r.ok) throw new Error('the event_outbox table is missing: run migrations/0004_event_outbox.sql as the owner (DATABASE_DIRECT_URL)');++ });return { outbox, reason: null };}@@@ -99,7 +99,7 @@ function setupJobs(o) {// tools.job.* → OpenVibe.Events through the `event_outbox` table in this same database (inert without EVENTS_URL).const events = outboxFromEnv({ db, sdk: o.sdk, log });if (events.reason && process.env.EVENTS_URL) console.warn(`[Jobs] ${o.service}: job events are not published — ${events.reason}`);-- if (events.outbox) events.outbox.ready.catch((err) => log.error(`[Jobs] ${o.service}: could not create the event outbox:`, err.message));++ if (events.outbox) events.outbox.ready.catch((err) => log.error(`[Jobs] ${o.service}: job events cannot be written:`, err.message));const system = createJobSystem({db, contracts: o.contracts, service: o.service, dataDir: o.dataDir,concurrency: envInt(`TOOLS_JOBS_CONCURRENCY_${o.service.toUpperCase()}`, envInt('TOOLS_JOBS_CONCURRENCY', 2)),@@@ -156,11 +156,27 @@ const settled = (s, id, state) => until(async () => { const r = await s.system.gawait system.start();const { job } = await system.submit({ owner: `user:${USER}`, type: 'test.upper', input: { text: 'x' } });await until(async () => (await system.get(job.id)).state === 'succeeded', 'inert job');-- assert.strictEqual((await db.prepare("SELECT COUNT(*) AS n FROM information_schema.tables WHERE table_name = 'event_outbox'").get()).n, 0, 'no outbox table without EVENTS_URL');++ assert.strictEqual(Number((await db.prepare('SELECT COUNT(*) AS n FROM event_outbox').get()).n), 0, 'nothing in the outbox without EVENTS_URL (the table itself is a migration)');assert.deepStrictEqual((await system.stats()).events, { enabled: false });system.stop();}++ // ── The outbox table is an owner migration, never runtime DDL ──++ {++ const db = await testDb(path.join(root, 'schema-db'));++ let ddl = 0;++ const spy = { ...sdk, events: { ...sdk.events, createPgOutbox: (d, o) => { const ob = sdk.events.createPgOutbox(d, o); ob.ensureSchema = async () => { ddl++; }; return ob; } } };++ const { outbox } = outboxFromEnv({ db, sdk: spy, env: ENV, fetch: fakeNetwork().fetch, log: silent });++ await outbox.ready;++ assert.strictEqual(ddl, 0, 'the runtime role never creates the outbox table');++ assert.ok(await db.maybe("SELECT to_regclass('event_outbox') AS t"), 'migrations/0004_event_outbox.sql created it');++ const missing = Object.create(db);++ missing.maybe = async () => ({ ok: false });++ const bare = outboxFromEnv({ db: missing, sdk: spy, env: ENV, fetch: fakeNetwork().fetch, log: silent }).outbox;++ await assert.rejects(bare.ready, /0004_event_outbox\.sql/, 'a missing table is reported, naming the migration');++ assert.strictEqual(ddl, 0);++ }++const net = fakeNetwork();// An outbox that can be told to fail right after it wrote a row (a failing write inside the transaction).let failOn = null;@@@ -4,7 +4,7 @@---- Not here on purpose (plan T8, decision 4):-- guard_salt, guard_day, the minute/burst token buckets -> Valkey (throttle state, worthless after a day)++-- the event outbox -> migrations/0004_event_outbox.sql (openvibe-sdk createPgOutbox's table, as owner)-- token_revocations -> openvibe-sdk/auth createPgRevocationStore creates its own table at boot---- `app` (the owning app, one row per app in the one database) has DEFAULT '' so scripts/migrate-to-postgres.js@@@ -10,8 +10,8 @@//// Every app's data/<guard|jobs|analytics>.db is imported into the shared tables, tagged with its app id// (`app` = the app name; the analytics tables already carry `service`). guard_salt and guard_day are not--// copied (Valkey, decision 4); the event outbox and token_revocations are left behind (the SDK re-creates--// its own tables at boot and a revocation cutoff is re-derivable). The SQLite files are opened read-only.++// copied (Valkey, decision 4); the event outbox's rows and token_revocations are left behind (the tables are++// migrations 0004 and 0003; an undelivered SQLite outbox row is not replayed, and a revocation cutoff is re-derivable). The SQLite files are opened read-only.//// It relies on the newer openvibe-sdk/db (0.25+) and better-sqlite3; OV_SDK_DIR points at an openvibe-sdk// checkout, else the gateway's node_modules is used. better-sqlite3 is scripts/' own dependency (no app
@@ -630,8 +630,10 @@ when the gateway's `/api/ready` or `/release.json` does not come up with every u
@@ -630,8 +630,10 @@ when the gateway's `/api/ready` or `/release.json` does not come up with every u
All eight apps move from `better-sqlite3` to **one `tools` PostgreSQL database** and to **Valkey** for the
All eight apps move from `better-sqlite3` to **one `tools` PostgreSQL database** and to **Valkey** for the
guard's throttle state. The schema is in `migrations/0001_tools.sql` (job store, `guard_abuse`) and
guard's throttle state. The schema is in `migrations/0001_tools.sql` (job store, `guard_abuse`) and
-`migrations/0002_analytics.sql` (request analytics, the DDL of `openvibe-shared/analytics/pg.js`); the event
-outbox and the revocation store create their own tables at boot. `scripts/migrate-to-postgres.js` imports
+`migrations/0002_analytics.sql` (request analytics, the DDL of `openvibe-shared/analytics/pg.js`),
+`migrations/0003_token_revocations.sql` and `migrations/0004_event_outbox.sql` (the tools.job.* outbox,
+`openvibe-sdk` createPgOutbox's table). All of them run as the owner (`DATABASE_DIRECT_URL`); nothing creates a
+table at runtime, so the runtime role needs no DDL rights. `scripts/migrate-to-postgres.js` imports
every `apps/*/data/{guard,jobs,analytics}.db` into that one database (each file's rows tagged with its app):
every `apps/*/data/{guard,jobs,analytics}.db` into that one database (each file's rows tagged with its app):
run it with `--pglite` for an in-memory rehearsal, `--dry-run --url <scratch>` for a scratch database, or
run it with `--pglite` for an in-memory rehearsal, `--dry-run --url <scratch>` for a scratch database, or
with `DATABASE_DIRECT_URL` set for the cutover; it prints a per-table count and checksum report.
with `DATABASE_DIRECT_URL` set for the cutover; it prints a per-table count and checksum report.
@@ -18,7 +18,8 @@
@@ -18,7 +18,8 @@
// output data or a browser session: a session-owned job has owner null.
// output data or a browser session: a session-owned job has owner null.
//
//
// Inert unless EVENTS_URL is set (and OV_OAUTH_CLIENT_SECRET, for the tools service token with
// Inert unless EVENTS_URL is set (and OV_OAUTH_CLIENT_SECRET, for the tools service token with
-// events.event.publish on openvibe.events): no outbox table, nothing written, nothing relayed.
+// events.event.publish on openvibe.events): nothing written, nothing relayed. The event_outbox table is
+// migrations/0004_event_outbox.sql (run as the owner), never created at runtime.
// EVENTS_PUBLISH=off turns it off with EVENTS_URL set.
// EVENTS_PUBLISH=off turns it off with EVENTS_URL set.
// ═══════════════════════════════════════════════════════════════
// ═══════════════════════════════════════════════════════════════
@@ -165,8 +166,10 @@ function createJobEvents({ contracts, service, outbox = null, referenceCount = a
@@ -165,8 +166,10 @@ function createJobEvents({ contracts, service, outbox = null, referenceCount = a
* OV_NETWORK_INTERNAL_URL token endpoint host (default http://127.0.0.1:4000)
* OV_NETWORK_INTERNAL_URL token endpoint host (default http://127.0.0.1:4000)
* EVENTS_RELAY_INTERVAL_MS relay poll (default 2000)
* EVENTS_RELAY_INTERVAL_MS relay poll (default 2000)
*
*
- * The outbox table is created at boot (createPgOutbox.ensureSchema); outbox.ready is that promise, for a
- * test (or a caller) that must know the table exists.
+ * The event_outbox table is an owner migration (migrations/0004_event_outbox.sql, DATABASE_DIRECT_URL):
+ * the runtime role (DATABASE_URL) may have no DDL rights, so createPgOutbox.ensureSchema is never called
+ * here. outbox.ready checks the table is there (after the app's migrations) and rejects, naming the
+ * migration, when it is not; index.js logs that.
*
*
* @param {object} o
* @param {object} o
* @param {object} o.db openvibe-sdk/db handle — the outbox table lives beside tool_jobs
* @param {object} o.db openvibe-sdk/db handle — the outbox table lives beside tool_jobs
@@ -203,7 +206,9 @@ function outboxFromEnv({ db, sdk, env = process.env, fetch: fetchImpl, log = con
@@ -203,7 +206,9 @@ function outboxFromEnv({ db, sdk, env = process.env, fetch: fetchImpl, log = con
},
},
});
});
outbox.url = url;
outbox.url = url;
- outbox.ready = outbox.ensureSchema();
+ outbox.ready = Promise.resolve(db.maybe("SELECT to_regclass('event_outbox') IS NOT NULL AS ok")).then((r) => {
+ if (!r || !r.ok) throw new Error('the event_outbox table is missing: run migrations/0004_event_outbox.sql as the owner (DATABASE_DIRECT_URL)');
+ });
return { outbox, reason: null };
return { outbox, reason: null };
}
}
@@ -99,7 +99,7 @@ function setupJobs(o) {
@@ -99,7 +99,7 @@ function setupJobs(o) {
// tools.job.* → OpenVibe.Events through the `event_outbox` table in this same database (inert without EVENTS_URL).
// tools.job.* → OpenVibe.Events through the `event_outbox` table in this same database (inert without EVENTS_URL).
const events = outboxFromEnv({ db, sdk: o.sdk, log });
const events = outboxFromEnv({ db, sdk: o.sdk, log });
if (events.reason && process.env.EVENTS_URL) console.warn(`[Jobs] ${o.service}: job events are not published — ${events.reason}`);
if (events.reason && process.env.EVENTS_URL) console.warn(`[Jobs] ${o.service}: job events are not published — ${events.reason}`);
- if (events.outbox) events.outbox.ready.catch((err) => log.error(`[Jobs] ${o.service}: could not create the event outbox:`, err.message));
+ if (events.outbox) events.outbox.ready.catch((err) => log.error(`[Jobs] ${o.service}: job events cannot be written:`, err.message));
const system = createJobSystem({
const system = createJobSystem({
db, contracts: o.contracts, service: o.service, dataDir: o.dataDir,
db, contracts: o.contracts, service: o.service, dataDir: o.dataDir,
concurrency: envInt(`TOOLS_JOBS_CONCURRENCY_${o.service.toUpperCase()}`, envInt('TOOLS_JOBS_CONCURRENCY', 2)),
concurrency: envInt(`TOOLS_JOBS_CONCURRENCY_${o.service.toUpperCase()}`, envInt('TOOLS_JOBS_CONCURRENCY', 2)),
@@ -156,11 +156,27 @@ const settled = (s, id, state) => until(async () => { const r = await s.system.g
@@ -156,11 +156,27 @@ const settled = (s, id, state) => until(async () => { const r = await s.system.g
await system.start();
await system.start();
const { job } = await system.submit({ owner: `user:${USER}`, type: 'test.upper', input: { text: 'x' } });
const { job } = await system.submit({ owner: `user:${USER}`, type: 'test.upper', input: { text: 'x' } });
await until(async () => (await system.get(job.id)).state === 'succeeded', 'inert job');
await until(async () => (await system.get(job.id)).state === 'succeeded', 'inert job');
- assert.strictEqual((await db.prepare("SELECT COUNT(*) AS n FROM information_schema.tables WHERE table_name = 'event_outbox'").get()).n, 0, 'no outbox table without EVENTS_URL');
+ assert.strictEqual(Number((await db.prepare('SELECT COUNT(*) AS n FROM event_outbox').get()).n), 0, 'nothing in the outbox without EVENTS_URL (the table itself is a migration)');
assert.deepStrictEqual((await system.stats()).events, { enabled: false });
assert.deepStrictEqual((await system.stats()).events, { enabled: false });
system.stop();
system.stop();
}
}
+ // ── The outbox table is an owner migration, never runtime DDL ──
+ {
+ const db = await testDb(path.join(root, 'schema-db'));
+ let ddl = 0;
+ const spy = { ...sdk, events: { ...sdk.events, createPgOutbox: (d, o) => { const ob = sdk.events.createPgOutbox(d, o); ob.ensureSchema = async () => { ddl++; }; return ob; } } };
+ const { outbox } = outboxFromEnv({ db, sdk: spy, env: ENV, fetch: fakeNetwork().fetch, log: silent });
+ await outbox.ready;
+ assert.strictEqual(ddl, 0, 'the runtime role never creates the outbox table');
+ assert.ok(await db.maybe("SELECT to_regclass('event_outbox') AS t"), 'migrations/0004_event_outbox.sql created it');
+ const missing = Object.create(db);
+ missing.maybe = async () => ({ ok: false });
+ const bare = outboxFromEnv({ db: missing, sdk: spy, env: ENV, fetch: fakeNetwork().fetch, log: silent }).outbox;
+ await assert.rejects(bare.ready, /0004_event_outbox\.sql/, 'a missing table is reported, naming the migration');
+ assert.strictEqual(ddl, 0);
+ }
+
const net = fakeNetwork();
const net = fakeNetwork();
// An outbox that can be told to fail right after it wrote a row (a failing write inside the transaction).
// An outbox that can be told to fail right after it wrote a row (a failing write inside the transaction).
let failOn = null;
let failOn = null;
@@ -4,7 +4,7 @@
@@ -4,7 +4,7 @@
--
--
-- Not here on purpose (plan T8, decision 4):
-- Not here on purpose (plan T8, decision 4):
-- guard_salt, guard_day, the minute/burst token buckets -> Valkey (throttle state, worthless after a day)
-- guard_salt, guard_day, the minute/burst token buckets -> Valkey (throttle state, worthless after a day)
+-- the event outbox -> migrations/0004_event_outbox.sql (openvibe-sdk createPgOutbox's table, as owner)
-- token_revocations -> openvibe-sdk/auth createPgRevocationStore creates its own table at boot
-- token_revocations -> openvibe-sdk/auth createPgRevocationStore creates its own table at boot
--
--
-- `app` (the owning app, one row per app in the one database) has DEFAULT '' so scripts/migrate-to-postgres.js
-- `app` (the owning app, one row per app in the one database) has DEFAULT '' so scripts/migrate-to-postgres.js
@@ -10,8 +10,8 @@
@@ -10,8 +10,8 @@
//
//
// Every app's data/<guard|jobs|analytics>.db is imported into the shared tables, tagged with its app id
// Every app's data/<guard|jobs|analytics>.db is imported into the shared tables, tagged with its app id
// (`app` = the app name; the analytics tables already carry `service`). guard_salt and guard_day are not
// (`app` = the app name; the analytics tables already carry `service`). guard_salt and guard_day are not
-// copied (Valkey, decision 4); the event outbox and token_revocations are left behind (the SDK re-creates
-// its own tables at boot and a revocation cutoff is re-derivable). The SQLite files are opened read-only.
+// copied (Valkey, decision 4); the event outbox's rows and token_revocations are left behind (the tables are
+// migrations 0004 and 0003; an undelivered SQLite outbox row is not replayed, and a revocation cutoff is re-derivable). The SQLite files are opened read-only.
//
//
// It relies on the newer openvibe-sdk/db (0.25+) and better-sqlite3; OV_SDK_DIR points at an openvibe-sdk
// It relies on the newer openvibe-sdk/db (0.25+) and better-sqlite3; OV_SDK_DIR points at an openvibe-sdk
// checkout, else the gateway's node_modules is used. better-sqlite3 is scripts/' own dependency (no app
// checkout, else the gateway's node_modules is used. better-sqlite3 is scripts/' own dependency (no app