openvibers/openvibe.sdk Commits
v0.20.0: db.afterCommit(fn) runs fn once the running transaction commits, never after a rollback; a savepoint that rolls back drops the hooks added inside it. Hooks run in order, outside the transaction, with the connection back in the pool; one that throws is logged and the rest still run. Outside a transaction fn runs on the next turn. It replaces the setImmediate-after-a-synchronous-transaction pattern of better-sqlite3 services (News' comment-thread visibility first). db.stats().open counts transactions in progress. tools/asyncify gains sqlite-schema-to-pg.js and g1pass.py, and asyncify.js treats { name: asyncFn } as an async method. docs/migrating-to-postgresql.md §7 has News' lessons (rowid tiebreaks, GREATEST, qualified ON CONFLICT targets, afterCommit, the PG inbox). npm test 41/41; db-ambient also through PgBouncer.
Changes
@@@ -8,6 +8,12 @@ release may change an API and says so here.++## 0.20.0 (2026-09-28)++++- **`db.afterCommit(fn)`** (and `t.afterCommit(fn)` on a transaction handle) runs `fn` once the running transaction commits. It never runs after a rollback, and a savepoint that rolls back drops the hooks added inside it. Hooks run in order, outside the transaction, with the connection already back in the pool; one that throws is logged and the rest still run. Outside a transaction, `fn` runs on the next turn. It replaces the better-sqlite3 pattern of a `setImmediate` after a synchronous transaction, for calls to other services that must see committed state (News' comment-thread visibility was the first user).++- **`db.stats().open`**: transactions in progress on this handle.++- `tools/asyncify` gains `sqlite-schema-to-pg.js` (a service's SQLite DDL → `migrations/0001_initial.sql`) and `g1pass.py` (the non-codemod edits of a Blog-shaped service). `asyncify.js` now treats `{ name: someAsyncFunction }` as an async method `name`. None of these are in the published package.++## 0.19.0 (2026-09-28)- **`openvibe-sdk/db` `runSqliteMigration({ service, sqlite, directUrl, migrations, tables, argv })`** is a service's `scripts/migrate-to-postgres.js` in one call. It:@@@ -3,7 +3,7 @@"stage": "alpha","code": true,"wave": "W2",-- "version": "0.19.0",++ "version": "0.20.0","plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30","deployed": false,"deployment": "none: a package released as git tags; services and apps pin a tag",@@@ -141,3 +141,10 @@ It changes nothing in the SQLite file.- **Claim an Idempotency-Key before running the handler**, with `INSERT … ON CONFLICT DO NOTHING RETURNING`. That way two concurrent requests with the same key run once.- **Clean text PostgreSQL refuses.** SQLite text can hold NUL and unpaired surrogates. Clean them in the importer's `map`, and report each column that needed it.- **pub/sub without Valkey.** `createPubSub` has no local fallback when Valkey is down. A hub that must keep delivering on one process falls back to local delivery itself.++- **News (2026-09-28)** added these:++ - **`rowid` tiebreaks.** `ORDER BY created_at, rowid` meant insertion order. A random ULID `id` is not, within one millisecond. Add `seq bigint GENERATED ALWAYS AS IDENTITY UNIQUE` to the table and order by `seq`. The importer fills it in the source's row order.++ - **Scalar `MAX(a, b)` / `MIN(a, b)`** are `GREATEST` / `LEAST`.++ - **`ON CONFLICT … DO UPDATE SET x = COALESCE(excluded.x, x)`** is ambiguous in PostgreSQL. Qualify the target: `COALESCE(excluded.x, <table>.x)`.++ - **Work after the commit.** A `setImmediate` after a synchronous better-sqlite3 transaction ran after the commit. Around an async transaction it does not. Use `db.afterCommit(fn)` (SDK ≥ 0.20).++ - **The inbox.** `createInbox` (better-sqlite3) becomes `createPgInbox`, awaited. Its `idempotency_receipts` table goes in `migrations/0001_initial.sql` (`inboxSchema()`), not an unawaited `ensureSchema()`.++ - **A script run as a child process in tests** cannot reach the test's PGlite. Call the function it wraps in-process, and `node --check` the script.@@@ -1,12 +1,12 @@{"name": "openvibe-sdk",-- "version": "0.19.0",++ "version": "0.20.0","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "openvibe-sdk",-- "version": "0.19.0",++ "version": "0.20.0","license": "MIT","devDependencies": {"@electric-sql/pglite": "^0.5.8",@@@ -1,6 +1,6 @@{"name": "openvibe-sdk",-- "version": "0.19.0",++ "version": "0.20.0","description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.","main": "index.js","types": "types/index.d.ts",@@@ -97,7 +97,7 @@ function createDb(o = {}) {const log = o.log || console;const service = o.service || process.env.OV_SERVICE || 'service';const slowMs = o.slowMs == null ? (o.pglite ? Infinity : 500) : o.slowMs; // PGlite's first query includes its WASM start-- const stats = { queries: 0, errors: 0, slow: 0, retries: 0, tx: 0 };++ const stats = { queries: 0, errors: 0, slow: 0, retries: 0, tx: 0, open: 0 };let hist = null;if (o.registry && typeof o.registry.histogram === 'function') {try { hist = o.registry.histogram({ name: 'db_query_seconds', help: 'Database query time by kind', labelNames: ['kind'], buckets: [0.001, 0.005, 0.02, 0.1, 0.5, 2, 10] }); } catch { hist = null; }@@@ -131,20 +131,21 @@ function createDb(o = {}) {};const inScope = (scope, fn) => (als ? als.run(scope, fn) : fn());-- async function tx(fn, opts = {}, conn = null, depth = 0) {++ async function tx(fn, opts = {}, conn = null, depth = 0, hooks = null) {if (!conn) {const cur = current();-- if (cur) { conn = cur.conn; depth = cur.depth; } // ambient: a savepoint in the running transaction++ if (cur) { conn = cur.conn; depth = cur.depth; hooks = cur.hooks; } // ambient: a savepoint in the running transaction}if (conn) {-- // Nested: a savepoint inside the running transaction.++ // Nested: a savepoint inside the running transaction. Its after-commit hooks go with it if it rolls back.const name = `sp_${depth}`;++ const mark = hooks.length;await timed(conn, `SAVEPOINT ${name}`, []);try {-- const out = await inScope({ conn, depth: depth + 1, done: false }, () => fn(txApi(conn, depth + 1)));++ const out = await inScope({ conn, depth: depth + 1, done: false, hooks }, () => fn(txApi(conn, depth + 1, hooks)));await timed(conn, `RELEASE SAVEPOINT ${name}`, []);return out;-- } catch (err) { await timed(conn, `ROLLBACK TO SAVEPOINT ${name}`, []).catch(() => {}); throw err; }++ } catch (err) { hooks.length = mark; await timed(conn, `ROLLBACK TO SAVEPOINT ${name}`, []).catch(() => {}); throw err; }}const iso = ISOLATION[opts.isolation || 'read committed'];if (!iso) throw new TypeError(`openvibe-sdk/db: isolation must be one of ${Object.keys(ISOLATION).join(', ')}`);@@@ -152,15 +153,23 @@ function createDb(o = {}) {for (let attempt = 0; ; attempt++) {const c = await adapter.acquire();stats.tx++;-- const scope = { conn: c, depth: 1, done: false };++ stats.open++;++ const scope = { conn: c, depth: 1, done: false, hooks: [] };++ let committed = false;try {await timed(c, `BEGIN ISOLATION LEVEL ${iso}${opts.readOnly ? ' READ ONLY' : ''}`, []);-- const out = await inScope(scope, () => fn(txApi(c, 1)));++ const out = await inScope(scope, () => fn(txApi(c, 1, scope.hooks)));scope.done = true;await timed(c, 'COMMIT', []);++ committed = true;++ stats.open--;++ adapter.release(c);++ await runHooks(scope.hooks);return out;} catch (err) {++ if (committed) throw err;scope.done = true;++ stats.open--;await timed(c, 'ROLLBACK', []).catch(() => {});const code = err && (err.code || (err.cause && err.cause.code));if (RETRYABLE.has(code) && attempt < retries) {@@@ -169,13 +178,26 @@ function createDb(o = {}) {continue;}throw err;-- } finally { adapter.release(c); }++ } finally { if (!committed) adapter.release(c); }}}-- function txApi(conn, depth) {++ /** After-commit hooks run in order, outside the transaction; one that throws is logged and the rest still run. */++ async function runHooks(hooks) {++ for (const h of hooks) {++ try { await h(); } catch (err) { log.warn(`[db] after-commit hook failed: ${err && err.message ? err.message : err}`); }++ }++ }++ function afterCommit(hooks, fn) {++ if (typeof fn !== 'function') throw new TypeError('openvibe-sdk/db: afterCommit(fn) needs a function');++ if (hooks) { hooks.push(fn); return; }++ setImmediate(() => runHooks([fn])); // no transaction: nothing to wait for but the caller's own turn++ }++++ function txApi(conn, depth, hooks) {const t = queryApi((text, values) => timed(conn, text, values), {});-- t.tx = (fn) => tx(fn, {}, conn, depth);++ t.tx = (fn) => tx(fn, {}, conn, depth, hooks);++ t.afterCommit = (fn) => afterCommit(hooks, fn);t.sql = sql;t.prepare = (text) => prepare(t, text);return t;@@@ -188,6 +210,14 @@ function createDb(o = {}) {db.prepare = (text) => prepare(db, text);/** True while this code runs inside db.tx (ambient mode). */db.inTransaction = () => Boolean(current());++ /**++ * Run fn once the running transaction commits (never if it rolls back; a savepoint that rolls back drops the hooks++ * added inside it). Outside a transaction, fn runs on the next turn. Hooks are awaited in order before db.tx++ * resolves, with the connection already back in the pool: a hook that must not delay the caller starts its work++ * and returns. For calls to other services that must see committed state (the setImmediate-after-a-synchronous-++ * better-sqlite3-transaction pattern).++ */++ db.afterCommit = (fn) => { const cur = current(); afterCommit(cur ? cur.hooks : null, fn); };/** Run fn outside any ambient transaction (its queries take a pool connection). */db.detached = (fn) => (als ? als.exit(fn) : fn());db.store = adapter.store;@@@ -31,6 +31,32 @@ function cases(label, open) {assert.equal(await count.get(), 1);} finally { await done(); }}],++ [`${label}: afterCommit runs after the commit, in order, never after a rollback; a failed savepoint drops its hooks`, async () => {++ const { db, done } = await open();++ try {++ const ins = db.prepare('INSERT INTO items (name, qty) VALUES (?, ?)');++ const seen = [];++ const committed = async (tag) => { seen.push([tag, db.inTransaction(), db.stats().open, await db.prepare('SELECT count(*)::int AS n FROM items').pluck().get()]); };++ await db.tx(async (t) => {++ await ins.run('a', 1);++ db.afterCommit(() => committed('ambient'));++ t.afterCommit(() => committed('handle'));++ await assert.rejects(db.tx(async () => { db.afterCommit(() => committed('dropped')); throw new Error('inner'); }), /inner/);++ await db.tx(async () => { db.afterCommit(() => committed('kept')); });++ db.afterCommit(() => { throw new Error('a failing hook is logged'); });++ assert.deepEqual(seen, [], 'nothing runs inside the transaction');++ assert.equal(db.stats().open, 1);++ });++ assert.deepEqual(seen, [['ambient', false, 0, 1], ['handle', false, 0, 1], ['kept', false, 0, 1]]);++ seen.length = 0;++ await assert.rejects(db.tx(async () => { await ins.run('b', 2); db.afterCommit(() => committed('rolled back')); throw new Error('boom'); }), /boom/);++ db.afterCommit(() => committed('outside'));++ assert.deepEqual(seen, [], 'outside a transaction: the next turn');++ await new Promise((r) => setImmediate(r));++ await new Promise((r) => setTimeout(r, 20));++ assert.deepEqual(seen, [['outside', false, 0, 1]]);++ } finally { await done(); }++ }],[`${label}: a db.tx inside is a savepoint; its failure undoes only itself`, async () => {const { db, done } = await open();try {@@@ -35,6 +35,16 @@ What stays for a person:- `RETURNING` in place of `lastInsertRowid`;- the boot (open, migrate, `createApp` async) and the test helpers (`openvibe-sdk/testing` `createTestDb`).++Also by hand: `db.transaction(fn)()` → `db.tx(fn)` (the codemod leaves it), scalar `MAX(a, b)` → `GREATEST`, `rowid` tiebreaks → an identity `seq` column, and a `setImmediate` meant to run after a commit → `db.afterCommit`. The lessons list in `docs/migrating-to-postgresql.md` §7 has the rest.++++## sqlite-schema-to-pg.js++++`toPg(sqliteDdl)` turns the SCHEMA string a service's `db.js` ran into the first migration: `INTEGER PRIMARY KEY AUTOINCREMENT` → identity, `TEXT` → `text COLLATE "C"`, `INTEGER` → `bigint`, `REAL` → `double precision`, `BLOB` → `bytea`, and no `IF NOT EXISTS`. It refuses triggers.++++## g1pass.py++++`g1pass.py <svc> <Name> <ENVPREFIX>`, run in a Blog-shaped service's checkout, makes the edits that are the same in every such service: config (`DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`), the PostgreSQL outbox, Valkey actor limits, readiness, and the test boot on `createTestDb`. Each edit checks its anchor and is skipped when already applied.++## awaitify-tests.pyThis is the earlier regex pass for test files (Wiki). `asyncify.js` over the server and the tests together supersedes it.@@@ -191,6 +191,10 @@ function main() {}// Every async function of the file (new or already) counts for callers.const all = new Set([...(asyncNames.local.get(f) || []), ...r.newlyAsyncNames]);++ // { name: someAsyncFunction } exports the function under another name: that name is async too.++ walk.full(parse(fs.readFileSync(f, 'utf8')), (n) => {++ if (n.type === 'Property' && n.value && n.value.type === 'Identifier' && all.has(n.value.name) && n.key) all.add(n.key.name || n.key.value);++ });walk.fullAncestor(parse(fs.readFileSync(f, 'utf8')), (node, _s, anc) => {if ((node.type === 'FunctionDeclaration' || node.type === 'FunctionExpression' || node.type === 'ArrowFunctionExpression') && node.async) {const parent = anc[anc.length - 2];nnew file mode 100644@@@ -0,0 +1,149 @@++#!/usr/bin/env python3++"""The shared, non-codemod edits of a content product's PostgreSQL pass (Blog-shaped services: server/app.js createApp,++server/events/outbox.js, server/http/actor-limits.js, server/observability.js, test/helpers/boot.js).++usage: g1pass.py <svc> <Name> <ENVPREFIX> (e.g. news News NEWS), run in the service's checkout. Idempotent-ish: each++edit asserts its anchor and skips when already applied."""++import json, re, sys, os++svc, Name, ENV = sys.argv[1], sys.argv[2], sys.argv[3]++def edit(p, pairs, required=True):++ if not os.path.exists(p): print('skip (missing)', p); return++ s = open(p).read(); o = s++ for a, b in pairs:++ if b in s and a not in s: continue++ if a not in s:++ if required: raise SystemExit(f'{p}: anchor not found: {a[:90]}')++ continue++ s = s.replace(a, b, 1)++ if s != o: open(p, 'w').write(s); print('edited', p)++# config++edit('server/config.js', [(f" dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',", f""" // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).++ db: {{ url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' }},++ valkey: {{ url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:{svc}:' }},++ // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.++ dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',""")])++# outbox++s = open('server/events/outbox.js').read()++edit('server/events/outbox.js', [++ ("const { createEventsClient, createOutbox } = require('openvibe-sdk/events');", "const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');"),++ (" const outbox = createOutbox(db, {", " // The PostgreSQL outbox: rows are written in the change's own transaction (enqueue(db, …) joins the ambient\n // transaction); several processes relay one table safely (leases).\n const outbox = createPgOutbox(db, {"),++ (" outbox.ensureSchema();\n", ""),++])++s = open('server/events/outbox.js').read()++s2 = re.sub(r'outbox\.enqueue\((?!db, )', 'outbox.enqueue(db, ', s)++if s2 != s: open('server/events/outbox.js', 'w').write(s2); print('enqueue(db, …)')++# app.js store++s = open('server/app.js').read()++m = re.search(r" const store = opts\.store \|\| openStore\(opts\.dbPath \|\| config\.dbPath, \{ now: opts\.now(, [^}]*)? \}\);", s)++if m:++ s = s.replace(m.group(0), " // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test, a script) hands in a store.\n const store = opts.store || await openStore(config, { now: opts.now, log });")++ s = s.replace('function createApp(opts = {}) {', 'async function createApp(opts = {}) {', 1)++ open('server/app.js', 'w').write(s); print('app store')++# limits + valkey++edit('server/http/actor-limits.js', [++ ("const { createActorLimiter, defaultActor } = require('openvibe-sdk/limits');", "const { createActorLimiter, createValkeyLimitStore, defaultActor } = require('openvibe-sdk/limits');"),++ ("function createActorLimits({ config, now = () => Date.now(), registry = null, log = console }) {", "function createActorLimits({ config, now = () => Date.now(), registry = null, log = console, valkey = null }) {"),++ (" now,\n", " now,\n // Shared across processes on Valkey (ADR-035) when VALKEY_URL is set; in-process otherwise.\n ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),\n"),++])++s = open('server/app.js').read()++m = re.search(r" ctx\.limits = createActorLimits\(\{ config, now: opts\.limitsNow \|\| \(\(\) => Date\.now\(\)\), registry: metrics\.registry, log \}\);", s)++if m:++ s = s.replace(m.group(0), " // Valkey (ADR-035): shared, never-authoritative state (per-actor limit counters). Optional.\n const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);\n ctx.valkey = valkey;\n ctx.limits = createActorLimits({ config, now: opts.limitsNow || (() => Date.now()), registry: metrics.registry, log, valkey });")++ open('server/app.js', 'w').write(s); print('app valkey')++# readiness++s = open('server/observability.js').read(); o = s++s = re.sub(r"(function create\w+Readiness\(\{ store, [^}]*?)(, release = null \}\))", lambda m: m.group(1) + (', valkey = null' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)++s = s.replace(""" {++ name: 'network_jwks', required: false,""", """ { name: 'valkey', required: false, check: async () => (valkey ? valkey.ready() : { skipped: 'VALKEY_URL not set: per-actor limits count in this process only' }) },++ {++ name: 'network_jwks', required: false,""", 1) if "name: 'valkey'" not in s else s++s = re.sub(r"""const names = new Set\((?:\(await )?db\.prepare\("SELECT name FROM sqlite_master WHERE type IN \('table','view'\)"\)\.all\(\)\)?\.map\(\(r\) => r\.name\)\);\n(\s*)const missing = CHARTER_TABLES\.filter\(\(t\) => !names\.has\(t\)\);\n\s*return missing\.length \? `missing \$\{missing\.join\(', '\)\}` : true;""",++ lambda m: f"""// A real round trip that names the store (postgresql / pglite), and the charter tables present.++{m.group(1)}const r = await db.ready();++{m.group(1)}if (!r.ok) return r.error;++{m.group(1)}const names = new Set((await db.prepare('SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()').all()).map((x) => x.name));++{m.group(1)}const missing = CHARTER_TABLES.filter((t) => !names.has(t));++{m.group(1)}return missing.length ? `missing ${{missing.join(', ')}} (migrations did not run)` : {{ ok: true, detail: r.detail }};""", s)++s = s.replace("check: () => {\n // A real round trip", "check: async () => {\n // A real round trip")++if s != o: open('server/observability.js', 'w').write(s); print('readiness')++s = open('server/app.js').read()++s2 = re.sub(r"(create\w+Readiness\(\{ store, [^}]*?release: release\.release)( \}\))", lambda m: m.group(1) + (', valkey: ctx.valkey' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)++if s2 != s: open('server/app.js', 'w').write(s2); print('app readiness')++# index.js: await createApp++s = open('server/index.js').read()++if 'await createApp()' not in s:++ s = s.replace("const { app, ctx } = createApp();", "(async () => {\nconst { app, ctx } = await createApp();", 1)++ s = s.replace(" try { ctx.store.close(); } catch { /* already closed */ }", " try { await ctx.store.close(); } catch { /* already closed */ }")++ s = s.replace(f"(db ${{config.dbPath}})", "(db ${ctx.store.db.store})")++ s = s.rstrip() + f"\n}})().catch((err) => {{ console.error('[{Name}] failed to start:', err); process.exit(1); }});\n"++ open('server/index.js', 'w').write(s); print('index')++# test helpers++os.makedirs('test/helpers', exist_ok=True)++open('test/helpers/db.js', 'w').write(f"""'use strict';++/**++ * A migrated database for one test run (ADR-035), from openvibe-sdk/testing: PGlite by default; with++ * {ENV}_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of this run's own.++ */++const {{ createTestDb, pgAvailable }} = require('openvibe-sdk/testing');++const {{ MIGRATIONS }} = require('../../server/db');++++const testDb = ({{ store = process.env.{ENV}_TEST_STORE || 'pglite', max = 4 }} = {{}}) => createTestDb({{ migrations: MIGRATIONS, store, service: '{svc}', max }});++++module.exports = {{ testDb, pgAvailable }};++""")++s = open('test/helpers/boot.js').read(); o = s++if "require('./db').testDb()" not in s:++ s = s.replace(" let server = null;\n let built = null;", " const { createStore } = require('../../server/db');\n // One database per boot (PGlite, or " + ENV + "_TEST_STORE=pg: the containers); a restart keeps it, like a file did.\n const testdb = await require('./db').testDb();\n let server = null;\n let built = null;", 1)++ s = re.sub(r"built = createApp\(\{ config, ", "built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), ", s, 1)++ s = re.sub(r"(await built\.ctx\.outbox\.stop\(\);) built\.ctx\.store\.close\(\);", r"\1", s)++ s = re.sub(r" function events\(type = null\) \{\n return t\.ctx\.store\.db\.prepare\('SELECT envelope FROM event_outbox ORDER BY id'\)\.all\(\)\.map\(\(r\) => JSON\.parse\(r\.envelope\)\)",++ " async function events(type = null) {\n return (await t.ctx.store.db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope))", s)++ s = s.replace(" async close() { await stop(); await network.close();", " async close() { await stop(); await testdb.close(); await network.close();", 1)++if s != o: open('test/helpers/boot.js', 'w').write(s); print('boot helper')++# CI, unit, env, import script++s = open('.github/workflows/ci.yml').read()++if 'test-services.sh' not in s:++ s = s.replace(f" contracts-service: {svc}\n", f""" contracts-service: {svc}++ # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk++ # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.++ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'++""", 1); open('.github/workflows/ci.yml', 'w').write(s); print('ci')++unit = f'deploy/systemd/openvibe-{svc}.service'++edit(unit, [("After=network.target\n", "After=network.target postgresql.service pgbouncer.service valkey-server.service\n"), ("Wants=openvibe-network.service\n", "Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service\n")], required=False)++edit('.env.example', [(f"{ENV}_DB_PATH=./data/{svc}.db", f"""# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh {svc}: the runtime role through PgBouncer,++# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).++# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.++DATABASE_URL=++DATABASE_DIRECT_URL=++VALKEY_URL=++VALKEY_PREFIX=++# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.++{ENV}_DB_PATH=./data/{svc}.db""")], required=False)++cfgmod = "require('../server/config')"++open('scripts/migrate-to-postgres.js', 'w').write(f"""#!/usr/bin/env node++'use strict';++/**++ * The one-time move of {Name}'s SQLite database ({ENV}_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is++ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).++ *++ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]++ *++ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts++ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its++ * name and columns.++ */++require('dotenv').config();++const {{ runSqliteMigration }} = require('openvibe-sdk/db');++const configLib = {cfgmod};++const {{ MIGRATIONS }} = require('../server/db');++++const TABLES = {{}};++++if (require.main === module) {{++ const config = configLib.load();++ runSqliteMigration({{ service: '{svc}', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES }})++ .then((code) => process.exit(code), (err) => {{ console.error(`migrate-to-postgres failed: ${{err.message}}`); process.exit(1); }});++}}++++module.exports = {{ TABLES }};++""")++print('done')nnew file mode 100644@@@ -0,0 +1,39 @@++{++ "name": "openvibe-asyncify",++ "lockfileVersion": 3,++ "requires": true,++ "packages": {++ "": {++ "name": "openvibe-asyncify",++ "license": "MIT",++ "dependencies": {++ "acorn": "^8.15.0",++ "acorn-walk": "^8.3.4"++ }++ },++ "node_modules/acorn": {++ "version": "8.18.0",++ "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.18.0.tgz",++ "integrity": "sha512-lGq+9yr1/GuAWaVYIHRjvvySG5/4VfKIvC8EWxStPdcDh/Ka7FG3twP6v4d5BkravUilhIAsG4Qj83t02LWUPQ==",++ "license": "MIT",++ "bin": {++ "acorn": "bin/acorn"++ },++ "engines": {++ "node": ">=0.4.0"++ }++ },++ "node_modules/acorn-walk": {++ "version": "8.3.5",++ "resolved": "https://registry.npmjs.org/acorn-walk/-/acorn-walk-8.3.5.tgz",++ "integrity": "sha512-HEHNfbars9v4pgpW6SO1KSPkfoS0xVOM/9UzkJltjlsHZmJasxg8aXkuZa7SMf8vKGIBhpUsPluQSqhJFCqebw==",++ "license": "MIT",++ "dependencies": {++ "acorn": "^8.11.0"++ },++ "engines": {++ "node": ">=0.4.0"++ }++ }++ }++}nnew file mode 100644@@@ -0,0 +1,28 @@++'use strict';++/**++ * A service's SQLite DDL (the SCHEMA string its db.js ran) → PostgreSQL DDL with the same meaning (ADR-035), for++ * migrations/0001_initial.sql:++ * INTEGER PRIMARY KEY AUTOINCREMENT → bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY (the importer keeps the ids)++ * TEXT → text COLLATE "C" (SQLite compares text bytewise; "C" sorts and compares the same way)++ * INTEGER → bigint (epoch milliseconds and counters; openvibe-sdk/db returns int8 as Number)++ * REAL → double precision, BLOB → bytea++ * CREATE … IF NOT EXISTS → CREATE … (a migration runs once)++ * Types only in column position (upper case, as SQLite schemas are written); comments are kept. Triggers++ * (RAISE(ABORT, …)) are refused: write them as PL/pgSQL by hand.++ *++ * const { toPg } = require('openvibe-sdk/tools/asyncify/sqlite-schema-to-pg'); (from a checkout)++ */++function toPg(sql) {++ if (/CREATE\s+TRIGGER/i.test(sql)) throw new Error('sqlite-schema-to-pg: triggers need writing as PL/pgSQL by hand');++ return sql++ .replace(/\bINTEGER PRIMARY KEY AUTOINCREMENT\b/g, 'bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY')++ .replace(/CREATE (UNIQUE )?INDEX IF NOT EXISTS/g, (m, u) => `CREATE ${u || ''}INDEX`)++ .replace(/CREATE TABLE IF NOT EXISTS/g, 'CREATE TABLE')++ .replace(/CREATE VIEW IF NOT EXISTS/g, 'CREATE VIEW')++ .replace(/^(\s*\w+\s+)TEXT\b/gm, '$1text COLLATE "C"')++ .replace(/^(\s*\w+\s+)INTEGER\b/gm, '$1bigint')++ .replace(/^(\s*\w+\s+)REAL\b/gm, '$1double precision')++ .replace(/^(\s*\w+\s+)BLOB\b/gm, '$1bytea');++}++++module.exports = { toPg };@@@ -43,6 +43,8 @@ export interface Tx extends Queryable {/** A nested savepoint: rolls back alone when fn throws. */tx<T>(fn: (t: Tx) => Promise<T>): Promise<T>;prepare(text: string): Statement;++ /** Run fn after the whole transaction commits (dropped if this savepoint or the transaction rolls back). */++ afterCommit(fn: () => unknown): void;}export interface TxOptions { isolation?: 'read committed' | 'repeatable read' | 'serializable'; retries?: number; readOnly?: boolean }export interface MigrateResult { applied: { id: string; name: string; phase: string; ms: number }[]; pending: { id: string; name: string; phase: string }[]; held: { id: string; reason: string }[] }@@@ -53,11 +55,13 @@ export interface Db extends Queryable {tx<T>(fn: (t: Tx) => Promise<T>, opts?: TxOptions): Promise<T>;prepare(text: string): Statement;inTransaction(): boolean;++ /** Run fn after the running transaction commits (dropped on rollback); outside one, on the next turn. */++ afterCommit(fn: () => unknown): void;/** Run fn outside any ambient transaction. */detached<T>(fn: () => T): T;ready(): Promise<{ ok: true; detail: { store: string; pool: { total: number; idle: number; waiting: number } } } | { ok: false; error: string }>;migrate(o: { dir: string; windowDays?: number; dryRun?: boolean; now?: () => number; log?: { log(msg: string): void } }): Promise<MigrateResult>;-- stats(): { queries: number; errors: number; slow: number; retries: number; tx: number; pool: { total: number; idle: number; waiting: number } };++ stats(): { queries: number; errors: number; slow: number; retries: number; tx: number; open: number; pool: { total: number; idle: number; waiting: number } };close(): Promise<void>;}export interface CreateDbOptions {
@@ -8,6 +8,12 @@ release may change an API and says so here.
@@ -8,6 +8,12 @@ release may change an API and says so here.
+## 0.20.0 (2026-09-28)
+
+- **`db.afterCommit(fn)`** (and `t.afterCommit(fn)` on a transaction handle) runs `fn` once the running transaction commits. It never runs after a rollback, and a savepoint that rolls back drops the hooks added inside it. Hooks run in order, outside the transaction, with the connection already back in the pool; one that throws is logged and the rest still run. Outside a transaction, `fn` runs on the next turn. It replaces the better-sqlite3 pattern of a `setImmediate` after a synchronous transaction, for calls to other services that must see committed state (News' comment-thread visibility was the first user).
+- **`db.stats().open`**: transactions in progress on this handle.
+- `tools/asyncify` gains `sqlite-schema-to-pg.js` (a service's SQLite DDL → `migrations/0001_initial.sql`) and `g1pass.py` (the non-codemod edits of a Blog-shaped service). `asyncify.js` now treats `{ name: someAsyncFunction }` as an async method `name`. None of these are in the published package.
+
## 0.19.0 (2026-09-28)
## 0.19.0 (2026-09-28)
- **`openvibe-sdk/db` `runSqliteMigration({ service, sqlite, directUrl, migrations, tables, argv })`** is a service's `scripts/migrate-to-postgres.js` in one call. It:
- **`openvibe-sdk/db` `runSqliteMigration({ service, sqlite, directUrl, migrations, tables, argv })`** is a service's `scripts/migrate-to-postgres.js` in one call. It:
@@ -3,7 +3,7 @@
@@ -3,7 +3,7 @@
"stage": "alpha",
"stage": "alpha",
"code": true,
"code": true,
"wave": "W2",
"wave": "W2",
- "version": "0.19.0",
+ "version": "0.20.0",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30",
"deployed": false,
"deployed": false,
"deployment": "none: a package released as git tags; services and apps pin a tag",
"deployment": "none: a package released as git tags; services and apps pin a tag",
@@ -141,3 +141,10 @@ It changes nothing in the SQLite file.
@@ -141,3 +141,10 @@ It changes nothing in the SQLite file.
- **Claim an Idempotency-Key before running the handler**, with `INSERT … ON CONFLICT DO NOTHING RETURNING`. That way two concurrent requests with the same key run once.
- **Claim an Idempotency-Key before running the handler**, with `INSERT … ON CONFLICT DO NOTHING RETURNING`. That way two concurrent requests with the same key run once.
- **Clean text PostgreSQL refuses.** SQLite text can hold NUL and unpaired surrogates. Clean them in the importer's `map`, and report each column that needed it.
- **Clean text PostgreSQL refuses.** SQLite text can hold NUL and unpaired surrogates. Clean them in the importer's `map`, and report each column that needed it.
- **pub/sub without Valkey.** `createPubSub` has no local fallback when Valkey is down. A hub that must keep delivering on one process falls back to local delivery itself.
- **pub/sub without Valkey.** `createPubSub` has no local fallback when Valkey is down. A hub that must keep delivering on one process falls back to local delivery itself.
+- **News (2026-09-28)** added these:
+ - **`rowid` tiebreaks.** `ORDER BY created_at, rowid` meant insertion order. A random ULID `id` is not, within one millisecond. Add `seq bigint GENERATED ALWAYS AS IDENTITY UNIQUE` to the table and order by `seq`. The importer fills it in the source's row order.
+ - **Scalar `MAX(a, b)` / `MIN(a, b)`** are `GREATEST` / `LEAST`.
+ - **`ON CONFLICT … DO UPDATE SET x = COALESCE(excluded.x, x)`** is ambiguous in PostgreSQL. Qualify the target: `COALESCE(excluded.x, <table>.x)`.
+ - **Work after the commit.** A `setImmediate` after a synchronous better-sqlite3 transaction ran after the commit. Around an async transaction it does not. Use `db.afterCommit(fn)` (SDK ≥ 0.20).
+ - **The inbox.** `createInbox` (better-sqlite3) becomes `createPgInbox`, awaited. Its `idempotency_receipts` table goes in `migrations/0001_initial.sql` (`inboxSchema()`), not an unawaited `ensureSchema()`.
+ - **A script run as a child process in tests** cannot reach the test's PGlite. Call the function it wraps in-process, and `node --check` the script.
@@ -1,12 +1,12 @@
@@ -1,12 +1,12 @@
{
{
"name": "openvibe-sdk",
"name": "openvibe-sdk",
- "version": "0.19.0",
+ "version": "0.20.0",
"lockfileVersion": 3,
"lockfileVersion": 3,
"requires": true,
"requires": true,
"packages": {
"packages": {
"": {
"": {
"name": "openvibe-sdk",
"name": "openvibe-sdk",
- "version": "0.19.0",
+ "version": "0.20.0",
"license": "MIT",
"license": "MIT",
"devDependencies": {
"devDependencies": {
"@electric-sql/pglite": "^0.5.8",
"@electric-sql/pglite": "^0.5.8",
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
{
{
"name": "openvibe-sdk",
"name": "openvibe-sdk",
- "version": "0.19.0",
+ "version": "0.20.0",
"description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.",
"description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.",
"main": "index.js",
"main": "index.js",
"types": "types/index.d.ts",
"types": "types/index.d.ts",
@@ -97,7 +97,7 @@ function createDb(o = {}) {
@@ -97,7 +97,7 @@ function createDb(o = {}) {
const log = o.log || console;
const log = o.log || console;
const service = o.service || process.env.OV_SERVICE || 'service';
const service = o.service || process.env.OV_SERVICE || 'service';
const slowMs = o.slowMs == null ? (o.pglite ? Infinity : 500) : o.slowMs; // PGlite's first query includes its WASM start
const slowMs = o.slowMs == null ? (o.pglite ? Infinity : 500) : o.slowMs; // PGlite's first query includes its WASM start
- const stats = { queries: 0, errors: 0, slow: 0, retries: 0, tx: 0 };
+ const stats = { queries: 0, errors: 0, slow: 0, retries: 0, tx: 0, open: 0 };
let hist = null;
let hist = null;
if (o.registry && typeof o.registry.histogram === 'function') {
if (o.registry && typeof o.registry.histogram === 'function') {
try { hist = o.registry.histogram({ name: 'db_query_seconds', help: 'Database query time by kind', labelNames: ['kind'], buckets: [0.001, 0.005, 0.02, 0.1, 0.5, 2, 10] }); } catch { hist = null; }
try { hist = o.registry.histogram({ name: 'db_query_seconds', help: 'Database query time by kind', labelNames: ['kind'], buckets: [0.001, 0.005, 0.02, 0.1, 0.5, 2, 10] }); } catch { hist = null; }
@@ -131,20 +131,21 @@ function createDb(o = {}) {
@@ -131,20 +131,21 @@ function createDb(o = {}) {
};
};
const inScope = (scope, fn) => (als ? als.run(scope, fn) : fn());
const inScope = (scope, fn) => (als ? als.run(scope, fn) : fn());
- async function tx(fn, opts = {}, conn = null, depth = 0) {
+ async function tx(fn, opts = {}, conn = null, depth = 0, hooks = null) {
if (!conn) {
if (!conn) {
const cur = current();
const cur = current();
- if (cur) { conn = cur.conn; depth = cur.depth; } // ambient: a savepoint in the running transaction
+ if (cur) { conn = cur.conn; depth = cur.depth; hooks = cur.hooks; } // ambient: a savepoint in the running transaction
}
}
if (conn) {
if (conn) {
- // Nested: a savepoint inside the running transaction.
+ // Nested: a savepoint inside the running transaction. Its after-commit hooks go with it if it rolls back.
const name = `sp_${depth}`;
const name = `sp_${depth}`;
+ const mark = hooks.length;
await timed(conn, `SAVEPOINT ${name}`, []);
await timed(conn, `SAVEPOINT ${name}`, []);
try {
try {
- const out = await inScope({ conn, depth: depth + 1, done: false }, () => fn(txApi(conn, depth + 1)));
+ const out = await inScope({ conn, depth: depth + 1, done: false, hooks }, () => fn(txApi(conn, depth + 1, hooks)));
await timed(conn, `RELEASE SAVEPOINT ${name}`, []);
await timed(conn, `RELEASE SAVEPOINT ${name}`, []);
return out;
return out;
- } catch (err) { await timed(conn, `ROLLBACK TO SAVEPOINT ${name}`, []).catch(() => {}); throw err; }
+ } catch (err) { hooks.length = mark; await timed(conn, `ROLLBACK TO SAVEPOINT ${name}`, []).catch(() => {}); throw err; }
}
}
const iso = ISOLATION[opts.isolation || 'read committed'];
const iso = ISOLATION[opts.isolation || 'read committed'];
if (!iso) throw new TypeError(`openvibe-sdk/db: isolation must be one of ${Object.keys(ISOLATION).join(', ')}`);
if (!iso) throw new TypeError(`openvibe-sdk/db: isolation must be one of ${Object.keys(ISOLATION).join(', ')}`);
@@ -152,15 +153,23 @@ function createDb(o = {}) {
@@ -152,15 +153,23 @@ function createDb(o = {}) {
for (let attempt = 0; ; attempt++) {
for (let attempt = 0; ; attempt++) {
const c = await adapter.acquire();
const c = await adapter.acquire();
stats.tx++;
stats.tx++;
- const scope = { conn: c, depth: 1, done: false };
+ stats.open++;
+ const scope = { conn: c, depth: 1, done: false, hooks: [] };
+ let committed = false;
try {
try {
await timed(c, `BEGIN ISOLATION LEVEL ${iso}${opts.readOnly ? ' READ ONLY' : ''}`, []);
await timed(c, `BEGIN ISOLATION LEVEL ${iso}${opts.readOnly ? ' READ ONLY' : ''}`, []);
- const out = await inScope(scope, () => fn(txApi(c, 1)));
+ const out = await inScope(scope, () => fn(txApi(c, 1, scope.hooks)));
scope.done = true;
scope.done = true;
await timed(c, 'COMMIT', []);
await timed(c, 'COMMIT', []);
+ committed = true;
+ stats.open--;
+ adapter.release(c);
+ await runHooks(scope.hooks);
return out;
return out;
} catch (err) {
} catch (err) {
+ if (committed) throw err;
scope.done = true;
scope.done = true;
+ stats.open--;
await timed(c, 'ROLLBACK', []).catch(() => {});
await timed(c, 'ROLLBACK', []).catch(() => {});
const code = err && (err.code || (err.cause && err.cause.code));
const code = err && (err.code || (err.cause && err.cause.code));
if (RETRYABLE.has(code) && attempt < retries) {
if (RETRYABLE.has(code) && attempt < retries) {
@@ -169,13 +178,26 @@ function createDb(o = {}) {
@@ -169,13 +178,26 @@ function createDb(o = {}) {
continue;
continue;
}
}
throw err;
throw err;
- } finally { adapter.release(c); }
+ } finally { if (!committed) adapter.release(c); }
}
}
}
}
- function txApi(conn, depth) {
+ /** After-commit hooks run in order, outside the transaction; one that throws is logged and the rest still run. */
+ async function runHooks(hooks) {
+ for (const h of hooks) {
+ try { await h(); } catch (err) { log.warn(`[db] after-commit hook failed: ${err && err.message ? err.message : err}`); }
+ }
+ }
+ function afterCommit(hooks, fn) {
+ if (typeof fn !== 'function') throw new TypeError('openvibe-sdk/db: afterCommit(fn) needs a function');
+ if (hooks) { hooks.push(fn); return; }
+ setImmediate(() => runHooks([fn])); // no transaction: nothing to wait for but the caller's own turn
+ }
+
+ function txApi(conn, depth, hooks) {
const t = queryApi((text, values) => timed(conn, text, values), {});
const t = queryApi((text, values) => timed(conn, text, values), {});
- t.tx = (fn) => tx(fn, {}, conn, depth);
+ t.tx = (fn) => tx(fn, {}, conn, depth, hooks);
+ t.afterCommit = (fn) => afterCommit(hooks, fn);
t.sql = sql;
t.sql = sql;
t.prepare = (text) => prepare(t, text);
t.prepare = (text) => prepare(t, text);
return t;
return t;
@@ -188,6 +210,14 @@ function createDb(o = {}) {
@@ -188,6 +210,14 @@ function createDb(o = {}) {
db.prepare = (text) => prepare(db, text);
db.prepare = (text) => prepare(db, text);
/** True while this code runs inside db.tx (ambient mode). */
/** True while this code runs inside db.tx (ambient mode). */
db.inTransaction = () => Boolean(current());
db.inTransaction = () => Boolean(current());
+ /**
+ * Run fn once the running transaction commits (never if it rolls back; a savepoint that rolls back drops the hooks
+ * added inside it). Outside a transaction, fn runs on the next turn. Hooks are awaited in order before db.tx
+ * resolves, with the connection already back in the pool: a hook that must not delay the caller starts its work
+ * and returns. For calls to other services that must see committed state (the setImmediate-after-a-synchronous-
+ * better-sqlite3-transaction pattern).
+ */
+ db.afterCommit = (fn) => { const cur = current(); afterCommit(cur ? cur.hooks : null, fn); };
/** Run fn outside any ambient transaction (its queries take a pool connection). */
/** Run fn outside any ambient transaction (its queries take a pool connection). */
db.detached = (fn) => (als ? als.exit(fn) : fn());
db.detached = (fn) => (als ? als.exit(fn) : fn());
db.store = adapter.store;
db.store = adapter.store;
@@ -31,6 +31,32 @@ function cases(label, open) {
@@ -31,6 +31,32 @@ function cases(label, open) {
assert.equal(await count.get(), 1);
assert.equal(await count.get(), 1);
} finally { await done(); }
} finally { await done(); }
}],
}],
+ [`${label}: afterCommit runs after the commit, in order, never after a rollback; a failed savepoint drops its hooks`, async () => {
+ const { db, done } = await open();
+ try {
+ const ins = db.prepare('INSERT INTO items (name, qty) VALUES (?, ?)');
+ const seen = [];
+ const committed = async (tag) => { seen.push([tag, db.inTransaction(), db.stats().open, await db.prepare('SELECT count(*)::int AS n FROM items').pluck().get()]); };
+ await db.tx(async (t) => {
+ await ins.run('a', 1);
+ db.afterCommit(() => committed('ambient'));
+ t.afterCommit(() => committed('handle'));
+ await assert.rejects(db.tx(async () => { db.afterCommit(() => committed('dropped')); throw new Error('inner'); }), /inner/);
+ await db.tx(async () => { db.afterCommit(() => committed('kept')); });
+ db.afterCommit(() => { throw new Error('a failing hook is logged'); });
+ assert.deepEqual(seen, [], 'nothing runs inside the transaction');
+ assert.equal(db.stats().open, 1);
+ });
+ assert.deepEqual(seen, [['ambient', false, 0, 1], ['handle', false, 0, 1], ['kept', false, 0, 1]]);
+ seen.length = 0;
+ await assert.rejects(db.tx(async () => { await ins.run('b', 2); db.afterCommit(() => committed('rolled back')); throw new Error('boom'); }), /boom/);
+ db.afterCommit(() => committed('outside'));
+ assert.deepEqual(seen, [], 'outside a transaction: the next turn');
+ await new Promise((r) => setImmediate(r));
+ await new Promise((r) => setTimeout(r, 20));
+ assert.deepEqual(seen, [['outside', false, 0, 1]]);
+ } finally { await done(); }
+ }],
[`${label}: a db.tx inside is a savepoint; its failure undoes only itself`, async () => {
[`${label}: a db.tx inside is a savepoint; its failure undoes only itself`, async () => {
const { db, done } = await open();
const { db, done } = await open();
try {
try {
@@ -35,6 +35,16 @@ What stays for a person:
@@ -35,6 +35,16 @@ What stays for a person:
- `RETURNING` in place of `lastInsertRowid`;
- `RETURNING` in place of `lastInsertRowid`;
- the boot (open, migrate, `createApp` async) and the test helpers (`openvibe-sdk/testing` `createTestDb`).
- the boot (open, migrate, `createApp` async) and the test helpers (`openvibe-sdk/testing` `createTestDb`).
+Also by hand: `db.transaction(fn)()` → `db.tx(fn)` (the codemod leaves it), scalar `MAX(a, b)` → `GREATEST`, `rowid` tiebreaks → an identity `seq` column, and a `setImmediate` meant to run after a commit → `db.afterCommit`. The lessons list in `docs/migrating-to-postgresql.md` §7 has the rest.
+
+## sqlite-schema-to-pg.js
+
+`toPg(sqliteDdl)` turns the SCHEMA string a service's `db.js` ran into the first migration: `INTEGER PRIMARY KEY AUTOINCREMENT` → identity, `TEXT` → `text COLLATE "C"`, `INTEGER` → `bigint`, `REAL` → `double precision`, `BLOB` → `bytea`, and no `IF NOT EXISTS`. It refuses triggers.
+
+## g1pass.py
+
+`g1pass.py <svc> <Name> <ENVPREFIX>`, run in a Blog-shaped service's checkout, makes the edits that are the same in every such service: config (`DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`), the PostgreSQL outbox, Valkey actor limits, readiness, and the test boot on `createTestDb`. Each edit checks its anchor and is skipped when already applied.
+
## awaitify-tests.py
## awaitify-tests.py
This is the earlier regex pass for test files (Wiki). `asyncify.js` over the server and the tests together supersedes it.
This is the earlier regex pass for test files (Wiki). `asyncify.js` over the server and the tests together supersedes it.
@@ -191,6 +191,10 @@ function main() {
@@ -191,6 +191,10 @@ function main() {
}
}
// Every async function of the file (new or already) counts for callers.
// Every async function of the file (new or already) counts for callers.
const all = new Set([...(asyncNames.local.get(f) || []), ...r.newlyAsyncNames]);
const all = new Set([...(asyncNames.local.get(f) || []), ...r.newlyAsyncNames]);
+ // { name: someAsyncFunction } exports the function under another name: that name is async too.
+ walk.full(parse(fs.readFileSync(f, 'utf8')), (n) => {
+ if (n.type === 'Property' && n.value && n.value.type === 'Identifier' && all.has(n.value.name) && n.key) all.add(n.key.name || n.key.value);
+ });
walk.fullAncestor(parse(fs.readFileSync(f, 'utf8')), (node, _s, anc) => {
walk.fullAncestor(parse(fs.readFileSync(f, 'utf8')), (node, _s, anc) => {
if ((node.type === 'FunctionDeclaration' || node.type === 'FunctionExpression' || node.type === 'ArrowFunctionExpression') && node.async) {
if ((node.type === 'FunctionDeclaration' || node.type === 'FunctionExpression' || node.type === 'ArrowFunctionExpression') && node.async) {
const parent = anc[anc.length - 2];
const parent = anc[anc.length - 2];
new file mode 100644
new file mode 100644
@@ -0,0 +1,149 @@
@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+"""The shared, non-codemod edits of a content product's PostgreSQL pass (Blog-shaped services: server/app.js createApp,
+server/events/outbox.js, server/http/actor-limits.js, server/observability.js, test/helpers/boot.js).
+usage: g1pass.py <svc> <Name> <ENVPREFIX> (e.g. news News NEWS), run in the service's checkout. Idempotent-ish: each
+edit asserts its anchor and skips when already applied."""
+import json, re, sys, os
+svc, Name, ENV = sys.argv[1], sys.argv[2], sys.argv[3]
+def edit(p, pairs, required=True):
+ if not os.path.exists(p): print('skip (missing)', p); return
+ s = open(p).read(); o = s
+ for a, b in pairs:
+ if b in s and a not in s: continue
+ if a not in s:
+ if required: raise SystemExit(f'{p}: anchor not found: {a[:90]}')
+ continue
+ s = s.replace(a, b, 1)
+ if s != o: open(p, 'w').write(s); print('edited', p)
+# config
+edit('server/config.js', [(f" dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',", f""" // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
+ db: {{ url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' }},
+ valkey: {{ url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:{svc}:' }},
+ // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
+ dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',""")])
+# outbox
+s = open('server/events/outbox.js').read()
+edit('server/events/outbox.js', [
+ ("const { createEventsClient, createOutbox } = require('openvibe-sdk/events');", "const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');"),
+ (" const outbox = createOutbox(db, {", " // The PostgreSQL outbox: rows are written in the change's own transaction (enqueue(db, …) joins the ambient\n // transaction); several processes relay one table safely (leases).\n const outbox = createPgOutbox(db, {"),
+ (" outbox.ensureSchema();\n", ""),
+])
+s = open('server/events/outbox.js').read()
+s2 = re.sub(r'outbox\.enqueue\((?!db, )', 'outbox.enqueue(db, ', s)
+if s2 != s: open('server/events/outbox.js', 'w').write(s2); print('enqueue(db, …)')
+# app.js store
+s = open('server/app.js').read()
+m = re.search(r" const store = opts\.store \|\| openStore\(opts\.dbPath \|\| config\.dbPath, \{ now: opts\.now(, [^}]*)? \}\);", s)
+if m:
+ s = s.replace(m.group(0), " // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test, a script) hands in a store.\n const store = opts.store || await openStore(config, { now: opts.now, log });")
+ s = s.replace('function createApp(opts = {}) {', 'async function createApp(opts = {}) {', 1)
+ open('server/app.js', 'w').write(s); print('app store')
+# limits + valkey
+edit('server/http/actor-limits.js', [
+ ("const { createActorLimiter, defaultActor } = require('openvibe-sdk/limits');", "const { createActorLimiter, createValkeyLimitStore, defaultActor } = require('openvibe-sdk/limits');"),
+ ("function createActorLimits({ config, now = () => Date.now(), registry = null, log = console }) {", "function createActorLimits({ config, now = () => Date.now(), registry = null, log = console, valkey = null }) {"),
+ (" now,\n", " now,\n // Shared across processes on Valkey (ADR-035) when VALKEY_URL is set; in-process otherwise.\n ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),\n"),
+])
+s = open('server/app.js').read()
+m = re.search(r" ctx\.limits = createActorLimits\(\{ config, now: opts\.limitsNow \|\| \(\(\) => Date\.now\(\)\), registry: metrics\.registry, log \}\);", s)
+if m:
+ s = s.replace(m.group(0), " // Valkey (ADR-035): shared, never-authoritative state (per-actor limit counters). Optional.\n const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);\n ctx.valkey = valkey;\n ctx.limits = createActorLimits({ config, now: opts.limitsNow || (() => Date.now()), registry: metrics.registry, log, valkey });")
+ open('server/app.js', 'w').write(s); print('app valkey')
+# readiness
+s = open('server/observability.js').read(); o = s
+s = re.sub(r"(function create\w+Readiness\(\{ store, [^}]*?)(, release = null \}\))", lambda m: m.group(1) + (', valkey = null' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)
+s = s.replace(""" {
+ name: 'network_jwks', required: false,""", """ { name: 'valkey', required: false, check: async () => (valkey ? valkey.ready() : { skipped: 'VALKEY_URL not set: per-actor limits count in this process only' }) },
+ {
+ name: 'network_jwks', required: false,""", 1) if "name: 'valkey'" not in s else s
+s = re.sub(r"""const names = new Set\((?:\(await )?db\.prepare\("SELECT name FROM sqlite_master WHERE type IN \('table','view'\)"\)\.all\(\)\)?\.map\(\(r\) => r\.name\)\);\n(\s*)const missing = CHARTER_TABLES\.filter\(\(t\) => !names\.has\(t\)\);\n\s*return missing\.length \? `missing \$\{missing\.join\(', '\)\}` : true;""",
+ lambda m: f"""// A real round trip that names the store (postgresql / pglite), and the charter tables present.
+{m.group(1)}const r = await db.ready();
+{m.group(1)}if (!r.ok) return r.error;
+{m.group(1)}const names = new Set((await db.prepare('SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()').all()).map((x) => x.name));
+{m.group(1)}const missing = CHARTER_TABLES.filter((t) => !names.has(t));
+{m.group(1)}return missing.length ? `missing ${{missing.join(', ')}} (migrations did not run)` : {{ ok: true, detail: r.detail }};""", s)
+s = s.replace("check: () => {\n // A real round trip", "check: async () => {\n // A real round trip")
+if s != o: open('server/observability.js', 'w').write(s); print('readiness')
+s = open('server/app.js').read()
+s2 = re.sub(r"(create\w+Readiness\(\{ store, [^}]*?release: release\.release)( \}\))", lambda m: m.group(1) + (', valkey: ctx.valkey' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)
+if s2 != s: open('server/app.js', 'w').write(s2); print('app readiness')
+# index.js: await createApp
+s = open('server/index.js').read()
+if 'await createApp()' not in s:
+ s = s.replace("const { app, ctx } = createApp();", "(async () => {\nconst { app, ctx } = await createApp();", 1)
+ s = s.replace(" try { ctx.store.close(); } catch { /* already closed */ }", " try { await ctx.store.close(); } catch { /* already closed */ }")
+ s = s.replace(f"(db ${{config.dbPath}})", "(db ${ctx.store.db.store})")
+ s = s.rstrip() + f"\n}})().catch((err) => {{ console.error('[{Name}] failed to start:', err); process.exit(1); }});\n"
+ open('server/index.js', 'w').write(s); print('index')
+# test helpers
+os.makedirs('test/helpers', exist_ok=True)
+open('test/helpers/db.js', 'w').write(f"""'use strict';
+/**
+ * A migrated database for one test run (ADR-035), from openvibe-sdk/testing: PGlite by default; with
+ * {ENV}_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of this run's own.
+ */
+const {{ createTestDb, pgAvailable }} = require('openvibe-sdk/testing');
+const {{ MIGRATIONS }} = require('../../server/db');
+
+const testDb = ({{ store = process.env.{ENV}_TEST_STORE || 'pglite', max = 4 }} = {{}}) => createTestDb({{ migrations: MIGRATIONS, store, service: '{svc}', max }});
+
+module.exports = {{ testDb, pgAvailable }};
+""")
+s = open('test/helpers/boot.js').read(); o = s
+if "require('./db').testDb()" not in s:
+ s = s.replace(" let server = null;\n let built = null;", " const { createStore } = require('../../server/db');\n // One database per boot (PGlite, or " + ENV + "_TEST_STORE=pg: the containers); a restart keeps it, like a file did.\n const testdb = await require('./db').testDb();\n let server = null;\n let built = null;", 1)
+ s = re.sub(r"built = createApp\(\{ config, ", "built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), ", s, 1)
+ s = re.sub(r"(await built\.ctx\.outbox\.stop\(\);) built\.ctx\.store\.close\(\);", r"\1", s)
+ s = re.sub(r" function events\(type = null\) \{\n return t\.ctx\.store\.db\.prepare\('SELECT envelope FROM event_outbox ORDER BY id'\)\.all\(\)\.map\(\(r\) => JSON\.parse\(r\.envelope\)\)",
+ " async function events(type = null) {\n return (await t.ctx.store.db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope))", s)
+ s = s.replace(" async close() { await stop(); await network.close();", " async close() { await stop(); await testdb.close(); await network.close();", 1)
+if s != o: open('test/helpers/boot.js', 'w').write(s); print('boot helper')
+# CI, unit, env, import script
+s = open('.github/workflows/ci.yml').read()
+if 'test-services.sh' not in s:
+ s = s.replace(f" contracts-service: {svc}\n", f""" contracts-service: {svc}
+ # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk
+ # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.
+ test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'
+""", 1); open('.github/workflows/ci.yml', 'w').write(s); print('ci')
+unit = f'deploy/systemd/openvibe-{svc}.service'
+edit(unit, [("After=network.target\n", "After=network.target postgresql.service pgbouncer.service valkey-server.service\n"), ("Wants=openvibe-network.service\n", "Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service\n")], required=False)
+edit('.env.example', [(f"{ENV}_DB_PATH=./data/{svc}.db", f"""# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh {svc}: the runtime role through PgBouncer,
+# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).
+# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
+DATABASE_URL=
+DATABASE_DIRECT_URL=
+VALKEY_URL=
+VALKEY_PREFIX=
+# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.
+{ENV}_DB_PATH=./data/{svc}.db""")], required=False)
+cfgmod = "require('../server/config')"
+open('scripts/migrate-to-postgres.js', 'w').write(f"""#!/usr/bin/env node
+'use strict';
+/**
+ * The one-time move of {Name}'s SQLite database ({ENV}_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
+ * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
+ *
+ * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
+ *
+ * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
+ * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
+ * name and columns.
+ */
+require('dotenv').config();
+const {{ runSqliteMigration }} = require('openvibe-sdk/db');
+const configLib = {cfgmod};
+const {{ MIGRATIONS }} = require('../server/db');
+
+const TABLES = {{}};
+
+if (require.main === module) {{
+ const config = configLib.load();
+ runSqliteMigration({{ service: '{svc}', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES }})
+ .then((code) => process.exit(code), (err) => {{ console.error(`migrate-to-postgres failed: ${{err.message}}`); process.exit(1); }});
+}}
+
+module.exports = {{ TABLES }};
+""")
+print('done')
new file mode 100644
new file mode 100644
@@ -0,0 +1,39 @@
@@ -0,0 +1,39 @@
+{
+ "name": "openvibe-asyncify",
+ "lockfileVersion": 3,
+ "requires": true,
+ "packages": {
+ "": {
+ "name": "openvibe-asyncify",
+ "license": "MIT",
+ "dependencies": {
+ "acorn": "^8.15.0",
+ "acorn-walk": "^8.3.4"
+ }
+ },
+ "node_modules/acorn": {
+ "version": "8.18.0",
+ "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.18.0.tgz",
+ "integrity": "sha512-lGq+9yr1/GuAWaVYIHRjvvySG5/4VfKIvC8EWxStPdcDh/Ka7FG3twP6v4d5BkravUilhIAsG4Qj83t02LWUPQ==",
+ "license": "MIT",
+ "bin": {
+ "acorn": "bin/acorn"
+ },
+ "engines": {
+ "node": ">=0.4.0"
+ }
+ },
+ "node_modules/acorn-walk": {
+ "version": "8.3.5",
+ "resolved": "https://registry.npmjs.org/acorn-walk/-/acorn-walk-8.3.5.tgz",
+ "integrity": "sha512-HEHNfbars9v4pgpW6SO1KSPkfoS0xVOM/9UzkJltjlsHZmJasxg8aXkuZa7SMf8vKGIBhpUsPluQSqhJFCqebw==",
+ "license": "MIT",
+ "dependencies": {
+ "acorn": "^8.11.0"
+ },
+ "engines": {
+ "node": ">=0.4.0"
+ }
+ }
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,28 @@
@@ -0,0 +1,28 @@
+'use strict';
+/**
+ * A service's SQLite DDL (the SCHEMA string its db.js ran) → PostgreSQL DDL with the same meaning (ADR-035), for
+ * migrations/0001_initial.sql:
+ * INTEGER PRIMARY KEY AUTOINCREMENT → bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY (the importer keeps the ids)
+ * TEXT → text COLLATE "C" (SQLite compares text bytewise; "C" sorts and compares the same way)
+ * INTEGER → bigint (epoch milliseconds and counters; openvibe-sdk/db returns int8 as Number)
+ * REAL → double precision, BLOB → bytea
+ * CREATE … IF NOT EXISTS → CREATE … (a migration runs once)
+ * Types only in column position (upper case, as SQLite schemas are written); comments are kept. Triggers
+ * (RAISE(ABORT, …)) are refused: write them as PL/pgSQL by hand.
+ *
+ * const { toPg } = require('openvibe-sdk/tools/asyncify/sqlite-schema-to-pg'); (from a checkout)
+ */
+function toPg(sql) {
+ if (/CREATE\s+TRIGGER/i.test(sql)) throw new Error('sqlite-schema-to-pg: triggers need writing as PL/pgSQL by hand');
+ return sql
+ .replace(/\bINTEGER PRIMARY KEY AUTOINCREMENT\b/g, 'bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY')
+ .replace(/CREATE (UNIQUE )?INDEX IF NOT EXISTS/g, (m, u) => `CREATE ${u || ''}INDEX`)
+ .replace(/CREATE TABLE IF NOT EXISTS/g, 'CREATE TABLE')
+ .replace(/CREATE VIEW IF NOT EXISTS/g, 'CREATE VIEW')
+ .replace(/^(\s*\w+\s+)TEXT\b/gm, '$1text COLLATE "C"')
+ .replace(/^(\s*\w+\s+)INTEGER\b/gm, '$1bigint')
+ .replace(/^(\s*\w+\s+)REAL\b/gm, '$1double precision')
+ .replace(/^(\s*\w+\s+)BLOB\b/gm, '$1bytea');
+}
+
+module.exports = { toPg };
@@ -43,6 +43,8 @@ export interface Tx extends Queryable {
@@ -43,6 +43,8 @@ export interface Tx extends Queryable {
/** A nested savepoint: rolls back alone when fn throws. */
/** A nested savepoint: rolls back alone when fn throws. */
tx<T>(fn: (t: Tx) => Promise<T>): Promise<T>;
tx<T>(fn: (t: Tx) => Promise<T>): Promise<T>;
prepare(text: string): Statement;
prepare(text: string): Statement;
+ /** Run fn after the whole transaction commits (dropped if this savepoint or the transaction rolls back). */
+ afterCommit(fn: () => unknown): void;
}
}
export interface TxOptions { isolation?: 'read committed' | 'repeatable read' | 'serializable'; retries?: number; readOnly?: boolean }
export interface TxOptions { isolation?: 'read committed' | 'repeatable read' | 'serializable'; retries?: number; readOnly?: boolean }
export interface MigrateResult { applied: { id: string; name: string; phase: string; ms: number }[]; pending: { id: string; name: string; phase: string }[]; held: { id: string; reason: string }[] }
export interface MigrateResult { applied: { id: string; name: string; phase: string; ms: number }[]; pending: { id: string; name: string; phase: string }[]; held: { id: string; reason: string }[] }
@@ -53,11 +55,13 @@ export interface Db extends Queryable {
@@ -53,11 +55,13 @@ export interface Db extends Queryable {
tx<T>(fn: (t: Tx) => Promise<T>, opts?: TxOptions): Promise<T>;
tx<T>(fn: (t: Tx) => Promise<T>, opts?: TxOptions): Promise<T>;
prepare(text: string): Statement;
prepare(text: string): Statement;
inTransaction(): boolean;
inTransaction(): boolean;
+ /** Run fn after the running transaction commits (dropped on rollback); outside one, on the next turn. */
+ afterCommit(fn: () => unknown): void;
/** Run fn outside any ambient transaction. */
/** Run fn outside any ambient transaction. */
detached<T>(fn: () => T): T;
detached<T>(fn: () => T): T;
ready(): Promise<{ ok: true; detail: { store: string; pool: { total: number; idle: number; waiting: number } } } | { ok: false; error: string }>;
ready(): Promise<{ ok: true; detail: { store: string; pool: { total: number; idle: number; waiting: number } } } | { ok: false; error: string }>;
migrate(o: { dir: string; windowDays?: number; dryRun?: boolean; now?: () => number; log?: { log(msg: string): void } }): Promise<MigrateResult>;
migrate(o: { dir: string; windowDays?: number; dryRun?: boolean; now?: () => number; log?: { log(msg: string): void } }): Promise<MigrateResult>;
- stats(): { queries: number; errors: number; slow: number; retries: number; tx: number; pool: { total: number; idle: number; waiting: number } };
+ stats(): { queries: number; errors: number; slow: number; retries: number; tx: number; open: number; pool: { total: number; idle: number; waiting: number } };
close(): Promise<void>;
close(): Promise<void>;
}
}
export interface CreateDbOptions {
export interface CreateDbOptions {