openvibers/openvibe.sdk Commits

Load test databases from a migrated PGlite snapshot in createTestDb (#12)

OP OpenVibers committed 5 Oct 2026 · 279bd46b95 · 13 files, +382 −29

* Load test databases from a migrated PGlite snapshot in createTestDb

Asked: Part of a cross-repository initiative (init-every-service-repository-s-f-1005T0017): Every service repository's full verify (test and test-pg) finishes in 10 minutes or less on this machine, starting with OpenVibe.Network (34 min now against a 20-min check limit). When done: ds-observe.js measures …
Harness job init-every-service-re-0017-s3-002045.

* Refresh migration ages on a snapshot hit and fix the review's findings

All findings verified and fixed. Summary:

**Finding 1 — contract migrations become eligible ~7 days after a snapshot is built: fixed.** `src/testing/db.js:160` now runs `UPDATE ov_migrations SET applied_at = now()` on a hit, before the behind-check, so a loaded snapshot looks freshly migrated and an ADR-028 contract stays held as it would on a fresh migrate. Regression test at `test/testing-db.test.js:163` (ages the expand in the snapshot via the seed, then asserts on a hit that the contract is not applied). Verified it fails without the fix (`contracted` was created).

**Finding 2 — `seedKey` alone collides with `seedKey` + `seed`: fixed.** `src/testing/db.js:95` now gates on `seed` first (`seed ? (seedKey ? key:… : fn:…) : 'none'`). Test at `test/testing-db.test.js:151` asserts the keys differ and that a seeded call is not served the unseeded snapshot. Verified it fails without the fix.

**Finding 3 — the lock's cross-process path is untested: fixed (test only).** The `takeLock` live-other-pid/wait path was correct; it had no coverage. `test/testing-db.test.js:84` forks a child that builds the same key (holding the lock through a slow seed) while this process waits and loads its file, asserting `hit`, that it waited rather than double-built, and that no lock/tmp file is left. No source change was needed.

**Finding 4 — `seed` has owner rights on PGlite but DML-only rights on pg: fixed.** `src/testing/db.js:251-257` now runs the pg seed on the owner connection (`OV_TEST_PG_DIRECT_URL`), matching PGlite, so owner-only ops (DDL, `TRUNCATE`, `ALTER SEQUENCE … RESTART`) behave the same on both stores. Documentation updated (`types/testing.d.ts:172`, module comment, CHANGELOG). Test added under `pgAvailable()` using `ALTER SEQUENCE … RESTART`; it is skipped in this checkout (no `OV_TEST_PG_URL`).

**Minor notes — not addressed (advisory, non-blocking).** The `snapshotDir` trust/umask and `pruneSnapshots`-on-a-user-supplied-dir remarks are hardening suggestions, not correctness bugs: `OV_TEST_SNAPSHOT_DIR` is by contract a dedicated snapshot directory, and the recommended `mode: 0o700` only applies at creation (it cannot re-secure a pre-existing directory), so I left the snapshot code unchanged rather than add unrelated behavior.

**Test results**
- All 13 cases in `test/testing-db.test.js` pass, including the three new tests (the pg-only case is skipped without containers, as the file already does).
- `ov test`: `21 passed, 12 skipped (unchanged since green) in 264 s`; `test/testing-db.test.js` ran (79.1 s) and passed. The pg-conditional seed test was not executed here because `OV_TEST_PG_URL`/`OV_TEST_PG_DIRECT_URL` are unset.

* Stop tracking the node_modules link and ignore it as a link too

* Pin openvibe-contracts v0.97.0 (the pin-drift check allows at most 10 minors behind)

* Re-copy the identity.service-token-claims types from openvibe-contracts v0.97.0

Changes

ddiff --git a/.gitignore b/.gitignore
iindex 2d98f97..631a0e7 100644
---- a/.gitignore
++++ b/.gitignore
@@@ -1,2 +1,3 @@
node_modules/
*.tgz
++node_modules
ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex 73147f8..54d406e 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -7,6 +7,22 @@ release may change an API and says so here.
++## 0.29.0 (2026-10-04)
++
++- **`createTestDb` reuses a migrated snapshot (`openvibe-sdk/testing`, minor):** on PGlite, the first test process
++ boots, migrates, runs the new `seed: async (db) => {}` hook and saves the data directory (`dumpDataDir`) under
++ `os.tmpdir()/openvibe-test-snapshots` (`OV_TEST_SNAPSHOT_DIR`); later processes load it (`loadDataDir`) instead of
++ migrating, refresh `ov_migrations.applied_at` (so an ADR-028 contract migration stays held as on a fresh migrate) and
++ run `migrate` only if the snapshot is behind. The key is the migrations' file names and contents, the
++ SDK and `@electric-sql/pglite` versions and `seedKey` (default: the seed function's source; a `seedKey` without a
++ `seed` is not the same snapshot as one with both). Parallel processes build
++ it once under a lock file; the others wait up to 60 s, then migrate on their own. A file that does not load is
++ deleted and rebuilt. The result adds `snapshot` (`'hit'`, `'built'` or `'off'`) and `setupMs`. Each database is
++ still the process's own: rows a test writes are never in the snapshot. **Opt out** with `OV_TEST_SNAPSHOT=0`.
++ On `store: 'pg'`, `seed` runs on the run's schema after the migrations with the schema owner's rights (as on PGlite)
++ and `setupMs` is reported; roles, privileges
++ and the lease are unchanged.
++
## 0.28.0 (2026-10-04)
- **`openvibe-sdk/commerce` (new, server; plan T5 step 13):** the Billing client Tips and VIP each carried a copy of.
ddiff --git a/README.md b/README.md
iindex cea14ae..7ba133d 100644
---- a/README.md
++++ b/README.md
@@@ -341,6 +341,8 @@ const platform = createMockPlatform({
const client = createClient({ fetch: platform.fetch, tokenProvider: createServiceTokenClient({ clientId: 'app_01K5WZX7S7Q4D2B8N3M6V1C9TR', clientSecret: 's', fetch: platform.fetch }) });
```
++For a service's own database, `createTestDb({ migrations, seed, seedKey })` returns a migrated PGlite (or, with `store: 'pg'`, a schema of its own on the test containers). The migrated and seeded PGlite is saved once as a snapshot, keyed by the migrations' contents, and later test processes load it instead of migrating (`snapshot: 'hit' | 'built' | 'off'`, `setupMs`; `OV_TEST_SNAPSHOT=0` turns it off, `OV_TEST_SNAPSHOT_DIR` moves it). See [docs/migrating-to-postgresql.md](docs/migrating-to-postgresql.md#5-tests).
++
The mock answers at the real public origins with real RS256 tokens and checks audience, capability, namespace and sandbox the way the services do. It covers:
- **Network:** discovery, `/oauth/token` (client credentials, authorization code with PKCE, refresh; developer apps), `GET /oauth/authorize` (consents automatically as `setAuthorization({ subjectId })`, or declines with `{ decision: 'deny' }`), the JWKS, the registry, `/api/v1/projects` (projects, members, apps, credentials, grants, quotas, audit), `/api/modules` and `/internal/modules`, and `/internal/identity`.
ddiff --git a/STATUS.json b/STATUS.json
iindex 078c712..015f6e4 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -3,7 +3,7 @@
"stage": "alpha",
"code": true,
"wave": "W2",
-- "version": "0.28.0",
++ "version": "0.29.0",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30",
"deployed": false,
"deployment": "none: a package released as git tags; services and apps pin a tag",
@@@ -19,6 +19,7 @@
"per-actor limiter for services (openvibe-sdk/limits)",
"the OpenVibe Frame for any app",
"mock platform for tests (openvibe-sdk/testing)",
++ "createTestDb: one migrated test database per process, loaded from a PGlite snapshot keyed by the migrations (OV_TEST_SNAPSHOT=0 turns it off)",
"ESM entries, TypeScript types and a browser bundle",
"db (ADR-035): the async PostgreSQL data layer (node-postgres through PgBouncer; PGlite in tests) with safe sql fragments, retrying transactions and savepoints, versioned expand/migrate/contract migrations, and importSqlite for the one-time SQLite move with verification",
"valkey, cache, queue, pubsub and a shared limits store on Valkey 9, each with an in-process twin for tests",
ddiff --git a/browser/openvibe-sdk.mjs b/browser/openvibe-sdk.mjs
iindex 1edf09a..9b81bd0 100644
---- a/browser/openvibe-sdk.mjs
++++ b/browser/openvibe-sdk.mjs
@@@ -1,4 +1,4 @@
--// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.28.0, MIT). Do not edit.
++// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.29.0, MIT). Do not edit.
// One self-contained ES module: core at the top level plus the browser-safe subpaths as namespaces
// (auth, registry, modules, realtime, media, community, search, chat, ai, jobs, tools, projects, geo). No server-only code: no service tokens, JWT verification, events, identity.
/* eslint-disable */
@@@ -41,7 +41,7 @@ const { paginate, offsetPager } = __load(7);
const semver = __load(6);
/** The package version (test/version.test.js keeps it equal to package.json). */
--const SDK_VERSION = '0.28.0';
++const SDK_VERSION = '0.29.0';
module.exports = {
createClient,
ddiff --git a/docs/migrating-to-postgresql.md b/docs/migrating-to-postgresql.md
iindex d41d1b6..ba3b2d2 100644
---- a/docs/migrating-to-postgresql.md
++++ b/docs/migrating-to-postgresql.md
@@@ -100,6 +100,7 @@ Anything held in a process that must be shared once there are two processes:
- a test that the hot routes run a bounded number of queries;
- a migration test (the files parse, apply in order, and apply twice without change);
- an import test that imports a small SQLite fixture with `importSqlite` and checks `report.ok`.
++- **One migrated database per test process, from a snapshot:** `const { db, close } = await createTestDb({ migrations, seed, seedKey })` (`openvibe-sdk/testing`). The first process builds the migrated (and seeded) PGlite and saves it under `os.tmpdir()/openvibe-test-snapshots` (`OV_TEST_SNAPSHOT_DIR`); every later one loads it in well under a second instead of migrating. Put the app's seed in `seed: async (db) => {}`, not after the call, and change `seedKey` when the seed changes (without one the seed function's source is the key). `snapshot` says `'hit'`, `'built'` or `'off'` and `setupMs` how long it took. `OV_TEST_SNAPSHOT=0` turns it off.
- **CI** starts the containers and runs the suite against them (`test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test'`, as the SDK does) when the service has pooler-sensitive code.
- **Skipping:** a test that cannot run something prints `<label>: skipped (<why>)`, and the shared runner reports it.
ddiff --git a/package-lock.json b/package-lock.json
iindex e086bdc..67303c0 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -1,18 +1,18 @@
{
"name": "openvibe-sdk",
-- "version": "0.28.0",
++ "version": "0.29.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "openvibe-sdk",
-- "version": "0.28.0",
++ "version": "0.29.0",
"license": "MIT",
"devDependencies": {
"@electric-sql/pglite": "^0.5.8",
"better-sqlite3": "^11.10.0",
"iovalkey": "^0.4.0",
-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.86.0",
++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.97.0",
"pg": "^8.23.0"
},
"engines": {
@@@ -459,9 +459,9 @@
}
},
"node_modules/openvibe-contracts": {
-- "version": "0.86.0",
-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.86.0",
-- "integrity": "sha512-3o3AILEBtQ2pK0KqK7GEAlN3HJjaKUl6vrbzEDiBbhK6Nr2hivgN/5MAmJ7WRkgaDkxNsZETs/fLZC2Q4oNCiA==",
++ "version": "0.97.0",
++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.97.0",
++ "integrity": "sha512-HPGBJYx6Dh2kaTg5B41WGhpw38zUdoU1VnWtkcsFkaU4E521uQOJxf6sIaLdHMsahqov7b9k2HKEFCnD9065MQ==",
"dev": true,
"license": "AGPL-3.0-only",
"dependencies": {
ddiff --git a/package.json b/package.json
iindex f306557..ddc691c 100644
---- a/package.json
++++ b/package.json
@@@ -1,6 +1,6 @@
{
"name": "openvibe-sdk",
-- "version": "0.28.0",
++ "version": "0.29.0",
"description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.",
"main": "index.js",
"types": "types/index.d.ts",
@@@ -276,7 +276,7 @@
"@electric-sql/pglite": "^0.5.8",
"better-sqlite3": "^11.10.0",
"iovalkey": "^0.4.0",
-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.86.0",
++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.97.0",
"pg": "^8.23.0"
},
"peerDependencies": {
ddiff --git a/src/core/index.js b/src/core/index.js
iindex d94ccf8..f956bd3 100644
---- a/src/core/index.js
++++ b/src/core/index.js
@@@ -8,7 +8,7 @@ const { paginate, offsetPager } = require('./paginate');
const semver = require('./semver');
/** The package version (test/version.test.js keeps it equal to package.json). */
--const SDK_VERSION = '0.28.0';
++const SDK_VERSION = '0.29.0';
module.exports = {
createClient,
ddiff --git a/src/testing/db.js b/src/testing/db.js
iindex 7bc371e..8efa8eb 100644
---- a/src/testing/db.js
++++ b/src/testing/db.js
@@@ -14,9 +14,21 @@
* concurrently updated"). close() ends the roles' backends (PgBouncer keeps idle server connections) and drops them.
* open(): another pooled handle on the same database (a second process).
*
++ * Snapshot (PGlite): a fresh PGlite plus the whole migration run costs seconds in every test process. The first
++ * process to need a database builds it (boot, migrate, the optional `seed(db)` hook), dumps its data directory to
++ * os.tmpdir()/openvibe-test-snapshots (OV_TEST_SNAPSHOT_DIR), and every later one loads that file instead. The key
++ * is the migrations' file names and contents, the SDK and @electric-sql/pglite versions and `seedKey` (else the
++ * seed function's source). Parallel processes build it once under a lock file (the others wait up to 60 s, then
++ * migrate on their own); a file that does not load is deleted and rebuilt. A hit refreshes `ov_migrations.applied_at`
++ * so an ADR-028 contract migration stays held exactly as it would on a fresh migrate. OV_TEST_SNAPSHOT=0 turns it off.
++ * The result says which happened (`snapshot`: 'hit' | 'built' | 'off') and how long setup took (`setupMs`).
++ *
* createTestValkey({ prefix }): the containers' Valkey (OV_TEST_VALKEY_URL) under a prefix no other run uses, or null.
*/
const crypto = require('crypto');
++const fs = require('fs');
++const os = require('os');
++const path = require('path');
const quiet = { log() {}, warn() {}, error: (...a) => console.error(...a) };
const SETUP_LOCK = 735_1_2026;
@@@ -54,13 +66,138 @@ async function sweepOrphans(su, { now = Date.now(), maxAge = ORPHAN_MS } = {}) {
}
const valkeyAvailable = () => !!process.env.OV_TEST_VALKEY_URL;
--async function createTestDb({ migrations, store = process.env.OV_TEST_STORE || 'pglite', service = 'test', max = 4, log = quiet } = {}) {
++// ── PGlite snapshot ────────────────────────────────────────────────────────────────────────
++const SNAPSHOT_WAIT_MS = 60_000; // a waiter migrates on its own after this
++const SNAPSHOT_LOCK_STALE_MS = 10 * 60e3; // a lock this old is a dead builder's
++const SNAPSHOT_KEEP_MS = 7 * 86400e3; // snapshots unused this long are removed when another is built
++const snapshotsOn = () => !/^(0|false|off|no)$/i.test(String(process.env.OV_TEST_SNAPSHOT || '').trim());
++const snapshotDir = () => process.env.OV_TEST_SNAPSHOT_DIR || path.join(os.tmpdir(), 'openvibe-test-snapshots');
++const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
++
++function pgliteVersion() {
++ try {
++ let d = path.dirname(require.resolve('@electric-sql/pglite'));
++ for (let i = 0; i < 5; i++, d = path.dirname(d)) {
++ const f = path.join(d, 'package.json');
++ if (fs.existsSync(f)) { const j = JSON.parse(fs.readFileSync(f, 'utf8')); if (j.name === '@electric-sql/pglite') return j.version; }
++ }
++ } catch { /* not installed: createDb says so */ }
++ return 'unknown';
++}
++
++/** The snapshot's name: what a migrated (and seeded) database depends on. */
++function snapshotKey({ migrations, seed, seedKey } = {}) {
++ const h = crypto.createHash('sha256');
++ h.update(`openvibe-test-snapshot:1\0sdk ${require('../../package.json').version}\0pglite ${pgliteVersion()}\0`);
++ if (migrations && fs.existsSync(migrations)) {
++ for (const f of fs.readdirSync(migrations).filter((n) => n.endsWith('.sql')).sort()) h.update(`${f}\0`).update(fs.readFileSync(path.join(migrations, f))).update('\0');
++ }
++ h.update(`seed\0${seed ? (seedKey != null ? `key:${seedKey}` : `fn:${seed.toString()}`) : 'none'}`);
++ return h.digest('hex').slice(0, 32);
++}
++
++const statOf = (f) => { try { return fs.statSync(f); } catch { return null; } };
++const sameFile = (a, b) => !!(a && b && a.ino === b.ino && a.size === b.size && a.mtimeMs === b.mtimeMs);
++
++/** Take the build lock (a file made exclusively), or break it when its builder is gone. */
++function takeLock(lock) {
++ try { fs.writeFileSync(lock, `${process.pid} ${Date.now()}`, { flag: 'wx' }); return true; } catch (e) { if (e.code !== 'EEXIST') throw e; }
++ const st = statOf(lock);
++ let pid = 0;
++ try { pid = parseInt(fs.readFileSync(lock, 'utf8'), 10) || 0; } catch { /* gone already */ }
++ let alive = true;
++ if (pid && pid !== process.pid) { try { process.kill(pid, 0); } catch (e) { alive = e.code === 'EPERM'; } }
++ if (!alive || (st && Date.now() - st.mtimeMs > SNAPSHOT_LOCK_STALE_MS)) fs.rmSync(lock, { force: true });
++ return false;
++}
++
++function pruneSnapshots(dir, keep) {
++ const now = Date.now();
++ for (const f of fs.readdirSync(dir)) {
++ const p = path.join(dir, f);
++ if (p === keep) continue;
++ const st = statOf(p);
++ if (!st) continue;
++ if ((f.endsWith('.pgdata') && now - st.mtimeMs > SNAPSHOT_KEEP_MS) || (f.endsWith('.tmp') && now - st.mtimeMs > 3600e3)) fs.rmSync(p, { force: true });
++ }
++}
++
++async function pgliteTestDb({ migrations, seed, seedKey, service, log }) {
const { createDb } = require('../db');
-- if (store !== 'pg') {
++ const t0 = Date.now();
++ const handle = (db, snapshot) => ({
++ db, store: 'pglite', snapshot, setupMs: Date.now() - t0, url: null, directUrl: null, open: null, close: () => db.close().catch(() => {}),
++ });
++ const build = async () => {
const db = createDb({ pglite: true, service: `${service}-test`, log });
-- if (migrations) await db.migrate({ dir: migrations, log });
-- return { db, store: 'pglite', url: null, directUrl: null, open: null, close: () => db.close().catch(() => {}) };
++ try {
++ if (migrations) await db.migrate({ dir: migrations, log });
++ if (seed) await seed(db);
++ } catch (e) { await db.close().catch(() => {}); throw e; }
++ return db;
++ };
++ if (!snapshotsOn()) return handle(await build(), 'off');
++ const dir = snapshotDir();
++ try { fs.mkdirSync(dir, { recursive: true }); } catch (e) { log.warn(`[test-db] snapshot directory ${dir}: ${e.message}; migrating instead`); return handle(await build(), 'off'); }
++ const file = path.join(dir, `${snapshotKey({ migrations, seed, seedKey })}.pgdata`);
++ const lock = `${file}.lock`;
++ const deadline = Date.now() + SNAPSHOT_WAIT_MS;
++ let bad = null; // the snapshot file that failed to load: rebuilt, never loaded again
++ for (;;) {
++ const st = statOf(file);
++ if (st && !sameFile(st, bad)) {
++ let inst = null;
++ try {
++ const { PGlite } = require('@electric-sql/pglite');
++ inst = new PGlite({ loadDataDir: new Blob([fs.readFileSync(file)]) });
++ await inst.waitReady;
++ const db = createDb({ pglite: inst, service: `${service}-test`, log });
++ if (migrations) {
++ // The snapshot froze ov_migrations.applied_at at build time. An ADR-028 contract migration is
++ // held until its expand has been applied for windowDays, so a reused snapshot would let it run
++ // ~7 days after it was built, while a fresh migrate holds it. Refresh the ages so the loaded
++ // database looks freshly migrated, as it did before snapshots.
++ await db.query('UPDATE ov_migrations SET applied_at = now()').catch(() => {});
++ // The key covers the migrations, so this is a check: migrate only if the snapshot is behind.
++ const applied = new Set((await db.many('SELECT id FROM ov_migrations').catch(() => [])).map((r) => r.id));
++ if (require('../db/migrate').parse(migrations).some((m) => !applied.has(m.id))) await db.migrate({ dir: migrations, log });
++ }
++ try { const now = new Date(); fs.utimesSync(file, now, now); } catch { /* kept a little less long */ }
++ return handle(db, 'hit');
++ } catch (e) {
++ if (inst) await inst.close().catch(() => {});
++ log.warn(`[test-db] snapshot ${path.basename(file)} did not load (${e.message}); rebuilding it`);
++ bad = st;
++ continue;
++ }
++ }
++ if (takeLock(lock)) {
++ let db;
++ try {
++ const now = statOf(file);
++ if (now && !sameFile(now, bad)) continue; // another process finished it meanwhile
++ if (now) fs.rmSync(file, { force: true });
++ db = await build();
++ const tmp = `${file}.${process.pid}.${crypto.randomBytes(4).toString('hex')}.tmp`;
++ try {
++ const blob = await db._adapter.instance.dumpDataDir('none');
++ fs.writeFileSync(tmp, Buffer.from(await blob.arrayBuffer()));
++ fs.renameSync(tmp, file);
++ pruneSnapshots(dir, file);
++ } catch (e) { fs.rmSync(tmp, { force: true }); log.warn(`[test-db] snapshot not saved: ${e.message}`); }
++ } finally { fs.rmSync(lock, { force: true }); }
++ return handle(db, 'built');
++ }
++ if (Date.now() > deadline) { log.warn(`[test-db] snapshot still being built after ${SNAPSHOT_WAIT_MS / 1000} s; migrating instead`); return handle(await build(), 'off'); }
++ await sleep(100);
}
++}
++
++async function createTestDb({ migrations, seed, seedKey, store = process.env.OV_TEST_STORE || 'pglite', service = 'test', max = 4, log = quiet } = {}) {
++ const { createDb } = require('../db');
++ if (seed != null && typeof seed !== 'function') throw new TypeError('createTestDb: seed must be an async function (db) => {}');
++ if (store !== 'pg') return pgliteTestDb({ migrations, seed, seedKey, service, log });
++ const t0 = Date.now();
if (!pgAvailable()) throw new Error('store pg needs OV_TEST_PG_URL and OV_TEST_PG_DIRECT_URL (openvibe-sdk scripts/test-services.sh up)');
const safe = String(service).replace(/[^a-z0-9]/g, '').slice(0, 12) || 'svc';
const name = runName(safe);
@@@ -111,9 +248,15 @@ async function createTestDb({ migrations, store = process.env.OV_TEST_STORE || '
try { await ownerDb.migrate({ dir: migrations, log }); } finally { await ownerDb.close(); }
}
db = createDb({ url: as(process.env.OV_TEST_PG_URL, name), service: `${service}-test`, max, log });
-- } catch (e) { await drop().catch(() => {}); throw e; } // a failed setup leaves nothing behind
++ if (seed) {
++ // On PGlite the seed runs on the migrated handle (the owner). The runtime role here is DML-only
++ // (no TRUNCATE, ALTER SEQUENCE or DDL), so run the pg seed as the owner too: both stores behave alike.
++ const seedDb = createDb({ url: as(process.env.OV_TEST_PG_DIRECT_URL, owner), service: `${service}-test-seed`, max: 1, log });
++ try { await seed(seedDb); } finally { await seedDb.close(); }
++ }
++ } catch (e) { if (db) await db.close().catch(() => {}); await drop().catch(() => {}); throw e; } // a failed setup leaves nothing behind
return {
-- db, store: 'postgresql', schema: name,
++ db, store: 'postgresql', schema: name, snapshot: 'off', setupMs: Date.now() - t0,
// For a process of its own (a worker the test spawns): DATABASE_URL (through PgBouncer, the service role) and
// DATABASE_DIRECT_URL (the owner, for its boot's migrate, which finds everything applied).
url: as(process.env.OV_TEST_PG_URL, name), directUrl: as(process.env.OV_TEST_PG_DIRECT_URL, owner),
@@@ -128,4 +271,4 @@ function createTestValkey({ prefix = 'test' } = {}) {
return createValkey({ url: process.env.OV_TEST_VALKEY_URL, prefix: `ov:${prefix}-test:${crypto.randomBytes(4).toString('hex')}:`, log: quiet });
}
--module.exports = { createTestDb, createTestValkey, pgAvailable, valkeyAvailable, runName, runTime, isRunSchema, leaseKey, sweepOrphans };
++module.exports = { createTestDb, createTestValkey, pgAvailable, valkeyAvailable, runName, runTime, isRunSchema, leaseKey, sweepOrphans, snapshotKey, snapshotDir };
ddiff --git a/test/testing-db.test.js b/test/testing-db.test.js
iindex ebf06c1..534ca1e 100644
---- a/test/testing-db.test.js
++++ b/test/testing-db.test.js
@@@ -1,25 +1,180 @@
'use strict';
/** openvibe-sdk/testing createTestDb: a migrated PGlite database; on the containers, parallel setups do not race. */
const assert = require('node:assert/strict');
++const { fork } = require('node:child_process');
const fs = require('fs');
const os = require('os');
const path = require('path');
--const { run } = require('./helpers');
++const { run, waitFor } = require('./helpers');
const { createTestDb, pgAvailable } = require('../src/testing');
--const { runName, runTime, isRunSchema, leaseKey, sweepOrphans } = require('../src/testing/db');
++const { runName, runTime, isRunSchema, leaseKey, sweepOrphans, snapshotKey } = require('../src/testing/db');
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sdk-testdb-'));
fs.writeFileSync(path.join(dir, '0001_initial.sql'), '-- phase: expand\nCREATE TABLE notes (id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, body text NOT NULL);\n');
--
++// Snapshots of this file's databases go to a directory of its own, removed at the end.
++const snaps = fs.mkdtempSync(path.join(os.tmpdir(), 'sdk-testdb-snap-'));
++process.env.OV_TEST_SNAPSHOT_DIR = snaps;
++delete process.env.OV_TEST_SNAPSHOT;
++const migrationsDir = (sql) => {
++ const d = fs.mkdtempSync(path.join(os.tmpdir(), 'sdk-testdb-'));
++ fs.writeFileSync(path.join(d, '0001_initial.sql'), `-- phase: expand\n${sql}\n`);
++ return d;
++};
++const snapshotFile = (o) => path.join(snaps, `${snapshotKey(o)}.pgdata`);
++const leftovers = () => fs.readdirSync(snaps).filter((f) => !f.endsWith('.pgdata'));
const tests = [
['pglite: migrated and usable', async () => {
const t = await createTestDb({ migrations: dir, service: 'sdk' });
try {
assert.equal(t.store, 'pglite');
++ assert.ok(['built', 'hit'].includes(t.snapshot), t.snapshot); assert.equal(typeof t.setupMs, 'number');
await t.db.prepare('INSERT INTO notes (body) VALUES (?)').run('hi');
assert.equal(await t.db.value('SELECT count(*)::int FROM notes'), 1);
} finally { await t.close(); }
}],
++ ['snapshot: the second database with the same migrations loads it; tables there, the first one\'s rows not', async () => {
++ const m = migrationsDir('CREATE TABLE items (id int PRIMARY KEY, label text NOT NULL);');
++ const a = await createTestDb({ migrations: m, service: 'sdk' });
++ try {
++ assert.equal(a.snapshot, 'built');
++ assert.ok(fs.existsSync(snapshotFile({ migrations: m })));
++ await a.db.query('INSERT INTO items VALUES (1, $1)', ['first']);
++ } finally { await a.close(); }
++ const b = await createTestDb({ migrations: m, service: 'sdk' });
++ try {
++ assert.equal(b.snapshot, 'hit');
++ assert.equal(await b.db.value('SELECT count(*)::int FROM items'), 0, 'data written by the first is not in the snapshot');
++ assert.equal(await b.db.value('SELECT count(*)::int FROM ov_migrations'), 1, 'migrations recorded');
++ await b.db.query('INSERT INTO items VALUES (2, $1)', ['second']);
++ assert.deepEqual(await b.db.many('SELECT id, label FROM items'), [{ id: 2, label: 'second' }]);
++ } finally { await b.close(); }
++ // Changed migration content is another key: built afresh, with the new schema.
++ fs.writeFileSync(path.join(m, '0001_initial.sql'), '-- phase: expand\nCREATE TABLE items (id int PRIMARY KEY, label text NOT NULL, extra text);\n');
++ const c = await createTestDb({ migrations: m, service: 'sdk' });
++ try {
++ assert.equal(c.snapshot, 'built');
++ await c.db.query("INSERT INTO items VALUES (3, 'c', 'x')");
++ } finally { await c.close(); }
++ // A new migration file misses too.
++ fs.writeFileSync(path.join(m, '0002_more.sql'), '-- phase: expand\nCREATE TABLE more (id int);\n');
++ const d = await createTestDb({ migrations: m, service: 'sdk' });
++ try { assert.equal(d.snapshot, 'built'); assert.equal(await d.db.value('SELECT count(*)::int FROM more'), 0); } finally { await d.close(); }
++ assert.deepEqual(leftovers(), [], 'no lock or temporary file left');
++ }],
++ ['snapshot: a corrupt file is deleted and rebuilt, never failing the test', async () => {
++ const m = migrationsDir('CREATE TABLE c (id int);');
++ fs.writeFileSync(snapshotFile({ migrations: m }), 'not a data directory');
++ const a = await createTestDb({ migrations: m, service: 'sdk' });
++ try { assert.equal(a.snapshot, 'built'); assert.equal(await a.db.value('SELECT count(*)::int FROM c'), 0); } finally { await a.close(); }
++ assert.ok(fs.statSync(snapshotFile({ migrations: m })).size > 1000, 'rebuilt');
++ const b = await createTestDb({ migrations: m, service: 'sdk' });
++ try { assert.equal(b.snapshot, 'hit'); } finally { await b.close(); }
++ }],
++ ['snapshot: concurrent builders make one file; the others wait and load it', async () => {
++ const m = migrationsDir('CREATE TABLE par (id int);');
++ const all = await Promise.all([1, 2, 3].map(() => createTestDb({ migrations: m, service: 'sdk' })));
++ try {
++ assert.deepEqual(all.map((t) => t.snapshot).sort(), ['built', 'hit', 'hit']);
++ await all[0].db.query('INSERT INTO par VALUES (1)');
++ for (const t of all.slice(1)) assert.equal(await t.db.value('SELECT count(*)::int FROM par'), 0, 'each its own database');
++ } finally { await Promise.all(all.map((t) => t.close())); }
++ assert.equal(fs.readdirSync(snaps).filter((f) => f.startsWith(snapshotKey({ migrations: m }))).length, 1);
++ assert.deepEqual(leftovers(), []);
++ }],
++ ['snapshot: a builder in another process holds the lock; this process waits and loads its file', async () => {
++ const m = migrationsDir('CREATE TABLE crossing (id int);');
++ const seed = async () => {}; const seedKey = 'slow-build';
++ const lock = `${snapshotFile({ migrations: m, seed, seedKey })}.lock`;
++ const script = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'sdk-testdb-child-')), 'build.js');
++ fs.writeFileSync(script, `'use strict';
++const { createTestDb } = require(${JSON.stringify(path.join(__dirname, '..', 'src', 'testing', 'db'))});
++(async () => {
++ const t = await createTestDb({ migrations: process.argv[2], seed: async () => { await new Promise((r) => setTimeout(r, 2500)); }, seedKey: 'slow-build', service: 'sdk' });
++ if (process.send) process.send({ snapshot: t.snapshot });
++ await t.close();
++})().catch((e) => { if (process.send) process.send({ error: e.message }); process.exit(1); });
++`);
++ const child = fork(script, [m], { stdio: ['ignore', 'ignore', 'inherit', 'ipc'] });
++ const built = new Promise((resolve, reject) => {
++ child.on('message', (msg) => { if (msg && msg.error) reject(new Error(msg.error)); else resolve(msg); });
++ child.on('exit', (code, signal) => reject(new Error(`snapshot builder exited (${code == null ? signal : code})`)));
++ });
++ built.catch(() => {}); // the waiter may fail before the child exits; keep that rejection handled
++ try {
++ await waitFor(() => fs.existsSync(lock), { timeoutMs: 30000 });
++ const t0 = Date.now();
++ const t = await createTestDb({ migrations: m, seed, seedKey, service: 'sdk' });
++ try {
++ assert.equal(t.snapshot, 'hit', 'the other process built it; this one loaded it');
++ assert.ok(Date.now() - t0 >= 100, 'this process waited for the live builder instead of breaking its lock');
++ } finally { await t.close(); }
++ assert.deepEqual(await built, { snapshot: 'built' });
++ } finally { child.kill('SIGKILL'); }
++ assert.deepEqual(leftovers(), []);
++ }],
++ ['snapshot: a lock left by a builder that died is broken', async () => {
++ const m = migrationsDir('CREATE TABLE stale (id int);');
++ fs.writeFileSync(`${snapshotFile({ migrations: m })}.lock`, `2147483646 ${Date.now()}`);
++ const t = await createTestDb({ migrations: m, service: 'sdk' });
++ try { assert.equal(t.snapshot, 'built'); } finally { await t.close(); }
++ assert.deepEqual(leftovers(), []);
++ }],
++ ['snapshot: OV_TEST_SNAPSHOT=0 migrates every time and writes nothing', async () => {
++ const m = migrationsDir('CREATE TABLE off (id int);');
++ process.env.OV_TEST_SNAPSHOT = '0';
++ try {
++ for (let i = 0; i < 2; i++) {
++ const t = await createTestDb({ migrations: m, service: 'sdk' });
++ try { assert.equal(t.snapshot, 'off'); assert.equal(await t.db.value('SELECT count(*)::int FROM off'), 0); } finally { await t.close(); }
++ }
++ assert.equal(fs.existsSync(snapshotFile({ migrations: m })), false);
++ } finally { delete process.env.OV_TEST_SNAPSHOT; }
++ }],
++ ['snapshot: the seed runs once per key and its rows are there after a hit', async () => {
++ const m = migrationsDir('CREATE TABLE users (id int PRIMARY KEY, name text NOT NULL);');
++ let runs = 0;
++ const seed = async (db) => { runs++; await db.query("INSERT INTO users VALUES (1, 'admin')"); };
++ for (const want of ['built', 'hit', 'hit']) {
++ const t = await createTestDb({ migrations: m, seed, seedKey: 'v1', service: 'sdk' });
++ try {
++ assert.equal(t.snapshot, want);
++ assert.deepEqual(await t.db.many('SELECT id, name FROM users'), [{ id: 1, name: 'admin' }]);
++ } finally { await t.close(); }
++ }
++ assert.equal(runs, 1);
++ const other = await createTestDb({ migrations: m, seed, seedKey: 'v2', service: 'sdk' });
++ try { assert.equal(other.snapshot, 'built'); } finally { await other.close(); }
++ assert.equal(runs, 2, 'another seedKey is another snapshot');
++ const plain = await createTestDb({ migrations: m, service: 'sdk' });
++ try { assert.equal(plain.snapshot, 'built'); assert.equal(await plain.db.value('SELECT count(*)::int FROM users'), 0, 'no seed, no rows'); } finally { await plain.close(); }
++ }],
++ ['snapshot: a seedKey without a seed is not the same snapshot as one with both', async () => {
++ const m = migrationsDir('CREATE TABLE keyed (id int PRIMARY KEY, name text);');
++ const seed = async (db) => { await db.query("INSERT INTO keyed VALUES (1, 'admin')"); };
++ assert.notEqual(snapshotKey({ migrations: m, seedKey: 'v1' }), snapshotKey({ migrations: m, seed, seedKey: 'v1' }), 'the keys differ');
++ const bare = await createTestDb({ migrations: m, seedKey: 'v1', service: 'sdk' });
++ try { assert.equal(bare.snapshot, 'built'); assert.equal(await bare.db.value('SELECT count(*)::int FROM keyed'), 0); } finally { await bare.close(); }
++ const seeded = await createTestDb({ migrations: m, seed, seedKey: 'v1', service: 'sdk' });
++ try {
++ assert.equal(seeded.snapshot, 'built', 'the unseeded snapshot is not served to a seeded call');
++ assert.equal(await seeded.db.value('SELECT count(*)::int FROM keyed'), 1, 'the seed ran');
++ } finally { await seeded.close(); }
++ }],
++ ['snapshot: a hit refreshes ov_migrations ages, so an ADR-028 contract stays held as on a fresh migrate', async () => {
++ const m = fs.mkdtempSync(path.join(os.tmpdir(), 'sdk-testdb-'));
++ fs.writeFileSync(path.join(m, '0001_expand.sql'), '-- phase: expand\nCREATE TABLE legacy (id int);\n');
++ fs.writeFileSync(path.join(m, '0002_contract.sql'), '-- phase: contract\n-- after: 0001\nCREATE TABLE contracted (id int);\n');
++ // The seed ages the expand; a snapshot built more than windowDays ago looks exactly like this to a hit.
++ const seed = async (db) => { await db.query("UPDATE ov_migrations SET applied_at = now() - interval '30 days'"); };
++ const a = await createTestDb({ migrations: m, seed, service: 'sdk' });
++ try { assert.equal(a.snapshot, 'built'); } finally { await a.close(); }
++ const b = await createTestDb({ migrations: m, seed, service: 'sdk' });
++ try {
++ assert.equal(b.snapshot, 'hit');
++ assert.equal(await b.db.value("SELECT to_regclass('contracted')"), null, 'the contract is still held');
++ assert.equal(await b.db.value("SELECT count(*)::int FROM ov_migrations WHERE id = '0002'"), 0, 'the contract is not recorded');
++ } finally { await b.close(); }
++ }],
['a run\'s schema name carries when it was made (so a killed run\'s leftovers can be swept); an older name has none', async () => {
const t0 = Date.now(), n = runName('sdk', t0);
assert.match(n, /^sdk_t[0-9a-z]{8,9}_[0-9a-f]{8}_l$/); assert.ok(n.length < 50);
@@@ -66,6 +221,19 @@ if (pgAvailable()) {
await su.close();
}
}]);
++ tests.push(['postgresql: seed runs with the schema owner\'s rights, as on PGlite', async () => {
++ const m = migrationsDir('CREATE TABLE seeded (id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, label text NOT NULL);');
++ const seed = async (db) => {
++ await db.query("INSERT INTO seeded (label) VALUES ('admin')");
++ await db.query('ALTER SEQUENCE seeded_id_seq RESTART WITH 100'); // owner-only: the runtime role may not
++ };
++ const t = await createTestDb({ migrations: m, store: 'pg', seed, service: 'sdk' });
++ try {
++ assert.equal(await t.db.value("SELECT nextval('seeded_id_seq')"), 100);
++ assert.equal(await t.db.value('SELECT count(*)::int FROM seeded'), 1);
++ } finally { await t.close(); }
++ }]);
} else console.log('createTestDb on the containers: skipped (OV_TEST_PG_URL not set; scripts/test-services.sh up)');
++tests.push(['(the snapshots made here are removed)', async () => { fs.rmSync(snaps, { recursive: true, force: true }); }]);
run(tests);
ddiff --git a/types/contracts.d.ts b/types/contracts.d.ts
iindex 33777cb..5839178 100644
---- a/types/contracts.d.ts
++++ b/types/contracts.d.ts
@@@ -33,9 +33,9 @@ export type SubjectRef =
id: string;
};
--/** [email protected] (owner: network) */
++/** [email protected] (owner: network) */
/**
-- * Claims of a short-lived RS256 client-credentials token issued by OpenVibe.Network to a service, app, mod or node principal. Replaces X-Internal-Key. App tokens (actor_type app) also carry project_id and env; receivers refuse env=sandbox unless they opted in.
++ * Claims of a short-lived RS256 client-credentials token issued by OpenVibe.Network to a service, app, mod, node or agent principal. Replaces X-Internal-Key. App tokens (actor_type app) also carry project_id and env; receivers refuse env=sandbox unless they opted in. Agent tokens (actor_type agent, sub agent:agt_<ULID>) are minted for the agent's host and also carry project_id, env and on_behalf_of (the owner), and optionally cap_confirm and act.
*/
export type ServiceTokenClaims = {
[k: string]: unknown | undefined;
@@@ -45,7 +45,7 @@ export type ServiceTokenClaims = {
*/
iss: string;
sub: string;
-- actor_type: "service" | "app" | "mod" | "node";
++ actor_type: "service" | "app" | "mod" | "node" | "agent";
/**
* @minItems 1
*/
@@@ -54,6 +54,10 @@ export type ServiceTokenClaims = {
* Granted capability ids; a trailing .* grants a family.
*/
cap: string[];
++ /**
++ * Agent tokens: capabilities the agent holds in confirm mode. Each use needs an approved network.confirmation-request@1, presented as OpenVibe-Confirmation: cnf_<ULID> and consumed by the owning service. Never also in cap, so a receiver that does not know agents refuses those actions.
++ */
++ cap_confirm?: string[];
/**
* Namespace constraints, e.g. live.* or mod.example.*. A developer app token carries its project_id (the whole project, as issued before namespaces were rows) and app.<project_id>.*: on Media, app.<project_id> and app.<project_id>.sandbox are the project's production and sandbox namespaces, with children below them.
*/
@@@ -62,17 +66,23 @@ export type ServiceTokenClaims = {
exp: number;
jti: string;
/**
-- * Developer project of an app principal (ADR-014). Services key tenancy by it. Absent on first-party service tokens.
++ * Developer project of an app or agent principal (ADR-014). Services key tenancy by it. Absent on first-party service tokens.
*/
project_id?: string;
/**
-- * Environment of an app principal. A receiver MUST refuse env=sandbox (401 token.sandbox_refused) unless it opted in to sandbox tokens. Absent on first-party service tokens, which are production.
++ * Environment of an app or agent principal. A receiver MUST refuse env=sandbox (401 token.sandbox_refused) unless it opted in to sandbox tokens. Absent on first-party service tokens, which are production.
*/
env?: "sandbox" | "production";
/**
-- * The person who authorized an app through the authorization-code flow. Absent on client_credentials tokens.
++ * The person who authorized an app through the authorization-code flow, or the owner an agent acts for (always present on agent tokens). Absent on client_credentials tokens.
*/
on_behalf_of?: string;
++ /**
++ * Agent tokens: the host that runs the agent and the token was minted for (RFC 8693 actor claim), a service or an app.
++ */
++ act?: {
++ sub: string;
++ };
[k: string]: unknown | undefined;
};
ddiff --git a/types/testing.d.ts b/types/testing.d.ts
iindex 3fb6bde..838e2ff 100644
---- a/types/testing.d.ts
++++ b/types/testing.d.ts
@@@ -160,10 +160,21 @@ export interface TestDb {
url: string | null;
/** DATABASE_DIRECT_URL for a spawned process (the owner, direct: its boot's migrate); null on PGlite. */
directUrl: string | null;
++ /** PGlite: loaded from the migrated snapshot ('hit'), made it ('built'), or migrated without one ('off'); always 'off' on store pg. */
++ snapshot: 'hit' | 'built' | 'off';
++ /** How long the setup took, in ms. */
++ setupMs: number;
close(): Promise<void>;
}
/** A migrated database for one test run: PGlite, or (store 'pg') roles and a schema of its own on the containers. */
--export declare function createTestDb(opts?: { migrations?: string; store?: 'pglite' | 'pg' | string; service?: string; max?: number; log?: object }): Promise<TestDb>;
++export declare function createTestDb(opts?: {
++ migrations?: string;
++ /** Runs once after the migrations with the schema owner's rights on both stores; on PGlite its rows are part of the snapshot. */
++ seed?: (db: Db) => Promise<void> | void;
++ /** Part of the snapshot key: change it when the seed changes (default: the seed function's source). */
++ seedKey?: string;
++ store?: 'pglite' | 'pg' | string; service?: string; max?: number; log?: object;
++}): Promise<TestDb>;
export declare function createTestValkey(opts?: { prefix?: string }): Valkey | null;
export declare function pgAvailable(): boolean;
export declare function valkeyAvailable(): boolean;