openvibers/openvibe.sdk Commits
openvibe-sdk 0.38.0: delete the one-time SQLite migration tooling (importSqlite, runSqliteMigration, asyncify) (#32)
Changes
@@@ -3,6 +3,15 @@All notable changes to `openvibe-sdk`. The package follows semver; while it is `0.x`, a minorrelease may change an API and says so here.++## 0.38.0 (2026-10-10)++++- **Removed: the one-time SQLite → PostgreSQL migration tooling** (plan T1, "Delete the migration era"). Every++ OpenVibe service runs on PostgreSQL and none calls them: `importSqlite` and `runSqliteMigration` (`openvibe-sdk/db`),++ `src/db/sqlite-import.js`, `src/db/sqlite-cli.js`, the `tools/asyncify/` conversion scripts and++ `docs/migrating-to-postgresql.md`. A consumer that still imported either function must drop it; nothing else changes.++- Kept: the better-sqlite3-shaped statements on PostgreSQL (`db.prepare`), and the SQLite outbox and inbox++ (`createOutbox`/`createInbox`) for a small app that keeps them in SQLite (OpenVibe.Examples' webhook consumer does).++## 0.37.2 (2026-10-10)- `openvibe-sdk/realtime`: each event's callback gets its opaque `cursor` (the SSE id Events sent) beside `seq`:@@@ -344,7 +344,7 @@ const platform = createMockPlatform({const client = createClient({ fetch: platform.fetch, tokenProvider: createServiceTokenClient({ clientId: 'app_01K5WZX7S7Q4D2B8N3M6V1C9TR', clientSecret: 's', fetch: platform.fetch }) });```--For a service's own database, `createTestDb({ migrations, seed, seedKey })` returns a migrated PGlite (or, with `store: 'pg'`, a schema of its own on the test containers). The migrated and seeded PGlite is saved once as a snapshot, keyed by the migrations' contents, and later test processes load it instead of migrating (`snapshot: 'hit' | 'built' | 'off'`, `setupMs`; `OV_TEST_SNAPSHOT=0` turns it off, `OV_TEST_SNAPSHOT_DIR` moves it). See [docs/migrating-to-postgresql.md](docs/migrating-to-postgresql.md#5-tests).++For a service's own database, `createTestDb({ migrations, seed, seedKey })` returns a migrated PGlite (or, with `store: 'pg'`, a schema of its own on the test containers). The migrated and seeded PGlite is saved once as a snapshot, keyed by the migrations' contents, and later test processes load it instead of migrating (`snapshot: 'hit' | 'built' | 'off'`, `setupMs`; `OV_TEST_SNAPSHOT=0` turns it off, `OV_TEST_SNAPSHOT_DIR` moves it).The mock answers at the real public origins with real RS256 tokens and checks audience, capability, namespace and sandbox the way the services do. It covers:@@@ -425,9 +425,8 @@ const thumbs = createQueue({ valkey, name: 'thumbs' });thumbs.process(async (job) => { … }, { concurrency: 4 });```--Install the drivers the service uses: `pg` and `iovalkey` (production), `@electric-sql/pglite` (tests), `better-sqlite3` (only for the one-time `importSqlite`).++Install the drivers the service uses: `pg` and `iovalkey` (production), `@electric-sql/pglite` (tests). `better-sqlite3` is only for a small app that keeps its outbox or inbox in SQLite (`createOutbox`/`createInbox`).--Moving an existing service from SQLite: [docs/migrating-to-postgresql.md](docs/migrating-to-postgresql.md) (the schema mapping, the code idioms, per-process state to Valkey, tests, the rehearsed import and the production switch).### Browser without a bundler@@@ -3,7 +3,7 @@"stage": "alpha","code": true,"wave": "W2",-- "version": "0.37.2",++ "version": "0.38.0","plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30","deployed": false,"deployment": "none: a package released as git tags; services and apps pin a tag",@@@ -22,7 +22,7 @@"mock platform for tests (openvibe-sdk/testing)","createTestDb: one migrated test database per process, loaded from a PGlite snapshot keyed by the migrations (OV_TEST_SNAPSHOT=0 turns it off)","ESM entries, TypeScript types and a browser bundle",-- "db (ADR-035): the async PostgreSQL data layer (node-postgres through PgBouncer; PGlite in tests) with safe sql fragments, retrying transactions and savepoints, versioned expand/migrate/contract migrations, and importSqlite for the one-time SQLite move with verification",++ "db (ADR-035): the async PostgreSQL data layer (node-postgres through PgBouncer; PGlite in tests) with safe sql fragments, retrying transactions and savepoints, versioned expand/migrate/contract migrations","valkey, cache, queue, pubsub and a shared limits store on Valkey 9, each with an in-process twin for tests","commerce (openvibe-sdk/commerce): the shared Billing client for Tips and VIP, with intentKey and receipt","events: the PostgreSQL outbox (lease-claimed, several relays per table) and inbox (handler in its transaction)",@@@ -1,4 +1,4 @@--// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.37.2, MIT). Do not edit.++// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.38.0, MIT). Do not edit.// One self-contained ES module: core at the top level plus the browser-safe subpaths as namespaces// (auth, registry, modules, realtime, media, community, bot, search, chat, ai, jobs, tools, projects, geo). No server-only code: no service tokens, JWT verification, events, identity./* eslint-disable */@@@ -42,7 +42,7 @@ const { paginate, offsetPager } = __load(7);const semver = __load(6);/** The package version (test/version.test.js keeps it equal to package.json). */--const SDK_VERSION = '0.37.2';++const SDK_VERSION = '0.38.0';module.exports = {createClient,ddeleted file mode 100644@@@ -1,192 +0,0 @@--# Moving a service from SQLite to PostgreSQL and Valkey----ADR-035 (and its amendment) moves every OpenVibe service from its own synchronous SQLite database to:--- **PostgreSQL 18**, through PgBouncer in transaction mode;--- **Valkey 9**, for shared, non-authoritative state;----and makes each service async from end to end. This guide is how every migration is done, so the results look the same everywhere. Read it with the roadmap's engineering standards (section 4B.7): async request paths, bounded pools, Valkey for shared state, indexes for every query shape, keyset pagination, no N+1 queries, and measured budgets.----## 0. What you get from the platform----- **The data role on the host** (OpenVibe.Host `roles/data/`): PostgreSQL, PgBouncer, pgBackRest and Valkey on loopback.-- - `sudo /opt/openvibe.host/roles/data/add-service.sh <svc>` gives the service a database `ov_<svc>`, an owner role, a pooled runtime role (DML only, 15 s statement timeout) and a Valkey user confined to `ov:<svc>:*`.-- - It writes four settings into `/etc/openvibe/<svc>.env`: `DATABASE_URL` (runtime role, through PgBouncer), `DATABASE_DIRECT_URL` (owner role: migrations), `VALKEY_URL` and `VALKEY_PREFIX`.--- **openvibe-sdk ≥ 0.15.0:**-- - `openvibe-sdk/db`: `createDb`, the `sql` tag, `tx`, `migrate` and `importSqlite`;-- - `openvibe-sdk/valkey`, `cache`, `queue` and `pubsub`, and `limits` with `createValkeyLimitStore`;-- - `openvibe-sdk/events`: `createPgOutbox`, `createPgInbox`, `outboxSchema` and `inboxSchema`.--- **Tests:** `@electric-sql/pglite` runs real PostgreSQL in-process. `openvibe-sdk/scripts/test-services.sh up` starts the production-shaped container set (PostgreSQL 18, PgBouncer and Valkey) for integration runs.----## 1. Dependencies----```--npm i pg iovalkey--npm i -D @electric-sql/pglite--# keep better-sqlite3 until the production import is done (the importer reads the old file); drop it after--```----Pin `openvibe-sdk` to `v0.15.0` or later. If the service uses `openvibe-publishing` or `openvibe-shared/config` / `analytics`, pin their PostgreSQL majors (see their CHANGELOGs).----## 2. The schema, as migrations----Create `migrations/0001_initial.sql` (`-- phase: expand`). It reproduces today's schema in PostgreSQL types, so the data can be imported as it is:----| SQLite today | PostgreSQL |--|---|---|--| `INTEGER PRIMARY KEY AUTOINCREMENT` | `bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY` (the importer keeps the values and moves the sequence) |--| `INTEGER` holding 0/1 | `boolean` (and change the code: `row.flag === 1` becomes `row.flag`) |--| `INTEGER` epoch milliseconds (`created_at INTEGER NOT NULL`) | **`bigint`**, kept in milliseconds, so the API still sends the same numbers |--| `DATETIME DEFAULT CURRENT_TIMESTAMP` / ISO text | `timestamptz NOT NULL DEFAULT now()`; the layer returns ISO-8601 strings (check what the API sent before and keep the format where clients depend on it) |--| `TEXT` holding JSON | `jsonb` (and stop calling `JSON.parse` on it: rows come back parsed) |--| `TEXT` (UUID, ULID, slug) | `text` with the same `CHECK`s; `citext` or a `lower()` index where the code compared `COLLATE NOCASE` |--| `REAL` | `double precision`; money stays integer cents (`bigint`), never float |--| `SUM(int)`, `SUM(bigint)`, `AVG(…)` | PostgreSQL returns `numeric`, which the layer gives as a **string**. Cast sums that must stay numbers: `SUM(amount)::bigint` (found in the Tips migration) |--| `BLOB` | `bytea` |--| `CHECK (x IN (…))` | the same `CHECK` (or an enum type only if it never changes) |----Then:--- **Indexes.** Add one for every query shape the routes run. That means every `WHERE` + `ORDER BY` pair on a list route, and a partial index for "due" or "unsent" queries (for example `WHERE sent_at IS NULL`).--- **Tables.** Put the outbox and inbox DDL in the migration: `outboxSchema('<svc>_event_outbox')` and `inboxSchema(…)`. Copy the SQL text into the file.--- **Triggers.** SQLite `RAISE(ABORT, …)` triggers become PL/pgSQL functions plus `CREATE TRIGGER … BEFORE UPDATE OR DELETE`.--- **Never edit a migration after it runs.** Add the next number instead. A change that drops or renames something is a `contract` migration naming its `expand` (`-- after: NNNN`), and it runs after the 7-day N-1 window.----## 3. The code: async from end to end----**The short path (SDK ≥ 0.18).**--- Keep the statements' shape. `db.prepare(text)` returns async `get`, `all`, `run` and `pluck`, with the same `?` and `@name` parameters, so `q.x.get(id)` becomes `await q.x.get(id)`.--- Keep functions that run inside a transaction calling `db`. With ambient transactions, `db.transaction(fn)()` becomes `await db.tx(async () => fn())`, and every `db` call inside joins it.--- What still needs a person is the SQL itself (the table below) and `lastInsertRowid`, which needs `RETURNING`.------| better-sqlite3 | openvibe-sdk/db |--|---|---|--| `db.prepare(q).get(a)` | `await db.maybe(sql\`… ${a}\`)` (or `one` when absence is a bug) |--| `db.prepare(q).all(a)` | `await db.many(sql\`…\`)` |--| `db.prepare(q).run(a)` / `.changes` | `await db.exec(sql\`…\`)` returns the count |--| `info.lastInsertRowid` | `INSERT … RETURNING id` with `db.one` |--| `db.transaction(fn)()` | `await db.tx(async (t) => { … })`: with ambient transactions (default) `db` calls inside join it; `t` also works |--| `INSERT OR IGNORE` | `INSERT … ON CONFLICT DO NOTHING` |--| `INSERT OR REPLACE` | `INSERT … ON CONFLICT (key) DO UPDATE SET …` (and `excluded.col` works) |--| `datetime('now')`, `strftime(…)`, `CURRENT_TIMESTAMP` | `now()`, `to_char(…)`, `extract(epoch from …)` |--| `IFNULL`, `GROUP_CONCAT`, `json_extract(x,'$.a')` | `COALESCE`, `string_agg`, `x->>'a'` |--| `LIKE` (case-insensitive for ASCII in SQLite) | `ILIKE`, or `lower(col) LIKE lower($1)` with an index |--| `LIMIT ? OFFSET ?` on large lists | keyset: `WHERE (created_at, id) < ($1, $2) ORDER BY created_at DESC, id DESC LIMIT $3` |--| a loop of single-row `SELECT`s | one query with `= ANY(${ids})` or a `JOIN` |--| a loop of single-row `INSERT`s | `sql.insert(rows)` (up to about 1000 rows a statement) |----- **Handlers.** Make every Express handler and every function that touches data `async`, and `await` it. Errors reach the error handler: wrap routes (`asyncHandler`) or use Express 5.--- **The pooler's rules** (transaction mode). On `DATABASE_URL`, no session state: no session `SET`, no `LISTEN`, and no advisory lock held outside a transaction (`pg_advisory_xact_lock` inside a `tx` is fine). Use `DATABASE_DIRECT_URL` only for migrations and the importer.--- **Money paths** use `db.tx(fn, { isolation: 'serializable' })`. Lock the rows you debit with `SELECT … FOR UPDATE`, in a fixed order, and use idempotency keys. The layer retries serialization failures.--- **Readiness.** Report `await db.ready()` (it names `postgresql`) and `await valkey.ready()` in `/ready`, so it never claims a store from configuration.--- **Boot.** `await createDb({ url: process.env.DATABASE_DIRECT_URL }).migrate({ dir })`, closed afterwards. Then `createDb({ service })` for serving. The advisory lock makes this safe when several processes start together.----## 4. Per-process state goes to Valkey----Anything held in a process that must be shared once there are two processes:--- **Per-actor limits:** `createActorLimiter({ …, store: createValkeyLimitStore(valkey) })`.--- **Caches** (maps, LRUs): `createCache({ valkey, namespace })`, with TTLs and tags.--- **Realtime fan-out** (a message received by one process must reach sockets on another): `createPubSub({ valkey })`.--- **Background work run from `setInterval`:**-- - If it must run once across processes, it becomes a `createQueue` job, or a job claimed with a lease row (`UPDATE … SET lease_until = … WHERE … FOR UPDATE SKIP LOCKED`).-- - A purely local chore (pruning a process's own memory) may stay a timer.--- **Nothing authoritative.** Money, accounts, grants, bans and entitlements live only in PostgreSQL.----`createValkey()` returns `null` without `VALKEY_URL`, and every module above then runs in-process. Tests need no Valkey unless they test sharing.----## 5. Tests----- **Per test, run real PostgreSQL:** `const db = createDb({ pglite: true }); await db.migrate({ dir })`. Keep test data in the same shape as before, then fix what the types change (booleans, parsed JSON, ISO timestamps).--- **What to add:**-- - a test that the hot routes run a bounded number of queries;-- - a migration test (the files parse, apply in order, and apply twice without change);-- - an import test that imports a small SQLite fixture with `importSqlite` and checks `report.ok`.--- **One migrated database per test process, from a snapshot:** `const { db, close } = await createTestDb({ migrations, seed, seedKey })` (`openvibe-sdk/testing`). The first process builds the migrated (and seeded) PGlite and saves it under `os.tmpdir()/openvibe-test-snapshots` (`OV_TEST_SNAPSHOT_DIR`); every later one loads it in well under a second instead of migrating. Put the app's seed in `seed: async (db) => {}`, not after the call, and change `seedKey` when the seed changes (without one the seed function's source is the key). `snapshot` says `'hit'`, `'built'` or `'off'` and `setupMs` how long it took. `OV_TEST_SNAPSHOT=0` turns it off.--- **CI** starts the containers and runs the suite against them (`test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test'`, as the SDK does) when the service has pooler-sensitive code.--- **Skipping:** a test that cannot run something prints `<label>: skipped (<why>)`, and the shared runner reports it.----## 6. The one-time import, and the production switch----Add `scripts/migrate-to-postgres.js`, a few lines around `runSqliteMigration` from `openvibe-sdk/db` (SDK ≥ 0.19; Wiki's is the example). It:--1. runs the migrations with `DATABASE_DIRECT_URL`;--2. runs `importSqlite({ sqlite: <the service's DB path>, db: owner, truncate: true, tables: { … dropColumns … } })`;--3. prints the report and exits 1 unless `report.ok`.----It changes nothing in the SQLite file.----1. **Rehearse** on a copy of the production database:-- - copy it (`sqlite3 <db> ".backup /tmp/rehearsal.db"` as the service user, then move it off the host) or use the drill copy;-- - run the script against a PGlite or local container database;-- - fix every problem it reports: dropped columns must be dropped on purpose, and dates it cannot read must be mapped;-- - record the per-table counts.--2. **In production:**-- 1. `sudo /opt/openvibe.host/roles/data/add-service.sh <svc>`.-- 2. Merge the PostgreSQL release, but do not deploy it yet.-- 3. Stop writes: `systemctl stop openvibe-<svc>` for a short stop, or the service's read-only switch when it has one.-- 4. Back up the SQLite file.-- 5. Run the script from the new release's directory, as the service user, with the service's environment (Host's recipe: guarded `export`, output filtered). It must end with `report.ok`.-- 6. Deploy the release (`sudo ovhost deploy <svc>`); it now reads `DATABASE_URL`.-- 7. Verify that `/ready` shows `store: postgresql`, the service's own smoke checks pass, and the error rate stays flat.--3. **Keep the SQLite file read-only for 7 days** (the N-1 window). It is the rollback: redeploy the previous release, which reads it, after the owner accepts losing the writes made since the switch, or replaying them from the PostgreSQL side. After the window, archive it to B2 (`ovhost archive push`) and delete it.--4. **Record the evidence** in the same change:-- - the import report (tables, rows, checksums);-- - `/ready` before and after;-- - the test counts;-- - `STATUS.json` (the database) and the README (Depends on: PostgreSQL and Valkey);-- - the roadmap plan's WS-X2 migration list.----**Rehearsing on the containers.** An embedded PGlite can break on very large rows (OpenVibe.AI's run payloads). When `--pglite` reports every table empty, import into a schema of the containers: `createTestDb({ migrations, store: 'pg' })`, then `importSqlite({ sqlite, db: t.db })`.----## 7. Lessons from the first migrations----- **Tips (2026-09-28)** is the reference: its `postgres` branch has the schema, the importer script, `test/helpers/db.js` and the multi-process integration test. Copy them.--- **Test roles on the shared containers.** End each per-run role's backends before dropping it, or PgBouncer keeps idle server connections and exhausts the container. `test/helpers/db.js` does this.--- **Replays compare bytes.** Keep a replayed response (idempotency records) as `text`: `jsonb` reorders keys.--- **Claim an Idempotency-Key before running the handler**, with `INSERT … ON CONFLICT DO NOTHING RETURNING`. That way two concurrent requests with the same key run once.--- **Clean text PostgreSQL refuses.** SQLite text can hold NUL and unpaired surrogates. Clean them in the importer's `map`, and report each column that needed it.--- **pub/sub without Valkey.** `createPubSub` has no local fallback when Valkey is down. A hub that must keep delivering on one process falls back to local delivery itself.--- **News (2026-09-28)** added these:-- - **`rowid` tiebreaks.** `ORDER BY created_at, rowid` meant insertion order. A random ULID `id` is not, within one millisecond. Add `seq bigint GENERATED ALWAYS AS IDENTITY UNIQUE` to the table and order by `seq`. The importer fills it in the source's row order.-- - **Scalar `MAX(a, b)` / `MIN(a, b)`** are `GREATEST` / `LEAST`.-- - **`ON CONFLICT … DO UPDATE SET x = COALESCE(excluded.x, x)`** is ambiguous in PostgreSQL. Qualify the target: `COALESCE(excluded.x, <table>.x)`.-- - **Work after the commit.** A `setImmediate` after a synchronous better-sqlite3 transaction ran after the commit. Around an async transaction it does not. Use `db.afterCommit(fn)` (SDK ≥ 0.20).-- - **The inbox.** `createInbox` (better-sqlite3) becomes `createPgInbox`, awaited. Its `idempotency_receipts` table goes in `migrations/0001_initial.sql` (`inboxSchema()`), not an unawaited `ensureSchema()`.-- - **A script run as a child process in tests** cannot reach the test's PGlite. Call the function it wraps in-process, and `node --check` the script.--- **Deals, Coupons, Trade and Reviews (2026-09-28)** added these:-- - **Higher-order wrappers.** A helper that takes a function (`conflictGuard(fn)`, `unique(fn)`, `staffAction(fn)`, a batch) must `await fn()`. Otherwise its `try` catches nothing and the caller races the write. Match PostgreSQL's error codes (`23505` unique, `23503` foreign key) with `err.table` and `err.detail`, not SQLite's message.-- - **Per-process counters around a transaction** (`batchDepth++ … finally batchDepth--`) break once requests interleave. Keep per-call state in `AsyncLocalStorage`.-- - **Factories stay synchronous.** A factory that seeded a row at construction became `async`, and every caller got a promise. Write such rows lazily on first use.-- - **SQL.** Name aliases are not allowed in `HAVING`: repeat the aggregate. A bare column next to `MAX()` in a `GROUP BY` takes its value from the max row in SQLite; use `DISTINCT ON … ORDER BY` instead. `SELECT DISTINCT` over a `LEFT JOIN` with `ORDER BY lower(x)` fails; use `WHERE … OR EXISTS (…)`. Cast a parameter tested only with `IS NULL` (`@p::bigint IS NULL`). Give a recursive CTE's seed the column's type and collation (`?::text COLLATE "C"`). `json_each(?)` becomes `jsonb_array_elements_text(?::jsonb)`.-- - **Values.** `jsonb` columns come back as objects, so do not `JSON.parse` them.-- - **Gauges.** A metrics `collect()` is synchronous. Read the database one scrape behind.-- - **`npm run test:pg` finds races PGlite hides.** PGlite has one connection, so an unawaited write finished before the next read. On a pool it does not.--- **Events (2026-09-28)** added these:-- - **A loop that read and then claimed in one synchronous step** (a delivery worker's "select due rows, mark them busy") can now run twice at once, from a timer and a kick, and send the same row twice. Serialize the passes: a call during a pass waits for one more pass after it.-- - **Register before the first `await`.** A connection counted against a cap, or joined to an in-process fan-out, after its catch-up queries lets the cap overrun and loses what was published meanwhile. Take the slot and join first; buffer live rows until the catch-up ends, then send those it did not cover.-- - **A function passed as an argument** (`flush(insertBatch)`) is async now; the codemod does not see through parameters. Await it by hand.-- - **No DDL at boot.** A module that ran `CREATE TABLE IF NOT EXISTS` on the serving handle fails on production roles (`permission denied for schema`) while PGlite allows it. The table goes in the migration; `sqlfix.py` reports `db.exec(`.-- - **Test boots that call `start()` directly** fall back to the development PGlite directory and share rows across runs. Pass every boot a `createTestDb` handle.-- - **Redaction.** PostgreSQL has no `secure_delete`: an overwritten row version stays until vacuum. Lower the table's `autovacuum_vacuum_scale_factor` and say so in the README.--- **Community (2026-09-28)** added these:-- - **A schema changed by `migrate()` after `SCHEMA`.** `gen-migration.js --open "require('./server/db').openDb(':memory:')"` takes the schema the SQLite release's own boot leaves (every ALTER applied, tables in dependency order, boot seeds as INSERTs). Run it before converting any code.-- - **SQLite text timestamps.** `DATETIME` columns hold `'YYYY-MM-DD HH:MM:SS'` text. Keep it: the migration gets `SQLITE_DATE_FUNCTIONS` (`ov_now()`, `datetime(t, modifier)`, `julianday(t)`), and `sqlfix.py` rewrites `CURRENT_TIMESTAMP` to `ov_now()` (PostgreSQL's would be a `timestamptz`, cast to different text).-- - **Type affinity.** SQLite stores text in an `INTEGER` column without complaint (a Chat room id); PostgreSQL refuses. Check the production copy (`tools/asyncify/typecheck.js <file>` lists columns holding another storage class) and read the code that writes each id column.-- - **`INTEGER PRIMARY KEY REFERENCES …`** is the other row's key, not a counter: `bigint PRIMARY KEY` (the converter does this now).-- - **camelCase aliases** fold to lower case (`AS totalViews` comes back as `totalviews`): quote them (`sqlfix.py` does).-- - **Explicit ids** into a `GENERATED ALWAYS` identity need `OVERRIDING SYSTEM VALUE` (`sqlfix.py` adds it where the migration says ALWAYS); `.lastInsertRowid` needs `RETURNING id`.-- - **`IMMEDIATE` transactions.** A read-then-write that relied on SQLite's write lock (a vote recount) locks its row first: `SELECT … FOR UPDATE`.-- - **Work kicked from inside a transaction** (`setImmediate(drain)`) runs before the commit now and misses the row: `db.afterCommit`, and a short delay when changes made together should go out as one pass.-- - **Event handlers that became async** (a WebSocket dispatch) can overtake each other: queue them in arrival order.-- - **Fire-and-forget helpers** (`tell(fn) { try { fn(x) } catch {} }`) catch nothing once `fn` is async: await it.-- - **Tests**: `const x = createFoo(…)` instances, `require('./x').fn()` calls and a test's own `rejects(promise)` helper are handled by the codemod now; a boot helper that re-requires the server gets its database passed in (`createApp({ db })`); SQLite-only upgrade tests (a `migrate()` rebuilding an old table) go, the migration ledger replaces them.--- **Media (2026-09-28)** added these:-- - **SQLite functions the code calls**: `json_valid`, `json_extract`, `json_type`, `instr`, `date('now', …)`, `strftime('%Y-%m-%dT%H:%M:%fZ', 'now', …)`: the migration defines SQLite-compatible `json_valid`/`json_extract`/`json_type`/`instr`, `ov_now_iso(modifier)`, and `sqlfix.py` rewrites `date(…)` to `substr(datetime(…), 1, 10)`.-- - **`SUM()` over bigint is `numeric`**, and the SDK returns numeric as exact text on purpose (money). Cast byte and count sums (`::bigint`); leave double-precision sums (durations) alone.-- - **Integers as booleans**: `WHERE 0`, `CASE WHEN is_public THEN`: write `FALSE`, `is_public = 1`.-- - **Your own `BEGIN`/`COMMIT` through `db.exec`** runs on whichever pooled connection answers: use `db.tx` (a nested `tx` is a savepoint; throw a marker to roll a dry run back).-- - **A generator that awaits** becomes `async function*`: its loops need `for await`.-- - **Callbacks a helper takes in an object** (`announce(app, event, { change, payload })`, `onRow`, `alsoInTx`) are awaited by hand; so are `x.start()` calls whose `start` became async (a fire-and-forget one needs `.catch`, or the rejection ends the process).-- - **Lazy accessors** (`function objects() { return require('../objects/model'); }`, `const m = objectsModel()`), inline `require('./x').fn()`, factory instances and `db` passed as a parameter (config `fileApis`) are resolved by the codemod now.-- - **Stores whose creation became async** (the Shared config store) get an `init()` at boot and keep a synchronous `get()`; anything they must answer synchronously (explicit keys) is cached on activation.-- - **The SQLite outbox refused to enqueue outside a transaction**; the PostgreSQL one takes any handle. Keep the guard in the service (`db.inTransaction()`), or an event can be written after the change committed.-- - **Tests**: one migrated database per test process through `node --import` (a preload), a script runner that runs operator scripts in the test process (a child process cannot reach an in-process PGlite), identities started at 100000 so tests' explicit small ids never collide, fault injection in JavaScript instead of `TEMP TRIGGER`s (the runtime role has no DDL under `test:pg`), and SQLite upgrade tests removed.-- - **Read-only reports** run in a `SET TRANSACTION READ ONLY` transaction instead of opening a read-only SQLite file.@@@ -2,7 +2,6 @@import cjs from '../src/db/index.js';export const {-- createDb, sql, DbError, importSqlite, runSqliteMigration, PARSERS,-- ISOLATION,++ createDb, sql, DbError, PARSERS, ISOLATION,} = cjs;export default cjs;@@@ -1,12 +1,12 @@{"name": "openvibe-sdk",-- "version": "0.37.2",++ "version": "0.38.0","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "openvibe-sdk",-- "version": "0.37.2",++ "version": "0.38.0","license": "MIT","devDependencies": {"@electric-sql/pglite": "^0.5.8",@@@ -1,6 +1,6 @@{"name": "openvibe-sdk",-- "version": "0.37.2",++ "version": "0.38.0","description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, Bot robots, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.","main": "index.js","types": "types/index.d.ts",@@@ -8,7 +8,7 @@ const { paginate, offsetPager } = require('./paginate');const semver = require('./semver');/** The package version (test/version.test.js keeps it equal to package.json). */--const SDK_VERSION = '0.37.2';++const SDK_VERSION = '0.38.0';module.exports = {createClient,@@@ -331,9 +331,4 @@ function pgliteAdapter(opt) {};}--/** The one-time SQLite → PostgreSQL move (loaded on use: it needs better-sqlite3). */--function importSqlite(opts) { return require('./sqlite-import').importSqlite(opts); }--/** A service's scripts/migrate-to-postgres.js in one call (./sqlite-cli.js). */--function runSqliteMigration(opts) { return require('./sqlite-cli').runSqliteMigration(opts); }----module.exports = { createDb, sql, DbError, importSqlite, runSqliteMigration, PARSERS, ISOLATION };++module.exports = { createDb, sql, DbError, PARSERS, ISOLATION };ddeleted file mode 100644@@@ -1,100 +0,0 @@--'use strict';--/**-- * The one-time move of a service's SQLite database into its PostgreSQL schema (ADR-035; the procedure is-- * docs/migrating-to-postgresql.md, section 6), shared by every service's scripts/migrate-to-postgres.js:-- *-- * const { runSqliteMigration } = require('openvibe-sdk/db');-- * runSqliteMigration({ service: 'wiki', sqlite: config.dbPath, directUrl: config.db.directUrl,-- * migrations: MIGRATIONS, tables: { new_name: { from: 'old_name' } } }).then((code) => process.exit(code));-- *-- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]-- *-- * 1. applies the migrations as the owner (directUrl, a direct connection); --pglite: an in-memory PostgreSQL-- * instead (a rehearsal with nothing to set up; the report is all it leaves);-- * 2. copies every table with importSqlite (batched, parents first, identities kept and sequences advanced)-- * into emptied tables (truncate: a rehearsal can be repeated);-- * 3. makes text PostgreSQL refuses storable, and reports it: a NUL character is dropped and an unpaired-- * surrogate becomes U+FFFD, in text and inside JSON values alike (SQLite took both);-- * 4. verifies each table's row count and a checksum of every row on both sides, prints the report, and-- * answers 1 unless report.ok. The SQLite file is opened read-only: nothing in it changes.-- * `tables` is importSqlite's per-table options (from, dropColumns, where, map), merged with the cleaning.-- */--const path = require('path');----const cleanText = (v) => v.toWellFormed().replace(/\u0000/g, '');--function cleanJson(v) {-- if (typeof v === 'string') return cleanText(v);-- if (Array.isArray(v)) return v.map(cleanJson);-- if (v && typeof v === 'object') return Object.fromEntries(Object.entries(v).map(([k, x]) => [cleanText(k), cleanJson(x)]));-- return v;--}----/** importSqlite's `tables` for the target schema: `tables`, plus a map per table making text storable. */--async function cleaningOptions(db, tables = {}) {-- const cols = await db.many(`SELECT table_name, column_name, udt_name FROM information_schema.columns-- WHERE table_schema = current_schema() AND table_name <> 'ov_migrations' AND udt_name IN ('text', 'varchar', 'json', 'jsonb')`);-- const byTable = new Map();-- for (const c of cols) { if (!byTable.has(c.table_name)) byTable.set(c.table_name, []); byTable.get(c.table_name).push(c); }-- const cleaned = new Map();-- const note = (t, c, v) => { const k = `${t}.${c}`; if (!cleaned.has(k)) cleaned.set(k, new Set()); cleaned.get(k).add(v); };-- const out = { ...tables };-- for (const [table, list] of byTable) {-- const base = tables[table] || {};-- out[table] = {-- ...base,-- map(row) {-- const r = base.map ? base.map(row) : row;-- if (!r) return r;-- for (const { column_name: c, udt_name: udt } of list) {-- const v = r[c];-- if (typeof v !== 'string') continue;-- if (udt === 'json' || udt === 'jsonb') {-- let parsed;-- try { parsed = JSON.parse(v); } catch { continue; } // left for the importer to report-- const fixed = JSON.stringify(cleanJson(parsed));-- if (fixed !== JSON.stringify(parsed)) { note(table, c, v); r[c] = fixed; }-- } else {-- const fixed = cleanText(v);-- if (fixed !== v) { note(table, c, v); r[c] = fixed; }-- }-- }-- return r;-- },-- };-- }-- return { tables: out, cleaned: () => [...cleaned].map(([column, values]) => ({ column, values: values.size })) };--}----async function runSqliteMigration({ service, sqlite, directUrl, migrations, tables = {}, skipSource = [], argv = process.argv.slice(2), out = console.log } = {}) {-- const { createDb, importSqlite } = require('./index');-- const opt = (name) => { const i = argv.indexOf(`--${name}`); return i >= 0 ? argv[i + 1] : null; };-- const flag = (name) => argv.includes(`--${name}`);-- const file = path.resolve(opt('sqlite') || sqlite || '');-- const quiet = { log() {}, warn: console.warn, error: console.error };-- let owner;-- if (flag('pglite')) {-- owner = createDb({ pglite: true, service: `${service}-import`, log: quiet });-- } else {-- if (!directUrl) throw new Error('DATABASE_DIRECT_URL is not set: the import runs as the owner on a direct connection');-- owner = createDb({ url: directUrl, service: `${service}-import`, max: 2, log: quiet });-- }-- try {-- await owner.migrate({ dir: migrations, log: quiet });-- const t0 = Date.now();-- const options = await cleaningOptions(owner, tables);-- const report = await importSqlite({ sqlite: file, db: owner, truncate: true, tables: options.tables, skipSource, log: quiet });-- const cleaned = options.cleaned();-- if (flag('json')) out(JSON.stringify({ sqlite: file, into: owner.store, ...report, cleaned }, null, 2));-- else {-- out(`import ${file} → ${owner.store} (${Date.now() - t0} ms): ${report.ok ? 'OK' : 'PROBLEMS'}`);-- for (const t of report.tables) out(` ${t.table.padEnd(30)} ${String(t.rows).padStart(8)} rows ${t.checksum || '-'}${t.source !== t.table ? ` (from ${t.source})` : ''}`);-- for (const c of cleaned) out(` cleaned: ${c.column}: ${c.values} value(s) with a NUL or an unpaired surrogate`);-- for (const p of report.problems) out(` problem: ${p.table}: ${p.problem}`);-- }-- return report.ok ? 0 : 1;-- } finally {-- await owner.close();-- }--}----module.exports = { runSqliteMigration, cleaningOptions, cleanText, cleanJson };ddeleted file mode 100644@@@ -1,283 +0,0 @@--'use strict';--/**-- * The one-time move of a service's SQLite data into its PostgreSQL schema (ADR-035 amendment, WS-X2 step 4).-- *-- * const { importSqlite } = require('openvibe-sdk/db');-- * const report = await importSqlite({ sqlite: '/opt/…/wiki.db', db: ownerDb, truncate: true });-- * if (!report.ok) throw new Error(JSON.stringify(report.problems));-- *-- * The target schema (created by the service's migrations) decides everything:-- * - which tables: every base table in `public` except ov_migrations that SQLite also has (or `tables`), copied-- * parents first (foreign-key order);-- * - which columns: those both sides have. A source column the target lacks is a problem unless the table lists-- * it in `dropColumns`; a target column the source lacks must have a default or be nullable;-- * - each value's conversion, by the target column's type: 0/1 → boolean, text or epoch → timestamptz (SQLite's-- * CURRENT_TIMESTAMP text is UTC), JSON text → jsonb (validated), JSON array text → PostgreSQL arrays,-- * Buffer → bytea, anything → text for text columns.-- * Rows go in batched multi-row INSERTs in one transaction (identity columns keep their values, then their-- * sequences move past the maximum). Verification compares each table's row count and a checksum of every row,-- * canonicalised the same way on both sides and walked in primary-key order (byte order for text keys).-- */--const crypto = require('crypto');--const { sql } = require('./sql');----function toIso(v, col) {-- if (v == null || v === '') return null;-- if (typeof v === 'number' || (typeof v === 'string' && /^\d{9,}(\.\d+)?$/.test(v))) {-- const n = Number(v);-- return new Date(n > 1e11 ? n : n * 1000).toISOString();-- }-- const s = String(v).trim();-- let d;-- if (/^\d{4}-\d{2}-\d{2}$/.test(s)) d = new Date(`${s}T00:00:00Z`);-- else if (/^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}(:\d{2}(\.\d+)?)?$/.test(s)) d = new Date(`${s.replace(' ', 'T')}Z`); // SQLite CURRENT_TIMESTAMP: UTC-- else d = new Date(s);-- if (Number.isNaN(d.getTime())) throw new Error(`${col}: "${s.slice(0, 40)}" is not a date/time`);-- return d.toISOString();--}----/** Convert one SQLite value for a PostgreSQL column type (information_schema udt_name). */--function convert(v, udt, col) {-- if (v === undefined || v === null) return null;-- if (udt.startsWith('_')) { // array-- if (Array.isArray(v)) return v;-- if (typeof v === 'string' && v.trim().startsWith('[')) { const a = JSON.parse(v); if (!Array.isArray(a)) throw new Error(`${col}: not a JSON array`); return a; }-- if (v === '') return [];-- throw new Error(`${col}: cannot read "${String(v).slice(0, 40)}" as an array (store it as JSON text)`);-- }-- switch (udt) {-- case 'bool':-- if (typeof v === 'boolean') return v;-- if (v === 1 || v === '1' || v === 'true' || v === 't') return true;-- if (v === 0 || v === '0' || v === 'false' || v === 'f') return false;-- throw new Error(`${col}: "${v}" is not a boolean`);-- case 'int2': case 'int4': case 'int8': {-- const n = typeof v === 'bigint' ? Number(v) : Number(v);-- if (!Number.isSafeInteger(n)) throw new Error(`${col}: "${v}" is not an integer`);-- return n;-- }-- case 'float4': case 'float8': { const n = Number(v); if (Number.isNaN(n)) throw new Error(`${col}: "${v}" is not a number`); return n; }-- case 'numeric': return String(v);-- case 'timestamptz': case 'timestamp': return toIso(v, col);-- case 'date': return String(v).slice(0, 10);-- case 'json': case 'jsonb':-- if (typeof v === 'string') { if (v === '') return null; JSON.parse(v); return v; }-- if (Buffer.isBuffer(v)) { const s = v.toString('utf8'); JSON.parse(s); return s; }-- return JSON.stringify(v);-- case 'bytea': return Buffer.isBuffer(v) ? v : Buffer.from(String(v));-- default: return Buffer.isBuffer(v) ? v.toString('utf8') : String(v);-- }--}----// The canonical text of a value for checksums (same rules on both sides).--function sortKeys(x) {-- if (Array.isArray(x)) return x.map(sortKeys);-- if (x && typeof x === 'object') return Object.keys(x).sort().reduce((o, k) => { o[k] = sortKeys(x[k]); return o; }, {});-- return x;--}--function canonical(v, udt) {-- if (v === null || v === undefined) return '\\N';-- if (udt === 'json' || udt === 'jsonb') {-- // JSON null and SQL NULL compare equal: a jsonb 'null' reads back as JS null, like a missing value.-- const parsed = typeof v === 'string' ? JSON.parse(v) : v;-- return parsed === null ? '\\N' : JSON.stringify(sortKeys(parsed));-- }-- if (udt === 'bytea') return Buffer.isBuffer(v) ? v.toString('hex') : Buffer.from(String(v)).toString('hex');-- if (udt === 'timestamptz' || udt === 'timestamp') return new Date(v).toISOString();-- if (udt.startsWith('_')) return JSON.stringify(v);-- if (udt === 'numeric') return String(Number(v));-- return String(v);--}----async function schemaOf(db) {-- const cols = await db.many(sql`-- SELECT c.table_name, c.column_name, c.udt_name, c.is_nullable = 'YES' AS nullable,-- c.column_default IS NOT NULL OR c.is_identity = 'YES' OR c.is_generated = 'ALWAYS' AS has_default,-- c.is_identity = 'YES' AS identity, c.is_generated = 'ALWAYS' AS generated-- FROM information_schema.columns c-- JOIN information_schema.tables t ON t.table_schema = c.table_schema AND t.table_name = c.table_name-- WHERE c.table_schema = current_schema() AND t.table_type = 'BASE TABLE' AND c.table_name <> 'ov_migrations'-- ORDER BY c.table_name, c.ordinal_position`);-- const pks = await db.many(sql`-- SELECT tc.relname AS table_name, a.attname AS column_name, array_position(i.indkey::int2[], a.attnum) AS pos-- FROM pg_index i JOIN pg_class tc ON tc.oid = i.indrelid JOIN pg_namespace n ON n.oid = tc.relnamespace-- JOIN pg_attribute a ON a.attrelid = tc.oid AND a.attnum = ANY(i.indkey)-- WHERE i.indisprimary AND n.nspname = current_schema() ORDER BY 1, 3`);-- const fks = await db.many(sql`-- SELECT c.conrelid::regclass::text AS child, c.confrelid::regclass::text AS parent-- FROM pg_constraint c JOIN pg_namespace n ON n.oid = c.connamespace-- WHERE c.contype = 'f' AND n.nspname = current_schema()`);-- const tables = new Map();-- for (const c of cols) {-- if (!tables.has(c.table_name)) tables.set(c.table_name, { name: c.table_name, columns: [], pk: [] });-- tables.get(c.table_name).columns.push(c);-- }-- for (const p of pks) if (tables.has(p.table_name)) tables.get(p.table_name).pk.push(p.column_name);-- // Parents first.-- const deps = new Map([...tables.keys()].map((t) => [t, new Set()]));-- for (const f of fks) { const ch = f.child.replace(/"/g, ''); const pa = f.parent.replace(/"/g, ''); if (deps.has(ch) && ch !== pa) deps.get(ch).add(pa); }-- const order = []; const state = new Map();-- const visit = (t) => { if (state.get(t) === 2) return; if (state.get(t) === 1) return; state.set(t, 1); for (const p of deps.get(t) || []) visit(p); state.set(t, 2); order.push(t); };-- for (const t of [...tables.keys()].sort()) visit(t);-- return order.map((t) => tables.get(t));--}----/**-- * @param {object} o-- * @param {string|object} o.sqlite a path, or an open better-sqlite3 Database-- * @param {object} o.db createDb() handle with write rights on the tables (the owner role)-- * @param {object} [o.tables] { [target]: { from?, dropColumns?: [], where?: 'SQL filter on the source', map?: (row) => row } }-- * @param {string[]} [o.only] import only these target tables-- * @param {boolean} [o.truncate] empty the target tables first (rehearsals)-- * @param {boolean} [o.verify] compare counts and checksums (default true)-- * @param {string[]} [o.skipSource] source tables deliberately not imported (every other source table must have a target)-- * → { ok, tables: [{ table, source, rows, ms, checksum }], problems: [] }-- */--async function importSqlite({ sqlite, db, tables: opts = {}, only = null, truncate = false, verify = true, skipSource = [], log = console } = {}) {-- let Database;-- const src = typeof sqlite === 'string' ? (() => { try { Database = require('better-sqlite3'); } catch { throw new Error('importSqlite: install better-sqlite3 to read the SQLite file'); } return new Database(sqlite, { readonly: true, fileMustExist: true }); })() : sqlite;-- const report = { ok: true, tables: [], problems: [] };-- const problem = (p) => { report.ok = false; report.problems.push(p); };-- try {-- const schema = (await schemaOf(db)).filter((t) => !only || only.includes(t.name));-- const srcTables = new Set(src.prepare("SELECT name FROM sqlite_master WHERE type = 'table'").all().map((r) => r.name));-- // Every source table must land somewhere: a table the target lacks would be dropped without a word (a table a-- // module created at runtime and the migration forgot). Skipped: SQLite's own, FTS5 virtual tables and their-- // shadow tables (an index is rebuilt, not copied), and skipSource.-- if (!only) {-- const virtuals = src.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND sql LIKE 'CREATE VIRTUAL TABLE%'").all().map((r) => r.name);-- const targets = new Set(schema.map((t) => (opts[t.name] && opts[t.name].from) || t.name));-- for (const name of srcTables) {-- if (targets.has(name) || name.startsWith('sqlite_') || skipSource.includes(name)) continue;-- if (virtuals.some((v) => name === v || name.startsWith(`${v}_`))) continue;-- problem({ table: name, problem: 'a source table with no target table (add it to the migrations, or list it in skipSource)' });-- }-- }-- const plan = [];-- for (const t of schema) {-- const o = opts[t.name] || {};-- const from = o.from || t.name;-- if (!srcTables.has(from)) { log.log && log.log(`[import] ${t.name}: no source table ${from} (left empty)`); continue; }-- const srcCols = src.prepare(`PRAGMA table_info("${from.replace(/"/g, '""')}")`).all().map((c) => c.name);-- const drop = new Set(o.dropColumns || []);-- const insertable = t.columns.filter((c) => !c.generated);-- const cols = insertable.filter((c) => srcCols.includes(c.column_name));-- const lost = srcCols.filter((c) => !t.columns.some((x) => x.column_name === c) && !drop.has(c));-- if (lost.length) problem({ table: t.name, problem: `source columns with no target column (list them in dropColumns to drop them): ${lost.join(', ')}` });-- const unfillable = insertable.filter((c) => !srcCols.includes(c.column_name) && !c.nullable && !c.has_default);-- if (unfillable.length) problem({ table: t.name, problem: `target columns the source cannot fill: ${unfillable.map((c) => c.column_name).join(', ')}` });-- plan.push({ t, from, cols, o });-- }-- if (!report.ok) return report;---- await db.tx(async (tx) => {-- if (truncate && plan.length) await tx.exec(sql`TRUNCATE ${sql.join(plan.map((p) => sql.ident(p.t.name)))} RESTART IDENTITY CASCADE`);-- for (const { t, from, cols, o } of plan) {-- const t0 = Date.now();-- const names = cols.map((c) => c.column_name);-- const hasIdentity = cols.some((c) => c.identity);-- const batch = Math.max(1, Math.min(1000, Math.floor(60000 / Math.max(1, names.length))));-- const stmt = src.prepare(`SELECT * FROM "${from.replace(/"/g, '""')}"${o.where ? ` WHERE ${o.where}` : ''}`);-- let rows = []; let n = 0;-- const flush = async () => {-- if (!rows.length) return;-- await tx.exec(sql`INSERT INTO ${sql.ident(t.name)} (${sql.join(names.map(sql.ident))}) ${hasIdentity ? sql`OVERRIDING SYSTEM VALUE ` : sql``}VALUES ${sql.join(rows.map((r) => sql`(${sql.join(r)})`))}`);-- n += rows.length; rows = [];-- };-- for (const raw of stmt.iterate()) {-- const row = o.map ? o.map({ ...raw }) : raw;-- if (!row) continue;-- rows.push(cols.map((c) => convert(row[c.column_name], c.udt_name, `${t.name}.${c.column_name}`)));-- if (rows.length >= batch) await flush();-- }-- await flush();-- for (const c of cols.filter((x) => x.identity || /nextval\(/.test(String(x.column_default || '')))) {-- await tx.query(sql`SELECT setval(pg_get_serial_sequence(${t.name}, ${c.column_name}), GREATEST((SELECT max(${sql.ident(c.column_name)}) FROM ${sql.ident(t.name)}), 1))`);-- }-- report.tables.push({ table: t.name, source: from, rows: n, ms: Date.now() - t0 });-- log.log && log.log(`[import] ${t.name}: ${n} rows in ${Date.now() - t0} ms`);-- }-- });---- if (verify) {-- for (const entry of report.tables) {-- const { t, from, cols, o } = plan.find((p) => p.t.name === entry.table);-- const result = await verifyTable({ src, db, t, from, cols, o });-- entry.checksum = result.pg.sum;-- if (result.sqlite.rows !== result.pg.rows || result.sqlite.sum !== result.pg.sum) {-- problem({ table: t.name, problem: `verification failed: SQLite ${result.sqlite.rows} rows (${result.sqlite.sum}), PostgreSQL ${result.pg.rows} rows (${result.pg.sum})` });-- }-- }-- }-- return report;-- } finally {-- if (typeof sqlite === 'string') src.close();-- }--}----/**-- * Stream both sides in primary-key order and hash each canonical row: SQLite iterates ORDER BY the key (its-- * BINARY collation is byte order, as COLLATE "C"), PostgreSQL reads keyset pages of 5000 (OFFSET pages for a-- * composite key). A table without a primary key is sorted in memory by every column.-- */--async function verifyTable({ src, db, t, from, cols, o }) {-- const names = cols.map((c) => c.column_name);-- const keys = t.pk.filter((k) => names.includes(k));-- const q = (x) => `"${String(x).replace(/"/g, '""')}"`;-- const text = new Set(cols.filter((c) => ['text', 'varchar', 'bpchar', 'citext', 'uuid'].includes(c.udt_name)).map((c) => c.column_name));-- const line = (vals) => vals.map((v, i) => canonical(v, cols[i].udt_name)).join('\t') + '\n';-- const convertRow = (row) => cols.map((c) => convert(row[c.column_name], c.udt_name, c.column_name));---- const h1 = crypto.createHash('sha256'); let c1 = 0;-- const where = o.where ? ` WHERE ${o.where}` : '';-- if (keys.length) {-- for (const raw of src.prepare(`SELECT * FROM ${q(from)}${where} ORDER BY ${keys.map(q).join(', ')}`).iterate()) {-- const row = o.map ? o.map({ ...raw }) : raw;-- if (!row) continue;-- h1.update(line(convertRow(row))); c1++;-- }-- } else {-- const all = src.prepare(`SELECT * FROM ${q(from)}${where}`).all().map((raw) => (o.map ? o.map({ ...raw }) : raw)).filter(Boolean).map(convertRow);-- all.sort((a, b) => cmpRows(a, b, names.map((_, i) => i)));-- for (const r of all) { h1.update(line(r)); c1++; }-- }---- const h2 = crypto.createHash('sha256'); let c2 = 0;-- const colList = sql.join(names.map(sql.ident));-- const orderBy = (ks) => sql.join(ks.map((k) => (text.has(k) ? sql`${sql.ident(k)} COLLATE "C"` : sql.ident(k))));-- const PAGE = 5000;-- if (keys.length === 1) {-- const k = keys[0]; let last = null;-- for (;;) {-- const cond = last === null ? sql`` : (text.has(k) ? sql`WHERE ${sql.ident(k)} COLLATE "C" > ${last}` : sql`WHERE ${sql.ident(k)} > ${last}`);-- const rows = await db.many(sql`SELECT ${colList} FROM ${sql.ident(t.name)} ${cond} ORDER BY ${orderBy([k])} LIMIT ${PAGE}`);-- for (const r of rows) { h2.update(line(names.map((n) => r[n]))); c2++; }-- if (rows.length < PAGE) break;-- last = rows[rows.length - 1][k];-- }-- } else {-- const ks = keys.length ? keys : names;-- for (let off = 0; ; off += PAGE) {-- const rows = await db.many(sql`SELECT ${colList} FROM ${sql.ident(t.name)} ORDER BY ${orderBy(ks)} LIMIT ${PAGE} OFFSET ${off}`);-- for (const r of rows) { h2.update(line(names.map((n) => r[n]))); c2++; }-- if (rows.length < PAGE) break;-- }-- }-- return { sqlite: { rows: c1, sum: h1.digest('hex').slice(0, 16) }, pg: { rows: c2, sum: h2.digest('hex').slice(0, 16) } };--}----/** Order rows the way PostgreSQL does with COLLATE "C" (byte order for text, numeric for numbers). */--function cmpRows(a, b, idx) {-- for (const i of idx) {-- const x = a[i]; const y = b[i];-- if (x === y) continue;-- if (x === null) return 1; if (y === null) return -1; // NULLS LAST, PostgreSQL's default for ASC-- if (typeof x === 'number' && typeof y === 'number') return x - y;-- const bx = Buffer.from(String(x)); const by = Buffer.from(String(y));-- const c = Buffer.compare(bx, by);-- if (c) return c;-- }-- return 0;--}----module.exports = { importSqlite, convert, canonical };@@@ -3,7 +3,7 @@* db (ADR-035): the sql builder binds every value; both adapters return the same row shapes (int8 → Number,* timestamps → ISO, dates → 'YYYY-MM-DD', numeric → exact text, json parsed); transactions commit, roll back* and nest as savepoints; serializable conflicts retry; migrations run once, in order, refuse edits, and hold a-- * contract migration inside the N-1 window; importSqlite copies a SQLite file into the schema parents first,++ * contract migration inside the N-1 window;* converts by target type, keeps identity values, and verifies counts and checksums. The suite runs on PGlite* always, and again through PgBouncer against PostgreSQL 18 when OV_TEST_PG_URL is set* (scripts/test-services.sh up).@@@ -13,7 +13,7 @@ const fs = require('node:fs');const os = require('node:os');const path = require('node:path');const { run } = require('./helpers');--const { createDb, sql, importSqlite } = require('../src/db');++const { createDb, sql } = require('../src/db');const { parse } = require('../src/db/migrate');const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ovsdk-db-'));@@@ -132,61 +132,6 @@ const tests = [await assert.rejects(db2.migrate({ dir: d2, log: quiet }), /older than applied migration 0005/);} finally { await db2.close(); }}],-- ['importSqlite: parents first, conversions by target type, identities kept, verified', async () => {-- const Database = require('better-sqlite3');-- const file = path.join(tmp, 'legacy.db');-- const s = new Database(file);-- s.exec(`CREATE TABLE people (id INTEGER PRIMARY KEY AUTOINCREMENT, username TEXT NOT NULL, is_admin INTEGER DEFAULT 0, settings TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP, legacy_note TEXT);-- CREATE TABLE notes (id INTEGER PRIMARY KEY, person_id INTEGER REFERENCES people(id), body TEXT, pinned INTEGER, posted INTEGER, tags TEXT);-- CREATE TABLE loose (k TEXT, v TEXT);-- CREATE TABLE forgotten (id INTEGER PRIMARY KEY, secret TEXT);-- CREATE VIRTUAL TABLE docs_fts USING fts5(title);`);-- const ins = s.prepare('INSERT INTO people (id, username, is_admin, settings, created_at, legacy_note) VALUES (?, ?, ?, ?, ?, ?)');-- ins.run(3, 'ann', 1, '{"theme":"blue","n":[1,2]}', '2026-09-01 10:00:00', 'drop me');-- ins.run(7, 'bob', 0, null, '2026-09-02T11:30:00.000Z', null);-- ins.run(12, 'zoë', 0, '{}', '2026-09-03 09:15:00', null);-- s.prepare('INSERT INTO notes VALUES (?, ?, ?, ?, ?, ?)').run(1, 3, 'hello', 1, 1790000000, '["a","b"]');-- s.prepare('INSERT INTO notes VALUES (?, ?, ?, ?, ?, ?)').run(2, 7, 'ms time', 0, 1790000000123, '[]');-- s.prepare('INSERT INTO loose VALUES (?, ?)').run('b', '2');-- s.prepare('INSERT INTO loose VALUES (?, ?)').run('a', '1');-- s.close();---- const db = createDb({ pglite: true });-- try {-- await db.migrate({ dir: mdir('imp', { '0001_s.sql': `-- phase: expand--CREATE TABLE people (id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, username text NOT NULL, is_admin boolean NOT NULL DEFAULT false, settings jsonb, created_at timestamptz NOT NULL DEFAULT now());--CREATE TABLE notes (id bigint PRIMARY KEY, person_id bigint REFERENCES people(id), body text, pinned boolean, posted timestamptz, tags text[] NOT NULL DEFAULT '{}');--CREATE TABLE loose (k text, v text);` }), log: { log() {} } });-- const quiet = { log() {} };-- // A source column with no target column is a problem until it is dropped on purpose.-- let r = await importSqlite({ sqlite: file, db, log: quiet });-- assert.equal(r.ok, false);-- assert.ok(r.problems.some((p) => /legacy_note/.test(p.problem)), JSON.stringify(r.problems));-- r = await importSqlite({ sqlite: file, db, log: quiet, tables: { people: { dropColumns: ['legacy_note'] } } });-- // A source table the target lacks would be dropped without a word: refused until skipped on purpose-- // (an FTS5 index and its shadow tables are skipped: an index is rebuilt, not copied).-- assert.equal(r.ok, false);-- assert.deepEqual(r.problems.map((p) => p.table), ['forgotten']);-- r = await importSqlite({ sqlite: file, db, log: quiet, truncate: true, skipSource: ['forgotten'], tables: { people: { dropColumns: ['legacy_note'] } } });-- assert.equal(r.ok, true, JSON.stringify(r.problems));-- assert.deepEqual(r.tables.map((t) => [t.table, t.rows]), [['loose', 2], ['people', 3], ['notes', 2]], 'parents before children');-- const ann = await db.one(sql`SELECT * FROM people WHERE id = 3`);-- assert.deepEqual([ann.username, ann.is_admin, ann.settings, ann.created_at], ['ann', true, { theme: 'blue', n: [1, 2] }, '2026-09-01T10:00:00.000Z']);-- const n2 = await db.one(sql`SELECT * FROM notes WHERE id = 2`);-- assert.deepEqual([n2.pinned, n2.posted, n2.tags], [false, new Date(1790000000123).toISOString(), []]);-- assert.equal((await db.one(sql`SELECT * FROM notes WHERE id = 1`)).posted, new Date(1790000000 * 1000).toISOString(), 'epoch seconds');-- const next = await db.one(sql`INSERT INTO people (username) VALUES ('new') RETURNING id`);-- assert.equal(next.id, 13, 'the identity continues after the imported maximum');-- // A second import onto the same rows fails loudly; truncate makes a rehearsal repeatable.-- r = await importSqlite({ sqlite: file, db, log: quiet, truncate: true, skipSource: ['forgotten'], tables: { people: { dropColumns: ['legacy_note'] } } });-- assert.equal(r.ok, true);-- // A transformation that differs between copy and check shows up as a failed verification.-- let flip = 0;-- r = await importSqlite({ sqlite: file, db, log: quiet, truncate: true, only: ['loose'], tables: { loose: { map: (row) => ({ ...row, v: String(flip++) }) } } });-- assert.equal(r.ok, false);-- assert.match(r.problems[0].problem, /verification failed/);-- } finally { await db.close(); }-- }],];if (process.env.OV_TEST_PG_URL && process.env.OV_TEST_PG_DIRECT_URL) tests.push(...suite('postgresql+pgbouncer', pgOpen));ddeleted file mode 100644@@@ -1,58 +0,0 @@--'use strict';--/** runSqliteMigration: migrate, import, clean what PostgreSQL refuses (NUL, unpaired surrogates) and report it,-- * verify, answer 0; a column the schema lacks is a problem and answers 1; --json prints the report. */--const assert = require('node:assert/strict');--const fs = require('fs');--const os = require('os');--const path = require('path');--const { run } = require('./helpers');--const { runSqliteMigration } = require('../src/db');----let Database;--try { Database = require('better-sqlite3'); } catch { Database = null; }----const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sdk-sqlitecli-'));--const migrations = path.join(dir, 'migrations');--fs.mkdirSync(migrations);--fs.writeFileSync(path.join(migrations, '0001_initial.sql'), `-- phase: expand--CREATE TABLE notes (id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, body text NOT NULL, meta jsonb, created_at bigint NOT NULL);--CREATE TABLE renamed (id text PRIMARY KEY, n integer NOT NULL);--`);----function sqliteFile(extraColumn = false) {-- const file = path.join(dir, `src-${extraColumn ? 'x' : 'ok'}.db`);-- const db = new Database(file);-- db.exec(`CREATE TABLE notes (id INTEGER PRIMARY KEY AUTOINCREMENT, body TEXT NOT NULL, meta TEXT, created_at INTEGER NOT NULL${extraColumn ? ', stray TEXT' : ''});-- CREATE TABLE old_name (id TEXT PRIMARY KEY, n INTEGER NOT NULL);`);-- const ins = db.prepare('INSERT INTO notes (body, meta, created_at) VALUES (?, ?, ?)');-- ins.run('plain', '{"a":1}', 1);-- ins.run('with\u0000nul', '{"k":"bad \\ud800 surrogate"}', 2);-- ins.run('none', null, 3);-- db.prepare('INSERT INTO old_name (id, n) VALUES (?, ?)').run('x', 7);-- db.close();-- return file;--}----const tests = [];--if (Database) {-- tests.push(['imports, cleans and reports, verifies and answers 0', async () => {-- const lines = [];-- const code = await runSqliteMigration({ service: 'sdk', sqlite: sqliteFile(), migrations, tables: { renamed: { from: 'old_name' } }, argv: ['--pglite'], out: (l) => lines.push(l) });-- assert.equal(code, 0, lines.join('\n'));-- assert.match(lines[0], /→ pglite .*: OK/);-- assert.ok(lines.some((l) => /notes .* 3 rows/.test(l)));-- assert.ok(lines.some((l) => /renamed .* 1 rows .*\(from old_name\)/.test(l)));-- assert.ok(lines.some((l) => /cleaned: notes\.body: 1 value/.test(l)), lines.join('\n'));-- assert.ok(lines.some((l) => /cleaned: notes\.meta: 1 value/.test(l)), lines.join('\n'));-- }]);-- tests.push(['a column the schema lacks is a problem: answers 1; --json prints the report', async () => {-- const lines = [];-- const code = await runSqliteMigration({ service: 'sdk', sqlite: sqliteFile(true), migrations, tables: { renamed: { from: 'old_name' } }, argv: ['--pglite', '--json'], out: (l) => lines.push(l) });-- assert.equal(code, 1);-- const report = JSON.parse(lines.join('\n'));-- assert.equal(report.ok, false);-- assert.ok(report.problems.some((p) => p.table === 'notes' && /stray/.test(p.problem)), JSON.stringify(report.problems));-- }]);--} else console.log('runSqliteMigration: skipped (better-sqlite3 not installed)');----run(tests);ddeleted file mode 100644@@@ -1,66 +0,0 @@--# asyncify: converting a service from better-sqlite3 to openvibe-sdk/db----This directory holds the tools used to convert Wiki and Blog (2026-09-28). They are not part of the published package.----```--cd tools/asyncify && npm install--node asyncify.js --config my-service.json $(git -C <service> ls-files 'server/*.js' 'server/**/*.js') <service>/test/*.test.js--```----## asyncify.js----It parses every file with acorn and does the following:----1. It adds `await` to calls that now return a promise:-- - prepared statements (`db.prepare(…).get/all/run`, `q.<name>.…`, and any name assigned from `.prepare(…)`);-- - Publishing store methods reached as `<x>.<store>.<method>(`;-- - `store.tx(`;-- - `asyncMethods` and `asyncGlobals` from the config;-- - methods of module APIs (`apis`: a variable name → the file whose functions became async).--2. It makes the enclosing functions `async`.--3. It repeats across all the files until nothing changes.----It leaves some calls unawaited:--- A promise used as a value (`.then`, `.catch`, `.finally`, `Promise.all/race([...])`, `assert.rejects(p)`) stays a promise.--- A call inside an array callback (`map`, `filter`, `forEach`, …) is **reported** (`MANUAL …`), never changed, because an async callback there silently breaks the result. The same goes for an async function passed by name to an array method.----Fix those by hand, with a `for … of` loop, `await Promise.all(xs.map(async …))`, or better a batch read (`revisions.getMany`, `citations.forRevisions`, `reviews.latestMany`, or `= ANY(?)`).----It also rewrites `assert.throws(async …)` as `await assert.rejects(async …)`. An edit that would break the syntax is refused and reported.----The config has these keys: `stores`, `bareStores`, `syncStoreMethods`, `apis`, `syncNames`, `asyncGlobals` and `asyncMethods`. See `example-blog.json`.----What stays for a person:--- the SQL dialect: `INSERT OR IGNORE/REPLACE` → `ON CONFLICT`, `COLLATE NOCASE` → `lower()`, `LIKE` → `ILIKE`, `IS ?` → `IS NOT DISTINCT FROM`, `DISTINCT` + `ORDER BY`, `PRAGMA` and `sqlite_master` → `information_schema`;--- `RETURNING` in place of `lastInsertRowid`;--- the boot (open, migrate, `createApp` async) and the test helpers (`openvibe-sdk/testing` `createTestDb`).----Also by hand: `db.transaction(fn)()` → `db.tx(fn)` (the codemod leaves it), scalar `MAX(a, b)` → `GREATEST`, `rowid` tiebreaks → an identity `seq` column, and a `setImmediate` meant to run after a commit → `db.afterCommit`. The lessons list in `docs/migrating-to-postgresql.md` §7 has the rest.----## The order for a Blog-shaped service----```--git checkout -b postgres--python3 g1pass.py <svc> <Name> <ENV> # package pins, config, outbox, limits, readiness, boot, CI, unit, env, import script--npm install--node gen-migration.js <Name> --publishing … # migrations/0001_initial.sql from the SQLite SCHEMA--python3 g1db.py <svc> <Name> # server/db.js: openDb / createStore / openStore--python3 sqlfix.py # the SQL rewrites (db.transaction(fn)() → db.tx(fn) too), and CHECK lines--node asyncify.js --config <svc>.json … # then await everything (db.tx included); wraps .map(async …) in Promise.all--npm test && npm run test:pg # then rehearse: scripts/migrate-to-postgres.js --sqlite <copy> --pglite--python3 g1docs.py <svc> <Name> <ENV> # README and STATUS.json--```----`asyncify.js` now turns `xs.map(cb)` with a callback that becomes async (or an async function passed by name) into `(await Promise.all(xs.map(…)))`, recursively outwards. It reports `.map(async …)` that is not awaited as a whole, and callbacks of other array methods.----## sqlite-schema-to-pg.js----`toPg(sqliteDdl)` turns the SCHEMA string a service's `db.js` ran into the first migration: `INTEGER PRIMARY KEY AUTOINCREMENT` → identity, `TEXT` → `text COLLATE "C"`, `INTEGER` → `bigint`, `REAL` → `double precision`, `BLOB` → `bytea`, and no `IF NOT EXISTS`. An index on `x COLLATE NOCASE` is on `lower(x)`. An immutable-row trigger (`BEFORE UPDATE [OF …]/DELETE … RAISE(ABORT, msg)`) becomes PL/pgSQL. Any other trigger is refused. `gen-migration.js` runs it on `server/db.js` at HEAD (it evaluates a SCHEMA with `${…}`) and adds the Publishing and SDK tables.----## g1pass.py----`g1pass.py <svc> <Name> <ENVPREFIX>`, run in a Blog-shaped service's checkout, makes the edits that are the same in every such service: config (`DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`), the PostgreSQL outbox, Valkey actor limits, readiness, and the test boot on `createTestDb`. Each edit checks its anchor and is skipped when already applied.----## awaitify-tests.py----This is the earlier regex pass for test files (Wiki). `asyncify.js` over the server and the tests together supersedes it.ddeleted file mode 100644@@@ -1,435 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * asyncify: the mechanical half of moving a service from better-sqlite3 to openvibe-sdk/db (ADR-035).-- *-- * node asyncify.js --config asyncify.json file.js ...-- *-- * config: {-- * "stores": ["revisions", ...], // publishing store property names reached as <x>.<store>.<method>(-- * "syncStoreMethods": ["..."], // methods of those stores that stay synchronous-- * "apis": { "blogs": "server/domain/blogs.js", ... }, // variable/property names that hold a module's API object-- * "syncNames": ["validHandle", ...], // functions that stay synchronous even when their module changes-- * "asyncGlobals": ["tx"] // extra callee names that are async (e.g. a local tx helper)-- * }-- *-- * Async calls: <stmt>.get/.all/.run(...) where <stmt> is db.prepare(...)/<x>.prepare(...)/q.<name>/<x>.q.<name>;-- * <x>.<store>.<method>(...); store.tx(...) / db.tx(...); calls to functions of these files that became async-- * (fixpoint, across files through `apis`). Each gets `await` (parenthesised when a member access follows), its-- * enclosing function becomes async. A function passed to map/filter/forEach/some/every/find/reduce/sort that would-- * become async is NOT changed: it is reported for a hand fix (an async callback there silently breaks semantics).-- */--const fs = require('fs');--const path = require('path');--const acorn = require('acorn');--const walk = require('acorn-walk');----const ARRAY_CB = new Set(['map', 'filter', 'forEach', 'some', 'every', 'find', 'findIndex', 'reduce', 'sort', 'flatMap']);----function parse(src) {-- return acorn.parse(src, { ecmaVersion: 'latest', sourceType: 'script', allowHashBang: true, allowReturnOutsideFunction: true, locations: true });--}----function memberPath(node) {-- const parts = [];-- let n = node;-- while (n) {-- if (n.type === 'MemberExpression' && !n.computed && n.property.type === 'Identifier') { parts.unshift(n.property.name); n = n.object; continue; }-- if (n.type === 'Identifier') { parts.unshift(n.name); break; }-- if (n.type === 'ThisExpression') { parts.unshift('this'); break; }-- if (n.type === 'CallExpression') { parts.unshift('()'); n = n.callee; continue; }-- return parts.length ? ['?', ...parts] : null;-- }-- return parts;--}----function functionName(fn, parent) {-- if (fn.id) return fn.id.name;-- if (parent && parent.type === 'VariableDeclarator' && parent.id.type === 'Identifier') return parent.id.name;-- if (parent && parent.type === 'Property' && parent.key) return parent.key.name || parent.key.value;-- if (parent && parent.type === 'AssignmentExpression' && parent.left.type === 'MemberExpression') return parent.left.property.name;-- return null;--}----function analyse(file, src, cfg, asyncNames) {-- const ast = parse(src);-- const ancestors = new Map();-- // parent links-- walk.fullAncestor(ast, (node, _state, anc) => { ancestors.set(node, anc.slice(0, -1)); });-- const isFn = (n) => n.type === 'FunctionDeclaration' || n.type === 'FunctionExpression' || n.type === 'ArrowFunctionExpression';-- const stores = new Set(cfg.stores || []);-- const syncStoreMethods = new Set(cfg.syncStoreMethods || []);-- const apis = { ...(cfg.apis || {}) };-- const syncNames = new Set(cfg.syncNames || []);-- const localAsync = new Set(asyncNames.local.get(file) || []);-- // const store = require('./store'): in this file, `store` is that module's API (the same name can mean another-- // module elsewhere, so a namespace require wins over the config's global apis entry).-- walk.full(parse(src), (n) => {-- if (n.type !== 'VariableDeclarator' || n.id.type !== 'Identifier' || !n.init || n.init.type !== 'CallExpression') return;-- const c = n.init;-- if (c.callee.type !== 'Identifier' || c.callee.name !== 'require' || !c.arguments[0] || typeof c.arguments[0].value !== 'string' || !c.arguments[0].value.startsWith('.')) return;-- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), c.arguments[0].value));-- const target = base.endsWith('.js') ? base : fs.existsSync(`${base}.js`) ? `${base}.js` : path.join(base, 'index.js');-- if (fs.existsSync(target)) apis[n.id.name] = target;-- });-- // const svc = createPasteService(…) with createPasteService imported from a local file (destructured): `svc` is that-- // file's API here (a factory's instance; its methods are the file's inner functions).-- {-- const imported = new Map();-- const tree = parse(src);-- walk.full(tree, (n) => {-- if (n.type !== 'VariableDeclarator' || n.id.type !== 'ObjectPattern' || !n.init || n.init.type !== 'CallExpression') return;-- const c = n.init;-- if (c.callee.type !== 'Identifier' || c.callee.name !== 'require' || !c.arguments[0] || typeof c.arguments[0].value !== 'string' || !c.arguments[0].value.startsWith('.')) return;-- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), c.arguments[0].value));-- const target = base.endsWith('.js') ? base : fs.existsSync(`${base}.js`) ? `${base}.js` : path.join(base, 'index.js');-- for (const prop of n.id.properties) {-- if (prop.type === 'Property' && prop.value && prop.value.type === 'Identifier') imported.set(prop.value.name, target);-- }-- });-- const factoryOf = (init) => {-- let x = init;-- if (x && x.type === 'AwaitExpression') x = x.argument;-- if (x && x.type === 'LogicalExpression') x = x.right; // opts.relay || createDiscordRelay(…)-- if (x && x.type === 'AwaitExpression') x = x.argument;-- return x && x.type === 'CallExpression' && x.callee.type === 'Identifier' && /^create[A-Z]/.test(x.callee.name) ? imported.get(x.callee.name) : null;-- };-- walk.full(tree, (n) => {-- const pairs = n.type === 'VariableDeclarator' && n.id.type === 'Identifier' ? [[n.id.name, n.init]]-- : n.type === 'AssignmentExpression' && n.left.type === 'Identifier' ? [[n.left.name, n.right]] : [];-- for (const [name, init] of pairs) { const f = factoryOf(init); if (f && fs.existsSync(f)) apis[name] = f; }-- });-- }-- // fileApis: { "server/comments/api.js": { "service": "server/comments/service.js" } } for a holder whose module-- // differs by file (a factory's instance passed in under a generic name).-- Object.assign(apis, (cfg.fileApis || {})[path.relative(process.cwd(), file)] || {});-- // const { seed } = require('./workflows/seed'): a function imported from a converted file is async when it is there.-- walk.full(ast, (n) => {-- if (n.type !== 'VariableDeclarator' || n.id.type !== 'ObjectPattern' || !n.init || n.init.type !== 'CallExpression') return;-- const c = n.init;-- if (c.callee.type !== 'Identifier' || c.callee.name !== 'require' || !c.arguments[0] || typeof c.arguments[0].value !== 'string' || !c.arguments[0].value.startsWith('.')) return;-- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), c.arguments[0].value));-- const set = asyncNames.api.get(base.endsWith('.js') ? base : `${base}.js`) || asyncNames.api.get(path.join(base, 'index.js'));-- if (!set) return;-- for (const prop of n.id.properties) {-- if (prop.type !== 'Property' || !prop.key) continue;-- const exported = prop.key.name || prop.key.value;-- const local = prop.value && prop.value.type === 'Identifier' ? prop.value.name : exported;-- if (set.has(exported) && !syncNames.has(exported)) localAsync.add(local);-- }-- });-- // const view = ops.providerView: an alias of an async module method is async too.-- walk.full(ast, (n) => {-- if (n.type !== 'VariableDeclarator' || n.id.type !== 'Identifier' || !n.init || n.init.type !== 'MemberExpression' || n.init.computed) return;-- const holder = n.init.object.type === 'Identifier' ? n.init.object.name : null;-- const set = holder && apis[holder] ? asyncNames.api.get(apis[holder]) : null;-- if (set && set.has(n.init.property.name) && !syncNames.has(n.init.property.name)) localAsync.add(n.id.name);-- });---- // Names that hold a prepared statement: const x = db.prepare(...), or { x: db.prepare(...) }.-- const stmtNames = new Set();-- const isPrepare = (n) => n && n.type === 'CallExpression' && n.callee.type === 'MemberExpression' && n.callee.property.name === 'prepare';-- walk.full(ast, (n) => {-- if (n.type === 'VariableDeclarator' && n.id.type === 'Identifier' && isPrepare(n.init)) stmtNames.add(n.id.name);-- if (n.type === 'Property' && n.key && isPrepare(n.value)) stmtNames.add(n.key.name || n.key.value);-- });---- // Higher-order helpers (conflictGuard(fn), waitFor(fn), once(id, fn), a route wrapper): when a local function is-- // passed an async function, its calls of that parameter return promises, so they are awaited too.-- const localFns = new Map();-- walk.full(ast, (n) => {-- if (n.type === 'FunctionDeclaration' && n.id) localFns.set(n.id.name, n);-- if (n.type === 'VariableDeclarator' && n.id.type === 'Identifier' && n.init && ['FunctionExpression', 'ArrowFunctionExpression'].includes(n.init.type)) localFns.set(n.id.name, n.init);-- });-- const asyncParamCalls = new Set();-- walk.full(ast, (n) => {-- if (n.type !== 'CallExpression' || n.callee.type !== 'Identifier' || !localFns.has(n.callee.name)) return;-- const target = localFns.get(n.callee.name);-- n.arguments.forEach((arg, i) => {-- const isAsyncFn = ['FunctionExpression', 'ArrowFunctionExpression'].includes(arg.type) && arg.async;-- const p = target.params[i];-- if (!isAsyncFn || !p || p.type !== 'Identifier') return;-- walk.full(target.body, (c) => { if (c.type === 'CallExpression' && c.callee.type === 'Identifier' && c.callee.name === p.name) asyncParamCalls.add(c); });-- });-- });---- // Lazy accessors: function objects() { return require('../objects/model'); } / const model = () => require('./model').-- const lazyAccessors = new Map();-- {-- const resolveRel = (lit) => {-- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), lit));-- return base.endsWith('.js') ? base : fs.existsSync(`${base}.js`) ? `${base}.js` : path.join(base, 'index.js');-- };-- const reqOf = (n) => (n && n.type === 'CallExpression' && n.callee.type === 'Identifier' && n.callee.name === 'require' && n.arguments[0]-- && typeof n.arguments[0].value === 'string' && n.arguments[0].value.startsWith('.') ? resolveRel(n.arguments[0].value) : null);-- const bodyReq = (f) => {-- if (!f || f.params.length) return null;-- if (f.body.type !== 'BlockStatement') return reqOf(f.body);-- const st = f.body.body;-- return st.length === 1 && st[0].type === 'ReturnStatement' ? reqOf(st[0].argument) : null;-- };-- for (const [name, f] of localFns) { const t = bodyReq(f); if (t && fs.existsSync(t)) lazyAccessors.set(name, t); }-- // const m = objectsModel(): m is that module's API here.-- walk.full(ast, (n) => {-- if (n.type !== 'VariableDeclarator' || n.id.type !== 'Identifier' || !n.init || n.init.type !== 'CallExpression') return;-- const c = n.init;-- if (c.callee.type === 'Identifier' && lazyAccessors.has(c.callee.name) && !c.arguments.length) apis[n.id.name] = lazyAccessors.get(c.callee.name);-- });-- }---- // Local functions that await a parameter themselves: rejects(p, …) → 'rejects:0'.-- const promiseTakers = new Set();-- for (const [name, f] of localFns) {-- const params = f.params.map((q) => (q.type === 'Identifier' ? q.name : null));-- walk.full(f.body, (x) => {-- if (x.type === 'AwaitExpression' && x.argument.type === 'Identifier' && params.includes(x.argument.name)) promiseTakers.add(`${name}:${params.indexOf(x.argument.name)}`);-- });-- }---- function asyncCall(call) {-- const c = call.callee;-- if (c.type === 'MemberExpression' && !c.computed) {-- const m = c.property.name;-- const obj = c.object;-- const p = memberPath(obj) || [];-- if (['get', 'all', 'run'].includes(m)) {-- if (obj.type === 'CallExpression' && obj.callee.type === 'MemberExpression' && ['prepare', 'pluck'].includes(obj.callee.property.name)) return true;-- if (p.length >= 2 && (p[p.length - 2] === 'q' || p[p.length - 2] === 'stmts' || p[p.length - 2] === 'Q')) return true;-- if (p.length && stmtNames.has(p[p.length - 1])) return true;-- }-- // x.tx(fn): openvibe-sdk/db's transaction on any handle (db, store.db, db.getDb(), a destructured one).-- if (m === 'tx' && !syncNames.has('tx')) return true;-- if (m === 'transaction' && p.length && ['store', 'db', 'this'].includes(p[p.length - 1])) return false;-- // objects().safeSync(: a local lazy accessor (function objects() { return require('../objects/model'); }).-- if (obj.type === 'CallExpression' && obj.callee.type === 'Identifier' && lazyAccessors.has(obj.callee.name) && obj.arguments.length === 0) {-- const set = asyncNames.api.get(lazyAccessors.get(obj.callee.name));-- if (set && set.has(m) && !syncNames.has(m)) return true;-- }-- // require('./events').pasteCreated(: a module's function reached through an inline require.-- if (obj.type === 'CallExpression' && obj.callee.type === 'Identifier' && obj.callee.name === 'require' && obj.arguments[0] && typeof obj.arguments[0].value === 'string' && obj.arguments[0].value.startsWith('.')) {-- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), obj.arguments[0].value));-- const set = asyncNames.api.get(base.endsWith('.js') ? base : `${base}.js`) || asyncNames.api.get(path.join(base, 'index.js'));-- if (set && set.has(m) && !syncNames.has(m)) return true;-- }-- // <x>.<store>.<method>(: a store reached through an object; a bare identifier only when listed in bareStores.-- if (p.length >= 2 && stores.has(p[p.length - 1]) && !syncStoreMethods.has(m)) return true;-- if (p.length === 1 && (cfg.bareStores || []).includes(p[0]) && !syncStoreMethods.has(m)) return true;-- if ((cfg.asyncMethods || []).includes(m)) return true;-- // <api>.<method>( across modules-- const holder = p[p.length - 1];-- if (apis[holder]) {-- const set = asyncNames.api.get(apis[holder]);-- if (set && set.has(m) && !syncNames.has(m)) return true;-- }-- if (p.length && (p[p.length - 1] === 'api' || p[p.length - 1] === 'svc' || p[p.length - 1] === 'self') && localAsync.has(m)) return true;-- }-- if (asyncParamCalls.has(call)) return true;-- if (c.type === 'Identifier') {-- if ((cfg.asyncGlobals || []).includes(c.name)) return true;-- if (localAsync.has(c.name) && !syncNames.has(c.name)) return true;-- }-- return false;-- }---- const edits = []; // { at, text } insertions-- const manual = [];-- const becameAsync = new Set();-- const newlyAsyncNames = new Set();-- const fnsToAsync = new Set();-- const wrappedMaps = new Set();-- const isFnN = (x) => x && (x.type === 'FunctionDeclaration' || x.type === 'FunctionExpression' || x.type === 'ArrowFunctionExpression');-- const inPromiseArg = (call) => { const p = ancestors.get(call) || []; const up = p[p.length - 1]; return up && up.type === 'CallExpression' && up.arguments[0] === call && up.callee.type === 'MemberExpression' && up.callee.object.name === 'Promise'; };-- /** fn will contain an await: make it async; when it is a .map() callback, wrap that map in (await Promise.all(…)) and-- * make its own enclosing function async too (recursively); another array method's callback needs a person. */-- function makeAsync(fn) {-- if (!fn || fnsToAsync.has(fn)) return;-- fnsToAsync.add(fn);-- const anc = ancestors.get(fn) || [];-- const call = anc[anc.length - 1];-- if (!(call && call.type === 'CallExpression' && call.arguments.includes(fn) && call.callee.type === 'MemberExpression' && ARRAY_CB.has(call.callee.property.name))) return;-- if (call.callee.property.name !== 'map') { manual.push(`${file}:${call.loc.start.line}: a .${call.callee.property.name}() callback becomes async`); return; }-- if (inPromiseArg(call) || wrappedMaps.has(call)) return;-- wrappedMaps.add(call);-- edits.push({ at: call.start, text: '(await Promise.all(', depth: 1 });-- edits.push({ at: call.end, text: '))', depth: 1 });-- makeAsync([...anc].reverse().find((x) => x !== fn && isFnN(x)));-- }-- // registry[k.list](): a computed call on a module API cannot be resolved here; a person awaits it if needed.-- walk.full(ast, (n) => {-- if (n.type !== 'CallExpression' || n.callee.type !== 'MemberExpression' || !n.callee.computed || n.callee.object.type !== 'Identifier' || !apis[n.callee.object.name]) return;-- const p = ancestors.get(n) || []; const up = p[p.length - 1];-- if (!(up && up.type === 'AwaitExpression')) manual.push(`${file}:${n.loc.start.line}: computed call ${n.callee.object.name}[…]() on a module API: await it if the method is async`);-- });-- // An async callback already there whose map result is not awaited as a whole is almost always a bug.-- walk.full(ast, (n) => {-- if (n.type === 'CallExpression' && n.callee.type === 'MemberExpression' && n.callee.property.name === 'map' && isFnN(n.arguments[0]) && n.arguments[0].async && !inPromiseArg(n)) {-- const p = ancestors.get(n) || []; const up = p[p.length - 1];-- if (!(up && up.type === 'VariableDeclarator')) manual.push(`${file}:${n.loc.start.line}: .map(async …) whose promises are not awaited together (Promise.all)`);-- }-- });---- // An async function passed by name to an array method: xs.map(asyncFn) / xs.map(api.asyncMethod) is wrapped in-- // (await Promise.all(…)) like an async callback; any other array method needs a person.-- const isFnNode = (x) => x.type === 'FunctionDeclaration' || x.type === 'FunctionExpression' || x.type === 'ArrowFunctionExpression';-- walk.full(ast, (n) => {-- if (n.type !== 'CallExpression' || n.callee.type !== 'MemberExpression' || !ARRAY_CB.has(n.callee.property.name) || !n.arguments[0]) return;-- const arg = n.arguments[0];-- let what = null;-- if (arg.type === 'Identifier' && localAsync.has(arg.name) && !syncNames.has(arg.name)) what = `async function ${arg.name}`;-- if (arg.type === 'MemberExpression' && !arg.computed && arg.object.type === 'Identifier' && apis[arg.object.name]) {-- const set = asyncNames.api.get(apis[arg.object.name]);-- if (set && set.has(arg.property.name) && !syncNames.has(arg.property.name)) what = `async method ${arg.object.name}.${arg.property.name}`;-- }-- if (!what) return;-- const p = ancestors.get(n) || [];-- const up = p[p.length - 1];-- if (n.callee.property.name === 'map' && up && up.type === 'CallExpression' && up.callee.type === 'MemberExpression' && up.callee.object.name === 'Promise') return;-- const outer = [...p].reverse().find(isFnNode);-- if (n.callee.property.name === 'map' && outer && !wrappedMaps.has(n)) {-- wrappedMaps.add(n);-- edits.push({ at: n.start, text: '(await Promise.all(', depth: 1 });-- edits.push({ at: n.end, text: '))', depth: 1 });-- makeAsync(outer);-- return;-- }-- manual.push(`${file}:${n.loc.start.line}: ${what} passed to .${n.callee.property.name}()`);-- });-- // Names awaited as a whole somewhere in the file (await first, await Promise.all([a, b]) excluded).-- const awaitedLater = new Set();-- walk.full(ast, (n) => { if (n.type === 'AwaitExpression' && n.argument && n.argument.type === 'Identifier') awaitedLater.add(n.argument.name); });-- walk.fullAncestor(ast, (node, _s, anc) => {-- if (node.type !== 'CallExpression' || !asyncCall(node)) return;-- const parents = anc.slice(0, -1);-- const parent = parents[parents.length - 1];-- if (parent && parent.type === 'AwaitExpression') return;-- // A promise used as a value stays a promise: p.then/catch/finally, or an element of Promise.all/race/allSettled/any.-- if (parent && parent.type === 'MemberExpression' && parent.object === node && ['then', 'catch', 'finally'].includes(parent.property.name)) return;-- if (parent && parent.type === 'ArrayExpression') {-- const gp = parents[parents.length - 2];-- if (gp && gp.type === 'CallExpression' && gp.callee.type === 'MemberExpression' && gp.callee.object.name === 'Promise') return;-- }-- if (parent && parent.type === 'CallExpression' && parent.callee.type === 'MemberExpression' && parent.callee.object.name === 'Promise') return;-- // const p = asyncCall(); … await p: the promise is started now on purpose and awaited later.-- if (parent && parent.type === 'VariableDeclarator' && parent.init === node && parent.id.type === 'Identifier' && awaitedLater.has(parent.id.name)) return;-- // assert.rejects(promise) / doesNotReject(promise) take the promise itself.-- if (parent && parent.type === 'CallExpression' && parent.arguments[0] === node && parent.callee.type === 'MemberExpression' && ['rejects', 'doesNotReject'].includes(parent.callee.property.name)) return;-- // So does a local helper that awaits that parameter itself (a test's rejects(p, status)).-- if (parent && parent.type === 'CallExpression' && parent.callee.type === 'Identifier' && promiseTakers.has(`${parent.callee.name}:${parent.arguments.indexOf(node)}`)) return;-- // Returned directly from an arrow body or a return statement of an async fn is fine too, but awaiting is harmless.-- const fnIdx = (() => { for (let i = parents.length - 1; i >= 0; i--) if (isFn(parents[i])) return i; return -1; })();-- const fn = fnIdx >= 0 ? parents[fnIdx] : null;-- if (fn) {-- const fnParent = parents[fnIdx - 1];-- const inPromiseAll = (call) => { const p = ancestors.get(call) || []; const up = p[p.length - 1]; return up && up.type === 'CallExpression' && up.arguments[0] === call && up.callee.type === 'MemberExpression' && up.callee.object.name === 'Promise'; };-- if (fnParent && fnParent.type === 'CallExpression' && fnParent.arguments.includes(fn) && fnParent.callee.type === 'MemberExpression' && ARRAY_CB.has(fnParent.callee.property.name)-- && !(fnParent.callee.property.name === 'map' && inPromiseAll(fnParent))) {-- // xs.map(cb) → (await Promise.all(xs.map(async cb))): same order, same length. Other array methods-- // change meaning with an async callback (filter keeps every promise), so they stay for a person.-- const outer = (() => { for (let i = fnIdx - 2; i >= 0; i--) if (isFn(parents[i])) return parents[i]; return null; })();-- if (fnParent.callee.property.name === 'map' && outer) {-- if (!wrappedMaps.has(fnParent)) {-- wrappedMaps.add(fnParent);-- edits.push({ at: fnParent.start, text: '(await Promise.all(', depth: 1 });-- edits.push({ at: fnParent.end, text: '))', depth: 1 });-- makeAsync(outer);-- }-- } else {-- manual.push(`${file}:${node.loc.start.line}: async call inside a .${fnParent.callee.property.name}() callback`);-- return;-- }-- }-- makeAsync(fn);-- }-- if (parent && parent.type === 'ExpressionStatement' && node.callee.type === 'Identifier' && (asyncNames.original.get(file) || new Set()).has(node.callee.name)-- && !/\bawait\s+$/.test(src.slice(Math.max(0, node.start - 12), node.start))) {-- manual.push(`${file}:${node.loc.start.line}: ${node.callee.name}() was already async and is called without await: left fire-and-forget (add .catch if it can reject)`);-- return;-- }-- const wrap = parent && parent.type === 'MemberExpression' && parent.object === node;-- // Never a second await in front of one already there (a call the parser saw inside another's callee).-- if (!wrap && /\bawait\s+$/.test(src.slice(Math.max(0, node.start - 12), node.start))) return;-- edits.push({ at: node.start, text: wrap ? '(await ' : 'await ' });-- if (wrap) edits.push({ at: node.end, text: ')' });-- });---- for (const fn of fnsToAsync) {-- if (fn.async) continue;-- const anc = ancestors.get(fn) || [];-- const parent = anc[anc.length - 1];-- becameAsync.add(fn);-- const name = functionName(fn, parent);-- if (name) newlyAsyncNames.add(name);-- if (parent && parent.type === 'Property' && (parent.method || parent.kind !== 'init')) edits.push({ at: parent.start, text: 'async ' });-- else if (parent && parent.type === 'MethodDefinition') edits.push({ at: parent.key.start, text: 'async ' });-- else edits.push({ at: fn.start, text: 'async ' });-- }-- return { edits, manual, newlyAsyncNames };--}----function apply(src, edits) {-- // At one position, closing text goes in first so it ends up after what opens there.-- // At one position: closers go in before openers (so an opener ends up left of a closer), an outer closer before an-- // inner one (it ends up right of it), an inner opener before an outer one (the outer ends up leftmost). A-- // (await Promise.all( wrap is outer (depth 1) to the await of the call it starts with (depth 0).-- const closer = (e) => e.text[0] === ')';-- const sorted = edits.slice().sort((a, b) => b.at - a.at-- || (closer(a) ? 0 : 1) - (closer(b) ? 0 : 1)-- || (closer(a) ? (b.depth || 0) - (a.depth || 0) : (a.depth || 0) - (b.depth || 0)));-- let out = src;-- for (const e of sorted) out = out.slice(0, e.at) + e.text + out.slice(e.at);-- return out;--}----function main() {-- const args = process.argv.slice(2);-- const ci = args.indexOf('--config');-- const cfg = ci >= 0 ? JSON.parse(fs.readFileSync(args[ci + 1], 'utf8')) : {};-- const files = args.filter((a, i) => a.endsWith('.js') && i !== ci + 1).map((f) => path.relative(process.cwd(), path.resolve(f)));-- const asyncNames = { local: new Map(), api: new Map(), original: new Map() };-- // Functions that were async before this run: a bare statement calling one was fire-and-forget on purpose (a worker-- // starting a job in its lane) and is left alone.-- for (const f of files) {-- const src = fs.readFileSync(f, 'utf8');-- asyncNames.original.set(f, new Set([...src.matchAll(/async function\s+(\w+)/g), ...src.matchAll(/(?:const|let)\s+(\w+)\s*=\s*async\b/g)].map((m) => m[1])));-- }-- let manual = [];-- for (let round = 0; round < 12; round++) {-- let changed = false;-- manual = [];-- for (const f of files) {-- const src = fs.readFileSync(f, 'utf8');-- let r;-- try { r = analyse(f, src, cfg, asyncNames); } catch (e) { console.error(`${f}: parse error ${e.message}`); process.exitCode = 1; continue; }-- manual.push(...r.manual);-- if (r.edits.length) {-- let next = apply(src, r.edits);-- // An assertion about a function that is async now: assert.throws cannot see a rejection.-- next = next.replace(/(^|[^.\w])assert\.throws\(async /g, '$1await assert.rejects(async ').replace(/(^|[^.\w])assert\.doesNotThrow\(async /g, '$1await assert.doesNotReject(async ');-- try { parse(next); } catch (e) { console.error(`${f}: the edits would break the syntax (${e.message}); left unchanged`); manual.push(`${f}: edits refused (syntax)`); continue; }-- fs.writeFileSync(f, next); changed = true;-- }-- // Every async function of the file (new or already) counts for callers.-- const all = new Set([...(asyncNames.local.get(f) || []), ...r.newlyAsyncNames]);-- // { name: someAsyncFunction } exports the function under another name: that name is async too.-- walk.full(parse(fs.readFileSync(f, 'utf8')), (n) => {-- if (n.type === 'Property' && n.value && n.value.type === 'Identifier' && all.has(n.value.name) && n.key) all.add(n.key.name || n.key.value);-- });-- walk.fullAncestor(parse(fs.readFileSync(f, 'utf8')), (node, _s, anc) => {-- if ((node.type === 'FunctionDeclaration' || node.type === 'FunctionExpression' || node.type === 'ArrowFunctionExpression') && node.async) {-- const parent = anc[anc.length - 2];-- const name = functionName(node, parent);-- if (name) all.add(name);-- }-- });-- const before = (asyncNames.local.get(f) || new Set()).size;-- asyncNames.local.set(f, all);-- asyncNames.api.set(f, all);-- if (all.size !== before) changed = true;-- }-- if (!changed) break;-- }-- for (const m of [...new Set(manual)]) console.log(`MANUAL ${m}`);-- for (const f of files) console.log(`async ${f}: ${[...(asyncNames.local.get(f) || [])].sort().join(', ')}`);--}----main();ddeleted file mode 100644@@@ -1,115 +0,0 @@--"""Add `await` to async service/db calls in test files converted from better-sqlite3 (openvibe-sdk/db ≥ 0.18).--Usage: python3 awaitify.py <sync-member,...> file... (sync members: svc members that stay synchronous)"""--import re, sys----def skip_string(s, j):-- q = s[j]; j += 1-- while j < len(s):-- c = s[j]-- if c == '\\': j += 2; continue-- if q == '`' and c == '$' and j + 1 < len(s) and s[j+1] == '{':-- j = skip_braces(s, j + 1); continue-- if c == q: return j + 1-- j += 1-- return j----def skip_braces(s, j): # s[j] == '{'-- depth = 0-- while j < len(s):-- c = s[j]-- if c in '\'"`': j = skip_string(s, j); continue-- if c == '{': depth += 1-- elif c == '}':-- depth -= 1-- if depth == 0: return j + 1-- j += 1-- return j----def call_end(s, i): # s[i] == '(' -> index after matching ')'-- depth = 0; j = i-- while j < len(s):-- c = s[j]-- if c in '\'"`': j = skip_string(s, j); continue-- if c == '/' and s[j+1:j+2] == '/': j = s.find('\n', j); j = len(s) if j < 0 else j; continue-- if c in '([{': depth += 1-- elif c in ')]}':-- depth -= 1-- if depth == 0: return j + 1-- j += 1-- return j----def transform(src, sync_members):-- out = []; i = 0; changed = 0-- pat = re.compile(r'((?:[A-Za-z_$][\w$]*\.)*(?:svc|db)\.(\w+)\(|(?:\bH\.)?\boutbox\(\s*h\w*\s*\)|(?:[A-Za-z_$][\w$]*\.)*(?:stores\.\w+|platform\.outbox)\.(\w+)\(|(?<![\w.$])(?:summarize|seedFn|seed)\()')-- while True:-- m = pat.search(src, i)-- if not m: out.append(src[i:]); break-- start = m.start()-- # not part of a longer member chain (e.g. foo.bar.svc handled by regex prefix); skip when preceded by '.'-- if start > 0 and src[start-1] in '.$':-- out.append(src[i:m.end()]); i = m.end(); continue-- before = src[:start].rstrip()-- text = m.group(0)-- if text.startswith('outbox') or text.startswith('H.outbox'):-- end = m.end()-- elif m.group(3) is not None or re.match(r'(summarize|seedFn|seed)\($', text):-- if m.group(3) in ('start', 'stop', 'counts'):-- out.append(src[i:m.end()]); i = m.end(); continue-- end = call_end(src, m.end() - 1)-- else:-- member = m.group(2)-- is_db = '.db.' in text or text.startswith('db.')-- if (not is_db and member in sync_members) or (is_db and member not in ('prepare', 'value', 'many', 'maybe', 'one', 'exec', 'query', 'tx')):-- out.append(src[i:m.end()]); i = m.end(); continue-- end = call_end(src, m.end() - 1)-- if is_db and member == 'prepare':-- # .pluck(...)? then .get/.all/.run(...)-- mm = re.match(r'\s*\.pluck\(', src[end:])-- if mm: end = call_end(src, end + mm.end() - 1)-- mm = re.match(r'\s*\.(get|all|run)\(', src[end:])-- if not mm:-- out.append(src[i:m.end()]); i = m.end(); continue-- end = call_end(src, end + mm.end() - 1)-- if before.endswith('await') or before.endswith('await ('):-- out.append(src[i:end]); i = end; continue-- head = src[start:m.end()]-- inner, _ = transform(src[m.end():end], sync_members) if end > m.end() else ('', 0)-- expr = head + inner-- nxt = src[end:end+1]-- rep = f'(await {expr})' if nxt in ('.', '[') else f'await {expr}'-- out.append(src[i:start]); out.append(rep); i = end; changed += 1-- return ''.join(out), changed----if __name__ == '__main__':-- sync = set(sys.argv[1].split(','))-- for f in sys.argv[2:]:-- s = open(f).read()-- t, n = transform(s, sync)-- # arrows/functions that now contain await on the same line: make them async-- t = re.sub(r'(const \w+ = )\(([^)]*)\) => (?=.*\bawait\b)', r'\1async (\2) => ', t)-- if n: open(f, 'w').write(t); print(f, n)----def await_helpers(src):-- names = set(re.findall(r'(?:const|let)\s+(\w+)\s*=\s*async\b', src)) | set(re.findall(r'async function\s+(\w+)\s*\(', src))-- names -= {'main', 'run'}-- if not names: return src, 0-- pat = re.compile(r'(?<![\w.$])(' + '|'.join(sorted(names, key=len, reverse=True)) + r')\(')-- out = []; i = 0; n = 0-- while True:-- m = pat.search(src, i)-- if not m: out.append(src[i:]); break-- start = m.start()-- before = src[:start].rstrip()-- # a definition, or already awaited-- if re.search(r'(function|const|let)\s*$', before) or re.search(r'(const|let)\s+$', src[:start]) or before.endswith('await') or re.search(r'function\s+$', src[:start]):-- out.append(src[i:m.end()]); i = m.end(); continue-- end = call_end(src, m.end() - 1)-- expr = src[start:end]; nxt = src[end:end+1]-- rep = f'(await {expr})' if nxt in ('.', '[') else f'await {expr}'-- out.append(src[i:start]); out.append(rep); i = end; n += 1-- return ''.join(out), n----if __name__ == '__main__' and len(sys.argv) > 1 and sys.argv[1] == '--helpers':-- for f in sys.argv[2:]:-- s = open(f).read(); t, n = await_helpers(s)-- if n: open(f, 'w').write(t); print(f, n)ddeleted file mode 100644@@@ -1,58 +0,0 @@--{-- "stores": [-- "revisions",-- "citations",-- "taxonomy",-- "scheduler",-- "attachments",-- "discussion",-- "reviews",-- "redirects",-- "sequencer",-- "outbox"-- ],-- "syncStoreMethods": [-- "start",-- "stop",-- "slugify",-- "enabled"-- ],-- "apis": {-- "blogs": "server/domain/blogs.js",-- "posts": "server/domain/posts.js",-- "publication": "server/domain/publication.js",-- "reading": "server/domain/reading.js",-- "access": "server/domain/access.js",-- "aiDrafts": "server/domain/ai-drafts.js",-- "changelog": "server/changelog.js",-- "effects": "server/domain/effects.js",-- "people": "server/clients/network.js",-- "worker": "server/worker.js",-- "entitlements": "server/domain/access.js",-- "ctx": "server/app.js",-- "cl": "server/changelog.js"-- },-- "syncNames": [-- "validHandle",-- "themeVars",-- "categoryVocabulary",-- "actorRef",-- "listOf",-- "truthy",-- "encodeCursor",-- "decodeCursor",-- "batchKey",-- "subjectOf",-- "start",-- "stop"-- ],-- "asyncGlobals": [-- "seedOfficialPost"-- ],-- "bareStores": [-- "outbox"-- ],-- "asyncMethods": [-- "events"-- ]--}\\ No newline at end of fileddeleted file mode 100644@@@ -1,64 +0,0 @@--#!/usr/bin/env python3--"""server/db.js of a Blog-shaped service, from better-sqlite3 to openvibe-sdk/db (ADR-035): openDb (DATABASE_URL through--PgBouncer, migrations as the owner, PGlite in development), createStore (the stores openStore returned, on the async--handle) and openStore(config). The SCHEMA literal goes (it is migrations/0001_initial.sql now: gen-migration.js).--usage: g1db.py <svc> <Name> run in the service's checkout after gen-migration.js."""--import re, sys--svc, Name = sys.argv[1], sys.argv[2]--p = 'server/db.js'--s = open(p).read()--if 'openvibe-sdk/db' in s: sys.exit('server/db.js is converted already')--m = re.search(r"\nfunction openStore\(dbPath, \{ now = \(\) => Date\.now\(\) \} = \{\}\) \{\n[\s\S]*?\n return \{\n([\s\S]*?)\n \};\n\}\n", s)--if not m: sys.exit('openStore(dbPath, { now }) with a returned object literal not found: convert by hand')--props = [l for l in m.group(1).split('\n') if not re.match(r"\s+(db|now|tx: .*|close: .*),?$", l)]--doc = s.rfind('/**', 0, m.start() + 1)--end = s.find('*/', doc) if doc != -1 else -1--# openStore's own doc comment only: the comment must end right before the function (whitespace between).--head = s[:doc] if doc != -1 and end != -1 and not s[end + 2:m.start() + 1].strip() else s[:m.start() + 1]--tail = s[m.end():]--head = re.sub(r"const Database = require\('better-sqlite3'\);\n", "const { createDb } = require('openvibe-sdk/db');\n", head)--head = re.sub(r"\nconst SCHEMA = `[\s\S]*?`;\n", "\nconst MIGRATIONS = path.join(__dirname, '..', 'migrations');\nconst DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');\n", head)--head = re.sub(r"(%s'?s?) own SQLite database(?: \(WAL\))?[:,] created on boot, idempotently\." % re.escape(Name),-- lambda x: x.group(1) + " own PostgreSQL database (ADR-035, roadmap WS-X2): the schema is migrations/NNNN_*.sql, applied at boot.", head)--body = f"""/**-- * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database-- * in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle.-- */--async function openDb(config, {{ log = console, registry }} = {{}}) {{-- if (!config.db.url) {{-- if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh {svc})');-- log.warn(`[{Name}] DATABASE_URL unset: embedded PGlite database in ${{DEV_PGLITE}} (development only, one process)`);-- fs.mkdirSync(DEV_PGLITE, {{ recursive: true }});-- const db = createDb({{ pglite: DEV_PGLITE, service: '{svc}', registry, log }});-- await db.migrate({{ dir: MIGRATIONS, log }});-- return db;-- }}-- if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');-- const owner = createDb({{ url: config.db.directUrl, service: '{svc}-migrate', max: 1, log }});-- try {{ await owner.migrate({{ dir: MIGRATIONS, log }}); }} finally {{ await owner.close(); }}-- return createDb({{ url: config.db.url, service: '{svc}', registry, log }});--}}----/**-- * Every store on a migrated database handle. opts.now — injectable clock (epoch ms), so tests and replays are-- * deterministic. store.tx(fn) is a transaction; inside it, plain db calls join it (ambient).-- */--function createStore(db, {{ now = () => Date.now() }} = {{}}) {{-- return {{-- db,-- now,--{chr(10).join(props)}-- tx: async (fn) => await db.tx(() => fn()),-- close: () => db.close(),-- }};--}}----/** openDb + createStore. */--async function openStore(config, {{ now, log }} = {{}}) {{-- return createStore(await openDb(config, {{ log }}), {{ now }});--}}--"""--tail = re.sub(r"module\.exports = \{ openStore(, )?", r"module.exports = { openDb, openStore, createStore, MIGRATIONS\1", tail)--s = head + body + tail--open(p, 'w').write(s)--print('server/db.js converted; stores kept:', ', '.join(re.findall(r"^\s+(\w+):", '\n'.join(props), re.M)) or 'none')ddeleted file mode 100644@@@ -1,42 +0,0 @@--#!/usr/bin/env python3--"""README.md and STATUS.json edits after a service's PostgreSQL pass (the wording Blog, News and Deals use).--usage: g1docs.py <svc> <Name> <ENVPREFIX> run in the service's checkout, after package.json has the new pins.--Each edit is skipped when its anchor is missing (it prints what it could not place, for a hand edit)."""--import json, re, sys--svc, Name, ENV = sys.argv[1], sys.argv[2], sys.argv[3]--pkg = json.load(open('package.json'))--ver = lambda n: (re.search(r'/v([0-9.]+)$', pkg['dependencies'].get(n, '')) or [None, None])[1]--s = open('README.md').read(); o = s--miss = []--def sub(pattern, repl, flags=0, what=''):-- global s-- n = re.subn(pattern, repl, s, count=1, flags=flags)-- if n[1]: s = n[0]-- else: miss.append(what or pattern[:60])--sub(r"live in %s'?s? own SQLite \(`%s_DB_PATH`\)" % (re.escape(Name), ENV),-- f"live in {Name}' own PostgreSQL database (`ov_{svc}` on the host's data role, ADR-035; schema in [migrations/](migrations/))", what='tables line')--if 'PostgreSQL 18 and Valkey 9' not in s:-- sub(r"\n- \*\*Packages\*\*", "\n- **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through\n `openvibe-sdk/db`; Valkey holds the per-actor limit counters (optional: without `VALKEY_URL` they count per process).\n- **Packages**", what='PG bullet')--for name in ['openvibe-publishing', 'openvibe-contracts', 'openvibe-sdk', 'openvibe-shared']:-- v = ver(name)-- if v: s = re.sub(r"(`%s`\s+)v[0-9.]+" % re.escape(name), r"\g<1>v" + v, s)--s = s.replace('(chrome, app icon', '(Frame, app icon')--unit = re.search(r"The unit is `openvibe-%s\.service` on `[^`]+`, the env file `/etc/openvibe/%s\.env`\." % (svc, svc), s)--if unit and 'add-service.sh' not in s:-- s = s[:unit.end()] + f""" The database is--`ov_{svc}` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh {svc}` writes its settings); the--release migrates it at boot. The one-time move from SQLite is `scripts/migrate-to-postgres.js` (openvibe-sdk--`runSqliteMigration`, with a `--pglite` rehearsal mode), run while the service is stopped; the old--`/var/lib/openvibe-{svc}/{svc}.db` stays read-only for 7 days as the rollback.""" + s[unit.end():]--elif not unit: miss.append('unit paragraph')--s = re.sub(r"Nothing blocks a rollback: the schema\s+code only adds tables and columns\.", "Migrations only add tables and columns.", s)--if s != o: open('README.md', 'w').write(s)--st = json.load(open('STATUS.json'))--st['runtime'] = re.sub(r'better-sqlite3 \d+', 'PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9', st.get('runtime', ''))--if ver('openvibe-contracts'): st['contracts'] = f"openvibe-contracts v{ver('openvibe-contracts')}"--if 'packages' in st:-- st['packages'] = [f"{n} v{ver(n)}" if ver(n) else p for p in st['packages'] for n in [p.split(' ')[0]]]--open('STATUS.json', 'w').write(json.dumps(st, indent=2, ensure_ascii=False) + '\n')--left = [l for l in s.splitlines() if re.search(r'SQLite|better-sqlite3', l)]--print('README/STATUS updated' + (f"; not placed: {', '.join(miss)}" if miss else ''))--for l in left: print(' still says SQLite:', l[:140])ddeleted file mode 100644@@@ -1,167 +0,0 @@--#!/usr/bin/env python3--"""The shared, non-codemod edits of a content product's PostgreSQL pass (Blog-shaped services: server/app.js createApp,--server/events/outbox.js, server/http/actor-limits.js, server/observability.js, test/helpers/boot.js).--usage: g1pass.py <svc> <Name> <ENVPREFIX> (e.g. news News NEWS), run in the service's checkout. Idempotent-ish: each--edit asserts its anchor and skips when already applied."""--import json, re, sys, os--svc, Name, ENV = sys.argv[1], sys.argv[2], sys.argv[3]--def edit(p, pairs, required=True):-- if not os.path.exists(p): print('skip (missing)', p); return-- s = open(p).read(); o = s-- for a, b in pairs:-- if b in s: continue # applied already (b may contain a)-- if a not in s:-- if required: raise SystemExit(f'{p}: anchor not found: {a[:90]}')-- continue-- s = s.replace(a, b, 1)-- if s != o: open(p, 'w').write(s); print('edited', p)--# package.json: the PostgreSQL stack and the pins it needs--pkg = json.load(open('package.json'))--tag = lambda repo, v: f'https://codeload.github.com/OpenVibers/{repo}/tar.gz/refs/tags/{v}'--d = pkg['dependencies']--d['openvibe-sdk'] = tag('OpenVibe.SDK', os.environ.get('G1_SDK', 'v0.20.0'))--if 'openvibe-publishing' in d: d['openvibe-publishing'] = tag('OpenVibe.Publishing', os.environ.get('G1_PUBLISHING', 'v1.0.0'))--if 'openvibe-contracts' in d: d['openvibe-contracts'] = tag('OpenVibe.Contracts', os.environ.get('G1_CONTRACTS', 'v0.76.0'))--d.setdefault('pg', '^8.23.0'); d.setdefault('iovalkey', '^0.4.0')--pkg['dependencies'] = dict(sorted(d.items()))--pkg.setdefault('devDependencies', {})['@electric-sql/pglite'] = '^0.5.8'--pkg['scripts']['test:pg'] = f'{ENV}_TEST_STORE=pg node test/run.js'--open('package.json', 'w').write(json.dumps(pkg, indent=2) + '\n'); print('package.json')--# config--edit('server/config.js', [(f" dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',", f""" // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).-- db: {{ url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' }},-- valkey: {{ url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:{svc}:' }},-- // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.-- dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',""")])--# outbox--if os.path.exists('server/events/outbox.js'):-- edit('server/events/outbox.js', [-- ("const { createEventsClient, createOutbox } = require('openvibe-sdk/events');", "const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');"),-- (" const outbox = createOutbox(db, {", " // The PostgreSQL outbox: rows are written in the change's own transaction (enqueue(db, …) joins the ambient\n // transaction); several processes relay one table safely (leases).\n const outbox = createPgOutbox(db, {"),--])--s = open('server/events/outbox.js').read()--s2 = re.sub(r"\n[ \t]*(await )?outbox\.ensureSchema\(\);[^\n]*", '', s)--if s2 != s: open('server/events/outbox.js', 'w').write(s2); print('outbox: schema is the migration now')--s = open('server/events/outbox.js').read()--s2 = re.sub(r'outbox\.enqueue\((?!db, )', 'outbox.enqueue(db, ', s)--if s2 != s: open('server/events/outbox.js', 'w').write(s2); print('enqueue(db, …)')--# app.js store--s = open('server/app.js').read()--m = re.search(r" const store = opts\.store \|\| openStore\(opts\.dbPath \|\| config\.dbPath, \{ now: opts\.now(, [^}]*)? \}\);", s)--if m:-- s = s.replace(m.group(0), " // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test, a script) hands in a store.\n const store = opts.store || await openStore(config, { now: opts.now, log });")-- s = s.replace('function createApp(opts = {}) {', 'async function createApp(opts = {}) {', 1)-- open('server/app.js', 'w').write(s); print('app store')--# limits + valkey--if os.path.exists('server/http/actor-limits.js'):-- s = open('server/http/actor-limits.js').read(); o = s-- if 'createValkeyLimitStore' not in s:-- s = re.sub(r"const \{ createActorLimiter(, defaultActor)? \} = require\('openvibe-sdk/limits'\);",-- lambda m: "const { createActorLimiter, createValkeyLimitStore" + (m.group(1) or '') + " } = require('openvibe-sdk/limits');", s, 1)-- s = re.sub(r"(function createActorLimits\(\{[^)]*?)( \}\) \{)", lambda m: m.group(1) + ', valkey = null' + m.group(2), s, 1)-- s = s.replace(" now,\n", " now,\n // Shared across processes on Valkey (ADR-035) when VALKEY_URL is set; in-process otherwise.\n ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),\n", 1)-- if s != o: open('server/http/actor-limits.js', 'w').write(s); print('edited server/http/actor-limits.js')--s = open('server/app.js').read()--m = re.search(r" ctx\.(limits|actorLimits) = createActorLimits\(\{ config, now: opts\.limitsNow \|\| \(\(\) => Date\.now\(\)\), registry: metrics\.registry, log \}\);", s)--if m:-- s = s.replace(m.group(0), " // Valkey (ADR-035): shared, never-authoritative state (per-actor limit counters). Optional.\n const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);\n ctx.valkey = valkey;\n ctx." + m.group(1) + " = createActorLimits({ config, now: opts.limitsNow || (() => Date.now()), registry: metrics.registry, log, valkey });")-- open('server/app.js', 'w').write(s); print('app valkey')--# readiness--s = open('server/observability.js').read(); o = s--s = re.sub(r"(function create\w+Readiness\(\{ store, [^}]*?)(, release = null \}\))", lambda m: m.group(1) + (', valkey = null' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)--s = s.replace(""" {-- name: 'network_jwks', required: false,""", """ { name: 'valkey', required: false, check: async () => (valkey ? valkey.ready() : { skipped: 'VALKEY_URL not set: per-actor limits count in this process only' }) },-- {-- name: 'network_jwks', required: false,""", 1) if "name: 'valkey'" not in s else s--s = re.sub(r"""const names = new Set\((?:\(await )?db\.prepare\("SELECT name FROM sqlite_master WHERE type IN \('table','view'\)"\)\.all\(\)\)?\.map\(\(r\) => r\.name\)\);\n(\s*)const missing = CHARTER_TABLES\.filter\(\(t\) => !names\.has\(t\)\);\n\s*return missing\.length \? `missing \$\{missing\.join\(', '\)\}` : true;""",-- lambda m: f"""// A real round trip that names the store (postgresql / pglite), and the charter tables present.--{m.group(1)}const r = await db.ready();--{m.group(1)}if (!r.ok) return r.error;--{m.group(1)}const names = new Set((await db.prepare('SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()').all()).map((x) => x.name));--{m.group(1)}const missing = CHARTER_TABLES.filter((t) => !names.has(t));--{m.group(1)}return missing.length ? `missing ${{missing.join(', ')}} (migrations did not run)` : {{ ok: true, detail: r.detail }};""", s)--s = s.replace("check: () => {\n // A real round trip", "check: async () => {\n // A real round trip")--if s != o: open('server/observability.js', 'w').write(s); print('readiness')--s = open('server/app.js').read()--s2 = re.sub(r"(create\w+Readiness\(\{ store, [^}]*?release: release\.release)( \}\))", lambda m: m.group(1) + (', valkey: ctx.valkey' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)--if s2 != s: open('server/app.js', 'w').write(s2); print('app readiness')--# index.js: await createApp--s = open('server/index.js').read()--if 'await createApp()' not in s:-- s = s.replace("const { app, ctx } = createApp();", "(async () => {\nconst { app, ctx } = await createApp();", 1)-- s = s.replace(" try { ctx.store.close(); } catch { /* already closed */ }", " try { await ctx.store.close(); } catch { /* already closed */ }")-- s = s.replace(f"(db ${{config.dbPath}})", "(db ${ctx.store.db.store})")-- s = s.rstrip() + f"\n}})().catch((err) => {{ console.error('[{Name}] failed to start:', err); process.exit(1); }});\n"-- open('server/index.js', 'w').write(s); print('index')--# test helpers--os.makedirs('test/helpers', exist_ok=True)--open('test/helpers/db.js', 'w').write(f"""'use strict';--/**-- * A migrated database for one test run (ADR-035), from openvibe-sdk/testing: PGlite by default; with-- * {ENV}_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of this run's own.-- */--const {{ createTestDb, pgAvailable }} = require('openvibe-sdk/testing');--const {{ MIGRATIONS }} = require('../../server/db');----const testDb = ({{ store = process.env.{ENV}_TEST_STORE || 'pglite', max = 4 }} = {{}}) => createTestDb({{ migrations: MIGRATIONS, store, service: '{svc}', max }});----module.exports = {{ testDb, pgAvailable }};--""")--s = open('test/helpers/boot.js').read(); o = s--if "require('./db').testDb()" not in s:-- s = s.replace(" let server = null;\n let built = null;", " const { createStore } = require('../../server/db');\n // One database per boot (PGlite, or " + ENV + "_TEST_STORE=pg: the containers); a restart keeps it, like a file did.\n const testdb = await require('./db').testDb();\n let server = null;\n let built = null;", 1)-- s = re.sub(r"built = createApp\(\{ config, ", "built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), ", s, 1)-- s = re.sub(r"(await built\.ctx\.outbox\.stop\(\);) built\.ctx\.store\.close\(\);", r"\1", s)-- s = re.sub(r" function events\(type = null\) \{\n return t\.ctx\.store\.db\.prepare\('SELECT envelope FROM event_outbox ORDER BY id'\)\.all\(\)\.map\(\(r\) => JSON\.parse\(r\.envelope\)\)",-- " async function events(type = null) {\n return (await t.ctx.store.db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope))", s)-- s = s.replace(" async close() { await stop(); await network.close();", " async close() { await stop(); await testdb.close(); await network.close();", 1)--if s != o: open('test/helpers/boot.js', 'w').write(s); print('boot helper')--# CI, unit, env, import script--s = open('.github/workflows/ci.yml').read()--if 'test-services.sh' not in s:-- s = s.replace(f" contracts-service: {svc}\n", f""" contracts-service: {svc}-- # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk-- # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.-- test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'--""", 1); open('.github/workflows/ci.yml', 'w').write(s); print('ci')--unit = f'deploy/systemd/openvibe-{svc}.service'--edit(unit, [("After=network.target\n", "After=network.target postgresql.service pgbouncer.service valkey-server.service\n"), ("Wants=openvibe-network.service\n", "Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service\n")], required=False)--edit('.env.example', [(f"{ENV}_DB_PATH=./data/{svc}.db", f"""# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh {svc}: the runtime role through PgBouncer,--# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).--# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.--DATABASE_URL=--DATABASE_DIRECT_URL=--VALKEY_URL=--VALKEY_PREFIX=--# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.--{ENV}_DB_PATH=./data/{svc}.db""")], required=False)--os.makedirs('scripts', exist_ok=True)--cfgmod = "require('../server/config')"--open('scripts/migrate-to-postgres.js', 'w').write(f"""#!/usr/bin/env node--'use strict';--/**-- * The one-time move of {Name}'s SQLite database ({ENV}_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is-- * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).-- *-- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]-- *-- * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts-- * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its-- * name and columns.-- */--require('dotenv').config();--const {{ runSqliteMigration }} = require('openvibe-sdk/db');--const configLib = {cfgmod};--const {{ MIGRATIONS }} = require('../server/db');----const TABLES = {{}};----if (require.main === module) {{-- const config = (configLib.load || configLib.loadConfig)();-- runSqliteMigration({{ service: '{svc}', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES }})-- .then((code) => process.exit(code), (err) => {{ console.error(`migrate-to-postgres failed: ${{err.message}}`); process.exit(1); }});--}}----module.exports = {{ TABLES }};--""")--print('done')ddeleted file mode 100644@@@ -1,172 +0,0 @@--#!/usr/bin/env node--'use strict';--/**-- * The first migration of a better-sqlite3 service (ADR-035), written once: the SCHEMA template literal of its-- * server/db.js through sqlite-schema-to-pg, then the openvibe-publishing stores it opens and the openvibe-sdk inbox-- * and outbox. Run in the service's checkout, after `npm install` of openvibe-publishing ≥ 1.0 and openvibe-sdk ≥ 0.18:-- *-- * node gen-migration.js <Name> [--publishing revisions:blog_post,citations:blog_post,index-hooks:blog,seo:blog_entity,…]-- * [--no-inbox] [--outbox-table t] [--inbox-table t] [--no-outbox] [--ensure a.js,b.js]-- * [--from server/store.js] (the module holding SCHEMA; default server/db.js)-- * [--open "require('./server/db').openDb(':memory:')"]-- * → migrations/0001_initial.sql-- *-- * --open: the schema as the service's own boot code leaves a fresh SQLite database (SCHEMA, then every migrate() ALTER-- * and seed, then the --ensure modules run on it), for a service whose migrate() changed tables after SCHEMA. Run it-- * before converting any code (the working copy is the SQLite release). Tables come out in dependency order, one column-- * per line; the rows the boot seeded become INSERTs; SQLite text timestamps (DATETIME, CURRENT_TIMESTAMP) bring-- * SQLITE_DATE_FUNCTIONS along.-- *-- * It lists every `rowid` the server code still orders by: those tables need a `seq bigint GENERATED ALWAYS AS-- * IDENTITY UNIQUE` column (a person adds it; see docs/migrating-to-postgresql.md §7).-- */--const fs = require('fs');--const path = require('path');--const { execFileSync } = require('child_process');--const { toPg, SQLITE_DATE_FUNCTIONS, PG_ONLY_RESERVED } = require('./sqlite-schema-to-pg');----const args = process.argv.slice(2);--const Name = args[0];--if (!Name) { console.error('usage: gen-migration.js <Name> [--publishing store:prefix,…] [--no-inbox]'); process.exit(2); }--const pi = args.indexOf('--publishing');--const pubs = pi >= 0 ? args[pi + 1].split(',').map((x) => x.split(':')) : [];--const req = (m) => require(require.resolve(m, { paths: [process.cwd()] }));----const opt = (name, def) => { const i = args.indexOf(name); return i >= 0 ? args[i + 1] : def; };--const ensureList = opt('--ensure', '') ? opt('--ensure', '').split(',') : [];----/** A fresh SQLite database opened by the service's code → PostgreSQL DDL (tables in dependency order) and seed rows. */--function fromOpenedDb(expr) {-- const { createRequire } = require('module');-- const creq = createRequire(path.resolve('package.json'));-- const sdb = new Function('require', `return (${expr});`)(creq);-- for (const f of ensureList) creq(path.resolve(f)).ensureSchema(sdb);-- const stripComments = (sql) => {-- let o = ''; let q = null;-- for (let i = 0; i < sql.length; i++) {-- const c = sql[i];-- if (q) { o += c; if (c === q) q = null; continue; }-- if (c === "'" || c === '"') { q = c; o += c; continue; }-- if (c === '-' && sql[i + 1] === '-') { while (i < sql.length && sql[i] !== '\n') i++; o += '\n'; continue; }-- o += c;-- }-- return o;-- };-- const splitTop = (body) => {-- const parts = []; let depth = 0; let q = null; let cur = '';-- for (const c of body) {-- if (q) { cur += c; if (c === q) q = null; continue; }-- if (c === "'" || c === '"') { q = c; cur += c; continue; }-- if (c === '(') depth++;-- if (c === ')') depth--;-- if (c === ',' && depth === 0) { parts.push(cur.trim()); cur = ''; continue; }-- cur += c;-- }-- if (cur.trim()) parts.push(cur.trim());-- return parts;-- };-- const objs = sdb.prepare("SELECT type, name, tbl_name, sql FROM sqlite_master WHERE sql IS NOT NULL AND name NOT LIKE 'sqlite_%' ORDER BY rowid").all();-- const tables = objs.filter((o) => o.type === 'table' && !/CREATE VIRTUAL/i.test(o.sql));-- const byName = new Map(tables.map((t) => [t.name, t]));-- const order = []; const done = new Set();-- const visit = (t, stack) => {-- if (done.has(t.name) || stack.has(t.name)) return;-- stack.add(t.name);-- for (const m of t.sql.matchAll(/REFERENCES\s+"?(\w+)"?/gi)) { const d = byName.get(m[1]); if (d && d.name !== t.name) visit(d, stack); }-- done.add(t.name); order.push(t);-- };-- for (const t of tables) visit(t, new Set());-- let ddl = '';-- const tsDefault = new Set(); const pk = new Map();-- for (const t of order) {-- const sql = stripComments(t.sql);-- const i = sql.indexOf('('); const j = sql.lastIndexOf(')');-- const cols = splitTop(sql.slice(i + 1, j)).map((x) => x.replace(/\s+/g, ' '));-- for (const c of cols) {-- const m = /^"?(\w+)"? /.exec(c);-- if (!m) continue;-- if (/DEFAULT (CURRENT_TIMESTAMP|\((datetime|strftime)\()/i.test(c)) tsDefault.add(`${t.name}.${m[1]}`);-- if (/^"?\w+"? INTEGER PRIMARY KEY/i.test(c)) pk.set(t.name, { col: m[1], always: /AUTOINCREMENT/i.test(c) });-- }-- const tail = sql.slice(j + 1).trim();-- ddl += `CREATE TABLE ${t.name} (\n ${cols.join(',\n ')}\n)${tail ? ` ${tail}` : ''};\n\n`;-- }-- for (const o of objs) if (o.type === 'index') ddl += `${stripComments(o.sql).replace(/\s+/g, ' ').trim()};\n`;-- for (const o of objs) if (o.type === 'trigger') ddl += `\n${o.sql.trim()};\n`;-- for (const o of objs) if (o.type === 'view') ddl += `\n${o.sql.trim()};\n`;-- let out = toPg(ddl);-- // Seed rows the boot wrote (spaces, categories, …); timestamp defaults fill themselves in.-- const lit = (v) => (v === null ? 'NULL' : typeof v === 'number' || typeof v === 'bigint' ? String(v)-- : Buffer.isBuffer(v) ? `'\\x${v.toString('hex')}'` : `'${String(v).replace(/'/g, "''")}'`);-- let seeds = '';-- for (const t of order) {-- const rows = sdb.prepare(`SELECT * FROM "${t.name}"`).all();-- if (!rows.length) continue;-- const cols = Object.keys(rows[0]).filter((c) => !tsDefault.has(`${t.name}.${c}`));-- const k = pk.get(t.name);-- const q = (c) => (PG_ONLY_RESERVED.includes(c.toLowerCase()) ? `"${c}"` : c);-- seeds += `INSERT INTO ${t.name} (${cols.map(q).join(', ')})${k && k.always && cols.includes(k.col) ? ' OVERRIDING SYSTEM VALUE' : ''} VALUES\n`-- + `${rows.map((r) => ` (${cols.map((c) => lit(r[c])).join(', ')})`).join(',\n')};\n`;-- if (k && cols.includes(k.col)) seeds += `SELECT setval(pg_get_serial_sequence('${t.name}', '${k.col}'), (SELECT MAX(${k.col}) FROM ${t.name}));\n`;-- }-- if (seeds) out += `\n-- Rows the SQLite boot seeded\n${seeds}`;-- if (/ov_now\(\)|ov_now_iso\(\)/.test(out)) out = `${SQLITE_DATE_FUNCTIONS}\n${out}`;-- try { sdb.close(); } catch { /* in-memory */ }-- return out;--}----// The schema as the SQLite release ran it: server/db.js at HEAD (the conversion edits the working copy). A literal with--// ${…} interpolations is evaluated by loading that file and reading its SCHEMA export.--const dbFile = (() => { const i = args.indexOf('--from'); return i >= 0 ? args[i + 1] : 'server/db.js'; })();--const openExpr = opt('--open', null);--const src = openExpr ? '' : execFileSync('git', ['show', `HEAD:${dbFile}`], { encoding: 'utf8' });--const m = openExpr ? [null, ''] : /const SCHEMA = `([\s\S]*?)`;/.exec(src);--if (!m) { console.error(`${dbFile} (HEAD) has no const SCHEMA = \`…\`;`); process.exit(1); }--if (!openExpr && m[1].includes('${')) {-- const tmp = path.resolve(path.dirname(dbFile), '.db-head-schema.js');-- fs.writeFileSync(tmp, src);-- try {-- const schema = require(tmp).SCHEMA;-- if (typeof schema !== 'string') throw new Error('it does not export SCHEMA');-- m[1] = schema;-- } catch (err) { console.error(`the SCHEMA literal has interpolations and could not be evaluated: ${err.message}`); process.exit(1); } finally { fs.unlinkSync(tmp); }--}--let out = `-- phase: expand--${(openExpr ? fromOpenedDb(openExpr) : toPg(m[1])).replace(/\n{3,}/g, '\n\n')}`;--// --ensure a.js,b.js: modules whose ensureSchema(db) created more tables at boot; their SQL is captured and converted too.--if (!openExpr) {-- const i = args.indexOf('--ensure');-- if (i >= 0) {-- for (const f of args[i + 1].split(',')) {-- const sqls = [];-- const fake = { exec: (q) => sqls.push(q), prepare: () => ({ run() {}, get() {}, all: () => [] }), pragma() {}, function() {} };-- req(path.resolve(f)).ensureSchema(fake);-- out += `\n-- ${f} ensureSchema()\n${toPg(sqls.join('\n').replace(/^ {4}/gm, '')).trim()}\n`;-- }-- }--}--const SCHEMA_OF = { 'index-hooks': (x, p) => x.sequencerSchema(p), seo: (x, p) => x.redirectsSchema(p) };--for (const [store, prefix] of pubs) {-- const mod = req(`openvibe-publishing/${store}`);-- out += `\n-- openvibe-publishing/${store} (prefix ${prefix})\n${(SCHEMA_OF[store] ? SCHEMA_OF[store](mod, prefix) : mod.schema(prefix)).trim()}\n`;--}--const ev = req('openvibe-sdk/events');--const inboxTable = opt('--inbox-table', 'idempotency_receipts');--const outboxTable = opt('--outbox-table', 'event_outbox');--if (!args.includes('--no-inbox')) out += `\n-- openvibe-sdk/events inbox: one receipt per (consumer, event) handled\n${ev.inboxSchema(inboxTable).trim()}\n`;--if (!args.includes('--no-outbox')) out += `\n-- openvibe-sdk/events PostgreSQL outbox\n${ev.outboxSchema(outboxTable).trim()}\n`;--// SQLite's own copies of the tables above (created by the packages at boot) would be defined twice.--for (const t of [outboxTable, inboxTable]) {-- const re = new RegExp(`\\nCREATE TABLE ${t} \\([\\s\\S]*?\\n\\);\\n`, 'g');-- if ((out.match(re) || []).length && out.includes(`CREATE TABLE IF NOT EXISTS ${t}`)) out = out.replace(re, '\n');--}--fs.mkdirSync('migrations', { recursive: true });--fs.writeFileSync(path.join('migrations', '0001_initial.sql'), out);--console.log(`migrations/0001_initial.sql: ${(out.match(/CREATE TABLE/g) || []).length} tables`);--try {-- const hits = execFileSync('git', ['grep', '-n', 'rowid', '--', 'server', 'scripts'], { encoding: 'utf8' }).trim();-- if (hits) console.log(`rowid still used (add a seq identity column to these tables and order by it):\n${hits}`);--} catch { /* no rowid */ }ddeleted file mode 100644@@@ -1,39 +0,0 @@--{-- "name": "openvibe-asyncify",-- "lockfileVersion": 3,-- "requires": true,-- "packages": {-- "": {-- "name": "openvibe-asyncify",-- "license": "MIT",-- "dependencies": {-- "acorn": "^8.15.0",-- "acorn-walk": "^8.3.4"-- }-- },-- "node_modules/acorn": {-- "version": "8.18.0",-- "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.18.0.tgz",-- "integrity": "sha512-lGq+9yr1/GuAWaVYIHRjvvySG5/4VfKIvC8EWxStPdcDh/Ka7FG3twP6v4d5BkravUilhIAsG4Qj83t02LWUPQ==",-- "license": "MIT",-- "bin": {-- "acorn": "bin/acorn"-- },-- "engines": {-- "node": ">=0.4.0"-- }-- },-- "node_modules/acorn-walk": {-- "version": "8.3.5",-- "resolved": "https://registry.npmjs.org/acorn-walk/-/acorn-walk-8.3.5.tgz",-- "integrity": "sha512-HEHNfbars9v4pgpW6SO1KSPkfoS0xVOM/9UzkJltjlsHZmJasxg8aXkuZa7SMf8vKGIBhpUsPluQSqhJFCqebw==",-- "license": "MIT",-- "dependencies": {-- "acorn": "^8.11.0"-- },-- "engines": {-- "node": ">=0.4.0"-- }-- }-- }--}ddeleted file mode 100644@@@ -1,10 +0,0 @@--{-- "name": "openvibe-asyncify",-- "private": true,-- "description": "The mechanical half of moving a service from better-sqlite3 to openvibe-sdk/db (ADR-035). Not part of the published SDK package.",-- "license": "MIT",-- "dependencies": {-- "acorn": "^8.15.0",-- "acorn-walk": "^8.3.4"-- }--}ddeleted file mode 100644@@@ -1,164 +0,0 @@--#!/usr/bin/env python3--"""The SQLite → PostgreSQL SQL rewrites that are safe to do by text (ADR-035), over a service's server/, scripts/ and--test/ files; and a report of what needs a person.--usage: sqlfix.py [files…] (default: git ls-files server scripts test, *.js)----Rewrites:-- INSERT OR IGNORE INTO t … → INSERT INTO t … ON CONFLICT DO NOTHING (at the end of that string literal)-- x COLLATE NOCASE → lower(x)-- LIKE / NOT LIKE → ILIKE / NOT ILIKE (SQLite's LIKE ignores ASCII case)-- json_each(?) → jsonb_array_elements_text(?::jsonb) (a JSON text parameter; column "value")-- IFNULL( → COALESCE(-- db.transaction(fn)() → db.tx(fn); db.inTransaction → db.inTransaction()-- sqlite_master table lists → information_schema.tables of the current schema-- pragma_table_info('t') → information_schema.columns of t-- sequencer.stamp(doc) → sequencer.stamp(store.db, doc) (openvibe-publishing 1.0)--Reported (fix by hand): INSERT OR REPLACE, rowid, scalar MAX(a, b)/MIN(a, b), HAVING on an alias, WITH RECURSIVE (the--seed needs the column's collation), datetime/strftime/julianday, IFNULL, GROUP_CONCAT, json_extract, lastInsertRowid,--.transaction(, setImmediate/nextTick near a transaction, createInbox, PRAGMA."""--import re, subprocess, sys--files = sys.argv[1:] or [f for f in subprocess.run(['git', 'ls-files', 'server', 'scripts', 'test'], capture_output=True, text=True).stdout.split() if f.endswith('.js')]--REPORT = [-- (r'INSERT OR REPLACE', 'INSERT OR REPLACE → ON CONFLICT (key) DO UPDATE SET … (qualify the target columns)'),-- (r'\browid\b', 'rowid → a seq identity column'),-- (r'\b(MAX|MIN)\(\s*[\w.]+\s*,', 'scalar MAX/MIN → GREATEST/LEAST'),-- (r'HAVING\s+[a-z_]+\s*[<>=!]', 'HAVING on an alias → repeat the aggregate'),-- (r'SELECT DISTINCT', 'SELECT DISTINCT with ORDER BY on an expression not selected fails; a DISTINCT over a LEFT JOIN is usually WHERE … OR EXISTS (…)'),-- (r'GROUP BY', 'GROUP BY: every selected bare column must be grouped (SQLite took a bare column from the MAX/MIN row: DISTINCT ON)'),-- (r'WITH RECURSIVE', 'recursive seed: cast ? to the column type and collation (?::text COLLATE "C")'),-- (r"datetime\(|strftime\(|julianday\(", 'SQLite date functions (with SQLITE_DATE_FUNCTIONS in the migration, datetime(t[, modifier]) and julianday(t) exist; strftime does not)'),-- (r'GROUP_CONCAT|json_extract|json_group_array|json_object\(', 'SQLite functions'),-- (r'lastInsertRowid', 'lastInsertRowid → RETURNING'),-- (r'\.transaction\(', 'db.transaction(fn)() → db.tx(fn)'),-- (r'setImmediate|process\.nextTick', 'deferred work: after a commit → db.afterCommit'),-- (r'createInbox\b', 'createInbox → createPgInbox (awaited; receipts table in the migration)'),-- (r'\bPRAGMA\b|\.pragma\(', 'PRAGMA'),-- (r'\b(db|this\.db)\.exec\(', 'DDL at boot on the serving handle: the serving role cannot create tables; the table belongs in a migration'),-- (r'\b(SUM|AVG)\((?![^)]*\bCASE\b)[^)]*\)(?!\s*::)(?!, *0\)::)', 'SUM/AVG of a bigint column is numeric, returned as text: cast it (COALESCE(SUM(x), 0)::bigint) unless the column is double precision'),-- (r'(@\w+|[^\w]\?) IS (NOT )?NULL(?!::)', 'a parameter tested with IS NULL may need a cast (@p::bigint IS NULL) when PostgreSQL cannot infer its type'),-- (r'COALESCE\(\?,', 'COALESCE(?, col): cast the parameter (?::text) when PostgreSQL cannot infer its type from the other arguments'),-- (r'UNIQUE constraint|constraint failed|SQLITE_[A-Z]', "SQLite error text → err.code ('23505' unique, '23503' foreign key, '23514' check) with err.table/err.constraint/err.detail"),--]--changed = []--# Table columns from the service's migrations, so upsert right-hand sides can be qualified even for columns the INSERT--# does not list (a counter bumped in place, a timestamp cleared).--TABLE_COLS = {}--ALWAYS_IDENTITY = {} # table → its GENERATED ALWAYS identity column--import glob as _glob--for mf in sorted(_glob.glob('migrations/*.sql')):-- ddl = open(mf).read()-- for tm in re.finditer(r'CREATE TABLE (?:IF NOT EXISTS )?(\w+) \(([\s\S]*?)\n\);', ddl):-- cols = [c.group(1).strip('"') for c in re.finditer(r'^\s*("?\w+"?)\s+(?:text|bigint|integer|double|boolean|bytea|jsonb|timestamptz|numeric|real)', tm.group(2), re.M)]-- TABLE_COLS.setdefault(tm.group(1), set()).update(cols)-- for am in re.finditer(r'ALTER TABLE (\w+) ADD COLUMN (\w+)', ddl):-- TABLE_COLS.setdefault(am.group(1), set()).add(am.group(2))-- for tm in re.finditer(r'CREATE TABLE (?:IF NOT EXISTS )?(\w+) \(([\s\S]*?)\n\);', ddl):-- for im in re.finditer(r'^\s*"?(\w+)"?\s+bigint GENERATED ALWAYS AS IDENTITY', tm.group(2), re.M):-- ALWAYS_IDENTITY.setdefault(tm.group(1), im.group(1))--# A migration carrying SQLITE_DATE_FUNCTIONS (text timestamps kept as SQLite wrote them): CURRENT_TIMESTAMP in code--# becomes ov_now() (PostgreSQL's would be a timestamptz, cast to other text), strftime's ISO 'now' ov_now_iso().--SQLITE_DATES = any('CREATE FUNCTION ov_now()' in open(mf).read() for mf in _glob.glob('migrations/*.sql'))--for p in files:-- s = open(p).read(); o = s-- if SQLITE_DATES:-- s = re.sub(r'\bCURRENT_TIMESTAMP\b', 'ov_now()', s)-- s = re.sub(r"strftime\('%Y-%m-%dT%H:%M:%fZ', ?'now'\)", 'ov_now_iso()', s)-- s = re.sub(r"strftime\('%Y-%m-%dT%H:%M:%fZ', ?'now', ?('[^']*'|\?)\)", r'ov_now_iso(\1)', s)-- s = re.sub(r"\bdate\('now', ?('[^']*'|\?)\)", r"substr(datetime('now', \1), 1, 10)", s)-- s = re.sub(r"(?<![\w.])date\(([\w.]+)\)(?= AS | >| <| =|,|\))", r'substr(datetime(\1), 1, 10)', s)-- while 'INSERT OR IGNORE INTO' in s:-- i = s.index('INSERT OR IGNORE INTO'); q = s[i - 1] if s[i - 1] in '\'"`' else '`'-- j = s.index(q, i)-- s = s[:i] + 'INSERT INTO' + s[i + len('INSERT OR IGNORE INTO'):j] + ' ON CONFLICT DO NOTHING' + s[j:]-- s = re.sub(r'(\w+\((?:[^()]|\([^()]*\))*\)) COLLATE NOCASE', r'lower(\1)', s) # an expression: COALESCE(a, b) COLLATE NOCASE-- s = re.sub(r'([\w.]+) COLLATE NOCASE', r'lower(\1)', s)-- s = re.sub(r'(?<![IN\w])LIKE (?=[@?:$\'(]|[a-z])', 'ILIKE ', s)-- s = s.replace('json_each(?)', 'jsonb_array_elements_text(?::jsonb)')-- s = re.sub(r'\bIFNULL\(', 'COALESCE(', s)-- # SQLite's null-safe x IS ? / x IS NOT ? (a parameter) are IS [NOT] DISTINCT FROM in PostgreSQL.-- s = re.sub(r'([\w.]+) IS NOT (\?|@\w+)', r'\1 IS DISTINCT FROM \2', s)-- s = re.sub(r'([\w.]+) IS (\?|@\w+)', r'\1 IS NOT DISTINCT FROM \2', s)-- # assert.throws(() => db.prepare(…).run(…), re): the statement rejects now.-- s = re.sub(r"assert\.throws\(\(\) => ((?:t\.)?[\w.]*db\.prepare\((?:[^()]|\([^()]*\))*\)\.(?:run|get|all)\((?:[^()]|\([^()]*\))*\)), ", r"await assert.rejects(\1, ", s)-- # A jsonb column (the SDK outbox's envelope) comes back as an object.-- s = re.sub(r"(?<!\? )JSON\.parse\((\w+)\.envelope\)", r"(typeof \1.envelope === 'string' ? JSON.parse(\1.envelope) : \1.envelope)", s)-- # db.transaction(fn)() → db.tx(fn) (balanced parentheses; the codemod then awaits it and makes fn async).-- out, i = [], 0-- while True:-- j = s.find('.transaction(', i)-- if j < 0: out.append(s[i:]); break-- k, depth = j + len('.transaction('), 1-- while k < len(s) and depth:-- depth += {'(': 1, ')': -1}.get(s[k], 0); k += 1-- if s[k:k + 2] == '()':-- out.append(s[i:j] + '.tx(' + s[j + len('.transaction('):k]); i = k + 2-- else:-- out.append(s[i:k]); i = k-- s = ''.join(out)-- # const f = db.transaction((a, b) => { … }) (called later as f(a, b); an immediately invoked one was rewritten just-- # above) → const f = (a, b) => db.tx(async () => { … })-- s = re.sub(r'const (\w+) = (\w+(?:\.\w+)?)\.transaction\(\(([^()]*)\) => \{', r'const \1 = (\3) => \2.tx(async () => {', s)-- # An unquoted alias is folded to lower case in PostgreSQL (AS totalViews comes back as totalviews): quote-- # camelCase aliases inside SQL string literals.-- def quote_aliases(lit):-- body = lit.group(0)-- if not re.search(r'\bSELECT\b', body): return body-- q = '\\"' if body[0] == '"' else '"'-- return re.sub(r'\b(AS|as) ([a-z]+[A-Z]\w*)\b', lambda m: f'{m.group(1)} {q}{m.group(2)}{q}', body)-- s = re.sub(r'`[^`]*`|\'(?:[^\'\\\n]|\\.)*\'|"(?:[^"\\\n]|\\.)*"', quote_aliases, s)-- # Column names PostgreSQL reserves (SQLite does not): quoted inside SQL string literals ("window", "user").-- def quote_reserved(lit):-- body = lit.group(0)-- if not re.search(r'\b(SELECT|INSERT|UPDATE|DELETE|CREATE)\b', body): return body-- q = '\\"' if body[0] == '"' else '"' # inside a double-quoted JS string the quotes are escaped-- return re.sub(r"(?<![.\w\"$\\'@:])(window|user)(?![\w\"(\\'])", lambda m: q + m.group(1) + q, body)-- s = re.sub(r'`[^`]*`|\'(?:[^\'\\\n]|\\.)*\'|"(?:[^"\\\n]|\\.)*"', quote_reserved, s)-- # better-sqlite3's db.inTransaction was a property; openvibe-sdk/db's is a function.-- s = re.sub(r'\b(db|this\.db|store\.db)\.inTransaction\b(?!\s*\()', r'\1.inTransaction()', s)-- # ON CONFLICT … DO UPDATE SET x = x + 1 / COALESCE(excluded.x, x): a bare column on the right is ambiguous in-- # PostgreSQL; qualify it with the table (columns of the INSERT list only; excluded.x and the targets stay).-- def qualify(m):-- table, cols, rest = m.group(1), [c.strip() for c in m.group(2).split(',')], m.group(3)-- cols = sorted(set(cols) | TABLE_COLS.get(table, set()), key=len, reverse=True)-- k = rest.index('DO UPDATE SET') + len('DO UPDATE SET')-- head, sets = rest[:k], rest[k:]-- def fix_rhs(a):-- lhs, eq, rhs = a.partition('=')-- if not eq: return a-- for c in cols:-- rhs = re.sub(r'(?<![.\w@:$])%s\b(?!\s*\()' % re.escape(c), f'{table}.{c}', rhs)-- return lhs + eq + rhs-- parts = re.split(r'(,\s*(?=\w+\s*=))', sets)-- return f'INSERT INTO {table} ({m.group(2)}){head}' + ''.join(fix_rhs(x) if not re.match(r',\s*$', x) else x for x in parts)-- s = re.sub(r'INSERT INTO (\w+) \(([^)]*)\)((?:(?!INSERT INTO)[^`\'"])*?DO UPDATE SET[^`\'"]*)', qualify, s)-- # INSERT INTO t (id, …) VALUES with an explicit id into a GENERATED ALWAYS identity (tests, fixtures, imports):-- # PostgreSQL refuses it without OVERRIDING SYSTEM VALUE.-- def override(m):-- table, cols = m.group(1), [c.strip().strip('"') for c in m.group(2).split(',')]-- return m.group(0) + ' OVERRIDING SYSTEM VALUE' if ALWAYS_IDENTITY.get(table) in cols else m.group(0)-- s = re.sub(r'INSERT INTO (\w+) \(([^)]*)\)(?! OVERRIDING)(?=\s*(?:VALUES|SELECT))', override, s)-- # (await db.prepare('INSERT …').run(…)).lastInsertRowid: the id comes back through RETURNING.-- s = re.sub(r"(prepare\()(`|'|\")(INSERT INTO (\w+)(?:(?!\2).)*?)(\2\)\s*\.run\((?:[^()]|\([^()]*\))*\)\)\s*\.lastInsertRowid)",-- lambda m: m.group(0) if 'RETURNING' in m.group(3) else m.group(1) + m.group(2) + m.group(3) + ' RETURNING id' + m.group(5), s)-- # Tests matching SQLite's constraint messages: PostgreSQL's say "duplicate key", "violates … constraint".-- s = re.sub(r'/UNIQUE constraint failed[^/\n]*/|/UNIQUE/', '/duplicate key/', s)-- s = re.sub(r'/CHECK constraint failed[^/\n]*/', '/violates check constraint/', s)-- s = re.sub(r'/NOT NULL constraint failed[^/\n]*/', '/violates not-null constraint/', s)-- s = re.sub(r'/FOREIGN KEY constraint failed[^/\n]*/', '/violates foreign key constraint/', s)-- # openvibe-publishing 1.0: the index sequencer stamps through a handle (the ambient transaction joins through store.db).-- handle = 'store.db' if 'store.db' in s else 'db'-- s = re.sub(r'((?:store\.)?sequencer\.stamp\()(?!store\.db|db,|t,)', lambda m: m.group(1) + handle + ', ', s)-- s = re.sub(r"(SELECT name FROM sqlite_master WHERE type (?:= 'table'|IN \('table', ?'view'\))) AND name NOT (?:I)?LIKE 'sqlite_%'", r"\1", s)-- s = re.sub(r"SELECT name FROM sqlite_master WHERE type (?:= 'table'|IN \('table', ?'view'\))",-- "SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()", s)-- # SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ? / 'x' → the table in the current schema-- s = re.sub(r"SELECT 1( AS \w+)? FROM sqlite_master WHERE type = 'table' AND name = (\?|'\w+')",-- r"SELECT 1\1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = \2", s)-- s = re.sub(r"SELECT name FROM pragma_table_info\('(\w+)'\)",-- r"SELECT column_name AS name FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = '\1' ORDER BY ordinal_position", s)-- if s != o: open(p, 'w').write(s); changed.append(p)-- for n, line in enumerate(s.split('\n'), 1):-- for pat, why in REPORT:-- if re.search(pat, line) and not line.lstrip().startswith(('//', '*')):-- print(f'CHECK {p}:{n}: {why}: {line.strip()[:120]}')--print('rewrote: ' + (', '.join(changed) or 'nothing'))ddeleted file mode 100644@@@ -1,98 +0,0 @@--'use strict';--/**-- * A service's SQLite DDL (the SCHEMA string its db.js ran) → PostgreSQL DDL with the same meaning (ADR-035), for-- * migrations/0001_initial.sql:-- * INTEGER PRIMARY KEY AUTOINCREMENT → bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY (the importer keeps the ids)-- * TEXT → text COLLATE "C" (SQLite compares text bytewise; "C" sorts and compares the same way)-- * INTEGER → bigint (epoch milliseconds and counters; openvibe-sdk/db returns int8 as Number)-- * REAL → double precision, BLOB → bytea-- * CREATE … IF NOT EXISTS → CREATE … (a migration runs once)-- * Types only in column position (upper case, as SQLite schemas are written); comments are kept. An immutable-row-- * trigger (BEFORE UPDATE/DELETE … RAISE(ABORT, msg)) becomes a PL/pgSQL function and trigger; any other trigger is-- * refused (write it by hand).-- *-- * const { toPg } = require('openvibe-sdk/tools/asyncify/sqlite-schema-to-pg'); (from a checkout)-- */--/** Words PostgreSQL reserves that SQLite takes as plain column names. */--const PG_ONLY_RESERVED = ['window', 'user', 'freeze', 'analyse', 'both', 'leading', 'trailing', 'symmetric', 'asymmetric', 'variadic',-- 'lateral', 'ilike', 'similar', 'verbose', 'tablesample', 'concurrently', 'overlaps', 'authorization', 'binary', 'collation',-- 'only', 'placing', 'array', 'fetch', 'current_role', 'session_user', 'current_user', 'current_catalog', 'current_schema'];----function toPg(sql) {-- // A SQLite row trigger → a PL/pgSQL function and a trigger. Its WHEN condition (which may hold a subquery, not-- // allowed in a PostgreSQL trigger's WHEN) goes inside the function as IF cond THEN body END IF; RAISE(ABORT, msg)-- // becomes RAISE EXCEPTION; a BEFORE trigger lets the row pass (NEW, or OLD for a delete), an AFTER one returns NULL.-- const cond = (c) => c.replace(/\s+/g, ' ').trim().replace(/ IS NOT (OLD|NEW)\./g, ' IS DISTINCT FROM $1.').replace(/(\b(?:OLD|NEW)\.\w+) IS (OLD|NEW)\./g, '$1 IS NOT DISTINCT FROM $2.');-- sql = sql.replace(/CREATE TRIGGER(?: IF NOT EXISTS)?\s+(\w+)\s+(BEFORE|AFTER)\s+(INSERT|UPDATE(?:\s+OF\s+[\w\s,]+?)?|DELETE)\s+ON\s+(\w+)(?:\s+FOR EACH ROW)?(?:\s+WHEN\s+([\s\S]+?))?\s+BEGIN\s+([\s\S]+?)\s*END;/g,-- (m, name, when, op, table, whenCond, body) => {-- const stmts = body.replace(/SELECT RAISE\(ABORT, '([^']*)'\);?/g, "RAISE EXCEPTION '$1';")-- .replace(/CURRENT_TIMESTAMP/g, 'ov_now()').trim().replace(/;?$/, ';');-- const row = when === 'AFTER' ? 'NULL' : /^DELETE/.test(op) ? 'OLD' : 'NEW';-- const inner = whenCond ? `IF ${cond(whenCond)} THEN ${stmts} END IF;` : stmts;-- return `CREATE FUNCTION ${name}() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN ${inner.replace(/\s+/g, ' ')} RETURN ${row}; END $$;\n`-- + `CREATE TRIGGER ${name} ${when} ${op.replace(/\s+/g, ' ')} ON ${table} FOR EACH ROW EXECUTE FUNCTION ${name}();`;-- });-- if (sql.split(';').some((st) => /CREATE\s+TRIGGER/i.test(st) && !/FOR EACH ROW[\s\S]*EXECUTE FUNCTION/.test(st) && !/\$\$/.test(st))) {-- throw new Error('sqlite-schema-to-pg: a trigger this converter does not read needs writing as PL/pgSQL by hand');-- }-- // COLLATE NOCASE: in an index, the index is on lower(col) (queries compare lower(col) too: sqlfix.py); on a column-- // definition it goes (a case-insensitive UNIQUE there needs a lower() unique index, written by hand).-- sql = sql.replace(/(CREATE (?:UNIQUE )?INDEX[^;]*?\()([^;]*?)\)/g, (m, head, cols) => head + cols.replace(/(\w+) COLLATE NOCASE/g, 'lower($1)') + ')');-- if (/^\s*\w+\s+TEXT\b[^\n]*COLLATE NOCASE[^\n]*UNIQUE|UNIQUE[^\n]*COLLATE NOCASE/m.test(sql)) throw new Error('sqlite-schema-to-pg: a case-insensitive UNIQUE column needs a unique index on lower(col), by hand');-- sql = sql.replace(/^(\s*\w+\s+TEXT\b[^\n]*?) COLLATE NOCASE/gm, '$1');-- // Column names PostgreSQL reserves (SQLite does not) are quoted: a column definition, and inside index/unique lists.-- // The service's queries must quote them too (Billing's settings.freeze).-- const R = PG_ONLY_RESERVED.join('|');-- sql = sql.replace(new RegExp(`^(\\s*)(${R})(\\s+[A-Za-z])`, 'gim'), '$1"$2"$3').replace(new RegExp(`([(,]\\s*)(${R})(?=\\s*[,)])`, 'gi'), '$1"$2"');-- return sql-- .replace(/\bINTEGER PRIMARY KEY AUTOINCREMENT\b/g, 'bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY')-- // An INTEGER PRIMARY KEY that references another table is that row's key, not a counter.-- .replace(/\bINTEGER PRIMARY KEY\b(?! AUTOINCREMENT)(?=[^,\n]*\bREFERENCES\b)/g, 'bigint PRIMARY KEY')-- // A bare INTEGER PRIMARY KEY is SQLite's rowid: code may insert without it or with an explicit value.-- .replace(/\bINTEGER PRIMARY KEY\b(?! AUTOINCREMENT)/g, 'bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY')-- .replace(/CREATE (UNIQUE )?INDEX IF NOT EXISTS/g, (m, u) => `CREATE ${u || ''}INDEX`)-- .replace(/CREATE TABLE IF NOT EXISTS/g, 'CREATE TABLE')-- .replace(/\)\s*WITHOUT ROWID\s*;/g, ');')-- .replace(/CREATE VIEW IF NOT EXISTS/g, 'CREATE VIEW')-- .replace(/^(\s*"?\w+"?\s+)TEXT\b/gm, '$1text COLLATE "C"')-- .replace(/^(\s*"?\w+"?\s+)INTEGER\b/gm, '$1bigint')-- .replace(/^(\s*"?\w+"?\s+)REAL\b/gm, '$1double precision')-- .replace(/^(\s*"?\w+"?\s+)BLOB\b/gm, '$1bytea')-- // SQLite text timestamps stay text in the same format; their defaults use SQLITE_DATE_FUNCTIONS below.-- .replace(/^(\s*"?\w+"?\s+)DATETIME\b/gm, '$1text COLLATE "C"')-- .replace(/DEFAULT CURRENT_TIMESTAMP\b/g, 'DEFAULT ov_now()')-- .replace(/DEFAULT \(datetime\('now'\)\)/g, 'DEFAULT ov_now()')-- .replace(/DEFAULT \(strftime\('%Y-%m-%dT%H:%M:%fZ', ?'now'\)\)/g, 'DEFAULT ov_now_iso()');--}----/**-- * SQLite's text timestamps and date functions on PostgreSQL, for a service whose columns hold 'YYYY-MM-DD HH:MM:SS'-- * (UTC) text: the stored values, their comparisons and what the API returns stay exactly as they were. ov_now() is-- * CURRENT_TIMESTAMP's text (per statement, as SQLite), ov_now_iso([modifier]) strftime('%Y-%m-%dT%H:%M:%fZ', 'now'[, modifier]);-- * datetime(t [, modifier]) and julianday(t) read 'now', SQLite text and ISO 8601 (a zone, or none for UTC) and give-- * NULL for what they cannot read, as SQLite does. A modifier is an interval ('-7 days', '+1 hour').-- */--const SQLITE_DATE_FUNCTIONS = `-- SQLite's text timestamps and date functions (openvibe-sdk tools/asyncify SQLITE_DATE_FUNCTIONS).--CREATE FUNCTION ov_ts(t text) RETURNS timestamp LANGUAGE plpgsql STABLE AS $$--BEGIN-- IF t IS NULL THEN RETURN NULL; END IF;-- IF t = 'now' THEN RETURN statement_timestamp() AT TIME ZONE 'UTC'; END IF;-- IF t ~ '\\d\\d:\\d\\d(:\\d\\d(\\.\\d+)?)?\\s*(Z|[+-]\\d\\d(:?\\d\\d)?)$' THEN RETURN t::timestamptz AT TIME ZONE 'UTC'; END IF;-- RETURN t::timestamp;--EXCEPTION WHEN others THEN RETURN NULL;--END $$;--CREATE FUNCTION ov_now() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD HH24:MI:SS') $$;--CREATE FUNCTION ov_now_iso() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;--CREATE FUNCTION ov_now_iso(modifier text) RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char((statement_timestamp() AT TIME ZONE 'UTC') + modifier::interval, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;--CREATE FUNCTION datetime(t text, modifier text DEFAULT NULL) RETURNS text LANGUAGE plpgsql STABLE AS $$--DECLARE ts timestamp := ov_ts(t);--BEGIN-- IF ts IS NULL THEN RETURN NULL; END IF;-- IF modifier IS NOT NULL THEN ts := ts + modifier::interval; END IF;-- RETURN to_char(ts, 'YYYY-MM-DD HH24:MI:SS');--EXCEPTION WHEN others THEN RETURN NULL;--END $$;--CREATE FUNCTION julianday(t text) RETURNS double precision LANGUAGE sql STABLE AS $$ SELECT extract(epoch FROM ov_ts(t))::double precision / 86400.0 + 2440587.5 $$;--`;----module.exports = { toPg, SQLITE_DATE_FUNCTIONS, PG_ONLY_RESERVED };ddeleted file mode 100644@@@ -1,14 +0,0 @@--#!/usr/bin/env node--'use strict';--// Every column declared INTEGER/REAL holding a value of another storage class (SQLite's affinity lets text in).--const Database = require(require.resolve('better-sqlite3', { paths: [process.cwd()] }));--const db = new Database(process.argv[2], { readonly: true });--for (const { name } of db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'").all()) {-- for (const c of db.prepare(`PRAGMA table_info("${name}")`).all()) {-- const t = String(c.type).toUpperCase();-- const want = /INT/.test(t) ? 'integer' : /REAL|FLOA|DOUB/.test(t) ? 'real' : null;-- if (!want) continue;-- const rows = db.prepare(`SELECT typeof("${c.name}") AS k, COUNT(*) AS n, MIN("${c.name}") AS ex FROM "${name}" WHERE "${c.name}" IS NOT NULL AND typeof("${c.name}") NOT IN ('${want}'${want === 'real' ? ", 'integer'" : ''}) GROUP BY 1`).all();-- for (const r of rows) console.log(`${name}.${c.name} (${t}): ${r.n} ${r.k} value(s), e.g. ${String(r.ex).slice(0, 40)}`);-- }--}
@@ -3,6 +3,15 @@
@@ -3,6 +3,15 @@
All notable changes to `openvibe-sdk`. The package follows semver; while it is `0.x`, a minor
All notable changes to `openvibe-sdk`. The package follows semver; while it is `0.x`, a minor
release may change an API and says so here.
release may change an API and says so here.
+## 0.38.0 (2026-10-10)
+
+- **Removed: the one-time SQLite → PostgreSQL migration tooling** (plan T1, "Delete the migration era"). Every
+ OpenVibe service runs on PostgreSQL and none calls them: `importSqlite` and `runSqliteMigration` (`openvibe-sdk/db`),
+ `src/db/sqlite-import.js`, `src/db/sqlite-cli.js`, the `tools/asyncify/` conversion scripts and
+ `docs/migrating-to-postgresql.md`. A consumer that still imported either function must drop it; nothing else changes.
+- Kept: the better-sqlite3-shaped statements on PostgreSQL (`db.prepare`), and the SQLite outbox and inbox
+ (`createOutbox`/`createInbox`) for a small app that keeps them in SQLite (OpenVibe.Examples' webhook consumer does).
+
## 0.37.2 (2026-10-10)
## 0.37.2 (2026-10-10)
- `openvibe-sdk/realtime`: each event's callback gets its opaque `cursor` (the SSE id Events sent) beside `seq`:
- `openvibe-sdk/realtime`: each event's callback gets its opaque `cursor` (the SSE id Events sent) beside `seq`:
@@ -344,7 +344,7 @@ const platform = createMockPlatform({
@@ -344,7 +344,7 @@ const platform = createMockPlatform({
const client = createClient({ fetch: platform.fetch, tokenProvider: createServiceTokenClient({ clientId: 'app_01K5WZX7S7Q4D2B8N3M6V1C9TR', clientSecret: 's', fetch: platform.fetch }) });
const client = createClient({ fetch: platform.fetch, tokenProvider: createServiceTokenClient({ clientId: 'app_01K5WZX7S7Q4D2B8N3M6V1C9TR', clientSecret: 's', fetch: platform.fetch }) });
```
```
-For a service's own database, `createTestDb({ migrations, seed, seedKey })` returns a migrated PGlite (or, with `store: 'pg'`, a schema of its own on the test containers). The migrated and seeded PGlite is saved once as a snapshot, keyed by the migrations' contents, and later test processes load it instead of migrating (`snapshot: 'hit' | 'built' | 'off'`, `setupMs`; `OV_TEST_SNAPSHOT=0` turns it off, `OV_TEST_SNAPSHOT_DIR` moves it). See [docs/migrating-to-postgresql.md](docs/migrating-to-postgresql.md#5-tests).
+For a service's own database, `createTestDb({ migrations, seed, seedKey })` returns a migrated PGlite (or, with `store: 'pg'`, a schema of its own on the test containers). The migrated and seeded PGlite is saved once as a snapshot, keyed by the migrations' contents, and later test processes load it instead of migrating (`snapshot: 'hit' | 'built' | 'off'`, `setupMs`; `OV_TEST_SNAPSHOT=0` turns it off, `OV_TEST_SNAPSHOT_DIR` moves it).
The mock answers at the real public origins with real RS256 tokens and checks audience, capability, namespace and sandbox the way the services do. It covers:
The mock answers at the real public origins with real RS256 tokens and checks audience, capability, namespace and sandbox the way the services do. It covers:
@@ -425,9 +425,8 @@ const thumbs = createQueue({ valkey, name: 'thumbs' });
@@ -425,9 +425,8 @@ const thumbs = createQueue({ valkey, name: 'thumbs' });
thumbs.process(async (job) => { … }, { concurrency: 4 });
thumbs.process(async (job) => { … }, { concurrency: 4 });
```
```
-Install the drivers the service uses: `pg` and `iovalkey` (production), `@electric-sql/pglite` (tests), `better-sqlite3` (only for the one-time `importSqlite`).
+Install the drivers the service uses: `pg` and `iovalkey` (production), `@electric-sql/pglite` (tests). `better-sqlite3` is only for a small app that keeps its outbox or inbox in SQLite (`createOutbox`/`createInbox`).
-Moving an existing service from SQLite: [docs/migrating-to-postgresql.md](docs/migrating-to-postgresql.md) (the schema mapping, the code idioms, per-process state to Valkey, tests, the rehearsed import and the production switch).
### Browser without a bundler
### Browser without a bundler
@@ -3,7 +3,7 @@
@@ -3,7 +3,7 @@
"stage": "alpha",
"stage": "alpha",
"code": true,
"code": true,
"wave": "W2",
"wave": "W2",
- "version": "0.37.2",
+ "version": "0.38.0",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30",
"plan": "OpenVibe End-to-End Realignment & Implementation Plan, revision 3 (20 Sep 2026), §3.2; roadmap §30",
"deployed": false,
"deployed": false,
"deployment": "none: a package released as git tags; services and apps pin a tag",
"deployment": "none: a package released as git tags; services and apps pin a tag",
@@ -22,7 +22,7 @@
@@ -22,7 +22,7 @@
"mock platform for tests (openvibe-sdk/testing)",
"mock platform for tests (openvibe-sdk/testing)",
"createTestDb: one migrated test database per process, loaded from a PGlite snapshot keyed by the migrations (OV_TEST_SNAPSHOT=0 turns it off)",
"createTestDb: one migrated test database per process, loaded from a PGlite snapshot keyed by the migrations (OV_TEST_SNAPSHOT=0 turns it off)",
"ESM entries, TypeScript types and a browser bundle",
"ESM entries, TypeScript types and a browser bundle",
- "db (ADR-035): the async PostgreSQL data layer (node-postgres through PgBouncer; PGlite in tests) with safe sql fragments, retrying transactions and savepoints, versioned expand/migrate/contract migrations, and importSqlite for the one-time SQLite move with verification",
+ "db (ADR-035): the async PostgreSQL data layer (node-postgres through PgBouncer; PGlite in tests) with safe sql fragments, retrying transactions and savepoints, versioned expand/migrate/contract migrations",
"valkey, cache, queue, pubsub and a shared limits store on Valkey 9, each with an in-process twin for tests",
"valkey, cache, queue, pubsub and a shared limits store on Valkey 9, each with an in-process twin for tests",
"commerce (openvibe-sdk/commerce): the shared Billing client for Tips and VIP, with intentKey and receipt",
"commerce (openvibe-sdk/commerce): the shared Billing client for Tips and VIP, with intentKey and receipt",
"events: the PostgreSQL outbox (lease-claimed, several relays per table) and inbox (handler in its transaction)",
"events: the PostgreSQL outbox (lease-claimed, several relays per table) and inbox (handler in its transaction)",
@@ -1,4 +1,4 @@
@@ -1,4 +1,4 @@
-// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.37.2, MIT). Do not edit.
+// Generated by scripts/browser-bundle.js from browser.js (openvibe-sdk 0.38.0, MIT). Do not edit.
// One self-contained ES module: core at the top level plus the browser-safe subpaths as namespaces
// One self-contained ES module: core at the top level plus the browser-safe subpaths as namespaces
// (auth, registry, modules, realtime, media, community, bot, search, chat, ai, jobs, tools, projects, geo). No server-only code: no service tokens, JWT verification, events, identity.
// (auth, registry, modules, realtime, media, community, bot, search, chat, ai, jobs, tools, projects, geo). No server-only code: no service tokens, JWT verification, events, identity.
/* eslint-disable */
/* eslint-disable */
@@ -42,7 +42,7 @@ const { paginate, offsetPager } = __load(7);
@@ -42,7 +42,7 @@ const { paginate, offsetPager } = __load(7);
const semver = __load(6);
const semver = __load(6);
/** The package version (test/version.test.js keeps it equal to package.json). */
/** The package version (test/version.test.js keeps it equal to package.json). */
-const SDK_VERSION = '0.37.2';
+const SDK_VERSION = '0.38.0';
module.exports = {
module.exports = {
createClient,
createClient,
deleted file mode 100644
deleted file mode 100644
@@ -1,192 +0,0 @@
@@ -1,192 +0,0 @@
-# Moving a service from SQLite to PostgreSQL and Valkey
-
-ADR-035 (and its amendment) moves every OpenVibe service from its own synchronous SQLite database to:
-- **PostgreSQL 18**, through PgBouncer in transaction mode;
-- **Valkey 9**, for shared, non-authoritative state;
-
-and makes each service async from end to end. This guide is how every migration is done, so the results look the same everywhere. Read it with the roadmap's engineering standards (section 4B.7): async request paths, bounded pools, Valkey for shared state, indexes for every query shape, keyset pagination, no N+1 queries, and measured budgets.
-
-## 0. What you get from the platform
-
-- **The data role on the host** (OpenVibe.Host `roles/data/`): PostgreSQL, PgBouncer, pgBackRest and Valkey on loopback.
- - `sudo /opt/openvibe.host/roles/data/add-service.sh <svc>` gives the service a database `ov_<svc>`, an owner role, a pooled runtime role (DML only, 15 s statement timeout) and a Valkey user confined to `ov:<svc>:*`.
- - It writes four settings into `/etc/openvibe/<svc>.env`: `DATABASE_URL` (runtime role, through PgBouncer), `DATABASE_DIRECT_URL` (owner role: migrations), `VALKEY_URL` and `VALKEY_PREFIX`.
-- **openvibe-sdk ≥ 0.15.0:**
- - `openvibe-sdk/db`: `createDb`, the `sql` tag, `tx`, `migrate` and `importSqlite`;
- - `openvibe-sdk/valkey`, `cache`, `queue` and `pubsub`, and `limits` with `createValkeyLimitStore`;
- - `openvibe-sdk/events`: `createPgOutbox`, `createPgInbox`, `outboxSchema` and `inboxSchema`.
-- **Tests:** `@electric-sql/pglite` runs real PostgreSQL in-process. `openvibe-sdk/scripts/test-services.sh up` starts the production-shaped container set (PostgreSQL 18, PgBouncer and Valkey) for integration runs.
-
-## 1. Dependencies
-
-```
-npm i pg iovalkey
-npm i -D @electric-sql/pglite
-# keep better-sqlite3 until the production import is done (the importer reads the old file); drop it after
-```
-
-Pin `openvibe-sdk` to `v0.15.0` or later. If the service uses `openvibe-publishing` or `openvibe-shared/config` / `analytics`, pin their PostgreSQL majors (see their CHANGELOGs).
-
-## 2. The schema, as migrations
-
-Create `migrations/0001_initial.sql` (`-- phase: expand`). It reproduces today's schema in PostgreSQL types, so the data can be imported as it is:
-
-| SQLite today | PostgreSQL |
-|---|---|
-| `INTEGER PRIMARY KEY AUTOINCREMENT` | `bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY` (the importer keeps the values and moves the sequence) |
-| `INTEGER` holding 0/1 | `boolean` (and change the code: `row.flag === 1` becomes `row.flag`) |
-| `INTEGER` epoch milliseconds (`created_at INTEGER NOT NULL`) | **`bigint`**, kept in milliseconds, so the API still sends the same numbers |
-| `DATETIME DEFAULT CURRENT_TIMESTAMP` / ISO text | `timestamptz NOT NULL DEFAULT now()`; the layer returns ISO-8601 strings (check what the API sent before and keep the format where clients depend on it) |
-| `TEXT` holding JSON | `jsonb` (and stop calling `JSON.parse` on it: rows come back parsed) |
-| `TEXT` (UUID, ULID, slug) | `text` with the same `CHECK`s; `citext` or a `lower()` index where the code compared `COLLATE NOCASE` |
-| `REAL` | `double precision`; money stays integer cents (`bigint`), never float |
-| `SUM(int)`, `SUM(bigint)`, `AVG(…)` | PostgreSQL returns `numeric`, which the layer gives as a **string**. Cast sums that must stay numbers: `SUM(amount)::bigint` (found in the Tips migration) |
-| `BLOB` | `bytea` |
-| `CHECK (x IN (…))` | the same `CHECK` (or an enum type only if it never changes) |
-
-Then:
-- **Indexes.** Add one for every query shape the routes run. That means every `WHERE` + `ORDER BY` pair on a list route, and a partial index for "due" or "unsent" queries (for example `WHERE sent_at IS NULL`).
-- **Tables.** Put the outbox and inbox DDL in the migration: `outboxSchema('<svc>_event_outbox')` and `inboxSchema(…)`. Copy the SQL text into the file.
-- **Triggers.** SQLite `RAISE(ABORT, …)` triggers become PL/pgSQL functions plus `CREATE TRIGGER … BEFORE UPDATE OR DELETE`.
-- **Never edit a migration after it runs.** Add the next number instead. A change that drops or renames something is a `contract` migration naming its `expand` (`-- after: NNNN`), and it runs after the 7-day N-1 window.
-
-## 3. The code: async from end to end
-
-**The short path (SDK ≥ 0.18).**
-- Keep the statements' shape. `db.prepare(text)` returns async `get`, `all`, `run` and `pluck`, with the same `?` and `@name` parameters, so `q.x.get(id)` becomes `await q.x.get(id)`.
-- Keep functions that run inside a transaction calling `db`. With ambient transactions, `db.transaction(fn)()` becomes `await db.tx(async () => fn())`, and every `db` call inside joins it.
-- What still needs a person is the SQL itself (the table below) and `lastInsertRowid`, which needs `RETURNING`.
-
-
-| better-sqlite3 | openvibe-sdk/db |
-|---|---|
-| `db.prepare(q).get(a)` | `await db.maybe(sql\`… ${a}\`)` (or `one` when absence is a bug) |
-| `db.prepare(q).all(a)` | `await db.many(sql\`…\`)` |
-| `db.prepare(q).run(a)` / `.changes` | `await db.exec(sql\`…\`)` returns the count |
-| `info.lastInsertRowid` | `INSERT … RETURNING id` with `db.one` |
-| `db.transaction(fn)()` | `await db.tx(async (t) => { … })`: with ambient transactions (default) `db` calls inside join it; `t` also works |
-| `INSERT OR IGNORE` | `INSERT … ON CONFLICT DO NOTHING` |
-| `INSERT OR REPLACE` | `INSERT … ON CONFLICT (key) DO UPDATE SET …` (and `excluded.col` works) |
-| `datetime('now')`, `strftime(…)`, `CURRENT_TIMESTAMP` | `now()`, `to_char(…)`, `extract(epoch from …)` |
-| `IFNULL`, `GROUP_CONCAT`, `json_extract(x,'$.a')` | `COALESCE`, `string_agg`, `x->>'a'` |
-| `LIKE` (case-insensitive for ASCII in SQLite) | `ILIKE`, or `lower(col) LIKE lower($1)` with an index |
-| `LIMIT ? OFFSET ?` on large lists | keyset: `WHERE (created_at, id) < ($1, $2) ORDER BY created_at DESC, id DESC LIMIT $3` |
-| a loop of single-row `SELECT`s | one query with `= ANY(${ids})` or a `JOIN` |
-| a loop of single-row `INSERT`s | `sql.insert(rows)` (up to about 1000 rows a statement) |
-
-- **Handlers.** Make every Express handler and every function that touches data `async`, and `await` it. Errors reach the error handler: wrap routes (`asyncHandler`) or use Express 5.
-- **The pooler's rules** (transaction mode). On `DATABASE_URL`, no session state: no session `SET`, no `LISTEN`, and no advisory lock held outside a transaction (`pg_advisory_xact_lock` inside a `tx` is fine). Use `DATABASE_DIRECT_URL` only for migrations and the importer.
-- **Money paths** use `db.tx(fn, { isolation: 'serializable' })`. Lock the rows you debit with `SELECT … FOR UPDATE`, in a fixed order, and use idempotency keys. The layer retries serialization failures.
-- **Readiness.** Report `await db.ready()` (it names `postgresql`) and `await valkey.ready()` in `/ready`, so it never claims a store from configuration.
-- **Boot.** `await createDb({ url: process.env.DATABASE_DIRECT_URL }).migrate({ dir })`, closed afterwards. Then `createDb({ service })` for serving. The advisory lock makes this safe when several processes start together.
-
-## 4. Per-process state goes to Valkey
-
-Anything held in a process that must be shared once there are two processes:
-- **Per-actor limits:** `createActorLimiter({ …, store: createValkeyLimitStore(valkey) })`.
-- **Caches** (maps, LRUs): `createCache({ valkey, namespace })`, with TTLs and tags.
-- **Realtime fan-out** (a message received by one process must reach sockets on another): `createPubSub({ valkey })`.
-- **Background work run from `setInterval`:**
- - If it must run once across processes, it becomes a `createQueue` job, or a job claimed with a lease row (`UPDATE … SET lease_until = … WHERE … FOR UPDATE SKIP LOCKED`).
- - A purely local chore (pruning a process's own memory) may stay a timer.
-- **Nothing authoritative.** Money, accounts, grants, bans and entitlements live only in PostgreSQL.
-
-`createValkey()` returns `null` without `VALKEY_URL`, and every module above then runs in-process. Tests need no Valkey unless they test sharing.
-
-## 5. Tests
-
-- **Per test, run real PostgreSQL:** `const db = createDb({ pglite: true }); await db.migrate({ dir })`. Keep test data in the same shape as before, then fix what the types change (booleans, parsed JSON, ISO timestamps).
-- **What to add:**
- - a test that the hot routes run a bounded number of queries;
- - a migration test (the files parse, apply in order, and apply twice without change);
- - an import test that imports a small SQLite fixture with `importSqlite` and checks `report.ok`.
-- **One migrated database per test process, from a snapshot:** `const { db, close } = await createTestDb({ migrations, seed, seedKey })` (`openvibe-sdk/testing`). The first process builds the migrated (and seeded) PGlite and saves it under `os.tmpdir()/openvibe-test-snapshots` (`OV_TEST_SNAPSHOT_DIR`); every later one loads it in well under a second instead of migrating. Put the app's seed in `seed: async (db) => {}`, not after the call, and change `seedKey` when the seed changes (without one the seed function's source is the key). `snapshot` says `'hit'`, `'built'` or `'off'` and `setupMs` how long it took. `OV_TEST_SNAPSHOT=0` turns it off.
-- **CI** starts the containers and runs the suite against them (`test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test'`, as the SDK does) when the service has pooler-sensitive code.
-- **Skipping:** a test that cannot run something prints `<label>: skipped (<why>)`, and the shared runner reports it.
-
-## 6. The one-time import, and the production switch
-
-Add `scripts/migrate-to-postgres.js`, a few lines around `runSqliteMigration` from `openvibe-sdk/db` (SDK ≥ 0.19; Wiki's is the example). It:
-1. runs the migrations with `DATABASE_DIRECT_URL`;
-2. runs `importSqlite({ sqlite: <the service's DB path>, db: owner, truncate: true, tables: { … dropColumns … } })`;
-3. prints the report and exits 1 unless `report.ok`.
-
-It changes nothing in the SQLite file.
-
-1. **Rehearse** on a copy of the production database:
- - copy it (`sqlite3 <db> ".backup /tmp/rehearsal.db"` as the service user, then move it off the host) or use the drill copy;
- - run the script against a PGlite or local container database;
- - fix every problem it reports: dropped columns must be dropped on purpose, and dates it cannot read must be mapped;
- - record the per-table counts.
-2. **In production:**
- 1. `sudo /opt/openvibe.host/roles/data/add-service.sh <svc>`.
- 2. Merge the PostgreSQL release, but do not deploy it yet.
- 3. Stop writes: `systemctl stop openvibe-<svc>` for a short stop, or the service's read-only switch when it has one.
- 4. Back up the SQLite file.
- 5. Run the script from the new release's directory, as the service user, with the service's environment (Host's recipe: guarded `export`, output filtered). It must end with `report.ok`.
- 6. Deploy the release (`sudo ovhost deploy <svc>`); it now reads `DATABASE_URL`.
- 7. Verify that `/ready` shows `store: postgresql`, the service's own smoke checks pass, and the error rate stays flat.
-3. **Keep the SQLite file read-only for 7 days** (the N-1 window). It is the rollback: redeploy the previous release, which reads it, after the owner accepts losing the writes made since the switch, or replaying them from the PostgreSQL side. After the window, archive it to B2 (`ovhost archive push`) and delete it.
-4. **Record the evidence** in the same change:
- - the import report (tables, rows, checksums);
- - `/ready` before and after;
- - the test counts;
- - `STATUS.json` (the database) and the README (Depends on: PostgreSQL and Valkey);
- - the roadmap plan's WS-X2 migration list.
-
-**Rehearsing on the containers.** An embedded PGlite can break on very large rows (OpenVibe.AI's run payloads). When `--pglite` reports every table empty, import into a schema of the containers: `createTestDb({ migrations, store: 'pg' })`, then `importSqlite({ sqlite, db: t.db })`.
-
-## 7. Lessons from the first migrations
-
-- **Tips (2026-09-28)** is the reference: its `postgres` branch has the schema, the importer script, `test/helpers/db.js` and the multi-process integration test. Copy them.
-- **Test roles on the shared containers.** End each per-run role's backends before dropping it, or PgBouncer keeps idle server connections and exhausts the container. `test/helpers/db.js` does this.
-- **Replays compare bytes.** Keep a replayed response (idempotency records) as `text`: `jsonb` reorders keys.
-- **Claim an Idempotency-Key before running the handler**, with `INSERT … ON CONFLICT DO NOTHING RETURNING`. That way two concurrent requests with the same key run once.
-- **Clean text PostgreSQL refuses.** SQLite text can hold NUL and unpaired surrogates. Clean them in the importer's `map`, and report each column that needed it.
-- **pub/sub without Valkey.** `createPubSub` has no local fallback when Valkey is down. A hub that must keep delivering on one process falls back to local delivery itself.
-- **News (2026-09-28)** added these:
- - **`rowid` tiebreaks.** `ORDER BY created_at, rowid` meant insertion order. A random ULID `id` is not, within one millisecond. Add `seq bigint GENERATED ALWAYS AS IDENTITY UNIQUE` to the table and order by `seq`. The importer fills it in the source's row order.
- - **Scalar `MAX(a, b)` / `MIN(a, b)`** are `GREATEST` / `LEAST`.
- - **`ON CONFLICT … DO UPDATE SET x = COALESCE(excluded.x, x)`** is ambiguous in PostgreSQL. Qualify the target: `COALESCE(excluded.x, <table>.x)`.
- - **Work after the commit.** A `setImmediate` after a synchronous better-sqlite3 transaction ran after the commit. Around an async transaction it does not. Use `db.afterCommit(fn)` (SDK ≥ 0.20).
- - **The inbox.** `createInbox` (better-sqlite3) becomes `createPgInbox`, awaited. Its `idempotency_receipts` table goes in `migrations/0001_initial.sql` (`inboxSchema()`), not an unawaited `ensureSchema()`.
- - **A script run as a child process in tests** cannot reach the test's PGlite. Call the function it wraps in-process, and `node --check` the script.
-- **Deals, Coupons, Trade and Reviews (2026-09-28)** added these:
- - **Higher-order wrappers.** A helper that takes a function (`conflictGuard(fn)`, `unique(fn)`, `staffAction(fn)`, a batch) must `await fn()`. Otherwise its `try` catches nothing and the caller races the write. Match PostgreSQL's error codes (`23505` unique, `23503` foreign key) with `err.table` and `err.detail`, not SQLite's message.
- - **Per-process counters around a transaction** (`batchDepth++ … finally batchDepth--`) break once requests interleave. Keep per-call state in `AsyncLocalStorage`.
- - **Factories stay synchronous.** A factory that seeded a row at construction became `async`, and every caller got a promise. Write such rows lazily on first use.
- - **SQL.** Name aliases are not allowed in `HAVING`: repeat the aggregate. A bare column next to `MAX()` in a `GROUP BY` takes its value from the max row in SQLite; use `DISTINCT ON … ORDER BY` instead. `SELECT DISTINCT` over a `LEFT JOIN` with `ORDER BY lower(x)` fails; use `WHERE … OR EXISTS (…)`. Cast a parameter tested only with `IS NULL` (`@p::bigint IS NULL`). Give a recursive CTE's seed the column's type and collation (`?::text COLLATE "C"`). `json_each(?)` becomes `jsonb_array_elements_text(?::jsonb)`.
- - **Values.** `jsonb` columns come back as objects, so do not `JSON.parse` them.
- - **Gauges.** A metrics `collect()` is synchronous. Read the database one scrape behind.
- - **`npm run test:pg` finds races PGlite hides.** PGlite has one connection, so an unawaited write finished before the next read. On a pool it does not.
-- **Events (2026-09-28)** added these:
- - **A loop that read and then claimed in one synchronous step** (a delivery worker's "select due rows, mark them busy") can now run twice at once, from a timer and a kick, and send the same row twice. Serialize the passes: a call during a pass waits for one more pass after it.
- - **Register before the first `await`.** A connection counted against a cap, or joined to an in-process fan-out, after its catch-up queries lets the cap overrun and loses what was published meanwhile. Take the slot and join first; buffer live rows until the catch-up ends, then send those it did not cover.
- - **A function passed as an argument** (`flush(insertBatch)`) is async now; the codemod does not see through parameters. Await it by hand.
- - **No DDL at boot.** A module that ran `CREATE TABLE IF NOT EXISTS` on the serving handle fails on production roles (`permission denied for schema`) while PGlite allows it. The table goes in the migration; `sqlfix.py` reports `db.exec(`.
- - **Test boots that call `start()` directly** fall back to the development PGlite directory and share rows across runs. Pass every boot a `createTestDb` handle.
- - **Redaction.** PostgreSQL has no `secure_delete`: an overwritten row version stays until vacuum. Lower the table's `autovacuum_vacuum_scale_factor` and say so in the README.
-- **Community (2026-09-28)** added these:
- - **A schema changed by `migrate()` after `SCHEMA`.** `gen-migration.js --open "require('./server/db').openDb(':memory:')"` takes the schema the SQLite release's own boot leaves (every ALTER applied, tables in dependency order, boot seeds as INSERTs). Run it before converting any code.
- - **SQLite text timestamps.** `DATETIME` columns hold `'YYYY-MM-DD HH:MM:SS'` text. Keep it: the migration gets `SQLITE_DATE_FUNCTIONS` (`ov_now()`, `datetime(t, modifier)`, `julianday(t)`), and `sqlfix.py` rewrites `CURRENT_TIMESTAMP` to `ov_now()` (PostgreSQL's would be a `timestamptz`, cast to different text).
- - **Type affinity.** SQLite stores text in an `INTEGER` column without complaint (a Chat room id); PostgreSQL refuses. Check the production copy (`tools/asyncify/typecheck.js <file>` lists columns holding another storage class) and read the code that writes each id column.
- - **`INTEGER PRIMARY KEY REFERENCES …`** is the other row's key, not a counter: `bigint PRIMARY KEY` (the converter does this now).
- - **camelCase aliases** fold to lower case (`AS totalViews` comes back as `totalviews`): quote them (`sqlfix.py` does).
- - **Explicit ids** into a `GENERATED ALWAYS` identity need `OVERRIDING SYSTEM VALUE` (`sqlfix.py` adds it where the migration says ALWAYS); `.lastInsertRowid` needs `RETURNING id`.
- - **`IMMEDIATE` transactions.** A read-then-write that relied on SQLite's write lock (a vote recount) locks its row first: `SELECT … FOR UPDATE`.
- - **Work kicked from inside a transaction** (`setImmediate(drain)`) runs before the commit now and misses the row: `db.afterCommit`, and a short delay when changes made together should go out as one pass.
- - **Event handlers that became async** (a WebSocket dispatch) can overtake each other: queue them in arrival order.
- - **Fire-and-forget helpers** (`tell(fn) { try { fn(x) } catch {} }`) catch nothing once `fn` is async: await it.
- - **Tests**: `const x = createFoo(…)` instances, `require('./x').fn()` calls and a test's own `rejects(promise)` helper are handled by the codemod now; a boot helper that re-requires the server gets its database passed in (`createApp({ db })`); SQLite-only upgrade tests (a `migrate()` rebuilding an old table) go, the migration ledger replaces them.
-- **Media (2026-09-28)** added these:
- - **SQLite functions the code calls**: `json_valid`, `json_extract`, `json_type`, `instr`, `date('now', …)`, `strftime('%Y-%m-%dT%H:%M:%fZ', 'now', …)`: the migration defines SQLite-compatible `json_valid`/`json_extract`/`json_type`/`instr`, `ov_now_iso(modifier)`, and `sqlfix.py` rewrites `date(…)` to `substr(datetime(…), 1, 10)`.
- - **`SUM()` over bigint is `numeric`**, and the SDK returns numeric as exact text on purpose (money). Cast byte and count sums (`::bigint`); leave double-precision sums (durations) alone.
- - **Integers as booleans**: `WHERE 0`, `CASE WHEN is_public THEN`: write `FALSE`, `is_public = 1`.
- - **Your own `BEGIN`/`COMMIT` through `db.exec`** runs on whichever pooled connection answers: use `db.tx` (a nested `tx` is a savepoint; throw a marker to roll a dry run back).
- - **A generator that awaits** becomes `async function*`: its loops need `for await`.
- - **Callbacks a helper takes in an object** (`announce(app, event, { change, payload })`, `onRow`, `alsoInTx`) are awaited by hand; so are `x.start()` calls whose `start` became async (a fire-and-forget one needs `.catch`, or the rejection ends the process).
- - **Lazy accessors** (`function objects() { return require('../objects/model'); }`, `const m = objectsModel()`), inline `require('./x').fn()`, factory instances and `db` passed as a parameter (config `fileApis`) are resolved by the codemod now.
- - **Stores whose creation became async** (the Shared config store) get an `init()` at boot and keep a synchronous `get()`; anything they must answer synchronously (explicit keys) is cached on activation.
- - **The SQLite outbox refused to enqueue outside a transaction**; the PostgreSQL one takes any handle. Keep the guard in the service (`db.inTransaction()`), or an event can be written after the change committed.
- - **Tests**: one migrated database per test process through `node --import` (a preload), a script runner that runs operator scripts in the test process (a child process cannot reach an in-process PGlite), identities started at 100000 so tests' explicit small ids never collide, fault injection in JavaScript instead of `TEMP TRIGGER`s (the runtime role has no DDL under `test:pg`), and SQLite upgrade tests removed.
- - **Read-only reports** run in a `SET TRANSACTION READ ONLY` transaction instead of opening a read-only SQLite file.
@@ -2,7 +2,6 @@
@@ -2,7 +2,6 @@
import cjs from '../src/db/index.js';
import cjs from '../src/db/index.js';
export const {
export const {
- createDb, sql, DbError, importSqlite, runSqliteMigration, PARSERS,
- ISOLATION,
+ createDb, sql, DbError, PARSERS, ISOLATION,
} = cjs;
} = cjs;
export default cjs;
export default cjs;
@@ -1,12 +1,12 @@
@@ -1,12 +1,12 @@
{
{
"name": "openvibe-sdk",
"name": "openvibe-sdk",
- "version": "0.37.2",
+ "version": "0.38.0",
"lockfileVersion": 3,
"lockfileVersion": 3,
"requires": true,
"requires": true,
"packages": {
"packages": {
"": {
"": {
"name": "openvibe-sdk",
"name": "openvibe-sdk",
- "version": "0.37.2",
+ "version": "0.38.0",
"license": "MIT",
"license": "MIT",
"devDependencies": {
"devDependencies": {
"@electric-sql/pglite": "^0.5.8",
"@electric-sql/pglite": "^0.5.8",
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
{
{
"name": "openvibe-sdk",
"name": "openvibe-sdk",
- "version": "0.37.2",
+ "version": "0.38.0",
"description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, Bot robots, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.",
"description": "Supported browser and server clients for the OpenVibe platform: auth (incl. developer apps), registry discovery, identity, user modules, events, realtime, media, community, Bot robots, search, chat, AI runs, Tools (registry, run API and jobs), VIP memberships and developer projects, the async PostgreSQL data layer (db) and the Valkey cache, limits store, queue and pub/sub, with deadlines, safe retries, problem errors and trace propagation.",
"main": "index.js",
"main": "index.js",
"types": "types/index.d.ts",
"types": "types/index.d.ts",
@@ -8,7 +8,7 @@ const { paginate, offsetPager } = require('./paginate');
@@ -8,7 +8,7 @@ const { paginate, offsetPager } = require('./paginate');
const semver = require('./semver');
const semver = require('./semver');
/** The package version (test/version.test.js keeps it equal to package.json). */
/** The package version (test/version.test.js keeps it equal to package.json). */
-const SDK_VERSION = '0.37.2';
+const SDK_VERSION = '0.38.0';
module.exports = {
module.exports = {
createClient,
createClient,
@@ -331,9 +331,4 @@ function pgliteAdapter(opt) {
@@ -331,9 +331,4 @@ function pgliteAdapter(opt) {
};
};
}
}
-/** The one-time SQLite → PostgreSQL move (loaded on use: it needs better-sqlite3). */
-function importSqlite(opts) { return require('./sqlite-import').importSqlite(opts); }
-/** A service's scripts/migrate-to-postgres.js in one call (./sqlite-cli.js). */
-function runSqliteMigration(opts) { return require('./sqlite-cli').runSqliteMigration(opts); }
-
-module.exports = { createDb, sql, DbError, importSqlite, runSqliteMigration, PARSERS, ISOLATION };
+module.exports = { createDb, sql, DbError, PARSERS, ISOLATION };
deleted file mode 100644
deleted file mode 100644
@@ -1,100 +0,0 @@
@@ -1,100 +0,0 @@
-'use strict';
-/**
- * The one-time move of a service's SQLite database into its PostgreSQL schema (ADR-035; the procedure is
- * docs/migrating-to-postgresql.md, section 6), shared by every service's scripts/migrate-to-postgres.js:
- *
- * const { runSqliteMigration } = require('openvibe-sdk/db');
- * runSqliteMigration({ service: 'wiki', sqlite: config.dbPath, directUrl: config.db.directUrl,
- * migrations: MIGRATIONS, tables: { new_name: { from: 'old_name' } } }).then((code) => process.exit(code));
- *
- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
- *
- * 1. applies the migrations as the owner (directUrl, a direct connection); --pglite: an in-memory PostgreSQL
- * instead (a rehearsal with nothing to set up; the report is all it leaves);
- * 2. copies every table with importSqlite (batched, parents first, identities kept and sequences advanced)
- * into emptied tables (truncate: a rehearsal can be repeated);
- * 3. makes text PostgreSQL refuses storable, and reports it: a NUL character is dropped and an unpaired
- * surrogate becomes U+FFFD, in text and inside JSON values alike (SQLite took both);
- * 4. verifies each table's row count and a checksum of every row on both sides, prints the report, and
- * answers 1 unless report.ok. The SQLite file is opened read-only: nothing in it changes.
- * `tables` is importSqlite's per-table options (from, dropColumns, where, map), merged with the cleaning.
- */
-const path = require('path');
-
-const cleanText = (v) => v.toWellFormed().replace(/\u0000/g, '');
-function cleanJson(v) {
- if (typeof v === 'string') return cleanText(v);
- if (Array.isArray(v)) return v.map(cleanJson);
- if (v && typeof v === 'object') return Object.fromEntries(Object.entries(v).map(([k, x]) => [cleanText(k), cleanJson(x)]));
- return v;
-}
-
-/** importSqlite's `tables` for the target schema: `tables`, plus a map per table making text storable. */
-async function cleaningOptions(db, tables = {}) {
- const cols = await db.many(`SELECT table_name, column_name, udt_name FROM information_schema.columns
- WHERE table_schema = current_schema() AND table_name <> 'ov_migrations' AND udt_name IN ('text', 'varchar', 'json', 'jsonb')`);
- const byTable = new Map();
- for (const c of cols) { if (!byTable.has(c.table_name)) byTable.set(c.table_name, []); byTable.get(c.table_name).push(c); }
- const cleaned = new Map();
- const note = (t, c, v) => { const k = `${t}.${c}`; if (!cleaned.has(k)) cleaned.set(k, new Set()); cleaned.get(k).add(v); };
- const out = { ...tables };
- for (const [table, list] of byTable) {
- const base = tables[table] || {};
- out[table] = {
- ...base,
- map(row) {
- const r = base.map ? base.map(row) : row;
- if (!r) return r;
- for (const { column_name: c, udt_name: udt } of list) {
- const v = r[c];
- if (typeof v !== 'string') continue;
- if (udt === 'json' || udt === 'jsonb') {
- let parsed;
- try { parsed = JSON.parse(v); } catch { continue; } // left for the importer to report
- const fixed = JSON.stringify(cleanJson(parsed));
- if (fixed !== JSON.stringify(parsed)) { note(table, c, v); r[c] = fixed; }
- } else {
- const fixed = cleanText(v);
- if (fixed !== v) { note(table, c, v); r[c] = fixed; }
- }
- }
- return r;
- },
- };
- }
- return { tables: out, cleaned: () => [...cleaned].map(([column, values]) => ({ column, values: values.size })) };
-}
-
-async function runSqliteMigration({ service, sqlite, directUrl, migrations, tables = {}, skipSource = [], argv = process.argv.slice(2), out = console.log } = {}) {
- const { createDb, importSqlite } = require('./index');
- const opt = (name) => { const i = argv.indexOf(`--${name}`); return i >= 0 ? argv[i + 1] : null; };
- const flag = (name) => argv.includes(`--${name}`);
- const file = path.resolve(opt('sqlite') || sqlite || '');
- const quiet = { log() {}, warn: console.warn, error: console.error };
- let owner;
- if (flag('pglite')) {
- owner = createDb({ pglite: true, service: `${service}-import`, log: quiet });
- } else {
- if (!directUrl) throw new Error('DATABASE_DIRECT_URL is not set: the import runs as the owner on a direct connection');
- owner = createDb({ url: directUrl, service: `${service}-import`, max: 2, log: quiet });
- }
- try {
- await owner.migrate({ dir: migrations, log: quiet });
- const t0 = Date.now();
- const options = await cleaningOptions(owner, tables);
- const report = await importSqlite({ sqlite: file, db: owner, truncate: true, tables: options.tables, skipSource, log: quiet });
- const cleaned = options.cleaned();
- if (flag('json')) out(JSON.stringify({ sqlite: file, into: owner.store, ...report, cleaned }, null, 2));
- else {
- out(`import ${file} → ${owner.store} (${Date.now() - t0} ms): ${report.ok ? 'OK' : 'PROBLEMS'}`);
- for (const t of report.tables) out(` ${t.table.padEnd(30)} ${String(t.rows).padStart(8)} rows ${t.checksum || '-'}${t.source !== t.table ? ` (from ${t.source})` : ''}`);
- for (const c of cleaned) out(` cleaned: ${c.column}: ${c.values} value(s) with a NUL or an unpaired surrogate`);
- for (const p of report.problems) out(` problem: ${p.table}: ${p.problem}`);
- }
- return report.ok ? 0 : 1;
- } finally {
- await owner.close();
- }
-}
-
-module.exports = { runSqliteMigration, cleaningOptions, cleanText, cleanJson };
deleted file mode 100644
deleted file mode 100644
@@ -1,283 +0,0 @@
@@ -1,283 +0,0 @@
-'use strict';
-/**
- * The one-time move of a service's SQLite data into its PostgreSQL schema (ADR-035 amendment, WS-X2 step 4).
- *
- * const { importSqlite } = require('openvibe-sdk/db');
- * const report = await importSqlite({ sqlite: '/opt/…/wiki.db', db: ownerDb, truncate: true });
- * if (!report.ok) throw new Error(JSON.stringify(report.problems));
- *
- * The target schema (created by the service's migrations) decides everything:
- * - which tables: every base table in `public` except ov_migrations that SQLite also has (or `tables`), copied
- * parents first (foreign-key order);
- * - which columns: those both sides have. A source column the target lacks is a problem unless the table lists
- * it in `dropColumns`; a target column the source lacks must have a default or be nullable;
- * - each value's conversion, by the target column's type: 0/1 → boolean, text or epoch → timestamptz (SQLite's
- * CURRENT_TIMESTAMP text is UTC), JSON text → jsonb (validated), JSON array text → PostgreSQL arrays,
- * Buffer → bytea, anything → text for text columns.
- * Rows go in batched multi-row INSERTs in one transaction (identity columns keep their values, then their
- * sequences move past the maximum). Verification compares each table's row count and a checksum of every row,
- * canonicalised the same way on both sides and walked in primary-key order (byte order for text keys).
- */
-const crypto = require('crypto');
-const { sql } = require('./sql');
-
-function toIso(v, col) {
- if (v == null || v === '') return null;
- if (typeof v === 'number' || (typeof v === 'string' && /^\d{9,}(\.\d+)?$/.test(v))) {
- const n = Number(v);
- return new Date(n > 1e11 ? n : n * 1000).toISOString();
- }
- const s = String(v).trim();
- let d;
- if (/^\d{4}-\d{2}-\d{2}$/.test(s)) d = new Date(`${s}T00:00:00Z`);
- else if (/^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}(:\d{2}(\.\d+)?)?$/.test(s)) d = new Date(`${s.replace(' ', 'T')}Z`); // SQLite CURRENT_TIMESTAMP: UTC
- else d = new Date(s);
- if (Number.isNaN(d.getTime())) throw new Error(`${col}: "${s.slice(0, 40)}" is not a date/time`);
- return d.toISOString();
-}
-
-/** Convert one SQLite value for a PostgreSQL column type (information_schema udt_name). */
-function convert(v, udt, col) {
- if (v === undefined || v === null) return null;
- if (udt.startsWith('_')) { // array
- if (Array.isArray(v)) return v;
- if (typeof v === 'string' && v.trim().startsWith('[')) { const a = JSON.parse(v); if (!Array.isArray(a)) throw new Error(`${col}: not a JSON array`); return a; }
- if (v === '') return [];
- throw new Error(`${col}: cannot read "${String(v).slice(0, 40)}" as an array (store it as JSON text)`);
- }
- switch (udt) {
- case 'bool':
- if (typeof v === 'boolean') return v;
- if (v === 1 || v === '1' || v === 'true' || v === 't') return true;
- if (v === 0 || v === '0' || v === 'false' || v === 'f') return false;
- throw new Error(`${col}: "${v}" is not a boolean`);
- case 'int2': case 'int4': case 'int8': {
- const n = typeof v === 'bigint' ? Number(v) : Number(v);
- if (!Number.isSafeInteger(n)) throw new Error(`${col}: "${v}" is not an integer`);
- return n;
- }
- case 'float4': case 'float8': { const n = Number(v); if (Number.isNaN(n)) throw new Error(`${col}: "${v}" is not a number`); return n; }
- case 'numeric': return String(v);
- case 'timestamptz': case 'timestamp': return toIso(v, col);
- case 'date': return String(v).slice(0, 10);
- case 'json': case 'jsonb':
- if (typeof v === 'string') { if (v === '') return null; JSON.parse(v); return v; }
- if (Buffer.isBuffer(v)) { const s = v.toString('utf8'); JSON.parse(s); return s; }
- return JSON.stringify(v);
- case 'bytea': return Buffer.isBuffer(v) ? v : Buffer.from(String(v));
- default: return Buffer.isBuffer(v) ? v.toString('utf8') : String(v);
- }
-}
-
-// The canonical text of a value for checksums (same rules on both sides).
-function sortKeys(x) {
- if (Array.isArray(x)) return x.map(sortKeys);
- if (x && typeof x === 'object') return Object.keys(x).sort().reduce((o, k) => { o[k] = sortKeys(x[k]); return o; }, {});
- return x;
-}
-function canonical(v, udt) {
- if (v === null || v === undefined) return '\\N';
- if (udt === 'json' || udt === 'jsonb') {
- // JSON null and SQL NULL compare equal: a jsonb 'null' reads back as JS null, like a missing value.
- const parsed = typeof v === 'string' ? JSON.parse(v) : v;
- return parsed === null ? '\\N' : JSON.stringify(sortKeys(parsed));
- }
- if (udt === 'bytea') return Buffer.isBuffer(v) ? v.toString('hex') : Buffer.from(String(v)).toString('hex');
- if (udt === 'timestamptz' || udt === 'timestamp') return new Date(v).toISOString();
- if (udt.startsWith('_')) return JSON.stringify(v);
- if (udt === 'numeric') return String(Number(v));
- return String(v);
-}
-
-async function schemaOf(db) {
- const cols = await db.many(sql`
- SELECT c.table_name, c.column_name, c.udt_name, c.is_nullable = 'YES' AS nullable,
- c.column_default IS NOT NULL OR c.is_identity = 'YES' OR c.is_generated = 'ALWAYS' AS has_default,
- c.is_identity = 'YES' AS identity, c.is_generated = 'ALWAYS' AS generated
- FROM information_schema.columns c
- JOIN information_schema.tables t ON t.table_schema = c.table_schema AND t.table_name = c.table_name
- WHERE c.table_schema = current_schema() AND t.table_type = 'BASE TABLE' AND c.table_name <> 'ov_migrations'
- ORDER BY c.table_name, c.ordinal_position`);
- const pks = await db.many(sql`
- SELECT tc.relname AS table_name, a.attname AS column_name, array_position(i.indkey::int2[], a.attnum) AS pos
- FROM pg_index i JOIN pg_class tc ON tc.oid = i.indrelid JOIN pg_namespace n ON n.oid = tc.relnamespace
- JOIN pg_attribute a ON a.attrelid = tc.oid AND a.attnum = ANY(i.indkey)
- WHERE i.indisprimary AND n.nspname = current_schema() ORDER BY 1, 3`);
- const fks = await db.many(sql`
- SELECT c.conrelid::regclass::text AS child, c.confrelid::regclass::text AS parent
- FROM pg_constraint c JOIN pg_namespace n ON n.oid = c.connamespace
- WHERE c.contype = 'f' AND n.nspname = current_schema()`);
- const tables = new Map();
- for (const c of cols) {
- if (!tables.has(c.table_name)) tables.set(c.table_name, { name: c.table_name, columns: [], pk: [] });
- tables.get(c.table_name).columns.push(c);
- }
- for (const p of pks) if (tables.has(p.table_name)) tables.get(p.table_name).pk.push(p.column_name);
- // Parents first.
- const deps = new Map([...tables.keys()].map((t) => [t, new Set()]));
- for (const f of fks) { const ch = f.child.replace(/"/g, ''); const pa = f.parent.replace(/"/g, ''); if (deps.has(ch) && ch !== pa) deps.get(ch).add(pa); }
- const order = []; const state = new Map();
- const visit = (t) => { if (state.get(t) === 2) return; if (state.get(t) === 1) return; state.set(t, 1); for (const p of deps.get(t) || []) visit(p); state.set(t, 2); order.push(t); };
- for (const t of [...tables.keys()].sort()) visit(t);
- return order.map((t) => tables.get(t));
-}
-
-/**
- * @param {object} o
- * @param {string|object} o.sqlite a path, or an open better-sqlite3 Database
- * @param {object} o.db createDb() handle with write rights on the tables (the owner role)
- * @param {object} [o.tables] { [target]: { from?, dropColumns?: [], where?: 'SQL filter on the source', map?: (row) => row } }
- * @param {string[]} [o.only] import only these target tables
- * @param {boolean} [o.truncate] empty the target tables first (rehearsals)
- * @param {boolean} [o.verify] compare counts and checksums (default true)
- * @param {string[]} [o.skipSource] source tables deliberately not imported (every other source table must have a target)
- * → { ok, tables: [{ table, source, rows, ms, checksum }], problems: [] }
- */
-async function importSqlite({ sqlite, db, tables: opts = {}, only = null, truncate = false, verify = true, skipSource = [], log = console } = {}) {
- let Database;
- const src = typeof sqlite === 'string' ? (() => { try { Database = require('better-sqlite3'); } catch { throw new Error('importSqlite: install better-sqlite3 to read the SQLite file'); } return new Database(sqlite, { readonly: true, fileMustExist: true }); })() : sqlite;
- const report = { ok: true, tables: [], problems: [] };
- const problem = (p) => { report.ok = false; report.problems.push(p); };
- try {
- const schema = (await schemaOf(db)).filter((t) => !only || only.includes(t.name));
- const srcTables = new Set(src.prepare("SELECT name FROM sqlite_master WHERE type = 'table'").all().map((r) => r.name));
- // Every source table must land somewhere: a table the target lacks would be dropped without a word (a table a
- // module created at runtime and the migration forgot). Skipped: SQLite's own, FTS5 virtual tables and their
- // shadow tables (an index is rebuilt, not copied), and skipSource.
- if (!only) {
- const virtuals = src.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND sql LIKE 'CREATE VIRTUAL TABLE%'").all().map((r) => r.name);
- const targets = new Set(schema.map((t) => (opts[t.name] && opts[t.name].from) || t.name));
- for (const name of srcTables) {
- if (targets.has(name) || name.startsWith('sqlite_') || skipSource.includes(name)) continue;
- if (virtuals.some((v) => name === v || name.startsWith(`${v}_`))) continue;
- problem({ table: name, problem: 'a source table with no target table (add it to the migrations, or list it in skipSource)' });
- }
- }
- const plan = [];
- for (const t of schema) {
- const o = opts[t.name] || {};
- const from = o.from || t.name;
- if (!srcTables.has(from)) { log.log && log.log(`[import] ${t.name}: no source table ${from} (left empty)`); continue; }
- const srcCols = src.prepare(`PRAGMA table_info("${from.replace(/"/g, '""')}")`).all().map((c) => c.name);
- const drop = new Set(o.dropColumns || []);
- const insertable = t.columns.filter((c) => !c.generated);
- const cols = insertable.filter((c) => srcCols.includes(c.column_name));
- const lost = srcCols.filter((c) => !t.columns.some((x) => x.column_name === c) && !drop.has(c));
- if (lost.length) problem({ table: t.name, problem: `source columns with no target column (list them in dropColumns to drop them): ${lost.join(', ')}` });
- const unfillable = insertable.filter((c) => !srcCols.includes(c.column_name) && !c.nullable && !c.has_default);
- if (unfillable.length) problem({ table: t.name, problem: `target columns the source cannot fill: ${unfillable.map((c) => c.column_name).join(', ')}` });
- plan.push({ t, from, cols, o });
- }
- if (!report.ok) return report;
-
- await db.tx(async (tx) => {
- if (truncate && plan.length) await tx.exec(sql`TRUNCATE ${sql.join(plan.map((p) => sql.ident(p.t.name)))} RESTART IDENTITY CASCADE`);
- for (const { t, from, cols, o } of plan) {
- const t0 = Date.now();
- const names = cols.map((c) => c.column_name);
- const hasIdentity = cols.some((c) => c.identity);
- const batch = Math.max(1, Math.min(1000, Math.floor(60000 / Math.max(1, names.length))));
- const stmt = src.prepare(`SELECT * FROM "${from.replace(/"/g, '""')}"${o.where ? ` WHERE ${o.where}` : ''}`);
- let rows = []; let n = 0;
- const flush = async () => {
- if (!rows.length) return;
- await tx.exec(sql`INSERT INTO ${sql.ident(t.name)} (${sql.join(names.map(sql.ident))}) ${hasIdentity ? sql`OVERRIDING SYSTEM VALUE ` : sql``}VALUES ${sql.join(rows.map((r) => sql`(${sql.join(r)})`))}`);
- n += rows.length; rows = [];
- };
- for (const raw of stmt.iterate()) {
- const row = o.map ? o.map({ ...raw }) : raw;
- if (!row) continue;
- rows.push(cols.map((c) => convert(row[c.column_name], c.udt_name, `${t.name}.${c.column_name}`)));
- if (rows.length >= batch) await flush();
- }
- await flush();
- for (const c of cols.filter((x) => x.identity || /nextval\(/.test(String(x.column_default || '')))) {
- await tx.query(sql`SELECT setval(pg_get_serial_sequence(${t.name}, ${c.column_name}), GREATEST((SELECT max(${sql.ident(c.column_name)}) FROM ${sql.ident(t.name)}), 1))`);
- }
- report.tables.push({ table: t.name, source: from, rows: n, ms: Date.now() - t0 });
- log.log && log.log(`[import] ${t.name}: ${n} rows in ${Date.now() - t0} ms`);
- }
- });
-
- if (verify) {
- for (const entry of report.tables) {
- const { t, from, cols, o } = plan.find((p) => p.t.name === entry.table);
- const result = await verifyTable({ src, db, t, from, cols, o });
- entry.checksum = result.pg.sum;
- if (result.sqlite.rows !== result.pg.rows || result.sqlite.sum !== result.pg.sum) {
- problem({ table: t.name, problem: `verification failed: SQLite ${result.sqlite.rows} rows (${result.sqlite.sum}), PostgreSQL ${result.pg.rows} rows (${result.pg.sum})` });
- }
- }
- }
- return report;
- } finally {
- if (typeof sqlite === 'string') src.close();
- }
-}
-
-/**
- * Stream both sides in primary-key order and hash each canonical row: SQLite iterates ORDER BY the key (its
- * BINARY collation is byte order, as COLLATE "C"), PostgreSQL reads keyset pages of 5000 (OFFSET pages for a
- * composite key). A table without a primary key is sorted in memory by every column.
- */
-async function verifyTable({ src, db, t, from, cols, o }) {
- const names = cols.map((c) => c.column_name);
- const keys = t.pk.filter((k) => names.includes(k));
- const q = (x) => `"${String(x).replace(/"/g, '""')}"`;
- const text = new Set(cols.filter((c) => ['text', 'varchar', 'bpchar', 'citext', 'uuid'].includes(c.udt_name)).map((c) => c.column_name));
- const line = (vals) => vals.map((v, i) => canonical(v, cols[i].udt_name)).join('\t') + '\n';
- const convertRow = (row) => cols.map((c) => convert(row[c.column_name], c.udt_name, c.column_name));
-
- const h1 = crypto.createHash('sha256'); let c1 = 0;
- const where = o.where ? ` WHERE ${o.where}` : '';
- if (keys.length) {
- for (const raw of src.prepare(`SELECT * FROM ${q(from)}${where} ORDER BY ${keys.map(q).join(', ')}`).iterate()) {
- const row = o.map ? o.map({ ...raw }) : raw;
- if (!row) continue;
- h1.update(line(convertRow(row))); c1++;
- }
- } else {
- const all = src.prepare(`SELECT * FROM ${q(from)}${where}`).all().map((raw) => (o.map ? o.map({ ...raw }) : raw)).filter(Boolean).map(convertRow);
- all.sort((a, b) => cmpRows(a, b, names.map((_, i) => i)));
- for (const r of all) { h1.update(line(r)); c1++; }
- }
-
- const h2 = crypto.createHash('sha256'); let c2 = 0;
- const colList = sql.join(names.map(sql.ident));
- const orderBy = (ks) => sql.join(ks.map((k) => (text.has(k) ? sql`${sql.ident(k)} COLLATE "C"` : sql.ident(k))));
- const PAGE = 5000;
- if (keys.length === 1) {
- const k = keys[0]; let last = null;
- for (;;) {
- const cond = last === null ? sql`` : (text.has(k) ? sql`WHERE ${sql.ident(k)} COLLATE "C" > ${last}` : sql`WHERE ${sql.ident(k)} > ${last}`);
- const rows = await db.many(sql`SELECT ${colList} FROM ${sql.ident(t.name)} ${cond} ORDER BY ${orderBy([k])} LIMIT ${PAGE}`);
- for (const r of rows) { h2.update(line(names.map((n) => r[n]))); c2++; }
- if (rows.length < PAGE) break;
- last = rows[rows.length - 1][k];
- }
- } else {
- const ks = keys.length ? keys : names;
- for (let off = 0; ; off += PAGE) {
- const rows = await db.many(sql`SELECT ${colList} FROM ${sql.ident(t.name)} ORDER BY ${orderBy(ks)} LIMIT ${PAGE} OFFSET ${off}`);
- for (const r of rows) { h2.update(line(names.map((n) => r[n]))); c2++; }
- if (rows.length < PAGE) break;
- }
- }
- return { sqlite: { rows: c1, sum: h1.digest('hex').slice(0, 16) }, pg: { rows: c2, sum: h2.digest('hex').slice(0, 16) } };
-}
-
-/** Order rows the way PostgreSQL does with COLLATE "C" (byte order for text, numeric for numbers). */
-function cmpRows(a, b, idx) {
- for (const i of idx) {
- const x = a[i]; const y = b[i];
- if (x === y) continue;
- if (x === null) return 1; if (y === null) return -1; // NULLS LAST, PostgreSQL's default for ASC
- if (typeof x === 'number' && typeof y === 'number') return x - y;
- const bx = Buffer.from(String(x)); const by = Buffer.from(String(y));
- const c = Buffer.compare(bx, by);
- if (c) return c;
- }
- return 0;
-}
-
-module.exports = { importSqlite, convert, canonical };
@@ -3,7 +3,7 @@
@@ -3,7 +3,7 @@
* db (ADR-035): the sql builder binds every value; both adapters return the same row shapes (int8 → Number,
* db (ADR-035): the sql builder binds every value; both adapters return the same row shapes (int8 → Number,
* timestamps → ISO, dates → 'YYYY-MM-DD', numeric → exact text, json parsed); transactions commit, roll back
* timestamps → ISO, dates → 'YYYY-MM-DD', numeric → exact text, json parsed); transactions commit, roll back
* and nest as savepoints; serializable conflicts retry; migrations run once, in order, refuse edits, and hold a
* and nest as savepoints; serializable conflicts retry; migrations run once, in order, refuse edits, and hold a
- * contract migration inside the N-1 window; importSqlite copies a SQLite file into the schema parents first,
+ * contract migration inside the N-1 window;
* converts by target type, keeps identity values, and verifies counts and checksums. The suite runs on PGlite
* converts by target type, keeps identity values, and verifies counts and checksums. The suite runs on PGlite
* always, and again through PgBouncer against PostgreSQL 18 when OV_TEST_PG_URL is set
* always, and again through PgBouncer against PostgreSQL 18 when OV_TEST_PG_URL is set
* (scripts/test-services.sh up).
* (scripts/test-services.sh up).
@@ -13,7 +13,7 @@ const fs = require('node:fs');
@@ -13,7 +13,7 @@ const fs = require('node:fs');
const os = require('node:os');
const os = require('node:os');
const path = require('node:path');
const path = require('node:path');
const { run } = require('./helpers');
const { run } = require('./helpers');
-const { createDb, sql, importSqlite } = require('../src/db');
+const { createDb, sql } = require('../src/db');
const { parse } = require('../src/db/migrate');
const { parse } = require('../src/db/migrate');
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ovsdk-db-'));
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'ovsdk-db-'));
@@ -132,61 +132,6 @@ const tests = [
@@ -132,61 +132,6 @@ const tests = [
await assert.rejects(db2.migrate({ dir: d2, log: quiet }), /older than applied migration 0005/);
await assert.rejects(db2.migrate({ dir: d2, log: quiet }), /older than applied migration 0005/);
} finally { await db2.close(); }
} finally { await db2.close(); }
}],
}],
- ['importSqlite: parents first, conversions by target type, identities kept, verified', async () => {
- const Database = require('better-sqlite3');
- const file = path.join(tmp, 'legacy.db');
- const s = new Database(file);
- s.exec(`CREATE TABLE people (id INTEGER PRIMARY KEY AUTOINCREMENT, username TEXT NOT NULL, is_admin INTEGER DEFAULT 0, settings TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP, legacy_note TEXT);
- CREATE TABLE notes (id INTEGER PRIMARY KEY, person_id INTEGER REFERENCES people(id), body TEXT, pinned INTEGER, posted INTEGER, tags TEXT);
- CREATE TABLE loose (k TEXT, v TEXT);
- CREATE TABLE forgotten (id INTEGER PRIMARY KEY, secret TEXT);
- CREATE VIRTUAL TABLE docs_fts USING fts5(title);`);
- const ins = s.prepare('INSERT INTO people (id, username, is_admin, settings, created_at, legacy_note) VALUES (?, ?, ?, ?, ?, ?)');
- ins.run(3, 'ann', 1, '{"theme":"blue","n":[1,2]}', '2026-09-01 10:00:00', 'drop me');
- ins.run(7, 'bob', 0, null, '2026-09-02T11:30:00.000Z', null);
- ins.run(12, 'zoë', 0, '{}', '2026-09-03 09:15:00', null);
- s.prepare('INSERT INTO notes VALUES (?, ?, ?, ?, ?, ?)').run(1, 3, 'hello', 1, 1790000000, '["a","b"]');
- s.prepare('INSERT INTO notes VALUES (?, ?, ?, ?, ?, ?)').run(2, 7, 'ms time', 0, 1790000000123, '[]');
- s.prepare('INSERT INTO loose VALUES (?, ?)').run('b', '2');
- s.prepare('INSERT INTO loose VALUES (?, ?)').run('a', '1');
- s.close();
-
- const db = createDb({ pglite: true });
- try {
- await db.migrate({ dir: mdir('imp', { '0001_s.sql': `-- phase: expand
-CREATE TABLE people (id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, username text NOT NULL, is_admin boolean NOT NULL DEFAULT false, settings jsonb, created_at timestamptz NOT NULL DEFAULT now());
-CREATE TABLE notes (id bigint PRIMARY KEY, person_id bigint REFERENCES people(id), body text, pinned boolean, posted timestamptz, tags text[] NOT NULL DEFAULT '{}');
-CREATE TABLE loose (k text, v text);` }), log: { log() {} } });
- const quiet = { log() {} };
- // A source column with no target column is a problem until it is dropped on purpose.
- let r = await importSqlite({ sqlite: file, db, log: quiet });
- assert.equal(r.ok, false);
- assert.ok(r.problems.some((p) => /legacy_note/.test(p.problem)), JSON.stringify(r.problems));
- r = await importSqlite({ sqlite: file, db, log: quiet, tables: { people: { dropColumns: ['legacy_note'] } } });
- // A source table the target lacks would be dropped without a word: refused until skipped on purpose
- // (an FTS5 index and its shadow tables are skipped: an index is rebuilt, not copied).
- assert.equal(r.ok, false);
- assert.deepEqual(r.problems.map((p) => p.table), ['forgotten']);
- r = await importSqlite({ sqlite: file, db, log: quiet, truncate: true, skipSource: ['forgotten'], tables: { people: { dropColumns: ['legacy_note'] } } });
- assert.equal(r.ok, true, JSON.stringify(r.problems));
- assert.deepEqual(r.tables.map((t) => [t.table, t.rows]), [['loose', 2], ['people', 3], ['notes', 2]], 'parents before children');
- const ann = await db.one(sql`SELECT * FROM people WHERE id = 3`);
- assert.deepEqual([ann.username, ann.is_admin, ann.settings, ann.created_at], ['ann', true, { theme: 'blue', n: [1, 2] }, '2026-09-01T10:00:00.000Z']);
- const n2 = await db.one(sql`SELECT * FROM notes WHERE id = 2`);
- assert.deepEqual([n2.pinned, n2.posted, n2.tags], [false, new Date(1790000000123).toISOString(), []]);
- assert.equal((await db.one(sql`SELECT * FROM notes WHERE id = 1`)).posted, new Date(1790000000 * 1000).toISOString(), 'epoch seconds');
- const next = await db.one(sql`INSERT INTO people (username) VALUES ('new') RETURNING id`);
- assert.equal(next.id, 13, 'the identity continues after the imported maximum');
- // A second import onto the same rows fails loudly; truncate makes a rehearsal repeatable.
- r = await importSqlite({ sqlite: file, db, log: quiet, truncate: true, skipSource: ['forgotten'], tables: { people: { dropColumns: ['legacy_note'] } } });
- assert.equal(r.ok, true);
- // A transformation that differs between copy and check shows up as a failed verification.
- let flip = 0;
- r = await importSqlite({ sqlite: file, db, log: quiet, truncate: true, only: ['loose'], tables: { loose: { map: (row) => ({ ...row, v: String(flip++) }) } } });
- assert.equal(r.ok, false);
- assert.match(r.problems[0].problem, /verification failed/);
- } finally { await db.close(); }
- }],
];
];
if (process.env.OV_TEST_PG_URL && process.env.OV_TEST_PG_DIRECT_URL) tests.push(...suite('postgresql+pgbouncer', pgOpen));
if (process.env.OV_TEST_PG_URL && process.env.OV_TEST_PG_DIRECT_URL) tests.push(...suite('postgresql+pgbouncer', pgOpen));
deleted file mode 100644
deleted file mode 100644
@@ -1,58 +0,0 @@
@@ -1,58 +0,0 @@
-'use strict';
-/** runSqliteMigration: migrate, import, clean what PostgreSQL refuses (NUL, unpaired surrogates) and report it,
- * verify, answer 0; a column the schema lacks is a problem and answers 1; --json prints the report. */
-const assert = require('node:assert/strict');
-const fs = require('fs');
-const os = require('os');
-const path = require('path');
-const { run } = require('./helpers');
-const { runSqliteMigration } = require('../src/db');
-
-let Database;
-try { Database = require('better-sqlite3'); } catch { Database = null; }
-
-const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sdk-sqlitecli-'));
-const migrations = path.join(dir, 'migrations');
-fs.mkdirSync(migrations);
-fs.writeFileSync(path.join(migrations, '0001_initial.sql'), `-- phase: expand
-CREATE TABLE notes (id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, body text NOT NULL, meta jsonb, created_at bigint NOT NULL);
-CREATE TABLE renamed (id text PRIMARY KEY, n integer NOT NULL);
-`);
-
-function sqliteFile(extraColumn = false) {
- const file = path.join(dir, `src-${extraColumn ? 'x' : 'ok'}.db`);
- const db = new Database(file);
- db.exec(`CREATE TABLE notes (id INTEGER PRIMARY KEY AUTOINCREMENT, body TEXT NOT NULL, meta TEXT, created_at INTEGER NOT NULL${extraColumn ? ', stray TEXT' : ''});
- CREATE TABLE old_name (id TEXT PRIMARY KEY, n INTEGER NOT NULL);`);
- const ins = db.prepare('INSERT INTO notes (body, meta, created_at) VALUES (?, ?, ?)');
- ins.run('plain', '{"a":1}', 1);
- ins.run('with\u0000nul', '{"k":"bad \\ud800 surrogate"}', 2);
- ins.run('none', null, 3);
- db.prepare('INSERT INTO old_name (id, n) VALUES (?, ?)').run('x', 7);
- db.close();
- return file;
-}
-
-const tests = [];
-if (Database) {
- tests.push(['imports, cleans and reports, verifies and answers 0', async () => {
- const lines = [];
- const code = await runSqliteMigration({ service: 'sdk', sqlite: sqliteFile(), migrations, tables: { renamed: { from: 'old_name' } }, argv: ['--pglite'], out: (l) => lines.push(l) });
- assert.equal(code, 0, lines.join('\n'));
- assert.match(lines[0], /→ pglite .*: OK/);
- assert.ok(lines.some((l) => /notes .* 3 rows/.test(l)));
- assert.ok(lines.some((l) => /renamed .* 1 rows .*\(from old_name\)/.test(l)));
- assert.ok(lines.some((l) => /cleaned: notes\.body: 1 value/.test(l)), lines.join('\n'));
- assert.ok(lines.some((l) => /cleaned: notes\.meta: 1 value/.test(l)), lines.join('\n'));
- }]);
- tests.push(['a column the schema lacks is a problem: answers 1; --json prints the report', async () => {
- const lines = [];
- const code = await runSqliteMigration({ service: 'sdk', sqlite: sqliteFile(true), migrations, tables: { renamed: { from: 'old_name' } }, argv: ['--pglite', '--json'], out: (l) => lines.push(l) });
- assert.equal(code, 1);
- const report = JSON.parse(lines.join('\n'));
- assert.equal(report.ok, false);
- assert.ok(report.problems.some((p) => p.table === 'notes' && /stray/.test(p.problem)), JSON.stringify(report.problems));
- }]);
-} else console.log('runSqliteMigration: skipped (better-sqlite3 not installed)');
-
-run(tests);
deleted file mode 100644
deleted file mode 100644
@@ -1,66 +0,0 @@
@@ -1,66 +0,0 @@
-# asyncify: converting a service from better-sqlite3 to openvibe-sdk/db
-
-This directory holds the tools used to convert Wiki and Blog (2026-09-28). They are not part of the published package.
-
-```
-cd tools/asyncify && npm install
-node asyncify.js --config my-service.json $(git -C <service> ls-files 'server/*.js' 'server/**/*.js') <service>/test/*.test.js
-```
-
-## asyncify.js
-
-It parses every file with acorn and does the following:
-
-1. It adds `await` to calls that now return a promise:
- - prepared statements (`db.prepare(…).get/all/run`, `q.<name>.…`, and any name assigned from `.prepare(…)`);
- - Publishing store methods reached as `<x>.<store>.<method>(`;
- - `store.tx(`;
- - `asyncMethods` and `asyncGlobals` from the config;
- - methods of module APIs (`apis`: a variable name → the file whose functions became async).
-2. It makes the enclosing functions `async`.
-3. It repeats across all the files until nothing changes.
-
-It leaves some calls unawaited:
-- A promise used as a value (`.then`, `.catch`, `.finally`, `Promise.all/race([...])`, `assert.rejects(p)`) stays a promise.
-- A call inside an array callback (`map`, `filter`, `forEach`, …) is **reported** (`MANUAL …`), never changed, because an async callback there silently breaks the result. The same goes for an async function passed by name to an array method.
-
-Fix those by hand, with a `for … of` loop, `await Promise.all(xs.map(async …))`, or better a batch read (`revisions.getMany`, `citations.forRevisions`, `reviews.latestMany`, or `= ANY(?)`).
-
-It also rewrites `assert.throws(async …)` as `await assert.rejects(async …)`. An edit that would break the syntax is refused and reported.
-
-The config has these keys: `stores`, `bareStores`, `syncStoreMethods`, `apis`, `syncNames`, `asyncGlobals` and `asyncMethods`. See `example-blog.json`.
-
-What stays for a person:
-- the SQL dialect: `INSERT OR IGNORE/REPLACE` → `ON CONFLICT`, `COLLATE NOCASE` → `lower()`, `LIKE` → `ILIKE`, `IS ?` → `IS NOT DISTINCT FROM`, `DISTINCT` + `ORDER BY`, `PRAGMA` and `sqlite_master` → `information_schema`;
-- `RETURNING` in place of `lastInsertRowid`;
-- the boot (open, migrate, `createApp` async) and the test helpers (`openvibe-sdk/testing` `createTestDb`).
-
-Also by hand: `db.transaction(fn)()` → `db.tx(fn)` (the codemod leaves it), scalar `MAX(a, b)` → `GREATEST`, `rowid` tiebreaks → an identity `seq` column, and a `setImmediate` meant to run after a commit → `db.afterCommit`. The lessons list in `docs/migrating-to-postgresql.md` §7 has the rest.
-
-## The order for a Blog-shaped service
-
-```
-git checkout -b postgres
-python3 g1pass.py <svc> <Name> <ENV> # package pins, config, outbox, limits, readiness, boot, CI, unit, env, import script
-npm install
-node gen-migration.js <Name> --publishing … # migrations/0001_initial.sql from the SQLite SCHEMA
-python3 g1db.py <svc> <Name> # server/db.js: openDb / createStore / openStore
-python3 sqlfix.py # the SQL rewrites (db.transaction(fn)() → db.tx(fn) too), and CHECK lines
-node asyncify.js --config <svc>.json … # then await everything (db.tx included); wraps .map(async …) in Promise.all
-npm test && npm run test:pg # then rehearse: scripts/migrate-to-postgres.js --sqlite <copy> --pglite
-python3 g1docs.py <svc> <Name> <ENV> # README and STATUS.json
-```
-
-`asyncify.js` now turns `xs.map(cb)` with a callback that becomes async (or an async function passed by name) into `(await Promise.all(xs.map(…)))`, recursively outwards. It reports `.map(async …)` that is not awaited as a whole, and callbacks of other array methods.
-
-## sqlite-schema-to-pg.js
-
-`toPg(sqliteDdl)` turns the SCHEMA string a service's `db.js` ran into the first migration: `INTEGER PRIMARY KEY AUTOINCREMENT` → identity, `TEXT` → `text COLLATE "C"`, `INTEGER` → `bigint`, `REAL` → `double precision`, `BLOB` → `bytea`, and no `IF NOT EXISTS`. An index on `x COLLATE NOCASE` is on `lower(x)`. An immutable-row trigger (`BEFORE UPDATE [OF …]/DELETE … RAISE(ABORT, msg)`) becomes PL/pgSQL. Any other trigger is refused. `gen-migration.js` runs it on `server/db.js` at HEAD (it evaluates a SCHEMA with `${…}`) and adds the Publishing and SDK tables.
-
-## g1pass.py
-
-`g1pass.py <svc> <Name> <ENVPREFIX>`, run in a Blog-shaped service's checkout, makes the edits that are the same in every such service: config (`DATABASE_URL`, `DATABASE_DIRECT_URL`, `VALKEY_URL`), the PostgreSQL outbox, Valkey actor limits, readiness, and the test boot on `createTestDb`. Each edit checks its anchor and is skipped when already applied.
-
-## awaitify-tests.py
-
-This is the earlier regex pass for test files (Wiki). `asyncify.js` over the server and the tests together supersedes it.
deleted file mode 100644
deleted file mode 100644
@@ -1,435 +0,0 @@
@@ -1,435 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * asyncify: the mechanical half of moving a service from better-sqlite3 to openvibe-sdk/db (ADR-035).
- *
- * node asyncify.js --config asyncify.json file.js ...
- *
- * config: {
- * "stores": ["revisions", ...], // publishing store property names reached as <x>.<store>.<method>(
- * "syncStoreMethods": ["..."], // methods of those stores that stay synchronous
- * "apis": { "blogs": "server/domain/blogs.js", ... }, // variable/property names that hold a module's API object
- * "syncNames": ["validHandle", ...], // functions that stay synchronous even when their module changes
- * "asyncGlobals": ["tx"] // extra callee names that are async (e.g. a local tx helper)
- * }
- *
- * Async calls: <stmt>.get/.all/.run(...) where <stmt> is db.prepare(...)/<x>.prepare(...)/q.<name>/<x>.q.<name>;
- * <x>.<store>.<method>(...); store.tx(...) / db.tx(...); calls to functions of these files that became async
- * (fixpoint, across files through `apis`). Each gets `await` (parenthesised when a member access follows), its
- * enclosing function becomes async. A function passed to map/filter/forEach/some/every/find/reduce/sort that would
- * become async is NOT changed: it is reported for a hand fix (an async callback there silently breaks semantics).
- */
-const fs = require('fs');
-const path = require('path');
-const acorn = require('acorn');
-const walk = require('acorn-walk');
-
-const ARRAY_CB = new Set(['map', 'filter', 'forEach', 'some', 'every', 'find', 'findIndex', 'reduce', 'sort', 'flatMap']);
-
-function parse(src) {
- return acorn.parse(src, { ecmaVersion: 'latest', sourceType: 'script', allowHashBang: true, allowReturnOutsideFunction: true, locations: true });
-}
-
-function memberPath(node) {
- const parts = [];
- let n = node;
- while (n) {
- if (n.type === 'MemberExpression' && !n.computed && n.property.type === 'Identifier') { parts.unshift(n.property.name); n = n.object; continue; }
- if (n.type === 'Identifier') { parts.unshift(n.name); break; }
- if (n.type === 'ThisExpression') { parts.unshift('this'); break; }
- if (n.type === 'CallExpression') { parts.unshift('()'); n = n.callee; continue; }
- return parts.length ? ['?', ...parts] : null;
- }
- return parts;
-}
-
-function functionName(fn, parent) {
- if (fn.id) return fn.id.name;
- if (parent && parent.type === 'VariableDeclarator' && parent.id.type === 'Identifier') return parent.id.name;
- if (parent && parent.type === 'Property' && parent.key) return parent.key.name || parent.key.value;
- if (parent && parent.type === 'AssignmentExpression' && parent.left.type === 'MemberExpression') return parent.left.property.name;
- return null;
-}
-
-function analyse(file, src, cfg, asyncNames) {
- const ast = parse(src);
- const ancestors = new Map();
- // parent links
- walk.fullAncestor(ast, (node, _state, anc) => { ancestors.set(node, anc.slice(0, -1)); });
- const isFn = (n) => n.type === 'FunctionDeclaration' || n.type === 'FunctionExpression' || n.type === 'ArrowFunctionExpression';
- const stores = new Set(cfg.stores || []);
- const syncStoreMethods = new Set(cfg.syncStoreMethods || []);
- const apis = { ...(cfg.apis || {}) };
- const syncNames = new Set(cfg.syncNames || []);
- const localAsync = new Set(asyncNames.local.get(file) || []);
- // const store = require('./store'): in this file, `store` is that module's API (the same name can mean another
- // module elsewhere, so a namespace require wins over the config's global apis entry).
- walk.full(parse(src), (n) => {
- if (n.type !== 'VariableDeclarator' || n.id.type !== 'Identifier' || !n.init || n.init.type !== 'CallExpression') return;
- const c = n.init;
- if (c.callee.type !== 'Identifier' || c.callee.name !== 'require' || !c.arguments[0] || typeof c.arguments[0].value !== 'string' || !c.arguments[0].value.startsWith('.')) return;
- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), c.arguments[0].value));
- const target = base.endsWith('.js') ? base : fs.existsSync(`${base}.js`) ? `${base}.js` : path.join(base, 'index.js');
- if (fs.existsSync(target)) apis[n.id.name] = target;
- });
- // const svc = createPasteService(…) with createPasteService imported from a local file (destructured): `svc` is that
- // file's API here (a factory's instance; its methods are the file's inner functions).
- {
- const imported = new Map();
- const tree = parse(src);
- walk.full(tree, (n) => {
- if (n.type !== 'VariableDeclarator' || n.id.type !== 'ObjectPattern' || !n.init || n.init.type !== 'CallExpression') return;
- const c = n.init;
- if (c.callee.type !== 'Identifier' || c.callee.name !== 'require' || !c.arguments[0] || typeof c.arguments[0].value !== 'string' || !c.arguments[0].value.startsWith('.')) return;
- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), c.arguments[0].value));
- const target = base.endsWith('.js') ? base : fs.existsSync(`${base}.js`) ? `${base}.js` : path.join(base, 'index.js');
- for (const prop of n.id.properties) {
- if (prop.type === 'Property' && prop.value && prop.value.type === 'Identifier') imported.set(prop.value.name, target);
- }
- });
- const factoryOf = (init) => {
- let x = init;
- if (x && x.type === 'AwaitExpression') x = x.argument;
- if (x && x.type === 'LogicalExpression') x = x.right; // opts.relay || createDiscordRelay(…)
- if (x && x.type === 'AwaitExpression') x = x.argument;
- return x && x.type === 'CallExpression' && x.callee.type === 'Identifier' && /^create[A-Z]/.test(x.callee.name) ? imported.get(x.callee.name) : null;
- };
- walk.full(tree, (n) => {
- const pairs = n.type === 'VariableDeclarator' && n.id.type === 'Identifier' ? [[n.id.name, n.init]]
- : n.type === 'AssignmentExpression' && n.left.type === 'Identifier' ? [[n.left.name, n.right]] : [];
- for (const [name, init] of pairs) { const f = factoryOf(init); if (f && fs.existsSync(f)) apis[name] = f; }
- });
- }
- // fileApis: { "server/comments/api.js": { "service": "server/comments/service.js" } } for a holder whose module
- // differs by file (a factory's instance passed in under a generic name).
- Object.assign(apis, (cfg.fileApis || {})[path.relative(process.cwd(), file)] || {});
- // const { seed } = require('./workflows/seed'): a function imported from a converted file is async when it is there.
- walk.full(ast, (n) => {
- if (n.type !== 'VariableDeclarator' || n.id.type !== 'ObjectPattern' || !n.init || n.init.type !== 'CallExpression') return;
- const c = n.init;
- if (c.callee.type !== 'Identifier' || c.callee.name !== 'require' || !c.arguments[0] || typeof c.arguments[0].value !== 'string' || !c.arguments[0].value.startsWith('.')) return;
- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), c.arguments[0].value));
- const set = asyncNames.api.get(base.endsWith('.js') ? base : `${base}.js`) || asyncNames.api.get(path.join(base, 'index.js'));
- if (!set) return;
- for (const prop of n.id.properties) {
- if (prop.type !== 'Property' || !prop.key) continue;
- const exported = prop.key.name || prop.key.value;
- const local = prop.value && prop.value.type === 'Identifier' ? prop.value.name : exported;
- if (set.has(exported) && !syncNames.has(exported)) localAsync.add(local);
- }
- });
- // const view = ops.providerView: an alias of an async module method is async too.
- walk.full(ast, (n) => {
- if (n.type !== 'VariableDeclarator' || n.id.type !== 'Identifier' || !n.init || n.init.type !== 'MemberExpression' || n.init.computed) return;
- const holder = n.init.object.type === 'Identifier' ? n.init.object.name : null;
- const set = holder && apis[holder] ? asyncNames.api.get(apis[holder]) : null;
- if (set && set.has(n.init.property.name) && !syncNames.has(n.init.property.name)) localAsync.add(n.id.name);
- });
-
- // Names that hold a prepared statement: const x = db.prepare(...), or { x: db.prepare(...) }.
- const stmtNames = new Set();
- const isPrepare = (n) => n && n.type === 'CallExpression' && n.callee.type === 'MemberExpression' && n.callee.property.name === 'prepare';
- walk.full(ast, (n) => {
- if (n.type === 'VariableDeclarator' && n.id.type === 'Identifier' && isPrepare(n.init)) stmtNames.add(n.id.name);
- if (n.type === 'Property' && n.key && isPrepare(n.value)) stmtNames.add(n.key.name || n.key.value);
- });
-
- // Higher-order helpers (conflictGuard(fn), waitFor(fn), once(id, fn), a route wrapper): when a local function is
- // passed an async function, its calls of that parameter return promises, so they are awaited too.
- const localFns = new Map();
- walk.full(ast, (n) => {
- if (n.type === 'FunctionDeclaration' && n.id) localFns.set(n.id.name, n);
- if (n.type === 'VariableDeclarator' && n.id.type === 'Identifier' && n.init && ['FunctionExpression', 'ArrowFunctionExpression'].includes(n.init.type)) localFns.set(n.id.name, n.init);
- });
- const asyncParamCalls = new Set();
- walk.full(ast, (n) => {
- if (n.type !== 'CallExpression' || n.callee.type !== 'Identifier' || !localFns.has(n.callee.name)) return;
- const target = localFns.get(n.callee.name);
- n.arguments.forEach((arg, i) => {
- const isAsyncFn = ['FunctionExpression', 'ArrowFunctionExpression'].includes(arg.type) && arg.async;
- const p = target.params[i];
- if (!isAsyncFn || !p || p.type !== 'Identifier') return;
- walk.full(target.body, (c) => { if (c.type === 'CallExpression' && c.callee.type === 'Identifier' && c.callee.name === p.name) asyncParamCalls.add(c); });
- });
- });
-
- // Lazy accessors: function objects() { return require('../objects/model'); } / const model = () => require('./model').
- const lazyAccessors = new Map();
- {
- const resolveRel = (lit) => {
- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), lit));
- return base.endsWith('.js') ? base : fs.existsSync(`${base}.js`) ? `${base}.js` : path.join(base, 'index.js');
- };
- const reqOf = (n) => (n && n.type === 'CallExpression' && n.callee.type === 'Identifier' && n.callee.name === 'require' && n.arguments[0]
- && typeof n.arguments[0].value === 'string' && n.arguments[0].value.startsWith('.') ? resolveRel(n.arguments[0].value) : null);
- const bodyReq = (f) => {
- if (!f || f.params.length) return null;
- if (f.body.type !== 'BlockStatement') return reqOf(f.body);
- const st = f.body.body;
- return st.length === 1 && st[0].type === 'ReturnStatement' ? reqOf(st[0].argument) : null;
- };
- for (const [name, f] of localFns) { const t = bodyReq(f); if (t && fs.existsSync(t)) lazyAccessors.set(name, t); }
- // const m = objectsModel(): m is that module's API here.
- walk.full(ast, (n) => {
- if (n.type !== 'VariableDeclarator' || n.id.type !== 'Identifier' || !n.init || n.init.type !== 'CallExpression') return;
- const c = n.init;
- if (c.callee.type === 'Identifier' && lazyAccessors.has(c.callee.name) && !c.arguments.length) apis[n.id.name] = lazyAccessors.get(c.callee.name);
- });
- }
-
- // Local functions that await a parameter themselves: rejects(p, …) → 'rejects:0'.
- const promiseTakers = new Set();
- for (const [name, f] of localFns) {
- const params = f.params.map((q) => (q.type === 'Identifier' ? q.name : null));
- walk.full(f.body, (x) => {
- if (x.type === 'AwaitExpression' && x.argument.type === 'Identifier' && params.includes(x.argument.name)) promiseTakers.add(`${name}:${params.indexOf(x.argument.name)}`);
- });
- }
-
- function asyncCall(call) {
- const c = call.callee;
- if (c.type === 'MemberExpression' && !c.computed) {
- const m = c.property.name;
- const obj = c.object;
- const p = memberPath(obj) || [];
- if (['get', 'all', 'run'].includes(m)) {
- if (obj.type === 'CallExpression' && obj.callee.type === 'MemberExpression' && ['prepare', 'pluck'].includes(obj.callee.property.name)) return true;
- if (p.length >= 2 && (p[p.length - 2] === 'q' || p[p.length - 2] === 'stmts' || p[p.length - 2] === 'Q')) return true;
- if (p.length && stmtNames.has(p[p.length - 1])) return true;
- }
- // x.tx(fn): openvibe-sdk/db's transaction on any handle (db, store.db, db.getDb(), a destructured one).
- if (m === 'tx' && !syncNames.has('tx')) return true;
- if (m === 'transaction' && p.length && ['store', 'db', 'this'].includes(p[p.length - 1])) return false;
- // objects().safeSync(: a local lazy accessor (function objects() { return require('../objects/model'); }).
- if (obj.type === 'CallExpression' && obj.callee.type === 'Identifier' && lazyAccessors.has(obj.callee.name) && obj.arguments.length === 0) {
- const set = asyncNames.api.get(lazyAccessors.get(obj.callee.name));
- if (set && set.has(m) && !syncNames.has(m)) return true;
- }
- // require('./events').pasteCreated(: a module's function reached through an inline require.
- if (obj.type === 'CallExpression' && obj.callee.type === 'Identifier' && obj.callee.name === 'require' && obj.arguments[0] && typeof obj.arguments[0].value === 'string' && obj.arguments[0].value.startsWith('.')) {
- const base = path.relative(process.cwd(), path.resolve(path.dirname(file), obj.arguments[0].value));
- const set = asyncNames.api.get(base.endsWith('.js') ? base : `${base}.js`) || asyncNames.api.get(path.join(base, 'index.js'));
- if (set && set.has(m) && !syncNames.has(m)) return true;
- }
- // <x>.<store>.<method>(: a store reached through an object; a bare identifier only when listed in bareStores.
- if (p.length >= 2 && stores.has(p[p.length - 1]) && !syncStoreMethods.has(m)) return true;
- if (p.length === 1 && (cfg.bareStores || []).includes(p[0]) && !syncStoreMethods.has(m)) return true;
- if ((cfg.asyncMethods || []).includes(m)) return true;
- // <api>.<method>( across modules
- const holder = p[p.length - 1];
- if (apis[holder]) {
- const set = asyncNames.api.get(apis[holder]);
- if (set && set.has(m) && !syncNames.has(m)) return true;
- }
- if (p.length && (p[p.length - 1] === 'api' || p[p.length - 1] === 'svc' || p[p.length - 1] === 'self') && localAsync.has(m)) return true;
- }
- if (asyncParamCalls.has(call)) return true;
- if (c.type === 'Identifier') {
- if ((cfg.asyncGlobals || []).includes(c.name)) return true;
- if (localAsync.has(c.name) && !syncNames.has(c.name)) return true;
- }
- return false;
- }
-
- const edits = []; // { at, text } insertions
- const manual = [];
- const becameAsync = new Set();
- const newlyAsyncNames = new Set();
- const fnsToAsync = new Set();
- const wrappedMaps = new Set();
- const isFnN = (x) => x && (x.type === 'FunctionDeclaration' || x.type === 'FunctionExpression' || x.type === 'ArrowFunctionExpression');
- const inPromiseArg = (call) => { const p = ancestors.get(call) || []; const up = p[p.length - 1]; return up && up.type === 'CallExpression' && up.arguments[0] === call && up.callee.type === 'MemberExpression' && up.callee.object.name === 'Promise'; };
- /** fn will contain an await: make it async; when it is a .map() callback, wrap that map in (await Promise.all(…)) and
- * make its own enclosing function async too (recursively); another array method's callback needs a person. */
- function makeAsync(fn) {
- if (!fn || fnsToAsync.has(fn)) return;
- fnsToAsync.add(fn);
- const anc = ancestors.get(fn) || [];
- const call = anc[anc.length - 1];
- if (!(call && call.type === 'CallExpression' && call.arguments.includes(fn) && call.callee.type === 'MemberExpression' && ARRAY_CB.has(call.callee.property.name))) return;
- if (call.callee.property.name !== 'map') { manual.push(`${file}:${call.loc.start.line}: a .${call.callee.property.name}() callback becomes async`); return; }
- if (inPromiseArg(call) || wrappedMaps.has(call)) return;
- wrappedMaps.add(call);
- edits.push({ at: call.start, text: '(await Promise.all(', depth: 1 });
- edits.push({ at: call.end, text: '))', depth: 1 });
- makeAsync([...anc].reverse().find((x) => x !== fn && isFnN(x)));
- }
- // registry[k.list](): a computed call on a module API cannot be resolved here; a person awaits it if needed.
- walk.full(ast, (n) => {
- if (n.type !== 'CallExpression' || n.callee.type !== 'MemberExpression' || !n.callee.computed || n.callee.object.type !== 'Identifier' || !apis[n.callee.object.name]) return;
- const p = ancestors.get(n) || []; const up = p[p.length - 1];
- if (!(up && up.type === 'AwaitExpression')) manual.push(`${file}:${n.loc.start.line}: computed call ${n.callee.object.name}[…]() on a module API: await it if the method is async`);
- });
- // An async callback already there whose map result is not awaited as a whole is almost always a bug.
- walk.full(ast, (n) => {
- if (n.type === 'CallExpression' && n.callee.type === 'MemberExpression' && n.callee.property.name === 'map' && isFnN(n.arguments[0]) && n.arguments[0].async && !inPromiseArg(n)) {
- const p = ancestors.get(n) || []; const up = p[p.length - 1];
- if (!(up && up.type === 'VariableDeclarator')) manual.push(`${file}:${n.loc.start.line}: .map(async …) whose promises are not awaited together (Promise.all)`);
- }
- });
-
- // An async function passed by name to an array method: xs.map(asyncFn) / xs.map(api.asyncMethod) is wrapped in
- // (await Promise.all(…)) like an async callback; any other array method needs a person.
- const isFnNode = (x) => x.type === 'FunctionDeclaration' || x.type === 'FunctionExpression' || x.type === 'ArrowFunctionExpression';
- walk.full(ast, (n) => {
- if (n.type !== 'CallExpression' || n.callee.type !== 'MemberExpression' || !ARRAY_CB.has(n.callee.property.name) || !n.arguments[0]) return;
- const arg = n.arguments[0];
- let what = null;
- if (arg.type === 'Identifier' && localAsync.has(arg.name) && !syncNames.has(arg.name)) what = `async function ${arg.name}`;
- if (arg.type === 'MemberExpression' && !arg.computed && arg.object.type === 'Identifier' && apis[arg.object.name]) {
- const set = asyncNames.api.get(apis[arg.object.name]);
- if (set && set.has(arg.property.name) && !syncNames.has(arg.property.name)) what = `async method ${arg.object.name}.${arg.property.name}`;
- }
- if (!what) return;
- const p = ancestors.get(n) || [];
- const up = p[p.length - 1];
- if (n.callee.property.name === 'map' && up && up.type === 'CallExpression' && up.callee.type === 'MemberExpression' && up.callee.object.name === 'Promise') return;
- const outer = [...p].reverse().find(isFnNode);
- if (n.callee.property.name === 'map' && outer && !wrappedMaps.has(n)) {
- wrappedMaps.add(n);
- edits.push({ at: n.start, text: '(await Promise.all(', depth: 1 });
- edits.push({ at: n.end, text: '))', depth: 1 });
- makeAsync(outer);
- return;
- }
- manual.push(`${file}:${n.loc.start.line}: ${what} passed to .${n.callee.property.name}()`);
- });
- // Names awaited as a whole somewhere in the file (await first, await Promise.all([a, b]) excluded).
- const awaitedLater = new Set();
- walk.full(ast, (n) => { if (n.type === 'AwaitExpression' && n.argument && n.argument.type === 'Identifier') awaitedLater.add(n.argument.name); });
- walk.fullAncestor(ast, (node, _s, anc) => {
- if (node.type !== 'CallExpression' || !asyncCall(node)) return;
- const parents = anc.slice(0, -1);
- const parent = parents[parents.length - 1];
- if (parent && parent.type === 'AwaitExpression') return;
- // A promise used as a value stays a promise: p.then/catch/finally, or an element of Promise.all/race/allSettled/any.
- if (parent && parent.type === 'MemberExpression' && parent.object === node && ['then', 'catch', 'finally'].includes(parent.property.name)) return;
- if (parent && parent.type === 'ArrayExpression') {
- const gp = parents[parents.length - 2];
- if (gp && gp.type === 'CallExpression' && gp.callee.type === 'MemberExpression' && gp.callee.object.name === 'Promise') return;
- }
- if (parent && parent.type === 'CallExpression' && parent.callee.type === 'MemberExpression' && parent.callee.object.name === 'Promise') return;
- // const p = asyncCall(); … await p: the promise is started now on purpose and awaited later.
- if (parent && parent.type === 'VariableDeclarator' && parent.init === node && parent.id.type === 'Identifier' && awaitedLater.has(parent.id.name)) return;
- // assert.rejects(promise) / doesNotReject(promise) take the promise itself.
- if (parent && parent.type === 'CallExpression' && parent.arguments[0] === node && parent.callee.type === 'MemberExpression' && ['rejects', 'doesNotReject'].includes(parent.callee.property.name)) return;
- // So does a local helper that awaits that parameter itself (a test's rejects(p, status)).
- if (parent && parent.type === 'CallExpression' && parent.callee.type === 'Identifier' && promiseTakers.has(`${parent.callee.name}:${parent.arguments.indexOf(node)}`)) return;
- // Returned directly from an arrow body or a return statement of an async fn is fine too, but awaiting is harmless.
- const fnIdx = (() => { for (let i = parents.length - 1; i >= 0; i--) if (isFn(parents[i])) return i; return -1; })();
- const fn = fnIdx >= 0 ? parents[fnIdx] : null;
- if (fn) {
- const fnParent = parents[fnIdx - 1];
- const inPromiseAll = (call) => { const p = ancestors.get(call) || []; const up = p[p.length - 1]; return up && up.type === 'CallExpression' && up.arguments[0] === call && up.callee.type === 'MemberExpression' && up.callee.object.name === 'Promise'; };
- if (fnParent && fnParent.type === 'CallExpression' && fnParent.arguments.includes(fn) && fnParent.callee.type === 'MemberExpression' && ARRAY_CB.has(fnParent.callee.property.name)
- && !(fnParent.callee.property.name === 'map' && inPromiseAll(fnParent))) {
- // xs.map(cb) → (await Promise.all(xs.map(async cb))): same order, same length. Other array methods
- // change meaning with an async callback (filter keeps every promise), so they stay for a person.
- const outer = (() => { for (let i = fnIdx - 2; i >= 0; i--) if (isFn(parents[i])) return parents[i]; return null; })();
- if (fnParent.callee.property.name === 'map' && outer) {
- if (!wrappedMaps.has(fnParent)) {
- wrappedMaps.add(fnParent);
- edits.push({ at: fnParent.start, text: '(await Promise.all(', depth: 1 });
- edits.push({ at: fnParent.end, text: '))', depth: 1 });
- makeAsync(outer);
- }
- } else {
- manual.push(`${file}:${node.loc.start.line}: async call inside a .${fnParent.callee.property.name}() callback`);
- return;
- }
- }
- makeAsync(fn);
- }
- if (parent && parent.type === 'ExpressionStatement' && node.callee.type === 'Identifier' && (asyncNames.original.get(file) || new Set()).has(node.callee.name)
- && !/\bawait\s+$/.test(src.slice(Math.max(0, node.start - 12), node.start))) {
- manual.push(`${file}:${node.loc.start.line}: ${node.callee.name}() was already async and is called without await: left fire-and-forget (add .catch if it can reject)`);
- return;
- }
- const wrap = parent && parent.type === 'MemberExpression' && parent.object === node;
- // Never a second await in front of one already there (a call the parser saw inside another's callee).
- if (!wrap && /\bawait\s+$/.test(src.slice(Math.max(0, node.start - 12), node.start))) return;
- edits.push({ at: node.start, text: wrap ? '(await ' : 'await ' });
- if (wrap) edits.push({ at: node.end, text: ')' });
- });
-
- for (const fn of fnsToAsync) {
- if (fn.async) continue;
- const anc = ancestors.get(fn) || [];
- const parent = anc[anc.length - 1];
- becameAsync.add(fn);
- const name = functionName(fn, parent);
- if (name) newlyAsyncNames.add(name);
- if (parent && parent.type === 'Property' && (parent.method || parent.kind !== 'init')) edits.push({ at: parent.start, text: 'async ' });
- else if (parent && parent.type === 'MethodDefinition') edits.push({ at: parent.key.start, text: 'async ' });
- else edits.push({ at: fn.start, text: 'async ' });
- }
- return { edits, manual, newlyAsyncNames };
-}
-
-function apply(src, edits) {
- // At one position, closing text goes in first so it ends up after what opens there.
- // At one position: closers go in before openers (so an opener ends up left of a closer), an outer closer before an
- // inner one (it ends up right of it), an inner opener before an outer one (the outer ends up leftmost). A
- // (await Promise.all( wrap is outer (depth 1) to the await of the call it starts with (depth 0).
- const closer = (e) => e.text[0] === ')';
- const sorted = edits.slice().sort((a, b) => b.at - a.at
- || (closer(a) ? 0 : 1) - (closer(b) ? 0 : 1)
- || (closer(a) ? (b.depth || 0) - (a.depth || 0) : (a.depth || 0) - (b.depth || 0)));
- let out = src;
- for (const e of sorted) out = out.slice(0, e.at) + e.text + out.slice(e.at);
- return out;
-}
-
-function main() {
- const args = process.argv.slice(2);
- const ci = args.indexOf('--config');
- const cfg = ci >= 0 ? JSON.parse(fs.readFileSync(args[ci + 1], 'utf8')) : {};
- const files = args.filter((a, i) => a.endsWith('.js') && i !== ci + 1).map((f) => path.relative(process.cwd(), path.resolve(f)));
- const asyncNames = { local: new Map(), api: new Map(), original: new Map() };
- // Functions that were async before this run: a bare statement calling one was fire-and-forget on purpose (a worker
- // starting a job in its lane) and is left alone.
- for (const f of files) {
- const src = fs.readFileSync(f, 'utf8');
- asyncNames.original.set(f, new Set([...src.matchAll(/async function\s+(\w+)/g), ...src.matchAll(/(?:const|let)\s+(\w+)\s*=\s*async\b/g)].map((m) => m[1])));
- }
- let manual = [];
- for (let round = 0; round < 12; round++) {
- let changed = false;
- manual = [];
- for (const f of files) {
- const src = fs.readFileSync(f, 'utf8');
- let r;
- try { r = analyse(f, src, cfg, asyncNames); } catch (e) { console.error(`${f}: parse error ${e.message}`); process.exitCode = 1; continue; }
- manual.push(...r.manual);
- if (r.edits.length) {
- let next = apply(src, r.edits);
- // An assertion about a function that is async now: assert.throws cannot see a rejection.
- next = next.replace(/(^|[^.\w])assert\.throws\(async /g, '$1await assert.rejects(async ').replace(/(^|[^.\w])assert\.doesNotThrow\(async /g, '$1await assert.doesNotReject(async ');
- try { parse(next); } catch (e) { console.error(`${f}: the edits would break the syntax (${e.message}); left unchanged`); manual.push(`${f}: edits refused (syntax)`); continue; }
- fs.writeFileSync(f, next); changed = true;
- }
- // Every async function of the file (new or already) counts for callers.
- const all = new Set([...(asyncNames.local.get(f) || []), ...r.newlyAsyncNames]);
- // { name: someAsyncFunction } exports the function under another name: that name is async too.
- walk.full(parse(fs.readFileSync(f, 'utf8')), (n) => {
- if (n.type === 'Property' && n.value && n.value.type === 'Identifier' && all.has(n.value.name) && n.key) all.add(n.key.name || n.key.value);
- });
- walk.fullAncestor(parse(fs.readFileSync(f, 'utf8')), (node, _s, anc) => {
- if ((node.type === 'FunctionDeclaration' || node.type === 'FunctionExpression' || node.type === 'ArrowFunctionExpression') && node.async) {
- const parent = anc[anc.length - 2];
- const name = functionName(node, parent);
- if (name) all.add(name);
- }
- });
- const before = (asyncNames.local.get(f) || new Set()).size;
- asyncNames.local.set(f, all);
- asyncNames.api.set(f, all);
- if (all.size !== before) changed = true;
- }
- if (!changed) break;
- }
- for (const m of [...new Set(manual)]) console.log(`MANUAL ${m}`);
- for (const f of files) console.log(`async ${f}: ${[...(asyncNames.local.get(f) || [])].sort().join(', ')}`);
-}
-
-main();
deleted file mode 100644
deleted file mode 100644
@@ -1,115 +0,0 @@
@@ -1,115 +0,0 @@
-"""Add `await` to async service/db calls in test files converted from better-sqlite3 (openvibe-sdk/db ≥ 0.18).
-Usage: python3 awaitify.py <sync-member,...> file... (sync members: svc members that stay synchronous)"""
-import re, sys
-
-def skip_string(s, j):
- q = s[j]; j += 1
- while j < len(s):
- c = s[j]
- if c == '\\': j += 2; continue
- if q == '`' and c == '$' and j + 1 < len(s) and s[j+1] == '{':
- j = skip_braces(s, j + 1); continue
- if c == q: return j + 1
- j += 1
- return j
-
-def skip_braces(s, j): # s[j] == '{'
- depth = 0
- while j < len(s):
- c = s[j]
- if c in '\'"`': j = skip_string(s, j); continue
- if c == '{': depth += 1
- elif c == '}':
- depth -= 1
- if depth == 0: return j + 1
- j += 1
- return j
-
-def call_end(s, i): # s[i] == '(' -> index after matching ')'
- depth = 0; j = i
- while j < len(s):
- c = s[j]
- if c in '\'"`': j = skip_string(s, j); continue
- if c == '/' and s[j+1:j+2] == '/': j = s.find('\n', j); j = len(s) if j < 0 else j; continue
- if c in '([{': depth += 1
- elif c in ')]}':
- depth -= 1
- if depth == 0: return j + 1
- j += 1
- return j
-
-def transform(src, sync_members):
- out = []; i = 0; changed = 0
- pat = re.compile(r'((?:[A-Za-z_$][\w$]*\.)*(?:svc|db)\.(\w+)\(|(?:\bH\.)?\boutbox\(\s*h\w*\s*\)|(?:[A-Za-z_$][\w$]*\.)*(?:stores\.\w+|platform\.outbox)\.(\w+)\(|(?<![\w.$])(?:summarize|seedFn|seed)\()')
- while True:
- m = pat.search(src, i)
- if not m: out.append(src[i:]); break
- start = m.start()
- # not part of a longer member chain (e.g. foo.bar.svc handled by regex prefix); skip when preceded by '.'
- if start > 0 and src[start-1] in '.$':
- out.append(src[i:m.end()]); i = m.end(); continue
- before = src[:start].rstrip()
- text = m.group(0)
- if text.startswith('outbox') or text.startswith('H.outbox'):
- end = m.end()
- elif m.group(3) is not None or re.match(r'(summarize|seedFn|seed)\($', text):
- if m.group(3) in ('start', 'stop', 'counts'):
- out.append(src[i:m.end()]); i = m.end(); continue
- end = call_end(src, m.end() - 1)
- else:
- member = m.group(2)
- is_db = '.db.' in text or text.startswith('db.')
- if (not is_db and member in sync_members) or (is_db and member not in ('prepare', 'value', 'many', 'maybe', 'one', 'exec', 'query', 'tx')):
- out.append(src[i:m.end()]); i = m.end(); continue
- end = call_end(src, m.end() - 1)
- if is_db and member == 'prepare':
- # .pluck(...)? then .get/.all/.run(...)
- mm = re.match(r'\s*\.pluck\(', src[end:])
- if mm: end = call_end(src, end + mm.end() - 1)
- mm = re.match(r'\s*\.(get|all|run)\(', src[end:])
- if not mm:
- out.append(src[i:m.end()]); i = m.end(); continue
- end = call_end(src, end + mm.end() - 1)
- if before.endswith('await') or before.endswith('await ('):
- out.append(src[i:end]); i = end; continue
- head = src[start:m.end()]
- inner, _ = transform(src[m.end():end], sync_members) if end > m.end() else ('', 0)
- expr = head + inner
- nxt = src[end:end+1]
- rep = f'(await {expr})' if nxt in ('.', '[') else f'await {expr}'
- out.append(src[i:start]); out.append(rep); i = end; changed += 1
- return ''.join(out), changed
-
-if __name__ == '__main__':
- sync = set(sys.argv[1].split(','))
- for f in sys.argv[2:]:
- s = open(f).read()
- t, n = transform(s, sync)
- # arrows/functions that now contain await on the same line: make them async
- t = re.sub(r'(const \w+ = )\(([^)]*)\) => (?=.*\bawait\b)', r'\1async (\2) => ', t)
- if n: open(f, 'w').write(t); print(f, n)
-
-def await_helpers(src):
- names = set(re.findall(r'(?:const|let)\s+(\w+)\s*=\s*async\b', src)) | set(re.findall(r'async function\s+(\w+)\s*\(', src))
- names -= {'main', 'run'}
- if not names: return src, 0
- pat = re.compile(r'(?<![\w.$])(' + '|'.join(sorted(names, key=len, reverse=True)) + r')\(')
- out = []; i = 0; n = 0
- while True:
- m = pat.search(src, i)
- if not m: out.append(src[i:]); break
- start = m.start()
- before = src[:start].rstrip()
- # a definition, or already awaited
- if re.search(r'(function|const|let)\s*$', before) or re.search(r'(const|let)\s+$', src[:start]) or before.endswith('await') or re.search(r'function\s+$', src[:start]):
- out.append(src[i:m.end()]); i = m.end(); continue
- end = call_end(src, m.end() - 1)
- expr = src[start:end]; nxt = src[end:end+1]
- rep = f'(await {expr})' if nxt in ('.', '[') else f'await {expr}'
- out.append(src[i:start]); out.append(rep); i = end; n += 1
- return ''.join(out), n
-
-if __name__ == '__main__' and len(sys.argv) > 1 and sys.argv[1] == '--helpers':
- for f in sys.argv[2:]:
- s = open(f).read(); t, n = await_helpers(s)
- if n: open(f, 'w').write(t); print(f, n)
deleted file mode 100644
deleted file mode 100644
@@ -1,58 +0,0 @@
@@ -1,58 +0,0 @@
-{
- "stores": [
- "revisions",
- "citations",
- "taxonomy",
- "scheduler",
- "attachments",
- "discussion",
- "reviews",
- "redirects",
- "sequencer",
- "outbox"
- ],
- "syncStoreMethods": [
- "start",
- "stop",
- "slugify",
- "enabled"
- ],
- "apis": {
- "blogs": "server/domain/blogs.js",
- "posts": "server/domain/posts.js",
- "publication": "server/domain/publication.js",
- "reading": "server/domain/reading.js",
- "access": "server/domain/access.js",
- "aiDrafts": "server/domain/ai-drafts.js",
- "changelog": "server/changelog.js",
- "effects": "server/domain/effects.js",
- "people": "server/clients/network.js",
- "worker": "server/worker.js",
- "entitlements": "server/domain/access.js",
- "ctx": "server/app.js",
- "cl": "server/changelog.js"
- },
- "syncNames": [
- "validHandle",
- "themeVars",
- "categoryVocabulary",
- "actorRef",
- "listOf",
- "truthy",
- "encodeCursor",
- "decodeCursor",
- "batchKey",
- "subjectOf",
- "start",
- "stop"
- ],
- "asyncGlobals": [
- "seedOfficialPost"
- ],
- "bareStores": [
- "outbox"
- ],
- "asyncMethods": [
- "events"
- ]
-}
\ No newline at end of file
\ No newline at end of file
deleted file mode 100644
deleted file mode 100644
@@ -1,64 +0,0 @@
@@ -1,64 +0,0 @@
-#!/usr/bin/env python3
-"""server/db.js of a Blog-shaped service, from better-sqlite3 to openvibe-sdk/db (ADR-035): openDb (DATABASE_URL through
-PgBouncer, migrations as the owner, PGlite in development), createStore (the stores openStore returned, on the async
-handle) and openStore(config). The SCHEMA literal goes (it is migrations/0001_initial.sql now: gen-migration.js).
-usage: g1db.py <svc> <Name> run in the service's checkout after gen-migration.js."""
-import re, sys
-svc, Name = sys.argv[1], sys.argv[2]
-p = 'server/db.js'
-s = open(p).read()
-if 'openvibe-sdk/db' in s: sys.exit('server/db.js is converted already')
-m = re.search(r"\nfunction openStore\(dbPath, \{ now = \(\) => Date\.now\(\) \} = \{\}\) \{\n[\s\S]*?\n return \{\n([\s\S]*?)\n \};\n\}\n", s)
-if not m: sys.exit('openStore(dbPath, { now }) with a returned object literal not found: convert by hand')
-props = [l for l in m.group(1).split('\n') if not re.match(r"\s+(db|now|tx: .*|close: .*),?$", l)]
-doc = s.rfind('/**', 0, m.start() + 1)
-end = s.find('*/', doc) if doc != -1 else -1
-# openStore's own doc comment only: the comment must end right before the function (whitespace between).
-head = s[:doc] if doc != -1 and end != -1 and not s[end + 2:m.start() + 1].strip() else s[:m.start() + 1]
-tail = s[m.end():]
-head = re.sub(r"const Database = require\('better-sqlite3'\);\n", "const { createDb } = require('openvibe-sdk/db');\n", head)
-head = re.sub(r"\nconst SCHEMA = `[\s\S]*?`;\n", "\nconst MIGRATIONS = path.join(__dirname, '..', 'migrations');\nconst DEV_PGLITE = path.join(__dirname, '..', 'data', 'pglite');\n", head)
-head = re.sub(r"(%s'?s?) own SQLite database(?: \(WAL\))?[:,] created on boot, idempotently\." % re.escape(Name),
- lambda x: x.group(1) + " own PostgreSQL database (ADR-035, roadmap WS-X2): the schema is migrations/NNNN_*.sql, applied at boot.", head)
-body = f"""/**
- * The serving handle (ADR-035): DATABASE_URL through PgBouncer; in development without it, an embedded PGlite database
- * in data/pglite. Migrations run first, as the owner (DATABASE_DIRECT_URL), or on the embedded handle.
- */
-async function openDb(config, {{ log = console, registry }} = {{}}) {{
- if (!config.db.url) {{
- if (config.isProduction) throw new Error('DATABASE_URL is not set: production serves from PostgreSQL (OpenVibe.Host roles/data add-service.sh {svc})');
- log.warn(`[{Name}] DATABASE_URL unset: embedded PGlite database in ${{DEV_PGLITE}} (development only, one process)`);
- fs.mkdirSync(DEV_PGLITE, {{ recursive: true }});
- const db = createDb({{ pglite: DEV_PGLITE, service: '{svc}', registry, log }});
- await db.migrate({{ dir: MIGRATIONS, log }});
- return db;
- }}
- if (!config.db.directUrl) throw new Error('DATABASE_DIRECT_URL is not set: migrations run with the owner role on a direct connection');
- const owner = createDb({{ url: config.db.directUrl, service: '{svc}-migrate', max: 1, log }});
- try {{ await owner.migrate({{ dir: MIGRATIONS, log }}); }} finally {{ await owner.close(); }}
- return createDb({{ url: config.db.url, service: '{svc}', registry, log }});
-}}
-
-/**
- * Every store on a migrated database handle. opts.now — injectable clock (epoch ms), so tests and replays are
- * deterministic. store.tx(fn) is a transaction; inside it, plain db calls join it (ambient).
- */
-function createStore(db, {{ now = () => Date.now() }} = {{}}) {{
- return {{
- db,
- now,
-{chr(10).join(props)}
- tx: async (fn) => await db.tx(() => fn()),
- close: () => db.close(),
- }};
-}}
-
-/** openDb + createStore. */
-async function openStore(config, {{ now, log }} = {{}}) {{
- return createStore(await openDb(config, {{ log }}), {{ now }});
-}}
-"""
-tail = re.sub(r"module\.exports = \{ openStore(, )?", r"module.exports = { openDb, openStore, createStore, MIGRATIONS\1", tail)
-s = head + body + tail
-open(p, 'w').write(s)
-print('server/db.js converted; stores kept:', ', '.join(re.findall(r"^\s+(\w+):", '\n'.join(props), re.M)) or 'none')
deleted file mode 100644
deleted file mode 100644
@@ -1,42 +0,0 @@
@@ -1,42 +0,0 @@
-#!/usr/bin/env python3
-"""README.md and STATUS.json edits after a service's PostgreSQL pass (the wording Blog, News and Deals use).
-usage: g1docs.py <svc> <Name> <ENVPREFIX> run in the service's checkout, after package.json has the new pins.
-Each edit is skipped when its anchor is missing (it prints what it could not place, for a hand edit)."""
-import json, re, sys
-svc, Name, ENV = sys.argv[1], sys.argv[2], sys.argv[3]
-pkg = json.load(open('package.json'))
-ver = lambda n: (re.search(r'/v([0-9.]+)$', pkg['dependencies'].get(n, '')) or [None, None])[1]
-s = open('README.md').read(); o = s
-miss = []
-def sub(pattern, repl, flags=0, what=''):
- global s
- n = re.subn(pattern, repl, s, count=1, flags=flags)
- if n[1]: s = n[0]
- else: miss.append(what or pattern[:60])
-sub(r"live in %s'?s? own SQLite \(`%s_DB_PATH`\)" % (re.escape(Name), ENV),
- f"live in {Name}' own PostgreSQL database (`ov_{svc}` on the host's data role, ADR-035; schema in [migrations/](migrations/))", what='tables line')
-if 'PostgreSQL 18 and Valkey 9' not in s:
- sub(r"\n- \*\*Packages\*\*", "\n- **PostgreSQL 18 and Valkey 9** (OpenVibe.Host `roles/data/`, ADR-035): every read and write is async through\n `openvibe-sdk/db`; Valkey holds the per-actor limit counters (optional: without `VALKEY_URL` they count per process).\n- **Packages**", what='PG bullet')
-for name in ['openvibe-publishing', 'openvibe-contracts', 'openvibe-sdk', 'openvibe-shared']:
- v = ver(name)
- if v: s = re.sub(r"(`%s`\s+)v[0-9.]+" % re.escape(name), r"\g<1>v" + v, s)
-s = s.replace('(chrome, app icon', '(Frame, app icon')
-unit = re.search(r"The unit is `openvibe-%s\.service` on `[^`]+`, the env file `/etc/openvibe/%s\.env`\." % (svc, svc), s)
-if unit and 'add-service.sh' not in s:
- s = s[:unit.end()] + f""" The database is
-`ov_{svc}` on the host's data role (`sudo /opt/openvibe.host/roles/data/add-service.sh {svc}` writes its settings); the
-release migrates it at boot. The one-time move from SQLite is `scripts/migrate-to-postgres.js` (openvibe-sdk
-`runSqliteMigration`, with a `--pglite` rehearsal mode), run while the service is stopped; the old
-`/var/lib/openvibe-{svc}/{svc}.db` stays read-only for 7 days as the rollback.""" + s[unit.end():]
-elif not unit: miss.append('unit paragraph')
-s = re.sub(r"Nothing blocks a rollback: the schema\s+code only adds tables and columns\.", "Migrations only add tables and columns.", s)
-if s != o: open('README.md', 'w').write(s)
-st = json.load(open('STATUS.json'))
-st['runtime'] = re.sub(r'better-sqlite3 \d+', 'PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9', st.get('runtime', ''))
-if ver('openvibe-contracts'): st['contracts'] = f"openvibe-contracts v{ver('openvibe-contracts')}"
-if 'packages' in st:
- st['packages'] = [f"{n} v{ver(n)}" if ver(n) else p for p in st['packages'] for n in [p.split(' ')[0]]]
-open('STATUS.json', 'w').write(json.dumps(st, indent=2, ensure_ascii=False) + '\n')
-left = [l for l in s.splitlines() if re.search(r'SQLite|better-sqlite3', l)]
-print('README/STATUS updated' + (f"; not placed: {', '.join(miss)}" if miss else ''))
-for l in left: print(' still says SQLite:', l[:140])
deleted file mode 100644
deleted file mode 100644
@@ -1,167 +0,0 @@
@@ -1,167 +0,0 @@
-#!/usr/bin/env python3
-"""The shared, non-codemod edits of a content product's PostgreSQL pass (Blog-shaped services: server/app.js createApp,
-server/events/outbox.js, server/http/actor-limits.js, server/observability.js, test/helpers/boot.js).
-usage: g1pass.py <svc> <Name> <ENVPREFIX> (e.g. news News NEWS), run in the service's checkout. Idempotent-ish: each
-edit asserts its anchor and skips when already applied."""
-import json, re, sys, os
-svc, Name, ENV = sys.argv[1], sys.argv[2], sys.argv[3]
-def edit(p, pairs, required=True):
- if not os.path.exists(p): print('skip (missing)', p); return
- s = open(p).read(); o = s
- for a, b in pairs:
- if b in s: continue # applied already (b may contain a)
- if a not in s:
- if required: raise SystemExit(f'{p}: anchor not found: {a[:90]}')
- continue
- s = s.replace(a, b, 1)
- if s != o: open(p, 'w').write(s); print('edited', p)
-# package.json: the PostgreSQL stack and the pins it needs
-pkg = json.load(open('package.json'))
-tag = lambda repo, v: f'https://codeload.github.com/OpenVibers/{repo}/tar.gz/refs/tags/{v}'
-d = pkg['dependencies']
-d['openvibe-sdk'] = tag('OpenVibe.SDK', os.environ.get('G1_SDK', 'v0.20.0'))
-if 'openvibe-publishing' in d: d['openvibe-publishing'] = tag('OpenVibe.Publishing', os.environ.get('G1_PUBLISHING', 'v1.0.0'))
-if 'openvibe-contracts' in d: d['openvibe-contracts'] = tag('OpenVibe.Contracts', os.environ.get('G1_CONTRACTS', 'v0.76.0'))
-d.setdefault('pg', '^8.23.0'); d.setdefault('iovalkey', '^0.4.0')
-pkg['dependencies'] = dict(sorted(d.items()))
-pkg.setdefault('devDependencies', {})['@electric-sql/pglite'] = '^0.5.8'
-pkg['scripts']['test:pg'] = f'{ENV}_TEST_STORE=pg node test/run.js'
-open('package.json', 'w').write(json.dumps(pkg, indent=2) + '\n'); print('package.json')
-# config
-edit('server/config.js', [(f" dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',", f""" // PostgreSQL (ADR-035): DATABASE_URL serves (PgBouncer), DATABASE_DIRECT_URL migrates (owner role).
- db: {{ url: env.DATABASE_URL || '', directUrl: env.DATABASE_DIRECT_URL || '' }},
- valkey: {{ url: env.VALKEY_URL || '', prefix: env.VALKEY_PREFIX || 'ov:{svc}:' }},
- // The SQLite file of releases before the switch: read once by scripts/migrate-to-postgres.js.
- dbPath: env.{ENV}_DB_PATH || './data/{svc}.db',""")])
-# outbox
-if os.path.exists('server/events/outbox.js'):
- edit('server/events/outbox.js', [
- ("const { createEventsClient, createOutbox } = require('openvibe-sdk/events');", "const { createEventsClient, createPgOutbox } = require('openvibe-sdk/events');"),
- (" const outbox = createOutbox(db, {", " // The PostgreSQL outbox: rows are written in the change's own transaction (enqueue(db, …) joins the ambient\n // transaction); several processes relay one table safely (leases).\n const outbox = createPgOutbox(db, {"),
-])
-s = open('server/events/outbox.js').read()
-s2 = re.sub(r"\n[ \t]*(await )?outbox\.ensureSchema\(\);[^\n]*", '', s)
-if s2 != s: open('server/events/outbox.js', 'w').write(s2); print('outbox: schema is the migration now')
-s = open('server/events/outbox.js').read()
-s2 = re.sub(r'outbox\.enqueue\((?!db, )', 'outbox.enqueue(db, ', s)
-if s2 != s: open('server/events/outbox.js', 'w').write(s2); print('enqueue(db, …)')
-# app.js store
-s = open('server/app.js').read()
-m = re.search(r" const store = opts\.store \|\| openStore\(opts\.dbPath \|\| config\.dbPath, \{ now: opts\.now(, [^}]*)? \}\);", s)
-if m:
- s = s.replace(m.group(0), " // PostgreSQL (ADR-035): opened and migrated here unless the caller (a test, a script) hands in a store.\n const store = opts.store || await openStore(config, { now: opts.now, log });")
- s = s.replace('function createApp(opts = {}) {', 'async function createApp(opts = {}) {', 1)
- open('server/app.js', 'w').write(s); print('app store')
-# limits + valkey
-if os.path.exists('server/http/actor-limits.js'):
- s = open('server/http/actor-limits.js').read(); o = s
- if 'createValkeyLimitStore' not in s:
- s = re.sub(r"const \{ createActorLimiter(, defaultActor)? \} = require\('openvibe-sdk/limits'\);",
- lambda m: "const { createActorLimiter, createValkeyLimitStore" + (m.group(1) or '') + " } = require('openvibe-sdk/limits');", s, 1)
- s = re.sub(r"(function createActorLimits\(\{[^)]*?)( \}\) \{)", lambda m: m.group(1) + ', valkey = null' + m.group(2), s, 1)
- s = s.replace(" now,\n", " now,\n // Shared across processes on Valkey (ADR-035) when VALKEY_URL is set; in-process otherwise.\n ...(valkey ? { store: createValkeyLimitStore(valkey) } : {}),\n", 1)
- if s != o: open('server/http/actor-limits.js', 'w').write(s); print('edited server/http/actor-limits.js')
-s = open('server/app.js').read()
-m = re.search(r" ctx\.(limits|actorLimits) = createActorLimits\(\{ config, now: opts\.limitsNow \|\| \(\(\) => Date\.now\(\)\), registry: metrics\.registry, log \}\);", s)
-if m:
- s = s.replace(m.group(0), " // Valkey (ADR-035): shared, never-authoritative state (per-actor limit counters). Optional.\n const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);\n ctx.valkey = valkey;\n ctx." + m.group(1) + " = createActorLimits({ config, now: opts.limitsNow || (() => Date.now()), registry: metrics.registry, log, valkey });")
- open('server/app.js', 'w').write(s); print('app valkey')
-# readiness
-s = open('server/observability.js').read(); o = s
-s = re.sub(r"(function create\w+Readiness\(\{ store, [^}]*?)(, release = null \}\))", lambda m: m.group(1) + (', valkey = null' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)
-s = s.replace(""" {
- name: 'network_jwks', required: false,""", """ { name: 'valkey', required: false, check: async () => (valkey ? valkey.ready() : { skipped: 'VALKEY_URL not set: per-actor limits count in this process only' }) },
- {
- name: 'network_jwks', required: false,""", 1) if "name: 'valkey'" not in s else s
-s = re.sub(r"""const names = new Set\((?:\(await )?db\.prepare\("SELECT name FROM sqlite_master WHERE type IN \('table','view'\)"\)\.all\(\)\)?\.map\(\(r\) => r\.name\)\);\n(\s*)const missing = CHARTER_TABLES\.filter\(\(t\) => !names\.has\(t\)\);\n\s*return missing\.length \? `missing \$\{missing\.join\(', '\)\}` : true;""",
- lambda m: f"""// A real round trip that names the store (postgresql / pglite), and the charter tables present.
-{m.group(1)}const r = await db.ready();
-{m.group(1)}if (!r.ok) return r.error;
-{m.group(1)}const names = new Set((await db.prepare('SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()').all()).map((x) => x.name));
-{m.group(1)}const missing = CHARTER_TABLES.filter((t) => !names.has(t));
-{m.group(1)}return missing.length ? `missing ${{missing.join(', ')}} (migrations did not run)` : {{ ok: true, detail: r.detail }};""", s)
-s = s.replace("check: () => {\n // A real round trip", "check: async () => {\n // A real round trip")
-if s != o: open('server/observability.js', 'w').write(s); print('readiness')
-s = open('server/app.js').read()
-s2 = re.sub(r"(create\w+Readiness\(\{ store, [^}]*?release: release\.release)( \}\))", lambda m: m.group(1) + (', valkey: ctx.valkey' if 'valkey' not in m.group(1) else '') + m.group(2), s, 1)
-if s2 != s: open('server/app.js', 'w').write(s2); print('app readiness')
-# index.js: await createApp
-s = open('server/index.js').read()
-if 'await createApp()' not in s:
- s = s.replace("const { app, ctx } = createApp();", "(async () => {\nconst { app, ctx } = await createApp();", 1)
- s = s.replace(" try { ctx.store.close(); } catch { /* already closed */ }", " try { await ctx.store.close(); } catch { /* already closed */ }")
- s = s.replace(f"(db ${{config.dbPath}})", "(db ${ctx.store.db.store})")
- s = s.rstrip() + f"\n}})().catch((err) => {{ console.error('[{Name}] failed to start:', err); process.exit(1); }});\n"
- open('server/index.js', 'w').write(s); print('index')
-# test helpers
-os.makedirs('test/helpers', exist_ok=True)
-open('test/helpers/db.js', 'w').write(f"""'use strict';
-/**
- * A migrated database for one test run (ADR-035), from openvibe-sdk/testing: PGlite by default; with
- * {ENV}_TEST_STORE=pg (npm run test:pg) the PostgreSQL + PgBouncer containers, with roles and a schema of this run's own.
- */
-const {{ createTestDb, pgAvailable }} = require('openvibe-sdk/testing');
-const {{ MIGRATIONS }} = require('../../server/db');
-
-const testDb = ({{ store = process.env.{ENV}_TEST_STORE || 'pglite', max = 4 }} = {{}}) => createTestDb({{ migrations: MIGRATIONS, store, service: '{svc}', max }});
-
-module.exports = {{ testDb, pgAvailable }};
-""")
-s = open('test/helpers/boot.js').read(); o = s
-if "require('./db').testDb()" not in s:
- s = s.replace(" let server = null;\n let built = null;", " const { createStore } = require('../../server/db');\n // One database per boot (PGlite, or " + ENV + "_TEST_STORE=pg: the containers); a restart keeps it, like a file did.\n const testdb = await require('./db').testDb();\n let server = null;\n let built = null;", 1)
- s = re.sub(r"built = createApp\(\{ config, ", "built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), ", s, 1)
- s = re.sub(r"(await built\.ctx\.outbox\.stop\(\);) built\.ctx\.store\.close\(\);", r"\1", s)
- s = re.sub(r" function events\(type = null\) \{\n return t\.ctx\.store\.db\.prepare\('SELECT envelope FROM event_outbox ORDER BY id'\)\.all\(\)\.map\(\(r\) => JSON\.parse\(r\.envelope\)\)",
- " async function events(type = null) {\n return (await t.ctx.store.db.prepare('SELECT envelope FROM event_outbox ORDER BY id').all()).map((r) => (typeof r.envelope === 'string' ? JSON.parse(r.envelope) : r.envelope))", s)
- s = s.replace(" async close() { await stop(); await network.close();", " async close() { await stop(); await testdb.close(); await network.close();", 1)
-if s != o: open('test/helpers/boot.js', 'w').write(s); print('boot helper')
-# CI, unit, env, import script
-s = open('.github/workflows/ci.yml').read()
-if 'test-services.sh' not in s:
- s = s.replace(f" contracts-service: {svc}\n", f""" contracts-service: {svc}
- # ADR-035: PostgreSQL 18 + PgBouncer (transaction mode) + Valkey 9 containers (openvibe-sdk
- # scripts/test-services.sh). npm test runs every file on PGlite; test:pg runs them all through PgBouncer.
- test: 'eval "$(node_modules/openvibe-sdk/scripts/test-services.sh up)" && npm test && npm run test:pg'
-""", 1); open('.github/workflows/ci.yml', 'w').write(s); print('ci')
-unit = f'deploy/systemd/openvibe-{svc}.service'
-edit(unit, [("After=network.target\n", "After=network.target postgresql.service pgbouncer.service valkey-server.service\n"), ("Wants=openvibe-network.service\n", "Wants=openvibe-network.service postgresql.service pgbouncer.service valkey-server.service\n")], required=False)
-edit('.env.example', [(f"{ENV}_DB_PATH=./data/{svc}.db", f"""# PostgreSQL (ADR-035), written by OpenVibe.Host roles/data/add-service.sh {svc}: the runtime role through PgBouncer,
-# the owner on a direct connection (migrations and the one-time import), and Valkey (shared limit counters).
-# Without DATABASE_URL, development uses an embedded PGlite database in data/pglite.
-DATABASE_URL=
-DATABASE_DIRECT_URL=
-VALKEY_URL=
-VALKEY_PREFIX=
-# The SQLite file of releases before PostgreSQL: read once by scripts/migrate-to-postgres.js.
-{ENV}_DB_PATH=./data/{svc}.db""")], required=False)
-os.makedirs('scripts', exist_ok=True)
-cfgmod = "require('../server/config')"
-open('scripts/migrate-to-postgres.js', 'w').write(f"""#!/usr/bin/env node
-'use strict';
-/**
- * The one-time move of {Name}'s SQLite database ({ENV}_DB_PATH) into its PostgreSQL schema (ADR-035; the procedure is
- * openvibe-sdk docs/migrating-to-postgresql.md, section 6, carried out by openvibe-sdk/db runSqliteMigration).
- *
- * node scripts/migrate-to-postgres.js [--sqlite <file>] [--pglite] [--json]
- *
- * Applies migrations/ as the owner (DATABASE_DIRECT_URL), copies every table into emptied tables, verifies row counts
- * and checksums, and exits 1 unless everything verified. The SQLite file is opened read-only. Every table keeps its
- * name and columns.
- */
-require('dotenv').config();
-const {{ runSqliteMigration }} = require('openvibe-sdk/db');
-const configLib = {cfgmod};
-const {{ MIGRATIONS }} = require('../server/db');
-
-const TABLES = {{}};
-
-if (require.main === module) {{
- const config = (configLib.load || configLib.loadConfig)();
- runSqliteMigration({{ service: '{svc}', sqlite: config.dbPath, directUrl: config.db.directUrl, migrations: MIGRATIONS, tables: TABLES }})
- .then((code) => process.exit(code), (err) => {{ console.error(`migrate-to-postgres failed: ${{err.message}}`); process.exit(1); }});
-}}
-
-module.exports = {{ TABLES }};
-""")
-print('done')
deleted file mode 100644
deleted file mode 100644
@@ -1,172 +0,0 @@
@@ -1,172 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-/**
- * The first migration of a better-sqlite3 service (ADR-035), written once: the SCHEMA template literal of its
- * server/db.js through sqlite-schema-to-pg, then the openvibe-publishing stores it opens and the openvibe-sdk inbox
- * and outbox. Run in the service's checkout, after `npm install` of openvibe-publishing ≥ 1.0 and openvibe-sdk ≥ 0.18:
- *
- * node gen-migration.js <Name> [--publishing revisions:blog_post,citations:blog_post,index-hooks:blog,seo:blog_entity,…]
- * [--no-inbox] [--outbox-table t] [--inbox-table t] [--no-outbox] [--ensure a.js,b.js]
- * [--from server/store.js] (the module holding SCHEMA; default server/db.js)
- * [--open "require('./server/db').openDb(':memory:')"]
- * → migrations/0001_initial.sql
- *
- * --open: the schema as the service's own boot code leaves a fresh SQLite database (SCHEMA, then every migrate() ALTER
- * and seed, then the --ensure modules run on it), for a service whose migrate() changed tables after SCHEMA. Run it
- * before converting any code (the working copy is the SQLite release). Tables come out in dependency order, one column
- * per line; the rows the boot seeded become INSERTs; SQLite text timestamps (DATETIME, CURRENT_TIMESTAMP) bring
- * SQLITE_DATE_FUNCTIONS along.
- *
- * It lists every `rowid` the server code still orders by: those tables need a `seq bigint GENERATED ALWAYS AS
- * IDENTITY UNIQUE` column (a person adds it; see docs/migrating-to-postgresql.md §7).
- */
-const fs = require('fs');
-const path = require('path');
-const { execFileSync } = require('child_process');
-const { toPg, SQLITE_DATE_FUNCTIONS, PG_ONLY_RESERVED } = require('./sqlite-schema-to-pg');
-
-const args = process.argv.slice(2);
-const Name = args[0];
-if (!Name) { console.error('usage: gen-migration.js <Name> [--publishing store:prefix,…] [--no-inbox]'); process.exit(2); }
-const pi = args.indexOf('--publishing');
-const pubs = pi >= 0 ? args[pi + 1].split(',').map((x) => x.split(':')) : [];
-const req = (m) => require(require.resolve(m, { paths: [process.cwd()] }));
-
-const opt = (name, def) => { const i = args.indexOf(name); return i >= 0 ? args[i + 1] : def; };
-const ensureList = opt('--ensure', '') ? opt('--ensure', '').split(',') : [];
-
-/** A fresh SQLite database opened by the service's code → PostgreSQL DDL (tables in dependency order) and seed rows. */
-function fromOpenedDb(expr) {
- const { createRequire } = require('module');
- const creq = createRequire(path.resolve('package.json'));
- const sdb = new Function('require', `return (${expr});`)(creq);
- for (const f of ensureList) creq(path.resolve(f)).ensureSchema(sdb);
- const stripComments = (sql) => {
- let o = ''; let q = null;
- for (let i = 0; i < sql.length; i++) {
- const c = sql[i];
- if (q) { o += c; if (c === q) q = null; continue; }
- if (c === "'" || c === '"') { q = c; o += c; continue; }
- if (c === '-' && sql[i + 1] === '-') { while (i < sql.length && sql[i] !== '\n') i++; o += '\n'; continue; }
- o += c;
- }
- return o;
- };
- const splitTop = (body) => {
- const parts = []; let depth = 0; let q = null; let cur = '';
- for (const c of body) {
- if (q) { cur += c; if (c === q) q = null; continue; }
- if (c === "'" || c === '"') { q = c; cur += c; continue; }
- if (c === '(') depth++;
- if (c === ')') depth--;
- if (c === ',' && depth === 0) { parts.push(cur.trim()); cur = ''; continue; }
- cur += c;
- }
- if (cur.trim()) parts.push(cur.trim());
- return parts;
- };
- const objs = sdb.prepare("SELECT type, name, tbl_name, sql FROM sqlite_master WHERE sql IS NOT NULL AND name NOT LIKE 'sqlite_%' ORDER BY rowid").all();
- const tables = objs.filter((o) => o.type === 'table' && !/CREATE VIRTUAL/i.test(o.sql));
- const byName = new Map(tables.map((t) => [t.name, t]));
- const order = []; const done = new Set();
- const visit = (t, stack) => {
- if (done.has(t.name) || stack.has(t.name)) return;
- stack.add(t.name);
- for (const m of t.sql.matchAll(/REFERENCES\s+"?(\w+)"?/gi)) { const d = byName.get(m[1]); if (d && d.name !== t.name) visit(d, stack); }
- done.add(t.name); order.push(t);
- };
- for (const t of tables) visit(t, new Set());
- let ddl = '';
- const tsDefault = new Set(); const pk = new Map();
- for (const t of order) {
- const sql = stripComments(t.sql);
- const i = sql.indexOf('('); const j = sql.lastIndexOf(')');
- const cols = splitTop(sql.slice(i + 1, j)).map((x) => x.replace(/\s+/g, ' '));
- for (const c of cols) {
- const m = /^"?(\w+)"? /.exec(c);
- if (!m) continue;
- if (/DEFAULT (CURRENT_TIMESTAMP|\((datetime|strftime)\()/i.test(c)) tsDefault.add(`${t.name}.${m[1]}`);
- if (/^"?\w+"? INTEGER PRIMARY KEY/i.test(c)) pk.set(t.name, { col: m[1], always: /AUTOINCREMENT/i.test(c) });
- }
- const tail = sql.slice(j + 1).trim();
- ddl += `CREATE TABLE ${t.name} (\n ${cols.join(',\n ')}\n)${tail ? ` ${tail}` : ''};\n\n`;
- }
- for (const o of objs) if (o.type === 'index') ddl += `${stripComments(o.sql).replace(/\s+/g, ' ').trim()};\n`;
- for (const o of objs) if (o.type === 'trigger') ddl += `\n${o.sql.trim()};\n`;
- for (const o of objs) if (o.type === 'view') ddl += `\n${o.sql.trim()};\n`;
- let out = toPg(ddl);
- // Seed rows the boot wrote (spaces, categories, …); timestamp defaults fill themselves in.
- const lit = (v) => (v === null ? 'NULL' : typeof v === 'number' || typeof v === 'bigint' ? String(v)
- : Buffer.isBuffer(v) ? `'\\x${v.toString('hex')}'` : `'${String(v).replace(/'/g, "''")}'`);
- let seeds = '';
- for (const t of order) {
- const rows = sdb.prepare(`SELECT * FROM "${t.name}"`).all();
- if (!rows.length) continue;
- const cols = Object.keys(rows[0]).filter((c) => !tsDefault.has(`${t.name}.${c}`));
- const k = pk.get(t.name);
- const q = (c) => (PG_ONLY_RESERVED.includes(c.toLowerCase()) ? `"${c}"` : c);
- seeds += `INSERT INTO ${t.name} (${cols.map(q).join(', ')})${k && k.always && cols.includes(k.col) ? ' OVERRIDING SYSTEM VALUE' : ''} VALUES\n`
- + `${rows.map((r) => ` (${cols.map((c) => lit(r[c])).join(', ')})`).join(',\n')};\n`;
- if (k && cols.includes(k.col)) seeds += `SELECT setval(pg_get_serial_sequence('${t.name}', '${k.col}'), (SELECT MAX(${k.col}) FROM ${t.name}));\n`;
- }
- if (seeds) out += `\n-- Rows the SQLite boot seeded\n${seeds}`;
- if (/ov_now\(\)|ov_now_iso\(\)/.test(out)) out = `${SQLITE_DATE_FUNCTIONS}\n${out}`;
- try { sdb.close(); } catch { /* in-memory */ }
- return out;
-}
-
-// The schema as the SQLite release ran it: server/db.js at HEAD (the conversion edits the working copy). A literal with
-// ${…} interpolations is evaluated by loading that file and reading its SCHEMA export.
-const dbFile = (() => { const i = args.indexOf('--from'); return i >= 0 ? args[i + 1] : 'server/db.js'; })();
-const openExpr = opt('--open', null);
-const src = openExpr ? '' : execFileSync('git', ['show', `HEAD:${dbFile}`], { encoding: 'utf8' });
-const m = openExpr ? [null, ''] : /const SCHEMA = `([\s\S]*?)`;/.exec(src);
-if (!m) { console.error(`${dbFile} (HEAD) has no const SCHEMA = \`…\`;`); process.exit(1); }
-if (!openExpr && m[1].includes('${')) {
- const tmp = path.resolve(path.dirname(dbFile), '.db-head-schema.js');
- fs.writeFileSync(tmp, src);
- try {
- const schema = require(tmp).SCHEMA;
- if (typeof schema !== 'string') throw new Error('it does not export SCHEMA');
- m[1] = schema;
- } catch (err) { console.error(`the SCHEMA literal has interpolations and could not be evaluated: ${err.message}`); process.exit(1); } finally { fs.unlinkSync(tmp); }
-}
-let out = `-- phase: expand
-${(openExpr ? fromOpenedDb(openExpr) : toPg(m[1])).replace(/\n{3,}/g, '\n\n')}`;
-// --ensure a.js,b.js: modules whose ensureSchema(db) created more tables at boot; their SQL is captured and converted too.
-if (!openExpr) {
- const i = args.indexOf('--ensure');
- if (i >= 0) {
- for (const f of args[i + 1].split(',')) {
- const sqls = [];
- const fake = { exec: (q) => sqls.push(q), prepare: () => ({ run() {}, get() {}, all: () => [] }), pragma() {}, function() {} };
- req(path.resolve(f)).ensureSchema(fake);
- out += `\n-- ${f} ensureSchema()\n${toPg(sqls.join('\n').replace(/^ {4}/gm, '')).trim()}\n`;
- }
- }
-}
-const SCHEMA_OF = { 'index-hooks': (x, p) => x.sequencerSchema(p), seo: (x, p) => x.redirectsSchema(p) };
-for (const [store, prefix] of pubs) {
- const mod = req(`openvibe-publishing/${store}`);
- out += `\n-- openvibe-publishing/${store} (prefix ${prefix})\n${(SCHEMA_OF[store] ? SCHEMA_OF[store](mod, prefix) : mod.schema(prefix)).trim()}\n`;
-}
-const ev = req('openvibe-sdk/events');
-const inboxTable = opt('--inbox-table', 'idempotency_receipts');
-const outboxTable = opt('--outbox-table', 'event_outbox');
-if (!args.includes('--no-inbox')) out += `\n-- openvibe-sdk/events inbox: one receipt per (consumer, event) handled\n${ev.inboxSchema(inboxTable).trim()}\n`;
-if (!args.includes('--no-outbox')) out += `\n-- openvibe-sdk/events PostgreSQL outbox\n${ev.outboxSchema(outboxTable).trim()}\n`;
-// SQLite's own copies of the tables above (created by the packages at boot) would be defined twice.
-for (const t of [outboxTable, inboxTable]) {
- const re = new RegExp(`\\nCREATE TABLE ${t} \\([\\s\\S]*?\\n\\);\\n`, 'g');
- if ((out.match(re) || []).length && out.includes(`CREATE TABLE IF NOT EXISTS ${t}`)) out = out.replace(re, '\n');
-}
-fs.mkdirSync('migrations', { recursive: true });
-fs.writeFileSync(path.join('migrations', '0001_initial.sql'), out);
-console.log(`migrations/0001_initial.sql: ${(out.match(/CREATE TABLE/g) || []).length} tables`);
-try {
- const hits = execFileSync('git', ['grep', '-n', 'rowid', '--', 'server', 'scripts'], { encoding: 'utf8' }).trim();
- if (hits) console.log(`rowid still used (add a seq identity column to these tables and order by it):\n${hits}`);
-} catch { /* no rowid */ }
deleted file mode 100644
deleted file mode 100644
@@ -1,39 +0,0 @@
@@ -1,39 +0,0 @@
-{
- "name": "openvibe-asyncify",
- "lockfileVersion": 3,
- "requires": true,
- "packages": {
- "": {
- "name": "openvibe-asyncify",
- "license": "MIT",
- "dependencies": {
- "acorn": "^8.15.0",
- "acorn-walk": "^8.3.4"
- }
- },
- "node_modules/acorn": {
- "version": "8.18.0",
- "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.18.0.tgz",
- "integrity": "sha512-lGq+9yr1/GuAWaVYIHRjvvySG5/4VfKIvC8EWxStPdcDh/Ka7FG3twP6v4d5BkravUilhIAsG4Qj83t02LWUPQ==",
- "license": "MIT",
- "bin": {
- "acorn": "bin/acorn"
- },
- "engines": {
- "node": ">=0.4.0"
- }
- },
- "node_modules/acorn-walk": {
- "version": "8.3.5",
- "resolved": "https://registry.npmjs.org/acorn-walk/-/acorn-walk-8.3.5.tgz",
- "integrity": "sha512-HEHNfbars9v4pgpW6SO1KSPkfoS0xVOM/9UzkJltjlsHZmJasxg8aXkuZa7SMf8vKGIBhpUsPluQSqhJFCqebw==",
- "license": "MIT",
- "dependencies": {
- "acorn": "^8.11.0"
- },
- "engines": {
- "node": ">=0.4.0"
- }
- }
- }
-}
deleted file mode 100644
deleted file mode 100644
@@ -1,10 +0,0 @@
@@ -1,10 +0,0 @@
-{
- "name": "openvibe-asyncify",
- "private": true,
- "description": "The mechanical half of moving a service from better-sqlite3 to openvibe-sdk/db (ADR-035). Not part of the published SDK package.",
- "license": "MIT",
- "dependencies": {
- "acorn": "^8.15.0",
- "acorn-walk": "^8.3.4"
- }
-}
deleted file mode 100644
deleted file mode 100644
@@ -1,164 +0,0 @@
@@ -1,164 +0,0 @@
-#!/usr/bin/env python3
-"""The SQLite → PostgreSQL SQL rewrites that are safe to do by text (ADR-035), over a service's server/, scripts/ and
-test/ files; and a report of what needs a person.
-usage: sqlfix.py [files…] (default: git ls-files server scripts test, *.js)
-
-Rewrites:
- INSERT OR IGNORE INTO t … → INSERT INTO t … ON CONFLICT DO NOTHING (at the end of that string literal)
- x COLLATE NOCASE → lower(x)
- LIKE / NOT LIKE → ILIKE / NOT ILIKE (SQLite's LIKE ignores ASCII case)
- json_each(?) → jsonb_array_elements_text(?::jsonb) (a JSON text parameter; column "value")
- IFNULL( → COALESCE(
- db.transaction(fn)() → db.tx(fn); db.inTransaction → db.inTransaction()
- sqlite_master table lists → information_schema.tables of the current schema
- pragma_table_info('t') → information_schema.columns of t
- sequencer.stamp(doc) → sequencer.stamp(store.db, doc) (openvibe-publishing 1.0)
-Reported (fix by hand): INSERT OR REPLACE, rowid, scalar MAX(a, b)/MIN(a, b), HAVING on an alias, WITH RECURSIVE (the
-seed needs the column's collation), datetime/strftime/julianday, IFNULL, GROUP_CONCAT, json_extract, lastInsertRowid,
-.transaction(, setImmediate/nextTick near a transaction, createInbox, PRAGMA."""
-import re, subprocess, sys
-files = sys.argv[1:] or [f for f in subprocess.run(['git', 'ls-files', 'server', 'scripts', 'test'], capture_output=True, text=True).stdout.split() if f.endswith('.js')]
-REPORT = [
- (r'INSERT OR REPLACE', 'INSERT OR REPLACE → ON CONFLICT (key) DO UPDATE SET … (qualify the target columns)'),
- (r'\browid\b', 'rowid → a seq identity column'),
- (r'\b(MAX|MIN)\(\s*[\w.]+\s*,', 'scalar MAX/MIN → GREATEST/LEAST'),
- (r'HAVING\s+[a-z_]+\s*[<>=!]', 'HAVING on an alias → repeat the aggregate'),
- (r'SELECT DISTINCT', 'SELECT DISTINCT with ORDER BY on an expression not selected fails; a DISTINCT over a LEFT JOIN is usually WHERE … OR EXISTS (…)'),
- (r'GROUP BY', 'GROUP BY: every selected bare column must be grouped (SQLite took a bare column from the MAX/MIN row: DISTINCT ON)'),
- (r'WITH RECURSIVE', 'recursive seed: cast ? to the column type and collation (?::text COLLATE "C")'),
- (r"datetime\(|strftime\(|julianday\(", 'SQLite date functions (with SQLITE_DATE_FUNCTIONS in the migration, datetime(t[, modifier]) and julianday(t) exist; strftime does not)'),
- (r'GROUP_CONCAT|json_extract|json_group_array|json_object\(', 'SQLite functions'),
- (r'lastInsertRowid', 'lastInsertRowid → RETURNING'),
- (r'\.transaction\(', 'db.transaction(fn)() → db.tx(fn)'),
- (r'setImmediate|process\.nextTick', 'deferred work: after a commit → db.afterCommit'),
- (r'createInbox\b', 'createInbox → createPgInbox (awaited; receipts table in the migration)'),
- (r'\bPRAGMA\b|\.pragma\(', 'PRAGMA'),
- (r'\b(db|this\.db)\.exec\(', 'DDL at boot on the serving handle: the serving role cannot create tables; the table belongs in a migration'),
- (r'\b(SUM|AVG)\((?![^)]*\bCASE\b)[^)]*\)(?!\s*::)(?!, *0\)::)', 'SUM/AVG of a bigint column is numeric, returned as text: cast it (COALESCE(SUM(x), 0)::bigint) unless the column is double precision'),
- (r'(@\w+|[^\w]\?) IS (NOT )?NULL(?!::)', 'a parameter tested with IS NULL may need a cast (@p::bigint IS NULL) when PostgreSQL cannot infer its type'),
- (r'COALESCE\(\?,', 'COALESCE(?, col): cast the parameter (?::text) when PostgreSQL cannot infer its type from the other arguments'),
- (r'UNIQUE constraint|constraint failed|SQLITE_[A-Z]', "SQLite error text → err.code ('23505' unique, '23503' foreign key, '23514' check) with err.table/err.constraint/err.detail"),
-]
-changed = []
-# Table columns from the service's migrations, so upsert right-hand sides can be qualified even for columns the INSERT
-# does not list (a counter bumped in place, a timestamp cleared).
-TABLE_COLS = {}
-ALWAYS_IDENTITY = {} # table → its GENERATED ALWAYS identity column
-import glob as _glob
-for mf in sorted(_glob.glob('migrations/*.sql')):
- ddl = open(mf).read()
- for tm in re.finditer(r'CREATE TABLE (?:IF NOT EXISTS )?(\w+) \(([\s\S]*?)\n\);', ddl):
- cols = [c.group(1).strip('"') for c in re.finditer(r'^\s*("?\w+"?)\s+(?:text|bigint|integer|double|boolean|bytea|jsonb|timestamptz|numeric|real)', tm.group(2), re.M)]
- TABLE_COLS.setdefault(tm.group(1), set()).update(cols)
- for am in re.finditer(r'ALTER TABLE (\w+) ADD COLUMN (\w+)', ddl):
- TABLE_COLS.setdefault(am.group(1), set()).add(am.group(2))
- for tm in re.finditer(r'CREATE TABLE (?:IF NOT EXISTS )?(\w+) \(([\s\S]*?)\n\);', ddl):
- for im in re.finditer(r'^\s*"?(\w+)"?\s+bigint GENERATED ALWAYS AS IDENTITY', tm.group(2), re.M):
- ALWAYS_IDENTITY.setdefault(tm.group(1), im.group(1))
-# A migration carrying SQLITE_DATE_FUNCTIONS (text timestamps kept as SQLite wrote them): CURRENT_TIMESTAMP in code
-# becomes ov_now() (PostgreSQL's would be a timestamptz, cast to other text), strftime's ISO 'now' ov_now_iso().
-SQLITE_DATES = any('CREATE FUNCTION ov_now()' in open(mf).read() for mf in _glob.glob('migrations/*.sql'))
-for p in files:
- s = open(p).read(); o = s
- if SQLITE_DATES:
- s = re.sub(r'\bCURRENT_TIMESTAMP\b', 'ov_now()', s)
- s = re.sub(r"strftime\('%Y-%m-%dT%H:%M:%fZ', ?'now'\)", 'ov_now_iso()', s)
- s = re.sub(r"strftime\('%Y-%m-%dT%H:%M:%fZ', ?'now', ?('[^']*'|\?)\)", r'ov_now_iso(\1)', s)
- s = re.sub(r"\bdate\('now', ?('[^']*'|\?)\)", r"substr(datetime('now', \1), 1, 10)", s)
- s = re.sub(r"(?<![\w.])date\(([\w.]+)\)(?= AS | >| <| =|,|\))", r'substr(datetime(\1), 1, 10)', s)
- while 'INSERT OR IGNORE INTO' in s:
- i = s.index('INSERT OR IGNORE INTO'); q = s[i - 1] if s[i - 1] in '\'"`' else '`'
- j = s.index(q, i)
- s = s[:i] + 'INSERT INTO' + s[i + len('INSERT OR IGNORE INTO'):j] + ' ON CONFLICT DO NOTHING' + s[j:]
- s = re.sub(r'(\w+\((?:[^()]|\([^()]*\))*\)) COLLATE NOCASE', r'lower(\1)', s) # an expression: COALESCE(a, b) COLLATE NOCASE
- s = re.sub(r'([\w.]+) COLLATE NOCASE', r'lower(\1)', s)
- s = re.sub(r'(?<![IN\w])LIKE (?=[@?:$\'(]|[a-z])', 'ILIKE ', s)
- s = s.replace('json_each(?)', 'jsonb_array_elements_text(?::jsonb)')
- s = re.sub(r'\bIFNULL\(', 'COALESCE(', s)
- # SQLite's null-safe x IS ? / x IS NOT ? (a parameter) are IS [NOT] DISTINCT FROM in PostgreSQL.
- s = re.sub(r'([\w.]+) IS NOT (\?|@\w+)', r'\1 IS DISTINCT FROM \2', s)
- s = re.sub(r'([\w.]+) IS (\?|@\w+)', r'\1 IS NOT DISTINCT FROM \2', s)
- # assert.throws(() => db.prepare(…).run(…), re): the statement rejects now.
- s = re.sub(r"assert\.throws\(\(\) => ((?:t\.)?[\w.]*db\.prepare\((?:[^()]|\([^()]*\))*\)\.(?:run|get|all)\((?:[^()]|\([^()]*\))*\)), ", r"await assert.rejects(\1, ", s)
- # A jsonb column (the SDK outbox's envelope) comes back as an object.
- s = re.sub(r"(?<!\? )JSON\.parse\((\w+)\.envelope\)", r"(typeof \1.envelope === 'string' ? JSON.parse(\1.envelope) : \1.envelope)", s)
- # db.transaction(fn)() → db.tx(fn) (balanced parentheses; the codemod then awaits it and makes fn async).
- out, i = [], 0
- while True:
- j = s.find('.transaction(', i)
- if j < 0: out.append(s[i:]); break
- k, depth = j + len('.transaction('), 1
- while k < len(s) and depth:
- depth += {'(': 1, ')': -1}.get(s[k], 0); k += 1
- if s[k:k + 2] == '()':
- out.append(s[i:j] + '.tx(' + s[j + len('.transaction('):k]); i = k + 2
- else:
- out.append(s[i:k]); i = k
- s = ''.join(out)
- # const f = db.transaction((a, b) => { … }) (called later as f(a, b); an immediately invoked one was rewritten just
- # above) → const f = (a, b) => db.tx(async () => { … })
- s = re.sub(r'const (\w+) = (\w+(?:\.\w+)?)\.transaction\(\(([^()]*)\) => \{', r'const \1 = (\3) => \2.tx(async () => {', s)
- # An unquoted alias is folded to lower case in PostgreSQL (AS totalViews comes back as totalviews): quote
- # camelCase aliases inside SQL string literals.
- def quote_aliases(lit):
- body = lit.group(0)
- if not re.search(r'\bSELECT\b', body): return body
- q = '\\"' if body[0] == '"' else '"'
- return re.sub(r'\b(AS|as) ([a-z]+[A-Z]\w*)\b', lambda m: f'{m.group(1)} {q}{m.group(2)}{q}', body)
- s = re.sub(r'`[^`]*`|\'(?:[^\'\\\n]|\\.)*\'|"(?:[^"\\\n]|\\.)*"', quote_aliases, s)
- # Column names PostgreSQL reserves (SQLite does not): quoted inside SQL string literals ("window", "user").
- def quote_reserved(lit):
- body = lit.group(0)
- if not re.search(r'\b(SELECT|INSERT|UPDATE|DELETE|CREATE)\b', body): return body
- q = '\\"' if body[0] == '"' else '"' # inside a double-quoted JS string the quotes are escaped
- return re.sub(r"(?<![.\w\"$\\'@:])(window|user)(?![\w\"(\\'])", lambda m: q + m.group(1) + q, body)
- s = re.sub(r'`[^`]*`|\'(?:[^\'\\\n]|\\.)*\'|"(?:[^"\\\n]|\\.)*"', quote_reserved, s)
- # better-sqlite3's db.inTransaction was a property; openvibe-sdk/db's is a function.
- s = re.sub(r'\b(db|this\.db|store\.db)\.inTransaction\b(?!\s*\()', r'\1.inTransaction()', s)
- # ON CONFLICT … DO UPDATE SET x = x + 1 / COALESCE(excluded.x, x): a bare column on the right is ambiguous in
- # PostgreSQL; qualify it with the table (columns of the INSERT list only; excluded.x and the targets stay).
- def qualify(m):
- table, cols, rest = m.group(1), [c.strip() for c in m.group(2).split(',')], m.group(3)
- cols = sorted(set(cols) | TABLE_COLS.get(table, set()), key=len, reverse=True)
- k = rest.index('DO UPDATE SET') + len('DO UPDATE SET')
- head, sets = rest[:k], rest[k:]
- def fix_rhs(a):
- lhs, eq, rhs = a.partition('=')
- if not eq: return a
- for c in cols:
- rhs = re.sub(r'(?<![.\w@:$])%s\b(?!\s*\()' % re.escape(c), f'{table}.{c}', rhs)
- return lhs + eq + rhs
- parts = re.split(r'(,\s*(?=\w+\s*=))', sets)
- return f'INSERT INTO {table} ({m.group(2)}){head}' + ''.join(fix_rhs(x) if not re.match(r',\s*$', x) else x for x in parts)
- s = re.sub(r'INSERT INTO (\w+) \(([^)]*)\)((?:(?!INSERT INTO)[^`\'"])*?DO UPDATE SET[^`\'"]*)', qualify, s)
- # INSERT INTO t (id, …) VALUES with an explicit id into a GENERATED ALWAYS identity (tests, fixtures, imports):
- # PostgreSQL refuses it without OVERRIDING SYSTEM VALUE.
- def override(m):
- table, cols = m.group(1), [c.strip().strip('"') for c in m.group(2).split(',')]
- return m.group(0) + ' OVERRIDING SYSTEM VALUE' if ALWAYS_IDENTITY.get(table) in cols else m.group(0)
- s = re.sub(r'INSERT INTO (\w+) \(([^)]*)\)(?! OVERRIDING)(?=\s*(?:VALUES|SELECT))', override, s)
- # (await db.prepare('INSERT …').run(…)).lastInsertRowid: the id comes back through RETURNING.
- s = re.sub(r"(prepare\()(`|'|\")(INSERT INTO (\w+)(?:(?!\2).)*?)(\2\)\s*\.run\((?:[^()]|\([^()]*\))*\)\)\s*\.lastInsertRowid)",
- lambda m: m.group(0) if 'RETURNING' in m.group(3) else m.group(1) + m.group(2) + m.group(3) + ' RETURNING id' + m.group(5), s)
- # Tests matching SQLite's constraint messages: PostgreSQL's say "duplicate key", "violates … constraint".
- s = re.sub(r'/UNIQUE constraint failed[^/\n]*/|/UNIQUE/', '/duplicate key/', s)
- s = re.sub(r'/CHECK constraint failed[^/\n]*/', '/violates check constraint/', s)
- s = re.sub(r'/NOT NULL constraint failed[^/\n]*/', '/violates not-null constraint/', s)
- s = re.sub(r'/FOREIGN KEY constraint failed[^/\n]*/', '/violates foreign key constraint/', s)
- # openvibe-publishing 1.0: the index sequencer stamps through a handle (the ambient transaction joins through store.db).
- handle = 'store.db' if 'store.db' in s else 'db'
- s = re.sub(r'((?:store\.)?sequencer\.stamp\()(?!store\.db|db,|t,)', lambda m: m.group(1) + handle + ', ', s)
- s = re.sub(r"(SELECT name FROM sqlite_master WHERE type (?:= 'table'|IN \('table', ?'view'\))) AND name NOT (?:I)?LIKE 'sqlite_%'", r"\1", s)
- s = re.sub(r"SELECT name FROM sqlite_master WHERE type (?:= 'table'|IN \('table', ?'view'\))",
- "SELECT table_name AS name FROM information_schema.tables WHERE table_schema = current_schema()", s)
- # SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ? / 'x' → the table in the current schema
- s = re.sub(r"SELECT 1( AS \w+)? FROM sqlite_master WHERE type = 'table' AND name = (\?|'\w+')",
- r"SELECT 1\1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = \2", s)
- s = re.sub(r"SELECT name FROM pragma_table_info\('(\w+)'\)",
- r"SELECT column_name AS name FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = '\1' ORDER BY ordinal_position", s)
- if s != o: open(p, 'w').write(s); changed.append(p)
- for n, line in enumerate(s.split('\n'), 1):
- for pat, why in REPORT:
- if re.search(pat, line) and not line.lstrip().startswith(('//', '*')):
- print(f'CHECK {p}:{n}: {why}: {line.strip()[:120]}')
-print('rewrote: ' + (', '.join(changed) or 'nothing'))
deleted file mode 100644
deleted file mode 100644
@@ -1,98 +0,0 @@
@@ -1,98 +0,0 @@
-'use strict';
-/**
- * A service's SQLite DDL (the SCHEMA string its db.js ran) → PostgreSQL DDL with the same meaning (ADR-035), for
- * migrations/0001_initial.sql:
- * INTEGER PRIMARY KEY AUTOINCREMENT → bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY (the importer keeps the ids)
- * TEXT → text COLLATE "C" (SQLite compares text bytewise; "C" sorts and compares the same way)
- * INTEGER → bigint (epoch milliseconds and counters; openvibe-sdk/db returns int8 as Number)
- * REAL → double precision, BLOB → bytea
- * CREATE … IF NOT EXISTS → CREATE … (a migration runs once)
- * Types only in column position (upper case, as SQLite schemas are written); comments are kept. An immutable-row
- * trigger (BEFORE UPDATE/DELETE … RAISE(ABORT, msg)) becomes a PL/pgSQL function and trigger; any other trigger is
- * refused (write it by hand).
- *
- * const { toPg } = require('openvibe-sdk/tools/asyncify/sqlite-schema-to-pg'); (from a checkout)
- */
-/** Words PostgreSQL reserves that SQLite takes as plain column names. */
-const PG_ONLY_RESERVED = ['window', 'user', 'freeze', 'analyse', 'both', 'leading', 'trailing', 'symmetric', 'asymmetric', 'variadic',
- 'lateral', 'ilike', 'similar', 'verbose', 'tablesample', 'concurrently', 'overlaps', 'authorization', 'binary', 'collation',
- 'only', 'placing', 'array', 'fetch', 'current_role', 'session_user', 'current_user', 'current_catalog', 'current_schema'];
-
-function toPg(sql) {
- // A SQLite row trigger → a PL/pgSQL function and a trigger. Its WHEN condition (which may hold a subquery, not
- // allowed in a PostgreSQL trigger's WHEN) goes inside the function as IF cond THEN body END IF; RAISE(ABORT, msg)
- // becomes RAISE EXCEPTION; a BEFORE trigger lets the row pass (NEW, or OLD for a delete), an AFTER one returns NULL.
- const cond = (c) => c.replace(/\s+/g, ' ').trim().replace(/ IS NOT (OLD|NEW)\./g, ' IS DISTINCT FROM $1.').replace(/(\b(?:OLD|NEW)\.\w+) IS (OLD|NEW)\./g, '$1 IS NOT DISTINCT FROM $2.');
- sql = sql.replace(/CREATE TRIGGER(?: IF NOT EXISTS)?\s+(\w+)\s+(BEFORE|AFTER)\s+(INSERT|UPDATE(?:\s+OF\s+[\w\s,]+?)?|DELETE)\s+ON\s+(\w+)(?:\s+FOR EACH ROW)?(?:\s+WHEN\s+([\s\S]+?))?\s+BEGIN\s+([\s\S]+?)\s*END;/g,
- (m, name, when, op, table, whenCond, body) => {
- const stmts = body.replace(/SELECT RAISE\(ABORT, '([^']*)'\);?/g, "RAISE EXCEPTION '$1';")
- .replace(/CURRENT_TIMESTAMP/g, 'ov_now()').trim().replace(/;?$/, ';');
- const row = when === 'AFTER' ? 'NULL' : /^DELETE/.test(op) ? 'OLD' : 'NEW';
- const inner = whenCond ? `IF ${cond(whenCond)} THEN ${stmts} END IF;` : stmts;
- return `CREATE FUNCTION ${name}() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN ${inner.replace(/\s+/g, ' ')} RETURN ${row}; END $$;\n`
- + `CREATE TRIGGER ${name} ${when} ${op.replace(/\s+/g, ' ')} ON ${table} FOR EACH ROW EXECUTE FUNCTION ${name}();`;
- });
- if (sql.split(';').some((st) => /CREATE\s+TRIGGER/i.test(st) && !/FOR EACH ROW[\s\S]*EXECUTE FUNCTION/.test(st) && !/\$\$/.test(st))) {
- throw new Error('sqlite-schema-to-pg: a trigger this converter does not read needs writing as PL/pgSQL by hand');
- }
- // COLLATE NOCASE: in an index, the index is on lower(col) (queries compare lower(col) too: sqlfix.py); on a column
- // definition it goes (a case-insensitive UNIQUE there needs a lower() unique index, written by hand).
- sql = sql.replace(/(CREATE (?:UNIQUE )?INDEX[^;]*?\()([^;]*?)\)/g, (m, head, cols) => head + cols.replace(/(\w+) COLLATE NOCASE/g, 'lower($1)') + ')');
- if (/^\s*\w+\s+TEXT\b[^\n]*COLLATE NOCASE[^\n]*UNIQUE|UNIQUE[^\n]*COLLATE NOCASE/m.test(sql)) throw new Error('sqlite-schema-to-pg: a case-insensitive UNIQUE column needs a unique index on lower(col), by hand');
- sql = sql.replace(/^(\s*\w+\s+TEXT\b[^\n]*?) COLLATE NOCASE/gm, '$1');
- // Column names PostgreSQL reserves (SQLite does not) are quoted: a column definition, and inside index/unique lists.
- // The service's queries must quote them too (Billing's settings.freeze).
- const R = PG_ONLY_RESERVED.join('|');
- sql = sql.replace(new RegExp(`^(\\s*)(${R})(\\s+[A-Za-z])`, 'gim'), '$1"$2"$3').replace(new RegExp(`([(,]\\s*)(${R})(?=\\s*[,)])`, 'gi'), '$1"$2"');
- return sql
- .replace(/\bINTEGER PRIMARY KEY AUTOINCREMENT\b/g, 'bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY')
- // An INTEGER PRIMARY KEY that references another table is that row's key, not a counter.
- .replace(/\bINTEGER PRIMARY KEY\b(?! AUTOINCREMENT)(?=[^,\n]*\bREFERENCES\b)/g, 'bigint PRIMARY KEY')
- // A bare INTEGER PRIMARY KEY is SQLite's rowid: code may insert without it or with an explicit value.
- .replace(/\bINTEGER PRIMARY KEY\b(?! AUTOINCREMENT)/g, 'bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY')
- .replace(/CREATE (UNIQUE )?INDEX IF NOT EXISTS/g, (m, u) => `CREATE ${u || ''}INDEX`)
- .replace(/CREATE TABLE IF NOT EXISTS/g, 'CREATE TABLE')
- .replace(/\)\s*WITHOUT ROWID\s*;/g, ');')
- .replace(/CREATE VIEW IF NOT EXISTS/g, 'CREATE VIEW')
- .replace(/^(\s*"?\w+"?\s+)TEXT\b/gm, '$1text COLLATE "C"')
- .replace(/^(\s*"?\w+"?\s+)INTEGER\b/gm, '$1bigint')
- .replace(/^(\s*"?\w+"?\s+)REAL\b/gm, '$1double precision')
- .replace(/^(\s*"?\w+"?\s+)BLOB\b/gm, '$1bytea')
- // SQLite text timestamps stay text in the same format; their defaults use SQLITE_DATE_FUNCTIONS below.
- .replace(/^(\s*"?\w+"?\s+)DATETIME\b/gm, '$1text COLLATE "C"')
- .replace(/DEFAULT CURRENT_TIMESTAMP\b/g, 'DEFAULT ov_now()')
- .replace(/DEFAULT \(datetime\('now'\)\)/g, 'DEFAULT ov_now()')
- .replace(/DEFAULT \(strftime\('%Y-%m-%dT%H:%M:%fZ', ?'now'\)\)/g, 'DEFAULT ov_now_iso()');
-}
-
-/**
- * SQLite's text timestamps and date functions on PostgreSQL, for a service whose columns hold 'YYYY-MM-DD HH:MM:SS'
- * (UTC) text: the stored values, their comparisons and what the API returns stay exactly as they were. ov_now() is
- * CURRENT_TIMESTAMP's text (per statement, as SQLite), ov_now_iso([modifier]) strftime('%Y-%m-%dT%H:%M:%fZ', 'now'[, modifier]);
- * datetime(t [, modifier]) and julianday(t) read 'now', SQLite text and ISO 8601 (a zone, or none for UTC) and give
- * NULL for what they cannot read, as SQLite does. A modifier is an interval ('-7 days', '+1 hour').
- */
-const SQLITE_DATE_FUNCTIONS = `-- SQLite's text timestamps and date functions (openvibe-sdk tools/asyncify SQLITE_DATE_FUNCTIONS).
-CREATE FUNCTION ov_ts(t text) RETURNS timestamp LANGUAGE plpgsql STABLE AS $$
-BEGIN
- IF t IS NULL THEN RETURN NULL; END IF;
- IF t = 'now' THEN RETURN statement_timestamp() AT TIME ZONE 'UTC'; END IF;
- IF t ~ '\\d\\d:\\d\\d(:\\d\\d(\\.\\d+)?)?\\s*(Z|[+-]\\d\\d(:?\\d\\d)?)$' THEN RETURN t::timestamptz AT TIME ZONE 'UTC'; END IF;
- RETURN t::timestamp;
-EXCEPTION WHEN others THEN RETURN NULL;
-END $$;
-CREATE FUNCTION ov_now() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD HH24:MI:SS') $$;
-CREATE FUNCTION ov_now_iso() RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char(statement_timestamp() AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;
-CREATE FUNCTION ov_now_iso(modifier text) RETURNS text LANGUAGE sql STABLE AS $$ SELECT to_char((statement_timestamp() AT TIME ZONE 'UTC') + modifier::interval, 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"') $$;
-CREATE FUNCTION datetime(t text, modifier text DEFAULT NULL) RETURNS text LANGUAGE plpgsql STABLE AS $$
-DECLARE ts timestamp := ov_ts(t);
-BEGIN
- IF ts IS NULL THEN RETURN NULL; END IF;
- IF modifier IS NOT NULL THEN ts := ts + modifier::interval; END IF;
- RETURN to_char(ts, 'YYYY-MM-DD HH24:MI:SS');
-EXCEPTION WHEN others THEN RETURN NULL;
-END $$;
-CREATE FUNCTION julianday(t text) RETURNS double precision LANGUAGE sql STABLE AS $$ SELECT extract(epoch FROM ov_ts(t))::double precision / 86400.0 + 2440587.5 $$;
-`;
-
-module.exports = { toPg, SQLITE_DATE_FUNCTIONS, PG_ONLY_RESERVED };
deleted file mode 100644
deleted file mode 100644
@@ -1,14 +0,0 @@
@@ -1,14 +0,0 @@
-#!/usr/bin/env node
-'use strict';
-// Every column declared INTEGER/REAL holding a value of another storage class (SQLite's affinity lets text in).
-const Database = require(require.resolve('better-sqlite3', { paths: [process.cwd()] }));
-const db = new Database(process.argv[2], { readonly: true });
-for (const { name } of db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'").all()) {
- for (const c of db.prepare(`PRAGMA table_info("${name}")`).all()) {
- const t = String(c.type).toUpperCase();
- const want = /INT/.test(t) ? 'integer' : /REAL|FLOA|DOUB/.test(t) ? 'real' : null;
- if (!want) continue;
- const rows = db.prepare(`SELECT typeof("${c.name}") AS k, COUNT(*) AS n, MIN("${c.name}") AS ex FROM "${name}" WHERE "${c.name}" IS NOT NULL AND typeof("${c.name}") NOT IN ('${want}'${want === 'real' ? ", 'integer'" : ''}) GROUP BY 1`).all();
- for (const r of rows) console.log(`${name}.${c.name} (${t}): ${r.n} ${r.k} value(s), e.g. ${String(r.ex).slice(0, 40)}`);
- }
-}