openvibers/openvibe.trade Commits
Trade: account export and deletion (ADR-033) — watchlists, alerts and drafts deleted; contributions made authorless (append-only rows only inside the erasure transaction); reviews kept; subscriptions at boot (SDK v0.37.0) (#21)
Changes
@@@ -2,5 +2,11 @@## Unreleased++- Account export and deletion (ADR-033) through `server/domain/account-data.js` over `openvibe-sdk/account-data` (SDK v0.37.0, was v0.35.0):++ - **Deleted:** watchlists, alert rules (with their deliveries) and drafts.++ - **Authorless:** contributed instruments, aliases, purges, context revisions and hand-recorded observations.++ - **Kept:** reviews.++ - **Migrations:** `0003_account_data.sql` adds `account_data_events`. `0004_account_erasure.sql` lets only the erasure transaction take a person's id out of the append-only revisions and observations.++ - The two subscriptions are created at boot. `test/account-data.test.js`.- openvibe-contracts moves to v0.97.0 (pin, lockfile and `node_modules`). All twelve `trade.*` ids are now defined by the release, so `server/auth/capabilities.js` drops its local fallback for proposed ids and sends every check through the library's grant rule; `test/capabilities.test.js` pins that every guarded id is defined and that exact, prefix, denied and unknown ids answer as `capabilities.check()` does. `server/index.js` moves its shutdown onto `openvibe-sdk/service`'s `gracefulStop` (drain with `Connection: close`, then the worker, the outbox and the store; SIGTERM/SIGINT replace the hand-rolled handlers and timer), pinned by the spawn-SIGTERM check in `test/service-kit.test.js`. README and STATUS.json name v0.97.0 and the pinned v0.26.0 SDK.- Every page is rendered through `openvibe-publishing/layout` (v1.2.0, on `openvibe-shared/shell` v2.6.0): the head, the Frame, the noscript navigation, the footer and its init come from the shared document; Trade keeps its site stylesheet, the disclaimers, the account bar and the "Recently shipped" line. Pins `openvibe-shared` v2.6.0 and `openvibe-publishing` v1.2.0.@@@ -128,7 +128,7 @@ deployment has no instruments (editors add them), no documents until the source| `/terms`, `/privacy`, `/dmca` | openvibe-shared legal pages || `/auth/*` | sign-in; the same session layer as Blog and Community || `/api/health`, `/api/ready`, `/release.json`, `POST /release-metrics`, `/metrics` | health, readiness, release, open tabs' update reports (counted in `/metrics`), metrics (loopback only) |--| `POST /internal/events` | signed OpenVibe.Events deliveries (`sources.*`), host-local |++| `POST /internal/events` | signed OpenVibe.Events deliveries (`sources.*`, and the two ADR-033 account events), host-local |### Discoverability (roadmap §32)@@@ -195,6 +195,22 @@ decided with the contracts library's matching rule (`server/auth/capabilities.jsConsumed: `sources.item.created|updated|removed` and `sources.fetch.failed` through the signedwebhook, as a wake-up for the cursor sync (the webhook is not durable truth; the sync is).++### Account export and deletion (ADR-033)++++`network.account.export_requested` and `network.account.deleted` arrive at `POST /internal/events` and are answered by++`server/domain/account-data.js` (`openvibe-sdk/account-data`).++++- **Export:** the person's watchlists and their items, alert rules and deliveries, context drafts, the revisions they++ wrote, the observations they recorded by hand, and their reviews.++- **Deleted:** watchlists (items cascade), alert rules with their deliveries, and drafts.++- **Made authorless:** instruments, aliases and purges they made. Context revisions and observations stay append-only:++ only inside the erasure transaction (`trade.account_erasure`, migration `0004_account_erasure.sql`) may a revision's++ author become NULL and their id leave `meta.authorship.authors`, or an observation's `recorded_by` become `deleted`.++ The text, the values and their provenance never change.++- **Kept, counted as retained:** reviews, since a person's approval is what lets reviewed text stay published.++- **Subscriptions:** the two are created at boot when missing (needs `EVENTS_URL` and `TRADE_EVENTS_WEBHOOK_SECRET`).++- **Receipts:** `account_data_events` keeps one per export and deletion.++### OpenVibe.AI seam (`trade.summarize_market_context`)1. The AI (or its caller) reads `GET /api/v1/instruments/:symbol/context/input` — the workflow input@@@ -242,7 +258,8 @@ services need to call Trade, is under [Grants the Network must hold](#grants-theEach grant is `[client, capability, audience]`:- `[trade, events.event.publish, openvibe.events]`--- `[trade, events.subscription.manage, openvibe.events]`++- `[trade, events.subscription.manage, openvibe.events]` (also the account subscriptions at boot)++- `[trade, network.account.export.contribute, openvibe.network]` and `[trade, network.account.deletion.confirm, openvibe.network]` (ADR-033; granted last, once this release is live)- `[trade, sources.item.read, openvibe.sources]`- `[trade, sources.source.read, openvibe.sources]`- For OpenVibe.AI to deliver drafts: `[ai, trade.context.read, openvibe.trade]` and@@@ -9,7 +9,7 @@"domain": "openvibe.trade","scope": "informational only (ADR-025): no custody, no order execution, no escrow, no listings between users, no personalised advice","contracts": "openvibe-contracts v0.112.0",-- "sdk": "openvibe-sdk v0.35.0",++ "sdk": "openvibe-sdk v0.37.0","shared": "openvibe-shared v2.15.0","packages": ["openvibe-contracts v0.97.0",nnew file mode 100644@@@ -0,0 +1,13 @@++-- phase: expand++-- OpenVibe.Trade: the receipts of the ADR-033 account export and deletion deliveries this service applied++-- (openvibe-sdk/account-data's ACCOUNT_DATA_SCHEMA), so a redelivered export or deletion changes nothing.++-- The row holds an export_id or deletion_id, the subject and the counts sent to Network; it is not a secret.++++CREATE TABLE IF NOT EXISTS account_data_events (++ id TEXT PRIMARY KEY,++ kind TEXT NOT NULL,++ subject TEXT NOT NULL,++ outcome JSONB,++ applied_at TIMESTAMPTZ NOT NULL DEFAULT now(),++ sent_at TIMESTAMPTZ++);nnew file mode 100644@@@ -0,0 +1,45 @@++-- phase: expand++-- ADR-033 account deletion (server/domain/account-data.js): context revisions and market observations stay++-- append-only, with one exception. Inside the account-erasure transaction, and only there (it sets++-- trade.account_erasure = 'on' with set_config(..., true), so the setting ends with the transaction), the person's++-- id may leave a row: a revision's author becomes NULL and their id leaves meta.authorship.authors (also on a++-- revision someone else wrote with them), and an++-- observation's recorded_by becomes 'deleted'. Every other column must stay as it was, so the text, the values and++-- their provenance never change. Deletes keep their old rules.++++CREATE OR REPLACE FUNCTION trade_context_revisions_guard() RETURNS trigger LANGUAGE plpgsql AS $$++BEGIN++ IF TG_OP = 'UPDATE' THEN++ IF current_setting('trade.account_erasure', true) = 'on'++ AND (NEW.author IS NULL OR NEW.author IS NOT DISTINCT FROM OLD.author)++ AND (NEW.id, NEW.entity_id, NEW.number, NEW.parent_id, NEW.parent_number, NEW.kind, NEW.reverted_to, NEW.content,++ NEW.fields, NEW.content_hash, NEW.message, NEW.created_at)++ IS NOT DISTINCT FROM++ (OLD.id, OLD.entity_id, OLD.number, OLD.parent_id, OLD.parent_number, OLD.kind, OLD.reverted_to, OLD.content,++ OLD.fields, OLD.content_hash, OLD.message, OLD.created_at)++ AND (NEW.meta - 'authorship') IS NOT DISTINCT FROM (OLD.meta - 'authorship')++ AND ((NEW.meta->'authorship') - 'authors') IS NOT DISTINCT FROM ((OLD.meta->'authorship') - 'authors') THEN++ RETURN NEW;++ END IF;++ RAISE EXCEPTION 'trade_context_revisions rows are immutable' USING ERRCODE = 'restrict_violation';++ END IF;++ IF NOT EXISTS (SELECT 1 FROM trade_context_revision_purges WHERE entity_id = OLD.entity_id) THEN++ RAISE EXCEPTION 'trade_context_revisions rows are never deleted outside a recorded purge' USING ERRCODE = 'restrict_violation';++ END IF;++ RETURN OLD;++END++$$;++++CREATE OR REPLACE FUNCTION trade_obs_no_update() RETURNS trigger LANGUAGE plpgsql AS $$++BEGIN++ IF current_setting('trade.account_erasure', true) = 'on' AND NEW.recorded_by = 'deleted'++ AND (NEW.id, NEW.instrument_id, NEW.metric, NEW.value, NEW.value_num, NEW.unit, NEW.currency, NEW.period, NEW.observed_at,++ NEW.source_key, NEW.source_item_id, NEW.source_url, NEW.source_ref, NEW.retrieved_at, NEW.max_age_sec, NEW.recorded_at)++ IS NOT DISTINCT FROM++ (OLD.id, OLD.instrument_id, OLD.metric, OLD.value, OLD.value_num, OLD.unit, OLD.currency, OLD.period, OLD.observed_at,++ OLD.source_key, OLD.source_item_id, OLD.source_url, OLD.source_ref, OLD.retrieved_at, OLD.max_age_sec, OLD.recorded_at) THEN++ RETURN NEW;++ END IF;++ RAISE EXCEPTION 'trade_market_observations rows are immutable';++END++$$;@@@ -17,7 +17,7 @@"iovalkey": "^0.4.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.112.0","openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.3.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.35.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.37.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.15.0","pg": "^8.23.0"},@@@ -907,9 +907,9 @@}},"node_modules/openvibe-sdk": {-- "version": "0.35.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.35.0",-- "integrity": "sha512-IF7XB97tU7Q+ccwssGx6QXXcfDPcVpl+mX3BjIr7C6NJlcmoqj9DbHiKrBxlVav0eG44Jf8zuXEwokrHDq3J0w==",++ "version": "0.37.0",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.37.0",++ "integrity": "sha512-afojId2KzOL4ZZrpwIeIAYUaX24ZOC3Nt+bdd1HSXm7aislt8WQNS62i8/d8W3udLr6rmSG29SjmnaRHUGfd4Q==","license": "MIT","engines": {"node": ">=20"@@@ -29,7 +29,7 @@"iovalkey": "^0.4.0","openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.112.0","openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.3.0",-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.35.0",++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.37.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.15.0","pg": "^8.23.0"},@@@ -33,6 +33,8 @@ const configLib = require('./config');const { openStore } = require('./db');const { createViewerResolver } = require('./auth/viewer');const { createWebhook } = require('./events/consumer');++const accountDataLib = require('./domain/account-data');++const { createNetworkSender } = require('openvibe-sdk/account-data');const { createUrls } = require('./domain/urls');const { createInstruments } = require('./domain/instruments');const { createFreshness } = require('./domain/source-status');@@@ -164,7 +166,12 @@ async function createApp(opts = {}) {app.use(machine);// ── Signed OpenVibe.Events deliveries (before any body parser) ──-- app.use(createWebhook({ store, config, ingest: ctx.sync, log }).router);++ // Account export and deletion (ADR-033, domain/account-data.js), pushed to Network with this service's own token.++ ctx.accountData = accountDataLib.create({ db: store.db, log });++ ctx.accountSend = opts.accountSend || (config.oauth.clientSecret++ ? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })++ : null);++ app.use(createWebhook({ store, config, ingest: ctx.sync, log, accountData: ctx.accountData, accountSend: ctx.accountSend }).router);// ── Sign-in (OAuth2 client of OpenVibe.Network) ─────────app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, standardHeaders: true, legacyHeaders: false }));nnew file mode 100644@@@ -0,0 +1,73 @@++'use strict';++++/**++ * Account export and deletion → Trade (ADR-033; openvibe-sdk/account-data). What Trade holds about a person:++ *++ * their own, deleted whole watchlists (their items go with them, ON DELETE CASCADE), alert rules and the++ * deliveries under them, and unsaved context drafts++ * contributed, made authorless instruments and aliases they added, context revisions they wrote (the author++ * column and their id in meta.authorship.authors), observations they recorded by hand++ * (recorded_by is NOT NULL, so it becomes 'deleted'), and purges they ran++ * kept context reviews: an approval by a person is what lets reviewed text stay published++ * (openvibe-publishing/authorship), so a reviewer's record stays (counted as retained)++ *++ * No secret or other person's data is held in any of these tables.++ */++const { createAccountData, TOPICS } = require('openvibe-sdk/account-data');++++const anonymize = { anonymize: {} };++++const TABLES = [++ { table: 'trade_watchlists', subject: 'owner_subject', file: 'watchlists.json', columns: ['id', 'name', 'created_at', 'updated_at'] },++ // Deliveries reference their rule, so they go first.++ { table: 'trade_alert_deliveries', subject: 'owner_subject', file: 'alert-deliveries.json', columns: ['id', 'rule_id', 'trigger_kind', 'trigger_id', 'summary', 'created_at'] },++ { table: 'trade_alert_rules', subject: 'owner_subject', file: 'alert-rules.json', columns: ['id', 'instrument_id', 'kind', 'metric', 'operator', 'threshold', 'unit', 'currency', 'form_types', 'status', 'created_at', 'updated_at'] },++ { table: 'trade_context_drafts', subject: 'owner', file: 'context-drafts.json', columns: ['entity_id', 'base_revision', 'content', 'fields', 'created_at', 'updated_at'] },++ { table: 'trade_instruments', subject: 'created_by', file: 'instruments.json', columns: ['id', 'symbol', 'name', 'kind', 'exchange', 'currency', 'created_at'], erase: anonymize },++ { table: 'trade_instrument_aliases', subject: 'added_by', file: null, erase: anonymize },++ { table: 'trade_context_revision_purges', subject: 'purged_by', file: null, erase: anonymize },++ { table: 'trade_context_reviews', subject: 'reviewer', file: 'context-reviews.json', columns: ['entity_id', 'revision', 'decision', 'note', 'reviewed_at'], order: 'reviewed_at', erase: { keep: 'a person\'s approval is what lets reviewed text stay published' } },++];++++/**++ * The watchlists' items (keyed by the list, not the person), the context revisions they wrote and the observations++ * they recorded by hand. Revisions and observations are append-only (database triggers), so they are erased in++ * extraErase under the trade.account_erasure setting (migrations/0004_account_erasure.sql), not by the table map.++ */++async function extraExport(db, subject) {++ const files = [];++ const revisions = await db.many(`SELECT entity_id, number, kind, content, fields, message, created_at FROM trade_context_revisions++ WHERE author = $1 ORDER BY created_at DESC LIMIT 5000`, [subject]);++ if (revisions.length) files.push({ name: 'context-revisions.json', content: revisions });++ const items = await db.many(`SELECT i.watchlist_id, i.instrument_id, i.note, i.added_at FROM trade_watchlist_items i++ JOIN trade_watchlists w ON w.id = i.watchlist_id WHERE w.owner_subject = $1 ORDER BY i.added_at DESC LIMIT 5000`, [subject]);++ if (items.length) files.push({ name: 'watchlist-items.json', content: items });++ const obs = await db.many(`SELECT id, instrument_id, metric, value, unit, currency, period, observed_at, source_key, source_url, recorded_at++ FROM trade_market_observations WHERE recorded_by = $1 ORDER BY recorded_at DESC LIMIT 5000`, [subject]);++ if (obs.length) files.push({ name: 'observations.json', content: obs });++ return files;++}++++async function extraErase(t, subjects, counts) {++ // Only this transaction may take a person's id out of the append-only rows; the setting ends with it.++ await t.value("SELECT set_config('trade.account_erasure', 'on', true)");++ counts.add(counts.retained, 'tombstones', await t.exec(`UPDATE trade_context_revisions SET author = NULL++ WHERE author = ANY($1::text[])`, [subjects]));++ counts.add(counts.retained, 'tombstones', await t.exec(`UPDATE trade_market_observations SET recorded_by = 'deleted'++ WHERE recorded_by = ANY($1::text[])`, [subjects]));++ counts.add(counts.retained, 'tombstones', await t.exec(`UPDATE trade_context_revisions++ SET meta = jsonb_set(meta, '{authorship,authors}', COALESCE((SELECT jsonb_agg(a) FROM jsonb_array_elements(meta->'authorship'->'authors') a++ WHERE NOT ((a #>> '{}') = ANY($1::text[]))), '[]'::jsonb))++ WHERE jsonb_typeof(meta->'authorship'->'authors') = 'array' AND (meta->'authorship'->'authors') ?| $1::text[]`, [subjects]));++ await t.value("SELECT set_config('trade.account_erasure', 'off', true)");++}++++/** The account-data handle for Trade's database (server/db.js, store.db). */++function create({ db, log = console } = {}) {++ return createAccountData({++ db, service: 'trade', tables: TABLES, extraExport, extraErase, log,++ note: 'Instruments, context text and observations you contributed stay without your name; reviews you approved stay attributed.',++ });++}++++module.exports = { create, TABLES, TOPICS };@@@ -1,7 +1,8 @@'use strict';/**-- * POST /internal/events — signed OpenVibe.Events webhook deliveries (subscription topic sources.*).++ * POST /internal/events — signed OpenVibe.Events webhook deliveries (subscription topic sources.*, and++ * network.account.export_requested / network.account.deleted for ADR-033, answered by domain/account-data.js).** The signature check, the ±300 s window and the exactly-once inbox are the ingest chassis* (openvibe-publishing/ingest, createEventConsumer); this route only maps its result onto HTTP and@@@ -14,8 +15,13 @@ const { createEventConsumer } = require('openvibe-publishing/ingest');const { http } = require('openvibe-contracts');const WAKE_RE = /^sources\.(item\.(created|updated|removed)|fetch\.failed)$/;++const { TOPICS: ACCOUNT_TOPICS } = require('openvibe-sdk/account-data');--function createWebhook({ store, config, ingest, log = console }) {++/**++ * accountData + accountSend: account export and deletion and its sender to Network. A failure throws, the inbox receipt++ * rolls back and Events redelivers; account-data keeps its own receipt per export and deletion id.++ */++function createWebhook({ store, config, ingest, log = console, accountData = null, accountSend = null }) {const router = express.Router();const consumer = createEventConsumer({ db: store.db, secrets: config.events.webhookSecret, consumer: 'sources', now: store.now });@@@ -25,6 +31,11 @@ function createWebhook({ store, config, ingest, log = console }) {let r;try {r = await consumer.apply(raw, req.headers, async (e) => {++ if (ACCOUNT_TOPICS.includes(e.event_type)) {++ if (!accountData || !accountSend) throw new Error('account export and deletion are not configured');++ await accountData.apply(e, { send: accountSend });++ return false;++ }const wake = WAKE_RE.test(e.event_type);if (wake) ingest.run().catch((err) => log.warn('[Trade] ingest after webhook failed:', err.message));return wake;@@@ -8,6 +8,7 @@* store, and exits 0 (the 5 s family's deadline exit code).*/const { gracefulStop } = require('openvibe-sdk/service');++const { startSubscriptions } = require('openvibe-sdk/account-data');const { createApp } = require('./app');(async () => {@@@ -21,10 +22,16 @@ const server = app.listen(config.port, config.host, () => {server.keepAliveTimeout = 65_000;ctx.outbox.start();ctx.worker.start();++// The two account subscriptions at OpenVibe.Events (ADR-033), created when missing; off without EVENTS_URL,++// TRADE_EVENTS_WEBHOOK_SECRET or the client secret. The sources.* one stays scripts/subscribe.js's.++const subscriptions = startSubscriptions({++ eventsUrl: config.events.url, endpoint: `http://127.0.0.1:${config.port}/internal/events`, secret: String(config.events.webhookSecret || '').split(',')[0].trim(),++ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,++});gracefulStop({name: 'Trade', server, deadlineExitCode: 0,-- stop: [() => ctx.worker.stop()],++ stop: [() => ctx.worker.stop(), () => { if (subscriptions) subscriptions.stop(); }],close: [() => ctx.outbox.stop(), () => ctx.store.close()],});})().catch((err) => { console.error('[Trade] failed to start:', err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,128 @@++'use strict';++/**++ * ADR-033: Trade's part of an account export and of an account deletion, through the signed /internal/events route++ * with a stand-in Network. Alice has a watchlist, an alert rule that fired, a draft, a context revision she wrote, an++ * observation she recorded by hand and a review she approved; Bob has a watchlist of his own. The export carries only++ * hers; the deletion removes what is hers alone, leaves her contributions without her id, keeps her review, and++ * confirms once.++ */++const assert = require('assert');++const http = require('http');++const { boot, check, done } = require('./helpers/boot');++const { createNetworkSender } = require('openvibe-sdk/account-data');++const { signDeliveryHeaders } = require('openvibe-sdk/events');++++const MIN = 60e3;++const EXP = 'exp_01JZ0000000000000000000EXP';++const DEL = 'del_01JZ0000000000000000000DEX';++++async function startNetworkStub() {++ const calls = [];++ const server = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', () => {++ const json = (status, obj) => { res.writeHead(status, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(obj)); };++ if (req.url === '/oauth/token') return json(200, { access_token: 'tok_trade', token_type: 'Bearer', expires_in: 300 });++ calls.push({ url: req.url, auth: req.headers.authorization, body: JSON.parse(Buffer.concat(chunks).toString() || 'null') });++ return json(201, {});++ });++ });++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ return { url: `http://127.0.0.1:${server.address().port}`, calls, close: () => new Promise((r) => server.close(r)) };++}++++const envelope = (id, type, payload) => ({ event: { event_id: id, event_type: type, source: 'network', version: 1, timestamp: new Date().toISOString(), actor: { type: 'service', id: 'network' }, subject: { type: 'account', id: payload.subject }, visibility: 'internal', payload }, seq: 1 });++++(async () => {++ const stub = await startNetworkStub();++ const t = await boot({ accountSend: createNetworkSender({ networkInternalUrl: stub.url, clientId: 'trade', clientSecret: 'shh' }) });++ const { alice, bob } = t;++ const db = t.ctx.store.db;++ const count = async (sql, args) => Number(await db.value(sql, args));++ const post = (body, secret = 'hook-secret') => t.get('/internal/events', { body, headers: { 'content-type': 'application/json', ...signDeliveryHeaders(body, secret) } });++ const acme = await t.instrument({ symbol: 'ACME', name: 'Acme Corp', cik: '1234567' });++++ try {++ await check('alice\'s rows are made through the app', async () => {++ for (const who of [alice, bob]) {++ const wl = await t.get('/api/v1/watchlists', { as: who, json: { name: `${who.username || 'list'} picks` } });++ assert.strictEqual(wl.status, 201, wl.text);++ assert.strictEqual((await t.get(`/api/v1/watchlists/${wl.json().watchlist.id}/items/ACME`, { method: 'PUT', as: who, json: { note: 'watch' } })).status, 200);++ }++ t.clock.advance(-10 * MIN);++ const rule = await t.get('/api/v1/alerts', { as: alice, json: { symbol: 'ACME', kind: 'threshold', metric: 'price.close', operator: 'above', threshold: '100', unit: 'USD', currency: 'USD' } });++ assert.strictEqual(rule.status, 201, rule.text);++ t.clock.set(t.T0 + 60 * MIN);++ await t.observe(acme, { metric: 'price.close', value: '101.00', source_ref: 'o1', observed_at: t.iso(t.T0 + MIN), retrieved_at: t.iso(t.T0 + MIN) });++ assert.strictEqual(await count('SELECT count(*) FROM trade_alert_deliveries WHERE owner_subject = $1', [alice.subject]), 1, 'the rule fired');++ await t.ctx.observations.record({ source_key: 'manual', unit: 'USD', currency: 'USD', metric: 'price.close', value: '99.00', source_ref: 'hand-1', observed_at: t.iso(t.T0 + 2 * MIN), retrieved_at: t.iso(t.T0 + 2 * MIN) }, acme, { recordedBy: alice.subject });++ await t.ctx.store.revisions.create({ entityId: acme.id, expectedRevision: 0, content: 'Acme makes anvils.', fields: {}, meta: { authorship: { mode: 'human', authors: [alice.subject, bob.subject] } }, author: alice.subject, message: 'first', allowUnchanged: true });++ await t.ctx.store.revisions.create({ entityId: acme.id, expectedRevision: 1, content: 'Acme makes anvils and rockets.', fields: {}, meta: { authorship: { mode: 'human', authors: [bob.subject, alice.subject] } }, author: bob.subject, message: 'second', allowUnchanged: true });++ await db.exec(`INSERT INTO trade_context_drafts (entity_id, owner, base_revision, content, created_at, updated_at) VALUES ($1, $2, 1, 'draft text', 1, 1)`, [acme.id, alice.subject]);++ await db.exec(`INSERT INTO trade_context_reviews (entity_id, revision, reviewer, decision, reviewed_at) VALUES ($1, 1, $2, 'approved', 1)`, [acme.id, alice.subject]);++ });++++ await check('an export part carries only alice\'s rows, with this service\'s token', async () => {++ const r = await post(JSON.stringify(envelope('evt_01JZ0000000000000000000E01', 'network.account.export_requested', { export_id: EXP, subject: alice.subject })));++ assert.strictEqual(r.status, 200, r.text);++ const part = stub.calls.find((c) => c.url === `/internal/account-exports/${EXP}/parts`);++ assert.strictEqual(part.auth, 'Bearer tok_trade');++ const files = Object.fromEntries(part.body.files.map((f) => [f.name, f.content]));++ assert.deepStrictEqual(Object.keys(files).sort(), ['alert-deliveries.json', 'alert-rules.json', 'context-drafts.json', 'context-reviews.json', 'context-revisions.json', 'observations.json', 'watchlist-items.json', 'watchlists.json']);++ assert.strictEqual(files['watchlists.json'].length, 1);++ assert.strictEqual(files['watchlist-items.json'].length, 1);++ assert.deepStrictEqual(files['observations.json'].map((o) => o.value), ['99.00'], 'only what she recorded by hand');++ assert.ok(!JSON.stringify(part.body).includes(bob.subject), 'nobody else\'s id');++ });++++ await check('a deletion removes what is hers, leaves her contributions without her id, keeps her review, and confirms once', async () => {++ const body = JSON.stringify(envelope('evt_01JZ0000000000000000000D01', 'network.account.deleted', { deletion_id: DEL, subject: alice.subject }));++ assert.strictEqual((await post(body)).status, 200);++ const a = [alice.subject];++ assert.strictEqual(await count('SELECT count(*) FROM trade_watchlists WHERE owner_subject = $1', a), 0);++ assert.strictEqual(await count('SELECT count(*) FROM trade_watchlists WHERE owner_subject = $1', [bob.subject]), 1, 'bob\'s list stays');++ assert.strictEqual(await count('SELECT count(*) FROM trade_watchlist_items'), 1, 'her items went with her list');++ assert.strictEqual(await count('SELECT count(*) FROM trade_alert_rules WHERE owner_subject = $1', a), 0);++ assert.strictEqual(await count('SELECT count(*) FROM trade_alert_deliveries WHERE owner_subject = $1', a), 0);++ assert.strictEqual(await count('SELECT count(*) FROM trade_context_drafts WHERE owner = $1', a), 0);++ assert.strictEqual(await count('SELECT count(*) FROM trade_context_revisions WHERE author IS NULL'), 1, 'her revision stays, authorless');++ const revs = await db.many('SELECT number, author, meta, content FROM trade_context_revisions WHERE entity_id = $1 ORDER BY number', [acme.id]);++ const authorsOf = (r) => (typeof r.meta === 'string' ? JSON.parse(r.meta) : r.meta).authorship.authors;++ assert.deepStrictEqual(revs.map((r) => [r.number, r.author, authorsOf(r)]), [[1, null, [bob.subject]], [2, bob.subject, [bob.subject]]], 'her id leaves both, bob\'s stays');++ assert.deepStrictEqual(revs.map((r) => r.content), ['Acme makes anvils.', 'Acme makes anvils and rockets.'], 'the text never changes');++ assert.strictEqual(await count("SELECT count(*) FROM trade_market_observations WHERE recorded_by = 'deleted'"), 1);++ assert.strictEqual(await count('SELECT count(*) FROM trade_market_observations WHERE recorded_by = $1', a), 0);++ assert.strictEqual(await count('SELECT count(*) FROM trade_context_reviews WHERE reviewer = $1', a), 1, 'her approval stays');++ const conf = stub.calls.filter((c) => c.url === `/internal/account-deletions/${DEL}/confirmations`);++ assert.strictEqual(conf.length, 1);++ assert.deepStrictEqual([conf[0].body.erased.trade_watchlists, conf[0].body.erased.trade_alert_rules, conf[0].body.erased.trade_alert_deliveries], [1, 1, 1]);++ assert.strictEqual(conf[0].body.retained.trade_context_reviews, 1);++ const again = await post(body);++ assert.strictEqual(again.json().duplicate, true);++ assert.strictEqual(stub.calls.filter((c) => c.url.includes('/confirmations')).length, 1, 'confirmed once');++ });++++ await check('the rows stay append-only outside the erasure, and inside it nothing but the person\'s id may change', async () => {++ await assert.rejects(db.exec("UPDATE trade_context_revisions SET author = NULL WHERE number = 2"), /immutable/);++ await assert.rejects(db.exec("UPDATE trade_market_observations SET recorded_by = 'deleted' WHERE source_ref = 'o1'"), /immutable/);++ await assert.rejects(db.tx(async (tx) => {++ await tx.value("SELECT set_config('trade.account_erasure', 'on', true)");++ await tx.exec("UPDATE trade_context_revisions SET content = 'rewritten', author = NULL WHERE number = 2");++ }), /immutable/);++ await assert.rejects(db.tx(async (tx) => {++ await tx.value("SELECT set_config('trade.account_erasure', 'on', true)");++ await tx.exec("UPDATE trade_market_observations SET recorded_by = 'deleted', value = '1.00' WHERE source_ref = 'o1'");++ }), /immutable/);++ });++++ await check('the route refuses a bad signature', async () => {++ const body = JSON.stringify(envelope('evt_01JZ0000000000000000000E02', 'network.account.export_requested', { export_id: EXP, subject: alice.subject }));++ assert.strictEqual((await post(body, 'not-the-secret')).status, 401);++ });++ } finally {++ await t.close();++ await stub.close();++ }++ done();++})();@@@ -49,7 +49,7 @@ async function boot(opts = {}) {let built = null;async function start() {const config = configLib.load(env);-- built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow, indexnow: opts.indexnow });++ built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow, indexnow: opts.indexnow, accountSend: opts.accountSend });await built.ctx.jwks.keys();server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });t.base = `http://127.0.0.1:${server.address().port}`;
@@ -2,5 +2,11 @@
@@ -2,5 +2,11 @@
## Unreleased
## Unreleased
+- Account export and deletion (ADR-033) through `server/domain/account-data.js` over `openvibe-sdk/account-data` (SDK v0.37.0, was v0.35.0):
+ - **Deleted:** watchlists, alert rules (with their deliveries) and drafts.
+ - **Authorless:** contributed instruments, aliases, purges, context revisions and hand-recorded observations.
+ - **Kept:** reviews.
+ - **Migrations:** `0003_account_data.sql` adds `account_data_events`. `0004_account_erasure.sql` lets only the erasure transaction take a person's id out of the append-only revisions and observations.
+ - The two subscriptions are created at boot. `test/account-data.test.js`.
- openvibe-contracts moves to v0.97.0 (pin, lockfile and `node_modules`). All twelve `trade.*` ids are now defined by the release, so `server/auth/capabilities.js` drops its local fallback for proposed ids and sends every check through the library's grant rule; `test/capabilities.test.js` pins that every guarded id is defined and that exact, prefix, denied and unknown ids answer as `capabilities.check()` does. `server/index.js` moves its shutdown onto `openvibe-sdk/service`'s `gracefulStop` (drain with `Connection: close`, then the worker, the outbox and the store; SIGTERM/SIGINT replace the hand-rolled handlers and timer), pinned by the spawn-SIGTERM check in `test/service-kit.test.js`. README and STATUS.json name v0.97.0 and the pinned v0.26.0 SDK.
- openvibe-contracts moves to v0.97.0 (pin, lockfile and `node_modules`). All twelve `trade.*` ids are now defined by the release, so `server/auth/capabilities.js` drops its local fallback for proposed ids and sends every check through the library's grant rule; `test/capabilities.test.js` pins that every guarded id is defined and that exact, prefix, denied and unknown ids answer as `capabilities.check()` does. `server/index.js` moves its shutdown onto `openvibe-sdk/service`'s `gracefulStop` (drain with `Connection: close`, then the worker, the outbox and the store; SIGTERM/SIGINT replace the hand-rolled handlers and timer), pinned by the spawn-SIGTERM check in `test/service-kit.test.js`. README and STATUS.json name v0.97.0 and the pinned v0.26.0 SDK.
- Every page is rendered through `openvibe-publishing/layout` (v1.2.0, on `openvibe-shared/shell` v2.6.0): the head, the Frame, the noscript navigation, the footer and its init come from the shared document; Trade keeps its site stylesheet, the disclaimers, the account bar and the "Recently shipped" line. Pins `openvibe-shared` v2.6.0 and `openvibe-publishing` v1.2.0.
- Every page is rendered through `openvibe-publishing/layout` (v1.2.0, on `openvibe-shared/shell` v2.6.0): the head, the Frame, the noscript navigation, the footer and its init come from the shared document; Trade keeps its site stylesheet, the disclaimers, the account bar and the "Recently shipped" line. Pins `openvibe-shared` v2.6.0 and `openvibe-publishing` v1.2.0.
@@ -128,7 +128,7 @@ deployment has no instruments (editors add them), no documents until the source
@@ -128,7 +128,7 @@ deployment has no instruments (editors add them), no documents until the source
| `/terms`, `/privacy`, `/dmca` | openvibe-shared legal pages |
| `/terms`, `/privacy`, `/dmca` | openvibe-shared legal pages |
| `/auth/*` | sign-in; the same session layer as Blog and Community |
| `/auth/*` | sign-in; the same session layer as Blog and Community |
| `/api/health`, `/api/ready`, `/release.json`, `POST /release-metrics`, `/metrics` | health, readiness, release, open tabs' update reports (counted in `/metrics`), metrics (loopback only) |
| `/api/health`, `/api/ready`, `/release.json`, `POST /release-metrics`, `/metrics` | health, readiness, release, open tabs' update reports (counted in `/metrics`), metrics (loopback only) |
-| `POST /internal/events` | signed OpenVibe.Events deliveries (`sources.*`), host-local |
+| `POST /internal/events` | signed OpenVibe.Events deliveries (`sources.*`, and the two ADR-033 account events), host-local |
### Discoverability (roadmap §32)
### Discoverability (roadmap §32)
@@ -195,6 +195,22 @@ decided with the contracts library's matching rule (`server/auth/capabilities.js
@@ -195,6 +195,22 @@ decided with the contracts library's matching rule (`server/auth/capabilities.js
Consumed: `sources.item.created|updated|removed` and `sources.fetch.failed` through the signed
Consumed: `sources.item.created|updated|removed` and `sources.fetch.failed` through the signed
webhook, as a wake-up for the cursor sync (the webhook is not durable truth; the sync is).
webhook, as a wake-up for the cursor sync (the webhook is not durable truth; the sync is).
+### Account export and deletion (ADR-033)
+
+`network.account.export_requested` and `network.account.deleted` arrive at `POST /internal/events` and are answered by
+`server/domain/account-data.js` (`openvibe-sdk/account-data`).
+
+- **Export:** the person's watchlists and their items, alert rules and deliveries, context drafts, the revisions they
+ wrote, the observations they recorded by hand, and their reviews.
+- **Deleted:** watchlists (items cascade), alert rules with their deliveries, and drafts.
+- **Made authorless:** instruments, aliases and purges they made. Context revisions and observations stay append-only:
+ only inside the erasure transaction (`trade.account_erasure`, migration `0004_account_erasure.sql`) may a revision's
+ author become NULL and their id leave `meta.authorship.authors`, or an observation's `recorded_by` become `deleted`.
+ The text, the values and their provenance never change.
+- **Kept, counted as retained:** reviews, since a person's approval is what lets reviewed text stay published.
+- **Subscriptions:** the two are created at boot when missing (needs `EVENTS_URL` and `TRADE_EVENTS_WEBHOOK_SECRET`).
+- **Receipts:** `account_data_events` keeps one per export and deletion.
+
### OpenVibe.AI seam (`trade.summarize_market_context`)
### OpenVibe.AI seam (`trade.summarize_market_context`)
1. The AI (or its caller) reads `GET /api/v1/instruments/:symbol/context/input` — the workflow input
1. The AI (or its caller) reads `GET /api/v1/instruments/:symbol/context/input` — the workflow input
@@ -242,7 +258,8 @@ services need to call Trade, is under [Grants the Network must hold](#grants-the
@@ -242,7 +258,8 @@ services need to call Trade, is under [Grants the Network must hold](#grants-the
Each grant is `[client, capability, audience]`:
Each grant is `[client, capability, audience]`:
- `[trade, events.event.publish, openvibe.events]`
- `[trade, events.event.publish, openvibe.events]`
-- `[trade, events.subscription.manage, openvibe.events]`
+- `[trade, events.subscription.manage, openvibe.events]` (also the account subscriptions at boot)
+- `[trade, network.account.export.contribute, openvibe.network]` and `[trade, network.account.deletion.confirm, openvibe.network]` (ADR-033; granted last, once this release is live)
- `[trade, sources.item.read, openvibe.sources]`
- `[trade, sources.item.read, openvibe.sources]`
- `[trade, sources.source.read, openvibe.sources]`
- `[trade, sources.source.read, openvibe.sources]`
- For OpenVibe.AI to deliver drafts: `[ai, trade.context.read, openvibe.trade]` and
- For OpenVibe.AI to deliver drafts: `[ai, trade.context.read, openvibe.trade]` and
@@ -9,7 +9,7 @@
@@ -9,7 +9,7 @@
"domain": "openvibe.trade",
"domain": "openvibe.trade",
"scope": "informational only (ADR-025): no custody, no order execution, no escrow, no listings between users, no personalised advice",
"scope": "informational only (ADR-025): no custody, no order execution, no escrow, no listings between users, no personalised advice",
"contracts": "openvibe-contracts v0.112.0",
"contracts": "openvibe-contracts v0.112.0",
- "sdk": "openvibe-sdk v0.35.0",
+ "sdk": "openvibe-sdk v0.37.0",
"shared": "openvibe-shared v2.15.0",
"shared": "openvibe-shared v2.15.0",
"packages": [
"packages": [
"openvibe-contracts v0.97.0",
"openvibe-contracts v0.97.0",
new file mode 100644
new file mode 100644
@@ -0,0 +1,13 @@
@@ -0,0 +1,13 @@
+-- phase: expand
+-- OpenVibe.Trade: the receipts of the ADR-033 account export and deletion deliveries this service applied
+-- (openvibe-sdk/account-data's ACCOUNT_DATA_SCHEMA), so a redelivered export or deletion changes nothing.
+-- The row holds an export_id or deletion_id, the subject and the counts sent to Network; it is not a secret.
+
+CREATE TABLE IF NOT EXISTS account_data_events (
+ id TEXT PRIMARY KEY,
+ kind TEXT NOT NULL,
+ subject TEXT NOT NULL,
+ outcome JSONB,
+ applied_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ sent_at TIMESTAMPTZ
+);
new file mode 100644
new file mode 100644
@@ -0,0 +1,45 @@
@@ -0,0 +1,45 @@
+-- phase: expand
+-- ADR-033 account deletion (server/domain/account-data.js): context revisions and market observations stay
+-- append-only, with one exception. Inside the account-erasure transaction, and only there (it sets
+-- trade.account_erasure = 'on' with set_config(..., true), so the setting ends with the transaction), the person's
+-- id may leave a row: a revision's author becomes NULL and their id leaves meta.authorship.authors (also on a
+-- revision someone else wrote with them), and an
+-- observation's recorded_by becomes 'deleted'. Every other column must stay as it was, so the text, the values and
+-- their provenance never change. Deletes keep their old rules.
+
+CREATE OR REPLACE FUNCTION trade_context_revisions_guard() RETURNS trigger LANGUAGE plpgsql AS $$
+BEGIN
+ IF TG_OP = 'UPDATE' THEN
+ IF current_setting('trade.account_erasure', true) = 'on'
+ AND (NEW.author IS NULL OR NEW.author IS NOT DISTINCT FROM OLD.author)
+ AND (NEW.id, NEW.entity_id, NEW.number, NEW.parent_id, NEW.parent_number, NEW.kind, NEW.reverted_to, NEW.content,
+ NEW.fields, NEW.content_hash, NEW.message, NEW.created_at)
+ IS NOT DISTINCT FROM
+ (OLD.id, OLD.entity_id, OLD.number, OLD.parent_id, OLD.parent_number, OLD.kind, OLD.reverted_to, OLD.content,
+ OLD.fields, OLD.content_hash, OLD.message, OLD.created_at)
+ AND (NEW.meta - 'authorship') IS NOT DISTINCT FROM (OLD.meta - 'authorship')
+ AND ((NEW.meta->'authorship') - 'authors') IS NOT DISTINCT FROM ((OLD.meta->'authorship') - 'authors') THEN
+ RETURN NEW;
+ END IF;
+ RAISE EXCEPTION 'trade_context_revisions rows are immutable' USING ERRCODE = 'restrict_violation';
+ END IF;
+ IF NOT EXISTS (SELECT 1 FROM trade_context_revision_purges WHERE entity_id = OLD.entity_id) THEN
+ RAISE EXCEPTION 'trade_context_revisions rows are never deleted outside a recorded purge' USING ERRCODE = 'restrict_violation';
+ END IF;
+ RETURN OLD;
+END
+$$;
+
+CREATE OR REPLACE FUNCTION trade_obs_no_update() RETURNS trigger LANGUAGE plpgsql AS $$
+BEGIN
+ IF current_setting('trade.account_erasure', true) = 'on' AND NEW.recorded_by = 'deleted'
+ AND (NEW.id, NEW.instrument_id, NEW.metric, NEW.value, NEW.value_num, NEW.unit, NEW.currency, NEW.period, NEW.observed_at,
+ NEW.source_key, NEW.source_item_id, NEW.source_url, NEW.source_ref, NEW.retrieved_at, NEW.max_age_sec, NEW.recorded_at)
+ IS NOT DISTINCT FROM
+ (OLD.id, OLD.instrument_id, OLD.metric, OLD.value, OLD.value_num, OLD.unit, OLD.currency, OLD.period, OLD.observed_at,
+ OLD.source_key, OLD.source_item_id, OLD.source_url, OLD.source_ref, OLD.retrieved_at, OLD.max_age_sec, OLD.recorded_at) THEN
+ RETURN NEW;
+ END IF;
+ RAISE EXCEPTION 'trade_market_observations rows are immutable';
+END
+$$;
@@ -17,7 +17,7 @@
@@ -17,7 +17,7 @@
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.112.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.112.0",
"openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.3.0",
"openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.3.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.35.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.37.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.15.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.15.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
},
},
@@ -907,9 +907,9 @@
@@ -907,9 +907,9 @@
}
}
},
},
"node_modules/openvibe-sdk": {
"node_modules/openvibe-sdk": {
- "version": "0.35.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.35.0",
- "integrity": "sha512-IF7XB97tU7Q+ccwssGx6QXXcfDPcVpl+mX3BjIr7C6NJlcmoqj9DbHiKrBxlVav0eG44Jf8zuXEwokrHDq3J0w==",
+ "version": "0.37.0",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.37.0",
+ "integrity": "sha512-afojId2KzOL4ZZrpwIeIAYUaX24ZOC3Nt+bdd1HSXm7aislt8WQNS62i8/d8W3udLr6rmSG29SjmnaRHUGfd4Q==",
"license": "MIT",
"license": "MIT",
"engines": {
"engines": {
"node": ">=20"
"node": ">=20"
@@ -29,7 +29,7 @@
@@ -29,7 +29,7 @@
"iovalkey": "^0.4.0",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.112.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.112.0",
"openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.3.0",
"openvibe-publishing": "https://codeload.github.com/OpenVibers/OpenVibe.Publishing/tar.gz/refs/tags/v1.3.0",
- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.35.0",
+ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.37.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.15.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v2.15.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
},
},
@@ -33,6 +33,8 @@ const configLib = require('./config');
@@ -33,6 +33,8 @@ const configLib = require('./config');
const { openStore } = require('./db');
const { openStore } = require('./db');
const { createViewerResolver } = require('./auth/viewer');
const { createViewerResolver } = require('./auth/viewer');
const { createWebhook } = require('./events/consumer');
const { createWebhook } = require('./events/consumer');
+const accountDataLib = require('./domain/account-data');
+const { createNetworkSender } = require('openvibe-sdk/account-data');
const { createUrls } = require('./domain/urls');
const { createUrls } = require('./domain/urls');
const { createInstruments } = require('./domain/instruments');
const { createInstruments } = require('./domain/instruments');
const { createFreshness } = require('./domain/source-status');
const { createFreshness } = require('./domain/source-status');
@@ -164,7 +166,12 @@ async function createApp(opts = {}) {
@@ -164,7 +166,12 @@ async function createApp(opts = {}) {
app.use(machine);
app.use(machine);
// ── Signed OpenVibe.Events deliveries (before any body parser) ──
// ── Signed OpenVibe.Events deliveries (before any body parser) ──
- app.use(createWebhook({ store, config, ingest: ctx.sync, log }).router);
+ // Account export and deletion (ADR-033, domain/account-data.js), pushed to Network with this service's own token.
+ ctx.accountData = accountDataLib.create({ db: store.db, log });
+ ctx.accountSend = opts.accountSend || (config.oauth.clientSecret
+ ? createNetworkSender({ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret, fetch: fetchImpl })
+ : null);
+ app.use(createWebhook({ store, config, ingest: ctx.sync, log, accountData: ctx.accountData, accountSend: ctx.accountSend }).router);
// ── Sign-in (OAuth2 client of OpenVibe.Network) ─────────
// ── Sign-in (OAuth2 client of OpenVibe.Network) ─────────
app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, standardHeaders: true, legacyHeaders: false }));
app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, standardHeaders: true, legacyHeaders: false }));
new file mode 100644
new file mode 100644
@@ -0,0 +1,73 @@
@@ -0,0 +1,73 @@
+'use strict';
+
+/**
+ * Account export and deletion → Trade (ADR-033; openvibe-sdk/account-data). What Trade holds about a person:
+ *
+ * their own, deleted whole watchlists (their items go with them, ON DELETE CASCADE), alert rules and the
+ * deliveries under them, and unsaved context drafts
+ * contributed, made authorless instruments and aliases they added, context revisions they wrote (the author
+ * column and their id in meta.authorship.authors), observations they recorded by hand
+ * (recorded_by is NOT NULL, so it becomes 'deleted'), and purges they ran
+ * kept context reviews: an approval by a person is what lets reviewed text stay published
+ * (openvibe-publishing/authorship), so a reviewer's record stays (counted as retained)
+ *
+ * No secret or other person's data is held in any of these tables.
+ */
+const { createAccountData, TOPICS } = require('openvibe-sdk/account-data');
+
+const anonymize = { anonymize: {} };
+
+const TABLES = [
+ { table: 'trade_watchlists', subject: 'owner_subject', file: 'watchlists.json', columns: ['id', 'name', 'created_at', 'updated_at'] },
+ // Deliveries reference their rule, so they go first.
+ { table: 'trade_alert_deliveries', subject: 'owner_subject', file: 'alert-deliveries.json', columns: ['id', 'rule_id', 'trigger_kind', 'trigger_id', 'summary', 'created_at'] },
+ { table: 'trade_alert_rules', subject: 'owner_subject', file: 'alert-rules.json', columns: ['id', 'instrument_id', 'kind', 'metric', 'operator', 'threshold', 'unit', 'currency', 'form_types', 'status', 'created_at', 'updated_at'] },
+ { table: 'trade_context_drafts', subject: 'owner', file: 'context-drafts.json', columns: ['entity_id', 'base_revision', 'content', 'fields', 'created_at', 'updated_at'] },
+ { table: 'trade_instruments', subject: 'created_by', file: 'instruments.json', columns: ['id', 'symbol', 'name', 'kind', 'exchange', 'currency', 'created_at'], erase: anonymize },
+ { table: 'trade_instrument_aliases', subject: 'added_by', file: null, erase: anonymize },
+ { table: 'trade_context_revision_purges', subject: 'purged_by', file: null, erase: anonymize },
+ { table: 'trade_context_reviews', subject: 'reviewer', file: 'context-reviews.json', columns: ['entity_id', 'revision', 'decision', 'note', 'reviewed_at'], order: 'reviewed_at', erase: { keep: 'a person\'s approval is what lets reviewed text stay published' } },
+];
+
+/**
+ * The watchlists' items (keyed by the list, not the person), the context revisions they wrote and the observations
+ * they recorded by hand. Revisions and observations are append-only (database triggers), so they are erased in
+ * extraErase under the trade.account_erasure setting (migrations/0004_account_erasure.sql), not by the table map.
+ */
+async function extraExport(db, subject) {
+ const files = [];
+ const revisions = await db.many(`SELECT entity_id, number, kind, content, fields, message, created_at FROM trade_context_revisions
+ WHERE author = $1 ORDER BY created_at DESC LIMIT 5000`, [subject]);
+ if (revisions.length) files.push({ name: 'context-revisions.json', content: revisions });
+ const items = await db.many(`SELECT i.watchlist_id, i.instrument_id, i.note, i.added_at FROM trade_watchlist_items i
+ JOIN trade_watchlists w ON w.id = i.watchlist_id WHERE w.owner_subject = $1 ORDER BY i.added_at DESC LIMIT 5000`, [subject]);
+ if (items.length) files.push({ name: 'watchlist-items.json', content: items });
+ const obs = await db.many(`SELECT id, instrument_id, metric, value, unit, currency, period, observed_at, source_key, source_url, recorded_at
+ FROM trade_market_observations WHERE recorded_by = $1 ORDER BY recorded_at DESC LIMIT 5000`, [subject]);
+ if (obs.length) files.push({ name: 'observations.json', content: obs });
+ return files;
+}
+
+async function extraErase(t, subjects, counts) {
+ // Only this transaction may take a person's id out of the append-only rows; the setting ends with it.
+ await t.value("SELECT set_config('trade.account_erasure', 'on', true)");
+ counts.add(counts.retained, 'tombstones', await t.exec(`UPDATE trade_context_revisions SET author = NULL
+ WHERE author = ANY($1::text[])`, [subjects]));
+ counts.add(counts.retained, 'tombstones', await t.exec(`UPDATE trade_market_observations SET recorded_by = 'deleted'
+ WHERE recorded_by = ANY($1::text[])`, [subjects]));
+ counts.add(counts.retained, 'tombstones', await t.exec(`UPDATE trade_context_revisions
+ SET meta = jsonb_set(meta, '{authorship,authors}', COALESCE((SELECT jsonb_agg(a) FROM jsonb_array_elements(meta->'authorship'->'authors') a
+ WHERE NOT ((a #>> '{}') = ANY($1::text[]))), '[]'::jsonb))
+ WHERE jsonb_typeof(meta->'authorship'->'authors') = 'array' AND (meta->'authorship'->'authors') ?| $1::text[]`, [subjects]));
+ await t.value("SELECT set_config('trade.account_erasure', 'off', true)");
+}
+
+/** The account-data handle for Trade's database (server/db.js, store.db). */
+function create({ db, log = console } = {}) {
+ return createAccountData({
+ db, service: 'trade', tables: TABLES, extraExport, extraErase, log,
+ note: 'Instruments, context text and observations you contributed stay without your name; reviews you approved stay attributed.',
+ });
+}
+
+module.exports = { create, TABLES, TOPICS };
@@ -1,7 +1,8 @@
@@ -1,7 +1,8 @@
'use strict';
'use strict';
/**
/**
- * POST /internal/events — signed OpenVibe.Events webhook deliveries (subscription topic sources.*).
+ * POST /internal/events — signed OpenVibe.Events webhook deliveries (subscription topic sources.*, and
+ * network.account.export_requested / network.account.deleted for ADR-033, answered by domain/account-data.js).
*
*
* The signature check, the ±300 s window and the exactly-once inbox are the ingest chassis
* The signature check, the ±300 s window and the exactly-once inbox are the ingest chassis
* (openvibe-publishing/ingest, createEventConsumer); this route only maps its result onto HTTP and
* (openvibe-publishing/ingest, createEventConsumer); this route only maps its result onto HTTP and
@@ -14,8 +15,13 @@ const { createEventConsumer } = require('openvibe-publishing/ingest');
@@ -14,8 +15,13 @@ const { createEventConsumer } = require('openvibe-publishing/ingest');
const { http } = require('openvibe-contracts');
const { http } = require('openvibe-contracts');
const WAKE_RE = /^sources\.(item\.(created|updated|removed)|fetch\.failed)$/;
const WAKE_RE = /^sources\.(item\.(created|updated|removed)|fetch\.failed)$/;
+const { TOPICS: ACCOUNT_TOPICS } = require('openvibe-sdk/account-data');
-function createWebhook({ store, config, ingest, log = console }) {
+/**
+ * accountData + accountSend: account export and deletion and its sender to Network. A failure throws, the inbox receipt
+ * rolls back and Events redelivers; account-data keeps its own receipt per export and deletion id.
+ */
+function createWebhook({ store, config, ingest, log = console, accountData = null, accountSend = null }) {
const router = express.Router();
const router = express.Router();
const consumer = createEventConsumer({ db: store.db, secrets: config.events.webhookSecret, consumer: 'sources', now: store.now });
const consumer = createEventConsumer({ db: store.db, secrets: config.events.webhookSecret, consumer: 'sources', now: store.now });
@@ -25,6 +31,11 @@ function createWebhook({ store, config, ingest, log = console }) {
@@ -25,6 +31,11 @@ function createWebhook({ store, config, ingest, log = console }) {
let r;
let r;
try {
try {
r = await consumer.apply(raw, req.headers, async (e) => {
r = await consumer.apply(raw, req.headers, async (e) => {
+ if (ACCOUNT_TOPICS.includes(e.event_type)) {
+ if (!accountData || !accountSend) throw new Error('account export and deletion are not configured');
+ await accountData.apply(e, { send: accountSend });
+ return false;
+ }
const wake = WAKE_RE.test(e.event_type);
const wake = WAKE_RE.test(e.event_type);
if (wake) ingest.run().catch((err) => log.warn('[Trade] ingest after webhook failed:', err.message));
if (wake) ingest.run().catch((err) => log.warn('[Trade] ingest after webhook failed:', err.message));
return wake;
return wake;
@@ -8,6 +8,7 @@
@@ -8,6 +8,7 @@
* store, and exits 0 (the 5 s family's deadline exit code).
* store, and exits 0 (the 5 s family's deadline exit code).
*/
*/
const { gracefulStop } = require('openvibe-sdk/service');
const { gracefulStop } = require('openvibe-sdk/service');
+const { startSubscriptions } = require('openvibe-sdk/account-data');
const { createApp } = require('./app');
const { createApp } = require('./app');
(async () => {
(async () => {
@@ -21,10 +22,16 @@ const server = app.listen(config.port, config.host, () => {
@@ -21,10 +22,16 @@ const server = app.listen(config.port, config.host, () => {
server.keepAliveTimeout = 65_000;
server.keepAliveTimeout = 65_000;
ctx.outbox.start();
ctx.outbox.start();
ctx.worker.start();
ctx.worker.start();
+// The two account subscriptions at OpenVibe.Events (ADR-033), created when missing; off without EVENTS_URL,
+// TRADE_EVENTS_WEBHOOK_SECRET or the client secret. The sources.* one stays scripts/subscribe.js's.
+const subscriptions = startSubscriptions({
+ eventsUrl: config.events.url, endpoint: `http://127.0.0.1:${config.port}/internal/events`, secret: String(config.events.webhookSecret || '').split(',')[0].trim(),
+ networkInternalUrl: config.networkInternalUrl, clientId: config.oauth.clientId, clientSecret: config.oauth.clientSecret,
+});
gracefulStop({
gracefulStop({
name: 'Trade', server, deadlineExitCode: 0,
name: 'Trade', server, deadlineExitCode: 0,
- stop: [() => ctx.worker.stop()],
+ stop: [() => ctx.worker.stop(), () => { if (subscriptions) subscriptions.stop(); }],
close: [() => ctx.outbox.stop(), () => ctx.store.close()],
close: [() => ctx.outbox.stop(), () => ctx.store.close()],
});
});
})().catch((err) => { console.error('[Trade] failed to start:', err); process.exit(1); });
})().catch((err) => { console.error('[Trade] failed to start:', err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,128 @@
@@ -0,0 +1,128 @@
+'use strict';
+/**
+ * ADR-033: Trade's part of an account export and of an account deletion, through the signed /internal/events route
+ * with a stand-in Network. Alice has a watchlist, an alert rule that fired, a draft, a context revision she wrote, an
+ * observation she recorded by hand and a review she approved; Bob has a watchlist of his own. The export carries only
+ * hers; the deletion removes what is hers alone, leaves her contributions without her id, keeps her review, and
+ * confirms once.
+ */
+const assert = require('assert');
+const http = require('http');
+const { boot, check, done } = require('./helpers/boot');
+const { createNetworkSender } = require('openvibe-sdk/account-data');
+const { signDeliveryHeaders } = require('openvibe-sdk/events');
+
+const MIN = 60e3;
+const EXP = 'exp_01JZ0000000000000000000EXP';
+const DEL = 'del_01JZ0000000000000000000DEX';
+
+async function startNetworkStub() {
+ const calls = [];
+ const server = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', () => {
+ const json = (status, obj) => { res.writeHead(status, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(obj)); };
+ if (req.url === '/oauth/token') return json(200, { access_token: 'tok_trade', token_type: 'Bearer', expires_in: 300 });
+ calls.push({ url: req.url, auth: req.headers.authorization, body: JSON.parse(Buffer.concat(chunks).toString() || 'null') });
+ return json(201, {});
+ });
+ });
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ return { url: `http://127.0.0.1:${server.address().port}`, calls, close: () => new Promise((r) => server.close(r)) };
+}
+
+const envelope = (id, type, payload) => ({ event: { event_id: id, event_type: type, source: 'network', version: 1, timestamp: new Date().toISOString(), actor: { type: 'service', id: 'network' }, subject: { type: 'account', id: payload.subject }, visibility: 'internal', payload }, seq: 1 });
+
+(async () => {
+ const stub = await startNetworkStub();
+ const t = await boot({ accountSend: createNetworkSender({ networkInternalUrl: stub.url, clientId: 'trade', clientSecret: 'shh' }) });
+ const { alice, bob } = t;
+ const db = t.ctx.store.db;
+ const count = async (sql, args) => Number(await db.value(sql, args));
+ const post = (body, secret = 'hook-secret') => t.get('/internal/events', { body, headers: { 'content-type': 'application/json', ...signDeliveryHeaders(body, secret) } });
+ const acme = await t.instrument({ symbol: 'ACME', name: 'Acme Corp', cik: '1234567' });
+
+ try {
+ await check('alice\'s rows are made through the app', async () => {
+ for (const who of [alice, bob]) {
+ const wl = await t.get('/api/v1/watchlists', { as: who, json: { name: `${who.username || 'list'} picks` } });
+ assert.strictEqual(wl.status, 201, wl.text);
+ assert.strictEqual((await t.get(`/api/v1/watchlists/${wl.json().watchlist.id}/items/ACME`, { method: 'PUT', as: who, json: { note: 'watch' } })).status, 200);
+ }
+ t.clock.advance(-10 * MIN);
+ const rule = await t.get('/api/v1/alerts', { as: alice, json: { symbol: 'ACME', kind: 'threshold', metric: 'price.close', operator: 'above', threshold: '100', unit: 'USD', currency: 'USD' } });
+ assert.strictEqual(rule.status, 201, rule.text);
+ t.clock.set(t.T0 + 60 * MIN);
+ await t.observe(acme, { metric: 'price.close', value: '101.00', source_ref: 'o1', observed_at: t.iso(t.T0 + MIN), retrieved_at: t.iso(t.T0 + MIN) });
+ assert.strictEqual(await count('SELECT count(*) FROM trade_alert_deliveries WHERE owner_subject = $1', [alice.subject]), 1, 'the rule fired');
+ await t.ctx.observations.record({ source_key: 'manual', unit: 'USD', currency: 'USD', metric: 'price.close', value: '99.00', source_ref: 'hand-1', observed_at: t.iso(t.T0 + 2 * MIN), retrieved_at: t.iso(t.T0 + 2 * MIN) }, acme, { recordedBy: alice.subject });
+ await t.ctx.store.revisions.create({ entityId: acme.id, expectedRevision: 0, content: 'Acme makes anvils.', fields: {}, meta: { authorship: { mode: 'human', authors: [alice.subject, bob.subject] } }, author: alice.subject, message: 'first', allowUnchanged: true });
+ await t.ctx.store.revisions.create({ entityId: acme.id, expectedRevision: 1, content: 'Acme makes anvils and rockets.', fields: {}, meta: { authorship: { mode: 'human', authors: [bob.subject, alice.subject] } }, author: bob.subject, message: 'second', allowUnchanged: true });
+ await db.exec(`INSERT INTO trade_context_drafts (entity_id, owner, base_revision, content, created_at, updated_at) VALUES ($1, $2, 1, 'draft text', 1, 1)`, [acme.id, alice.subject]);
+ await db.exec(`INSERT INTO trade_context_reviews (entity_id, revision, reviewer, decision, reviewed_at) VALUES ($1, 1, $2, 'approved', 1)`, [acme.id, alice.subject]);
+ });
+
+ await check('an export part carries only alice\'s rows, with this service\'s token', async () => {
+ const r = await post(JSON.stringify(envelope('evt_01JZ0000000000000000000E01', 'network.account.export_requested', { export_id: EXP, subject: alice.subject })));
+ assert.strictEqual(r.status, 200, r.text);
+ const part = stub.calls.find((c) => c.url === `/internal/account-exports/${EXP}/parts`);
+ assert.strictEqual(part.auth, 'Bearer tok_trade');
+ const files = Object.fromEntries(part.body.files.map((f) => [f.name, f.content]));
+ assert.deepStrictEqual(Object.keys(files).sort(), ['alert-deliveries.json', 'alert-rules.json', 'context-drafts.json', 'context-reviews.json', 'context-revisions.json', 'observations.json', 'watchlist-items.json', 'watchlists.json']);
+ assert.strictEqual(files['watchlists.json'].length, 1);
+ assert.strictEqual(files['watchlist-items.json'].length, 1);
+ assert.deepStrictEqual(files['observations.json'].map((o) => o.value), ['99.00'], 'only what she recorded by hand');
+ assert.ok(!JSON.stringify(part.body).includes(bob.subject), 'nobody else\'s id');
+ });
+
+ await check('a deletion removes what is hers, leaves her contributions without her id, keeps her review, and confirms once', async () => {
+ const body = JSON.stringify(envelope('evt_01JZ0000000000000000000D01', 'network.account.deleted', { deletion_id: DEL, subject: alice.subject }));
+ assert.strictEqual((await post(body)).status, 200);
+ const a = [alice.subject];
+ assert.strictEqual(await count('SELECT count(*) FROM trade_watchlists WHERE owner_subject = $1', a), 0);
+ assert.strictEqual(await count('SELECT count(*) FROM trade_watchlists WHERE owner_subject = $1', [bob.subject]), 1, 'bob\'s list stays');
+ assert.strictEqual(await count('SELECT count(*) FROM trade_watchlist_items'), 1, 'her items went with her list');
+ assert.strictEqual(await count('SELECT count(*) FROM trade_alert_rules WHERE owner_subject = $1', a), 0);
+ assert.strictEqual(await count('SELECT count(*) FROM trade_alert_deliveries WHERE owner_subject = $1', a), 0);
+ assert.strictEqual(await count('SELECT count(*) FROM trade_context_drafts WHERE owner = $1', a), 0);
+ assert.strictEqual(await count('SELECT count(*) FROM trade_context_revisions WHERE author IS NULL'), 1, 'her revision stays, authorless');
+ const revs = await db.many('SELECT number, author, meta, content FROM trade_context_revisions WHERE entity_id = $1 ORDER BY number', [acme.id]);
+ const authorsOf = (r) => (typeof r.meta === 'string' ? JSON.parse(r.meta) : r.meta).authorship.authors;
+ assert.deepStrictEqual(revs.map((r) => [r.number, r.author, authorsOf(r)]), [[1, null, [bob.subject]], [2, bob.subject, [bob.subject]]], 'her id leaves both, bob\'s stays');
+ assert.deepStrictEqual(revs.map((r) => r.content), ['Acme makes anvils.', 'Acme makes anvils and rockets.'], 'the text never changes');
+ assert.strictEqual(await count("SELECT count(*) FROM trade_market_observations WHERE recorded_by = 'deleted'"), 1);
+ assert.strictEqual(await count('SELECT count(*) FROM trade_market_observations WHERE recorded_by = $1', a), 0);
+ assert.strictEqual(await count('SELECT count(*) FROM trade_context_reviews WHERE reviewer = $1', a), 1, 'her approval stays');
+ const conf = stub.calls.filter((c) => c.url === `/internal/account-deletions/${DEL}/confirmations`);
+ assert.strictEqual(conf.length, 1);
+ assert.deepStrictEqual([conf[0].body.erased.trade_watchlists, conf[0].body.erased.trade_alert_rules, conf[0].body.erased.trade_alert_deliveries], [1, 1, 1]);
+ assert.strictEqual(conf[0].body.retained.trade_context_reviews, 1);
+ const again = await post(body);
+ assert.strictEqual(again.json().duplicate, true);
+ assert.strictEqual(stub.calls.filter((c) => c.url.includes('/confirmations')).length, 1, 'confirmed once');
+ });
+
+ await check('the rows stay append-only outside the erasure, and inside it nothing but the person\'s id may change', async () => {
+ await assert.rejects(db.exec("UPDATE trade_context_revisions SET author = NULL WHERE number = 2"), /immutable/);
+ await assert.rejects(db.exec("UPDATE trade_market_observations SET recorded_by = 'deleted' WHERE source_ref = 'o1'"), /immutable/);
+ await assert.rejects(db.tx(async (tx) => {
+ await tx.value("SELECT set_config('trade.account_erasure', 'on', true)");
+ await tx.exec("UPDATE trade_context_revisions SET content = 'rewritten', author = NULL WHERE number = 2");
+ }), /immutable/);
+ await assert.rejects(db.tx(async (tx) => {
+ await tx.value("SELECT set_config('trade.account_erasure', 'on', true)");
+ await tx.exec("UPDATE trade_market_observations SET recorded_by = 'deleted', value = '1.00' WHERE source_ref = 'o1'");
+ }), /immutable/);
+ });
+
+ await check('the route refuses a bad signature', async () => {
+ const body = JSON.stringify(envelope('evt_01JZ0000000000000000000E02', 'network.account.export_requested', { export_id: EXP, subject: alice.subject }));
+ assert.strictEqual((await post(body, 'not-the-secret')).status, 401);
+ });
+ } finally {
+ await t.close();
+ await stub.close();
+ }
+ done();
+})();
@@ -49,7 +49,7 @@ async function boot(opts = {}) {
@@ -49,7 +49,7 @@ async function boot(opts = {}) {
let built = null;
let built = null;
async function start() {
async function start() {
const config = configLib.load(env);
const config = configLib.load(env);
- built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow, indexnow: opts.indexnow });
+ built = await createApp({ config, store: createStore(testdb.db, { now: clock.now }), now: clock.now, log: opts.log || quiet, limitsNow: opts.limitsNow, indexnow: opts.indexnow, accountSend: opts.accountSend });
await built.ctx.jwks.keys();
await built.ctx.jwks.keys();
server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
t.base = `http://127.0.0.1:${server.address().port}`;
t.base = `http://127.0.0.1:${server.address().port}`;