openvibers/openvibe.connect Commits
Durable webhooks: 72 hours, 1,000 a share, sealed at rest, in order, with a delivery log (CON-15) (#43)
Changes
@@@ -4,6 +4,11 @@ What changed in OpenVibe Connect, newest first.## Unreleased — 2026-11-03++- Durable webhooks, hardened (CON-15): a share holds at most 1,000 waiting webhooks for 72 hours, every body is++ sealed at rest with a key derived from the session secret, and the app gets `x-openvibe-received-at` so it keeps the++ original arrival time. The owner gets a delivery log (page and `GET /api/v1/shares/:id/hooks/:hook/deliveries`).++ Events' retries now span the same 72 hours. Migration `0012_hook_deliveries`.++- MCP OAuth 2.1 (CON-16): a share's address works as a connector in Claude or any MCP client. The client is sent toConnect to sign the person in with OpenVibe: protected-resource metadata on the share, then registration, authorizewith PKCE S256 and consent, then tokens bound to that share and rotated on refresh. The owner sees and signs out each@@@ -42,25 +42,29 @@ instead" lets the computer go and shows a new key once (ADR-057 decision 4, CON-### Webhooks that wait for your computerOn a share's page, "Make a webhook address" gives you `https://connect.openvibe.website/h/hk_…` and a path on your app--(`/webhooks/github`). Give that address to GitHub, Stripe or anything else that sends webhooks. Each webhook is kept for--up to 7 days and handed to your app, at that path, as soon as your computer is connected, so a laptop that sleeps--misses nothing (CON-15, ADR-057 decision 7):----- The address takes POST, PUT and PATCH, up to 1 MB and 120 a minute, and answers `202` at once.--- Connect keeps the method, the provider's headers (`content-type`, `user-agent`, `x-…`, `stripe-signature`,-- `svix-…`, `webhook-…`; never cookies, authorization or forwarding headers) and the body. It publishes only a-- pointer, `connect.hook.received`, to OpenVibe.Events, in the same transaction.++(`/webhooks/github`). Give that address to GitHub, Stripe or anything else that sends webhooks. Each webhook is sealed,++kept for up to 72 hours and handed to your app, at that path, as soon as your computer is connected, so a laptop that++sleeps misses nothing (CON-15, ADR-057 decision 7):++++- The address takes POST, PUT and PATCH, up to 1 MB and 120 a minute, and answers `202` at once. A share holds at most++ 1,000 waiting webhooks; after that the address answers `429` and the provider retries.++- Connect keeps the method, the provider's headers minus hop-by-hop, credential and forwarding headers, and the body.++ The body is sealed at rest with a key derived from the service's session secret and is never readable from the++ database alone. It publishes only a pointer, `connect.hook.received`, to OpenVibe.Events, in the same transaction.- The moment the computer connects, Connect hands it every webhook that waited, oldest first. Events also delivers the-- pointer back to Connect's own subscription, retried for about seven days, which finds it settled. Connect hands the-- webhook to your app with `x-openvibe-hook` and `x-openvibe-event-id` (dedupe on it). Your app's answer below 500 settles it, and-- its body is dropped. While the computer is offline, or the app answers 5xx, Connect says 503 and Events tries again.++ pointer back to Connect's own subscription, retried for about 72 hours, which finds it settled. Connect hands the++ webhook to your app with `x-openvibe-hook`, `x-openvibe-event-id` (dedupe on it) and `x-openvibe-received-at` (when++ the provider's webhook first arrived). Your app's answer below 500 settles it, and its body is dropped. While the++ computer is offline, or the app answers 5xx, Connect says 503 and Events tries again.++- The share's page and the API's delivery log show what arrived, what the app answered and when (never a body).- Nobody signs in to send a webhook, so check the provider's own signature in your app.| Call | What ||---|---|--| `GET /api/v1/shares/:id/hooks` | its webhook addresses, with `received`, `delivered` and `waiting` |++| `GET /api/v1/shares/:id/hooks` | its webhook addresses, with `received`, `delivered`, `waiting` and the last deliveries || `POST /api/v1/shares/:id/hooks` `{ path, label? }` | a new address → `201 { hook }` (at most 10 a share) || `DELETE /api/v1/shares/:id/hooks/:hook` | remove it; webhooks still waiting are dropped |++| `GET /api/v1/shares/:id/hooks/:hook/deliveries` | the delivery log: method, path, status, error and when |## What your app sees@@@ -187,7 +191,8 @@ browser ──https──▶ Cloudflare ──▶ nginx ──▶ Connect (5240)no bodies; a restart forgets them.- `server/auth/node-auth.js` — a computer as OpenVibe Node: its node token, and the single-use tunnel tokens (CON-4).- `server/machines.js` — the owner's computers, read from Network with the owner's own token.--- `server/hooks.js`, `server/http/hooks.js` — webhook addresses, the kept webhooks, and Events' deliveries (CON-15).++- `server/hooks.js`, `server/http/hooks.js` — webhook addresses, the sealed kept webhooks, the 1,000-deep queue, the++delivery log and Events' deliveries (CON-15).- `public/ov-connect.js` — the client.The v0 transport is a WebSocket on 443 through Cloudflare (ADR-057's fallback path), so nothing needs a UDP port or a@@@ -3,7 +3,7 @@"version": "0.1.0","stage": "product","code": true,-- "plan": "Website Connect v0 (plan 2026-10-28 §5.9b, ADR-057 reviewed 2026-11-02): share one port on your computer at https://<name>--<user>.openvibe.website behind OpenVibe sign-in",++ "plan": "Website Connect v0 (plan 2026-10-28 \u00a75.9b, ADR-057 reviewed 2026-11-02): share one port on your computer at https://<name>--<user>.openvibe.website behind OpenVibe sign-in","runtime": "node 22, express 4, ws 8, PostgreSQL 18 (openvibe-sdk/db, async)","port": "5240","domain": "connect.openvibe.website",@@@ -15,11 +15,13 @@"openvibe-sdk v0.91.0","openvibe-shared v4.115.1"],-- "summary": "OpenVibe Connect puts one port on a person's computer on the web at its own HTTPS address, behind OpenVibe sign-in. A one-file client (Node 22, no dependencies) opens one outgoing WebSocket through Cloudflare and carries requests to localhost: HTTP with streaming both ways, server-sent events and WebSocket upgrades. Every visitor signs in and must be allowed by the share's policy (only the owner, named people, or anyone signed in); the share host keeps its own host-only signed cookie. Nothing that passes through is stored.",++ "summary": "OpenVibe Connect puts one port on a person's computer on the web at its own HTTPS address, behind OpenVibe sign-in. A one-file client (Node 22, no dependencies) opens one outgoing WebSocket through Cloudflare and carries requests to localhost: HTTP with streaming both ways, server-sent events and WebSocket upgrades. Every visitor signs in and must be allowed by the share's policy (only the owner, named people, or anyone signed in); the share host keeps its own host-only signed cookie. Nothing that passes through a share is stored; a webhook addressed to one is sealed and kept for up to 72 hours until its computer takes it.","works": ["Shares at <name>--<user>.openvibe.website, never colliding with Space sites; five per person; a closed share keeps its address","The client: hello with a hashed-at-rest key, streams both ways with backpressure, SSE, WebSocket upgrades as a byte pipe, reconnect, takeover, stop on a bad key","The gate: /grant with OpenVibe sign-in, one-time codes, __Host-ovc bound to the share and its policy version",++ "Durable webhooks (CON-15): an address on each share keeps up to 1,000 webhooks for 72 hours, sealed at rest, replays them in order with x-openvibe-received-at and a delivery log the owner can read",++ "The OpenVibe Node plugin (CON-4): a share served by the owner's paired computer, no key, with `openvibe-node share <port>` and the share in `openvibe-node status`","What the app gets: localhost Host, X-Forwarded-Host, X-OpenVibe-User/Subject; Set-Cookie Domain removed; no CDN caching","Pages (front door, your shares, a share's page), the API for a person's token, account export/deletion","Checks on PGlite and PostgreSQL, including a share end to end through the real client"@@@ -29,9 +31,7 @@"TCP/UDP shares (CON-11/12)","Custom domains (CON-7)","The QUIC edge (CON-3)",-- "The private mesh (CON-13)",-- "Durable webhooks (CON-15)",-- "The OpenVibe Node plugin (CON-4)"++ "The private mesh (CON-13)"],"capabilitiesRegistered": false,"serviceManifestRegistered": true,nnew file mode 100644@@@ -0,0 +1,26 @@++-- phase: expand++-- Arrival order: a share's waiting webhooks replay oldest first, including two that arrive in the same millisecond.++ALTER TABLE hook_requests ADD COLUMN IF NOT EXISTS seq bigserial;++CREATE INDEX IF NOT EXISTS hook_requests_seq ON hook_requests (hook_id, seq);++++-- The owner's delivery log (CON-15): what arrived at a webhook address, what the computer's app answered (or why it++-- could not), and when. It never keeps a body or a provider header: the log says that a delivery happened, not what++-- the provider said. Rows are removed 72 hours after they are made, with the webhooks they describe.++CREATE TABLE IF NOT EXISTS hook_deliveries (++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,++ share_id text NOT NULL REFERENCES shares(id) ON DELETE CASCADE,++ hook_id text NOT NULL REFERENCES share_hooks(id) ON DELETE CASCADE,++ owner_subject text NOT NULL, -- usr_…: the person's account export names these rows++ request_id text, -- hb_… of the kept webhook; gone once it is delivered++ event_id text, -- evt_… the app also sees in x-openvibe-event-id++ method text NOT NULL,++ path text NOT NULL, -- the path on the computer's app++ status integer, -- the app's answer; null when it did not answer++ error text, -- why a delivery failed, at most 200 characters++ received_at timestamptz, -- when the provider's webhook first arrived++ delivered_at timestamptz, -- null for a failed attempt++ created_at timestamptz NOT NULL DEFAULT now()++);++CREATE INDEX IF NOT EXISTS hook_deliveries_share ON hook_deliveries (share_id, created_at DESC);++CREATE INDEX IF NOT EXISTS hook_deliveries_hook ON hook_deliveries (hook_id, created_at DESC);++CREATE INDEX IF NOT EXISTS hook_deliveries_owner ON hook_deliveries (owner_subject, created_at DESC);@@@ -146,6 +146,12 @@.cn-keyuse { display: grid; gap: 2px; }.cn-hooks .cn-keyuse-list { list-style: none; margin: 0; padding: 0; display: flex; flex-wrap: wrap; gap: 2px 12px; font-size: 13px; }.cn-hooks .cn-keyuse-list li { display: inline; padding: 0; border: 0; border-radius: 0; background: none; color: var(--ovk-muted); }++.cn-hooks .cn-delivery-log { flex-basis: 100%; font-size: 13px; color: var(--ovk-muted); }++.cn-hooks .cn-delivery-log summary { cursor: pointer; }++.cn-hooks .cn-delivery-log ul { list-style: none; margin: 6px 0 0; padding: 0; display: grid; gap: 3px; }++.cn-hooks .cn-delivery-log li { padding: 0; border: 0; border-radius: 0; background: none; justify-content: flex-start; }++.cn-hooks .cn-delivery-log b { color: var(--ovk-text); font-weight: 600; }++.cn-muted { color: var(--ovk-muted); }.cn-hook-add { display: flex; flex-wrap: wrap; align-items: end; gap: 10px 14px; }.cn-hook-add .cn-field { flex: 1 1 200px; }@@@ -104,7 +104,7 @@ async function createApp(opts = {}) {table: 'service_outbox', eventTypes: ['connect.hook.received'],validate: (env) => contracts.validate('events.event-envelope@1', env),});-- const hooks = createHooks(s, { config, outbox, now: s.now });++ const hooks = createHooks(s, { config, outbox, now: s.now, log });const ctx = { config, s, keys, sso, principal, shares, tunnels, gate, proxy, apiKeys, oauth, notifier, nodeAuth, machines, hooks, outbox, accountData, accountSend, log };ctx.probe = createProbe({ shares, proxy, config, log, now: s.now });@@@ -113,10 +113,16 @@ async function createApp(opts = {}) {/** A share's computer is connected: its waiting webhooks go to its app, one at a time, oldest first. → delivered count */ctx.deliverWaiting = async (share) => {let n = 0;-- for (const q of await hooks.waiting(share.id)) {-- const out = await proxy.deliverHook(share, { method: q.method, path: q.path, headers: q.headers, body: q.body, hookId: q.hookId, eventId: q.eventId || q.ref });++ for (const ref of await hooks.pending(share.id)) {++ const q = await hooks.request(ref);++ if (!q) continue;++ const out = await proxy.deliverHook(share, { method: q.method, path: q.path, headers: q.headers, body: q.body, hookId: q.hookId, eventId: q.eventId || q.ref, receivedAt: q.receivedAt });if (out.offline) break;-- if (out.status && out.status < 500 && await hooks.settle(q.ref, out.status)) n++;++ if (!out.status || out.status >= 500) {++ await hooks.fail(q.ref, out.status || null, out.error || `the app answered ${out.status}`);++ continue;++ }++ if (await hooks.settle(q.ref, out.status)) n++;}if (n) log.log(`[OpenVibe.Connect] ${n} waiting webhook(s) handed to ${share.host}`);return n;@@@ -128,8 +134,8 @@ async function createApp(opts = {}) {try { due = await shares.expireDue(); } catch (err) { log.warn('[OpenVibe.Connect] expiry sweep failed:', err.message); return []; }due.forEach((r) => { tunnels.close(r.id, 4003, 'the share ended'); proxy.forget(r.host); proxy.inspector.forget(r.id); });if (due.length) log.log(`[OpenVibe.Connect] ${due.length} share(s) ended on time`);-- // Webhooks kept past their 7 days go, delivered or not (CON-15).-- try { const gone = await hooks.sweep(); if (gone) log.log(`[OpenVibe.Connect] ${gone} webhook(s) past 7 days removed`); } catch (err) { log.warn('[OpenVibe.Connect] webhook sweep failed:', err.message); }++ // Webhooks and delivery-log rows past their 72 hours go, delivered or not (CON-15).++ try { const gone = await hooks.sweep(); if (gone) log.log(`[OpenVibe.Connect] ${gone} webhook(s) past 72 hours removed`); } catch (err) { log.warn('[OpenVibe.Connect] webhook sweep failed:', err.message); }return due;};if (opts.expirySweep !== false) {@@@ -455,10 +455,11 @@ function createProxy({ config, shares, tunnels, gate, apiKeys = null, oauth = nu/*** One stored webhook to the share's computer (CON-15): the request as it arrived, at the hook's path, with-- * x-openvibe-hook and x-openvibe-event-id so the app can tell and dedupe. The answer's body is read and dropped.++ * x-openvibe-hook, x-openvibe-event-id and x-openvibe-received-at (the original arrival time) so the app can tell,++ * dedupe and preserve order. The answer's body is read and dropped.* → { status } when the app answered, { offline: true } with no computer, { error } when it broke on the way.*/-- function deliverHook(share, { method, path, headers = [], body, hookId, eventId }) {++ function deliverHook(share, { method, path, headers = [], body, hookId, eventId, receivedAt }) {const tunnel = tunnels.get(share.id);if (!tunnel) return Promise.resolve({ offline: true });const port = (tunnel.clientInfo && tunnel.clientInfo.port) || share.local_port;@@@ -478,7 +479,9 @@ function createProxy({ config, shares, tunnels, gate, apiKeys = null, oauth = nutimer.unref();const out = (Array.isArray(headers) ? headers : []).filter((h) => Array.isArray(h) && h.length === 2 && !/^(host|content-length|x-openvibe-)/i.test(String(h[0])));out.push(['host', `localhost:${port}`], ['x-forwarded-host', share.host], ['x-forwarded-proto', config.shares.scheme],-- ['x-openvibe-hook', String(hookId || '')], ['x-openvibe-event-id', String(eventId || '')], ['content-length', String(body ? body.length : 0)]);++ ['x-openvibe-hook', String(hookId || '')], ['x-openvibe-event-id', String(eventId || '')],++ ['x-openvibe-received-at', receivedAt ? new Date(receivedAt).toISOString() : new Date().toISOString()],++ ['content-length', String(body ? body.length : 0)]);tunnel.send({ t: 'req', id, method, path, headers: out });if (body && body.length) {rec.in += body.length;@@@ -3,15 +3,16 @@/*** Hooks that wait for your computer (CON-15, ADR-057 decision 7). A hook is an address a provider sends webhooks to,* https://connect.openvibe.website/h/<hk_…>, bound to one share and a path on the computer's app. A webhook is kept-- * (7 days) and a pointer to it is published as connect.hook.received through the outbox, in one transaction. Events-- * delivers it to Connect's own subscription (retried for up to seven days), and Connect hands it to the share's-- * computer when it is connected. The computer may be asleep for days and still get every webhook, in order of arrival-- * or not, each once it answers.++ * (72 hours) and a pointer to it is published as connect.hook.received through the outbox, in one transaction. Events++ * delivers it to Connect's own subscription (retried for up to 72 hours), and Connect hands it to the share's++ * computer when it is connected. The computer may be asleep for days and still get every webhook, in order of arrival,++ * each once it answers. A share's queue holds at most 1,000 undelivered webhooks, and each body is sealed at rest.** create(owner, share, { path, label }) → hook list(shareId) → hooks remove(owner, id)* get(id) receive(hook, { method, headers, body }) → { ref }-- * request(ref) → the stored request with its hook settle(ref, status) sweep() → rows removed-- * waiting(shareId) → its undelivered requests, oldest first (handed over at once when its computer connects)++ * request(ref) → the stored request with its hook settle(ref, status) fail(ref, status, error)++ * pending(shareId) → its undelivered refs, oldest first (handed over one at a time when its computer connects)++ * deliveries(shareId, hookId?) → the owner's delivery log sweep() → rows removed*/const crypto = require('crypto');const { ids } = require('openvibe-contracts');@@@ -20,24 +21,61 @@ const HOOK_ID = /^hk_[0-9A-HJKMNP-TV-Z]{26}$/;const REF_RE = /^hb_[A-Za-z0-9_-]{16,64}$/;const PATH_RE = /^\/[\x21-\x7e]{0,499}$/;const METHODS = ['POST', 'PUT', 'PATCH'];--const KEEP_MS = 7 * 86400_000;++const KEEP_MS = 72 * 3600_000;const MAX_BODY = 1024 * 1024;++const MAX_WAITING = 1000;const PER_SHARE = 10;--// The provider headers a webhook keeps: what apps check signatures and events with. Never cookies, authorization,--// forwarding or Cloudflare headers, and never Connect's own.--const DROP = /^(x-forwarded-|x-real-ip$|x-openvibe-|cf-|x-amz-cf-)/;--const KEEP = /^(content-type|content-encoding|user-agent|stripe-signature|svix-.+|webhook-.+|x-.+)$/;++// The request headers a webhook keeps: everything but the hop-by-hop headers, forwarding and Cloudflare headers, and++// credentials (a provider's cookie or Authorization header is not the app's). Signatures and event names are kept.++const DROP = new Set(['connection', 'keep-alive', 'proxy-authenticate', 'proxy-authorization', 'te', 'trailer', 'transfer-encoding',++ 'upgrade', 'host', 'content-length', 'cookie', 'cookie2', 'authorization', 'forwarded', 'x-real-ip', 'cdn-loop', 'true-client-ip']);++const DROP_RE = /^(x-forwarded-|x-openvibe-|cf-|x-amz-cf-)/;++++// A sealed body starts with this magic; a row written before sealing has no magic and is read as it is until the next++// sweep seals it.++const SEAL_MAGIC = Buffer.from('OVC1');++const SEAL_IV = 12;++const SEAL_TAG = 16;class HookError extends Error {constructor(status, code, message) { super(message); this.status = status; this.code = code; }}++/** The key that seals a webhook body: derived from the session secret, never stored. */++function bodyKey(secret) {++ if (!secret) throw new Error('CONNECT_SESSION_SECRET is not set: a webhook body cannot be sealed');++ return Buffer.from(crypto.hkdfSync('sha256', Buffer.from(String(secret)), Buffer.from('openvibe-connect'), 'hook-body-v1', 32));++}++++function sealBody(key, hookId, ref, body) {++ const iv = crypto.randomBytes(SEAL_IV);++ const cipher = crypto.createCipheriv('aes-256-gcm', key, iv);++ cipher.setAAD(Buffer.from(`connect.hook|${hookId}|${ref}`));++ const sealed = Buffer.concat([cipher.update(body), cipher.final()]);++ return Buffer.concat([SEAL_MAGIC, iv, cipher.getAuthTag(), sealed]);++}++++/** → { body, sealed }. A body written before sealing has no magic and is returned as it is. */++function openBody(key, hookId, ref, blob) {++ const b = Buffer.from(blob || []);++ if (b.length < SEAL_MAGIC.length + SEAL_IV + SEAL_TAG || !b.subarray(0, SEAL_MAGIC.length).equals(SEAL_MAGIC)) {++ return { body: b, sealed: false };++ }++ const iv = b.subarray(SEAL_MAGIC.length, SEAL_MAGIC.length + SEAL_IV);++ const tag = b.subarray(SEAL_MAGIC.length + SEAL_IV, SEAL_MAGIC.length + SEAL_IV + SEAL_TAG);++ const sealed = b.subarray(SEAL_MAGIC.length + SEAL_IV + SEAL_TAG);++ const decipher = crypto.createDecipheriv('aes-256-gcm', key, iv);++ decipher.setAAD(Buffer.from(`connect.hook|${hookId}|${ref}`));++ decipher.setAuthTag(tag);++ return { body: Buffer.concat([decipher.update(sealed), decipher.final()]), sealed: true };++}++/** The headers a stored webhook keeps: at most 40, each at most 2 KB. → [[name, value]] */function keepHeaders(headers) {const out = [];for (const [k, v] of Object.entries(headers || {})) {const name = String(k).toLowerCase();-- if (DROP.test(name) || !KEEP.test(name)) continue;++ if (DROP.has(name) || DROP_RE.test(name)) continue;const value = Array.isArray(v) ? v.join(', ') : String(v);if (value.length > 2048) continue;out.push([name, value]);@@@ -46,15 +84,22 @@ function keepHeaders(headers) {return out;}--function createHooks(s, { config, outbox = null, now = () => Date.now() }) {++function createHooks(s, { config, outbox = null, now = () => Date.now(), log = console } = {}) {const { db } = s;const base = config.baseUrl;++ const key = bodyKey(config.shares && config.shares.sessionSecret);const iso = (v) => (v ? new Date(v).toISOString() : null);-- const view = (r) => r && ({++ const view = (r, recent) => r && ({id: r.id, share_id: r.share_id, url: `${base}/h/${r.id}`, local_path: r.local_path, label: r.label || null, status: r.status,created_at: iso(r.created_at), received: Number(r.received || 0), delivered: Number(r.delivered || 0),last_received_at: iso(r.last_received_at), last_delivered_at: iso(r.last_delivered_at), last_status: r.last_status == null ? null : Number(r.last_status),waiting: r.waiting == null ? undefined : Number(r.waiting),++ deliveries: recent === undefined ? undefined : recent,++ });++ const delivery = (d) => ({++ id: Number(d.id), hook_id: d.hook_id, request_id: d.request_id || null, event_id: d.event_id || null,++ method: d.method, path: d.path, status: d.status == null ? null : Number(d.status), error: d.error || null,++ received_at: iso(d.received_at), delivered_at: iso(d.delivered_at), created_at: iso(d.created_at),});async function get(id) {@@@ -65,7 +110,15 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {async function list(shareId) {const rows = await db.many(`SELECT h.*, (SELECT count(*)::int FROM hook_requests r WHERE r.hook_id = h.id AND r.delivered_at IS NULL AND r.expires_at > now()) AS waitingFROM share_hooks h WHERE h.share_id = $1 AND h.status = 'active' ORDER BY h.created_at`, [shareId]);-- return rows.map(view);++ const recent = await db.many(`SELECT id, hook_id, request_id, event_id, method, path, status, error, received_at, delivered_at, created_at++ FROM hook_deliveries WHERE share_id = $1 ORDER BY created_at DESC, id DESC LIMIT 100`, [shareId]);++ const byHook = new Map();++ for (const d of recent) {++ const arr = byHook.get(d.hook_id) || [];++ if (arr.length < 5) arr.push(delivery(d));++ byHook.set(d.hook_id, arr);++ }++ return rows.map((r) => view(r, byHook.get(r.id) || []));}async function create(owner, share, input = {}) {@@@ -89,7 +142,7 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {return view({ ...hook, status: 'removed' });}-- /** A webhook arrived: kept, and its pointer published, together. → { ref } */++ /** A webhook arrived: sealed, queued under the share's cap, and its pointer published, together. → { ref } */async function receive(hook, { method, headers, body }) {if (!METHODS.includes(method)) throw new HookError(405, 'hook.method', 'A webhook address takes POST, PUT or PATCH.');const buf = Buffer.isBuffer(body) ? body : Buffer.from(body == null ? '' : String(body));@@@ -101,9 +154,15 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {// The event's id is chosen here and kept with the request: the app sees the same x-openvibe-event-id however// the webhook reaches it (handed over on connect, or Events' retry).const eventId = ids.newId('event');++ const sealed = sealBody(key, hook.id, ref, buf);await s.tx(async (t) => {++ // One row lock per share makes the queue cap exact while two providers post at once.++ await t.exec('SELECT id FROM shares WHERE id = $1 FOR UPDATE', [hook.share_id]);++ const q = await t.one(`SELECT count(*)::int AS n FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_id++ WHERE h.share_id = $1 AND r.delivered_at IS NULL AND r.expires_at > $2`, [hook.share_id, at.toISOString()]);++ if (q.n >= MAX_WAITING) throw new HookError(429, 'hook.queue_full', `This share is holding ${MAX_WAITING} webhooks already; they go to the computer first.`);await t.exec('INSERT INTO hook_requests (id, hook_id, method, headers, body, received_at, expires_at, event_id) VALUES ($1, $2, $3, $4::jsonb, $5, $6, $7, $8)',-- [ref, hook.id, method, JSON.stringify(kept), buf, at.toISOString(), new Date(at.getTime() + KEEP_MS).toISOString(), eventId]);++ [ref, hook.id, method, JSON.stringify(kept), sealed, at.toISOString(), new Date(at.getTime() + KEEP_MS).toISOString(), eventId]);await t.exec('UPDATE share_hooks SET received = received + 1, last_received_at = $2 WHERE id = $1', [hook.id, at.toISOString()]);if (outbox) {await outbox.emitIn(t, {@@@ -118,40 +177,91 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {return { ref };}-- /** The stored request a delivery points at, with its hook; null when it was delivered, removed or expired. */++ /** The stored request a delivery points at, with its hook and its original arrival time; null when it is gone. */async function request(ref) {if (!REF_RE.test(String(ref || ''))) return null;const r = await db.maybe(`SELECT r.*, h.share_id, h.local_path, h.status AS hook_status FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_idWHERE r.id = $1`, [ref]);if (!r || r.hook_status !== 'active' || r.delivered_at || new Date(r.expires_at).getTime() <= now()) return null;-- return { ref: r.id, hookId: r.hook_id, shareId: r.share_id, path: r.local_path, method: r.method, headers: Array.isArray(r.headers) ? r.headers : [], body: r.body, eventId: r.event_id || null };++ let body;++ try { body = openBody(key, r.hook_id, r.id, r.body).body; }++ catch (err) {++ log.warn('[OpenVibe.Connect] a webhook body cannot be opened:', err && err.message);++ return null;++ }++ return { ref: r.id, hookId: r.hook_id, shareId: r.share_id, path: r.local_path, method: r.method, headers: Array.isArray(r.headers) ? r.headers : [], body, eventId: r.event_id || null, receivedAt: iso(r.received_at) };}-- /** A share's webhooks still waiting for its computer, oldest first (at most `limit`). */-- async function waiting(shareId, limit = 100) {++ /**++ * A share's webhooks still waiting for its computer, oldest first, up to the share's 1,000 cap. It returns refs,++ * not bodies: the caller opens one at a time, so a full queue never sits in memory at once.++ */++ async function pending(shareId, limit = MAX_WAITING) {const rows = await db.many(`SELECT r.id FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_idWHERE h.share_id = $1 AND h.status = 'active' AND r.delivered_at IS NULL AND r.expires_at > $2-- ORDER BY r.received_at LIMIT $3`, [shareId, new Date(now()).toISOString(), limit]);-- const out = [];-- for (const r of rows) { const q = await request(r.id); if (q) out.push(q); }-- return out;++ ORDER BY r.seq LIMIT $3`, [shareId, new Date(now()).toISOString(), Math.min(Number(limit) || MAX_WAITING, MAX_WAITING)]);++ return rows.map((r) => r.id);}-- /** The computer's app answered: the request is delivered (its body is dropped) and the hook counts it. */++ /** The computer's app answered: the request is delivered (its body is dropped), logged and counted. */async function settle(ref, status) {++ let settled = false;++ await s.tx(async (t) => {++ const at = new Date(now()).toISOString();++ const r = await t.maybe('UPDATE hook_requests SET delivered_at = $2, status = $3, body = \'\'::bytea WHERE id = $1 AND delivered_at IS NULL RETURNING hook_id, method, event_id, received_at', [ref, at, status]);++ if (!r) return;++ const h = await t.maybe('SELECT share_id, owner_subject, local_path FROM share_hooks WHERE id = $1', [r.hook_id]);++ if (!h) return;++ await t.exec(`INSERT INTO hook_deliveries (share_id, hook_id, owner_subject, request_id, event_id, method, path, status, received_at, delivered_at, created_at)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $9, $9)`,++ [h.share_id, r.hook_id, h.owner_subject, ref, r.event_id || null, r.method, h.local_path, status, r.received_at || at]);++ await t.exec('UPDATE share_hooks SET delivered = delivered + 1, last_delivered_at = $2, last_status = $3 WHERE id = $1', [r.hook_id, at, status]);++ settled = true;++ });++ return settled;++ }++++ /** The computer's app refused a webhook: the request stays queued for another try, and the owner sees why. */++ async function fail(ref, status, error) {++ const r = await db.maybe('SELECT hook_id, method, event_id, received_at FROM hook_requests WHERE id = $1', [ref]);++ if (!r) return false;++ const h = await db.maybe('SELECT share_id, owner_subject, local_path FROM share_hooks WHERE id = $1', [r.hook_id]);++ if (!h) return false;const at = new Date(now()).toISOString();-- const r = await db.maybe("UPDATE hook_requests SET delivered_at = $2, status = $3, body = ''::bytea WHERE id = $1 AND delivered_at IS NULL RETURNING hook_id", [ref, at, status]);-- if (r) await db.exec('UPDATE share_hooks SET delivered = delivered + 1, last_delivered_at = $2, last_status = $3 WHERE id = $1', [r.hook_id, at, status]);-- return Boolean(r);++ const why = String(error || '').slice(0, 200) || `the app answered ${status || 'nothing'}`;++ await db.exec(`INSERT INTO hook_deliveries (share_id, hook_id, owner_subject, request_id, event_id, method, path, status, error, received_at, created_at)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $10)`,++ [h.share_id, r.hook_id, h.owner_subject, ref, r.event_id || null, r.method, h.local_path, status || null, why, r.received_at || at]);++ await db.exec('UPDATE share_hooks SET last_status = $2 WHERE id = $1', [r.hook_id, status || null]);++ return true;++ }++++ /** The owner's delivery log, newest first: what arrived, what the app answered, and when. */++ async function deliveries(shareId, hookId = null, limit = 50) {++ const rows = hookId++ ? await db.many('SELECT * FROM hook_deliveries WHERE share_id = $1 AND hook_id = $2 ORDER BY created_at DESC, id DESC LIMIT $3', [shareId, hookId, Math.max(1, Math.min(Math.trunc(Number(limit)) || 50, 200))])++ : await db.many('SELECT * FROM hook_deliveries WHERE share_id = $1 ORDER BY created_at DESC, id DESC LIMIT $2', [shareId, Math.max(1, Math.min(Math.trunc(Number(limit)) || 50, 200))]);++ return rows.map(delivery);++ }++++ /** Bodies written before sealing are sealed now, so nothing readable is left at rest. → sealed count */++ async function sealLegacy() {++ const rows = await db.many("SELECT id, hook_id, body FROM hook_requests WHERE body <> ''::bytea AND encode(substr(body, 1, 4), 'hex') <> '4f564331'");++ for (const r of rows) {++ await db.exec('UPDATE hook_requests SET body = $2 WHERE id = $1', [r.id, sealBody(key, r.hook_id, r.id, Buffer.from(r.body))]);++ }++ return rows.length;}-- /** Requests older than 7 days go, delivered or not. */++ /** Requests and delivery-log rows older than 72 hours go, delivered or not. → removed request count */async function sweep() {++ await sealLegacy();++ await db.exec('DELETE FROM hook_deliveries WHERE created_at <= $1', [new Date(now() - KEEP_MS).toISOString()]);const r = await db.many('DELETE FROM hook_requests WHERE expires_at <= $1 RETURNING id', [new Date(now()).toISOString()]);return r.length;}-- return { get, list, create, remove, receive, request, waiting, settle, sweep, KEEP_MS, MAX_BODY };++ return { get, list, create, remove, receive, request, pending, settle, fail, deliveries, sealLegacy, sweep, KEEP_MS, MAX_BODY, MAX_WAITING };}--module.exports = { createHooks, HookError, keepHeaders, HOOK_ID, METHODS, MAX_BODY, KEEP_MS };++module.exports = { createHooks, HookError, keepHeaders, bodyKey, HOOK_ID, METHODS, MAX_BODY, MAX_WAITING, KEEP_MS };@@@ -15,9 +15,10 @@* PUT /shares/:id/node { node: 'nod_…' } one of your computers serves it, as OpenVibe Node (CON-4): the key* stops working → { share }; { node: null } lets the computer go → { share, key } (a new key)* GET /machines your computers that can serve a share (Network's list of your machines)-- * GET /shares/:id/hooks its webhook addresses (CON-15): each kept webhook waits up to 7 days for the computer++ * GET /shares/:id/hooks its webhook addresses (CON-15): each kept webhook waits up to 72 hours for the computer* POST /shares/:id/hooks { path, label? } → 201 { hook }: https://connect.openvibe.website/h/<id> → that path* DELETE /shares/:id/hooks/:hook removes it; webhooks still waiting are dropped++ * GET /shares/:id/hooks/:hook/deliveries the owner's log: what the app answered, and when* GET /shares/:id/visits who opened it in the last 7 days* GET /shares/:id/requests the last 100 requests through it, newest first (?after=<seq>: only newer ones);* memory only, query values shown as "…" (edge/inspector.js)@@@ -137,6 +138,13 @@ function createApi(ctx) {if (!hook || hook.share_id !== share.id) return fail(res, req, 404, 'hook.not_found', 'No such webhook address.');return res.json({ hook: await hooks.remove(owner(req), hook.id) });}));++ r.get('/shares/:id/hooks/:hook/deliveries', guard(async (req, res) => {++ const share = await mine(req, res);++ if (!share) return undefined;++ const hook = await hooks.get(req.params.hook);++ if (!hook || hook.share_id !== share.id) return fail(res, req, 404, 'hook.not_found', 'No such webhook address.');++ return res.json({ deliveries: await hooks.deliveries(share.id, hook.id, req.query.limit) });++ }));// API keys (CON-16 v0): a program passes the gate with one. The key is in the answer that makes it, never again.r.get('/shares/:id/api-keys', guard(async (req, res) => {@@@ -3,11 +3,11 @@/*** Hooks that wait for your computer (CON-15): the address providers send to, and Events' deliveries back.*-- * POST|PUT|PATCH /h/:id a provider's webhook (1 MB): kept 7 days, its pointer published → 202++ * POST|PUT|PATCH /h/:id a provider's webhook (1 MB): sealed, kept 72 hours, its pointer published → 202* POST /internal/hook-deliveries Events, signed v2, loopback only (nginx answers 404 for /internal/): the* webhook goes to the share's computer. 200 once its app answered (any status* below 500) or when the webhook is gone; 503 while the computer is offline or-- * its app fails, so Events tries again (up to seven days, server/index.js).++ * its app fails, so Events tries again (up to 72 hours, server/index.js).*/const express = require('express');const { parseDelivery } = require('openvibe-sdk/events');@@@ -39,9 +39,12 @@ function createHookRoutes(ctx) {if (tooMany(hook.id)) return answer(res.set('Retry-After', '60'), 429, { error: 'rate_limited', detail: `At most ${PER_HOOK_MINUTE} webhooks a minute to one address.` });try {const { ref } = await hooks.receive(hook, { method: req.method, headers: req.headers, body: Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0) });-- return answer(res, 202, { received: true, id: ref, delivered_when: 'the computer is connected (kept up to 7 days)' });++ return answer(res, 202, { received: true, id: ref, delivered_when: 'the computer is connected (kept up to 72 hours)' });} catch (err) {-- if (err instanceof HookError) return answer(res, err.status, { error: err.code, detail: err.message });++ if (err instanceof HookError) {++ if (err.code === 'hook.queue_full') res.set('Retry-After', '300');++ return answer(res, err.status, { error: err.code, detail: err.message });++ }log.warn('[OpenVibe.Connect] webhook not kept:', err && err.message);return answer(res.set('Retry-After', '30'), 503, { error: 'unavailable', detail: 'Try again shortly.' });}@@@ -61,9 +64,12 @@ function createHookRoutes(ctx) {if (!stored) return res.status(200).json({ gone: true });const share = await shares.get(stored.shareId);if (!share || share.status !== 'active') return res.status(200).json({ gone: true });-- const out = await proxy.deliverHook(share, { method: stored.method, path: stored.path, headers: stored.headers, body: stored.body, hookId: stored.hookId, eventId: e.event_id });++ const out = await proxy.deliverHook(share, { method: stored.method, path: stored.path, headers: stored.headers, body: stored.body, hookId: stored.hookId, eventId: e.event_id, receivedAt: stored.receivedAt });if (out.offline) return res.status(503).set('Retry-After', '300').json({ waiting: 'the computer is not connected' });-- if (!out.status || out.status >= 500) return res.status(503).json({ retry: out.error || `the app answered ${out.status}` });++ if (!out.status || out.status >= 500) {++ await hooks.fail(stored.ref, out.status || null, out.error || `the app answered ${out.status}`);++ return res.status(503).json({ retry: out.error || `the app answered ${out.status}` });++ }await hooks.settle(stored.ref, out.status);return res.status(200).json({ delivered: true, status: out.status });});@@@ -18,6 +18,9 @@ const TABLES = [// the account is taken off it.{ table: 'share_reports', subject: 'reporter_subject', value: (usr) => usr, file: 'reports.json', erase: { anonymize: {} },columns: ['id', 'host', 'reason', 'details', 'status', 'created_at'] },++ // The delivery log of a person's webhook addresses (CON-15): method, path and the app's answer, never a body.++ { table: 'hook_deliveries', subject: 'owner_subject', value: (usr) => usr, file: 'webhook-deliveries.json', erase: 'delete',++ columns: ['id', 'share_id', 'hook_id', 'request_id', 'event_id', 'method', 'path', 'status', 'error', 'received_at', 'delivered_at', 'created_at'] },// Webhook addresses on a person's shares (CON-15). Their waiting webhooks (hook_requests) go with them, and with// the shares (ON DELETE CASCADE); the export names the addresses and their counts, never a webhook's body.{ table: 'share_hooks', subject: 'owner_subject', value: (usr) => usr, file: 'webhooks.json', erase: 'delete',@@@ -41,7 +44,7 @@ function create({ db, log = console } = {}) {db,service: 'connect',tables: TABLES,-- note: 'Your shares on OpenVibe Connect, their webhook addresses, and your visits to other people\'s shares. Connect never stores what passed through a share; a webhook to one of your addresses is kept only until your computer takes it, at most 7 days.',++ note: 'Your shares on OpenVibe Connect, their webhook addresses, and your visits to other people\'s shares. Connect never stores what passed through a share; a webhook to one of your addresses is kept only until your computer takes it, at most 72 hours.',log,});}@@@ -61,7 +61,7 @@ async function start() {log: ctx.log || console,});// Hooks (CON-15): the outbox relay publishes webhook pointers, and Connect's own subscription brings them back to-- // /internal/hook-deliveries, retried for about seven days while a computer is offline. Off in drill mode and++ // /internal/hook-deliveries, retried for about 72 hours while a computer is offline. Off in drill mode and// without the events URL, the delivery secret and the client secret.const hookSubs = config.drill ? null : ensureHookSubscription({ config, log: ctx.log || console });if (!config.drill && ctx.outbox) ctx.outbox.start();@@@ -70,10 +70,10 @@ async function start() {return { server, ctx };}--// About seven days of retries (Events allows 20 attempts, each wait at most a day): 5 s, 30 s, 2 min, 10 min, 30 min,--// 1, 2, 3, 4, 6, 8 hours, four of 12 hours and four of a day.++// About 72 hours of retries (Events allows 20 attempts): 5 s, 30 s, 2 min, 10 min, 30 min, 1, 2, 3, 4, 6, 8 hours,++// four of 4 hours, three of 6 hours and two of 8 hours.const HOOK_BACKOFF_MS = [5e3, 30e3, 120e3, 600e3, 1800e3, 3600e3, 7200e3, 10800e3, 14400e3, 21600e3, 28800e3,-- 43200e3, 43200e3, 43200e3, 43200e3, 86400e3, 86400e3, 86400e3, 86400e3];++ 14400e3, 14400e3, 14400e3, 14400e3, 21600e3, 21600e3, 21600e3, 28800e3, 28800e3];/** Connect's subscription to its own connect.hook.received, made once (409 means it is there); tried again every 10 minutes until it is. */function ensureHookSubscription({ config, log = console, fetchImpl = globalThis.fetch }) {@@@ -264,12 +264,17 @@ function nodeBlock(share, machines, now) {* was handed over, and how many wait; the form for another. Plain forms, no script.*/function hooksBlock(share, hooks, now) {++ const log = (h) => {++ const rows = Array.isArray(h.deliveries) ? h.deliveries.slice(0, 5) : [];++ if (!rows.length) return '';++ return `<details class="cn-delivery-log"><summary>Delivery log</summary><ul>${rows.map((d) => `<li><b>${d.status == null ? 'failed' : esc(d.status)}</b> ${esc(d.method)} ${esc(d.path)} · ${esc(ago(d.delivered_at || d.created_at, now))}${d.error ? ` · ${esc(d.error)}` : ''}</li>`).join('')}</ul></details>`;++ };const rows = hooks.length ? `<ul class="cn-hooks">${hooks.map((h, i) => `<li><div class="cn-hook-main"><div class="cn-copy"><code id="cn-hook-${i}">${esc(h.url)}</code><button type="button" class="cn-btn cn-btn-quiet" data-ovk-copy="#cn-hook-${i}" hidden>${kit.icon('copy')} Copy</button></div>--<small>→ localhost:${esc(share.local_port)}${esc(h.local_path)}${h.label ? ` · ${esc(h.label)}` : ''} · ${esc(h.received)} received, ${esc(h.delivered)} handed over${h.waiting ? `, <b>${esc(h.waiting)} waiting for your computer</b>` : ''}${h.last_received_at ? ` · last ${esc(ago(h.last_received_at, now))}` : ''}${h.last_status ? ` · your app answered ${esc(h.last_status)}` : ''}</small></div>++<small>→ localhost:${esc(share.local_port)}${esc(h.local_path)}${h.label ? ` · ${esc(h.label)}` : ''} · ${esc(h.received)} received, ${esc(h.delivered)} handed over${h.waiting ? `, <b>${esc(h.waiting)} waiting for your computer</b>` : ''}${h.last_received_at ? ` · last ${esc(ago(h.last_received_at, now))}` : ''}${h.last_status ? ` · your app answered ${esc(h.last_status)}` : ''}</small>${log(h)}</div><form method="post" action="/shares/${esc(share.id)}/hooks/${esc(h.id)}/remove"><button class="cn-btn cn-btn-quiet" type="submit">${kit.icon('close')} Remove</button></form></li>`).join('')}</ul>`: '<p class="cn-empty">No webhook addresses yet.</p>';return `<section class="cn-hooks-sec" aria-labelledby="cn-hooks-h"><h2 id="cn-hooks-h">Webhooks that wait for your computer</h2>--<p>Give a service (GitHub, Stripe, anything that sends webhooks) an address here instead of your share's. Each webhook is kept for up to 7 days and handed to your app, at the path you pick, as soon as your computer is connected: a laptop that sleeps misses nothing. Nobody has to sign in to send one, so check the provider's signature in your app.</p>++<p>Give a service (GitHub, Stripe, anything that sends webhooks) an address here instead of your share's. Each webhook is sealed, kept for up to 72 hours, and handed to your app, at the path you pick, as soon as your computer is connected: a laptop that sleeps misses nothing. A share holds at most 1,000 waiting webhooks; the log below says what your app answered. Nobody has to sign in to send one, so check the provider's signature in your app.</p>${rows}<form class="cn-form cn-hook-add" method="post" action="/shares/${esc(share.id)}/hooks"><label class="cn-field"><span>Path on your app</span><input name="path" maxlength="500" value="/webhooks" placeholder="/webhooks/github" autocapitalize="none" autocomplete="off" spellcheck="false"></label>@@@ -2,12 +2,13 @@/*** Hooks that wait for your computer (CON-15): a webhook address on a share, a provider's webhook kept with only the* provider's headers and its pointer in the outbox (one transaction), Events' signed delivery handed to the computer-- * once it is connected (503 while it is not, so Events retries), once only, and the sweep after 7 days.++ * once it is connected (503 while it is not, so Events retries), once only, and the sweep after 72 hours.* node test/hooks.test.js*/const assert = require('assert');const contracts = require('openvibe-contracts');const { signDeliveryHeaders } = require('openvibe-sdk/events');++const { keepHeaders } = require('../server/hooks');const { boot, startLocalApp } = require('./helpers/boot');const { check, done } = require('./helpers/check');@@@ -22,6 +23,15 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';let hook;let envelope;++ const settled = async (ref) => {++ for (let i = 0; i < 200; i++) {++ const row = await t.ctx.s.db.maybe('SELECT delivered_at FROM hook_requests WHERE id = $1', [ref]);++ if (!row || row.delivered_at) return true;++ await new Promise((r) => setTimeout(r, 10));++ }++ return false;++ };++const deliver = async (env, secret = SECRET) => {const raw = Buffer.from(JSON.stringify({ event: env }));const headers = { 'content-type': 'application/json', ...signDeliveryHeaders(raw, secret), 'x-openvibe-subscription-id': 'sub_x', 'x-openvibe-delivery-attempt': '1' };@@@ -44,18 +54,30 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';assert.ok(page.text.includes(hook.url));});-- await check('a provider\'s webhook: 202, kept with its headers (never cookies or forwarding), its pointer in the outbox', async () => {++ await check('a provider\'s webhook: 202, sealed, kept with its headers minus hop-by-hop and credentials, its pointer in the outbox', async () => {const body = JSON.stringify({ ref: 'refs/heads/main', commits: 3 });-- const r = await t.get(`/h/${hook.id}`, { method: 'POST', body, headers: { 'content-type': 'application/json', 'x-github-event': 'push', 'x-hub-signature-256': 'sha256=abc', cookie: 'session=secret', 'x-forwarded-for': '203.0.113.9', authorization: 'Bearer nope' } });++ const r = await t.get(`/h/${hook.id}`, { method: 'POST', body, headers: {++ 'content-type': 'application/json', 'x-github-event': 'push', 'x-hub-signature-256': 'sha256=abc',++ cookie: 'session=secret', 'x-forwarded-for': '203.0.113.9', authorization: 'Bearer nope', 'x-openvibe-user': 'mallory',++ } });assert.strictEqual(r.status, 202, r.text);const ref = r.json().id;assert.match(ref, /^hb_/);-- const row = await t.ctx.s.db.one('SELECT method, headers, body FROM hook_requests WHERE id = $1', [ref]);++ const row = await t.ctx.s.db.one('SELECT method, headers, body, received_at, expires_at FROM hook_requests WHERE id = $1', [ref]);assert.strictEqual(row.method, 'POST');-- assert.strictEqual(Buffer.from(row.body).toString(), body);++ const kept = t.ctx.hooks.request(ref);++ assert.strictEqual((await kept).body.toString(), body, 'the body is there for the computer, not for the database');++ assert.ok(!Buffer.from(row.body).toString().includes('refs/heads/main'), 'the body is sealed at rest');++ assert.match(Buffer.from(row.body).subarray(0, 4).toString(), /^OVC1$/);const names = row.headers.map(([k]) => k);assert.ok(names.includes('x-github-event') && names.includes('x-hub-signature-256') && names.includes('content-type'), names.join(','));-- assert.ok(!names.includes('cookie') && !names.includes('x-forwarded-for') && !names.includes('authorization'), names.join(','));++ for (const gone of ['cookie', 'x-forwarded-for', 'authorization', 'x-openvibe-user']) {++ assert.ok(!names.includes(gone), `${gone} must not be kept: ${names.join(',')}`);++ }++ const hop = keepHeaders({ connection: 'keep-alive', te: 'trailers', 'proxy-authorization': 'Basic nope', upgrade: 'h2c', 'transfer-encoding': 'chunked', 'x-github-event': 'push' }).map(([k]) => k);++ assert.deepStrictEqual(hop, ['x-github-event'], 'hop-by-hop headers are dropped');++ const lifetime = await t.ctx.s.db.one('SELECT extract(epoch FROM expires_at - received_at)::int AS s FROM hook_requests WHERE id = $1', [ref]);++ assert.strictEqual(lifetime.s, 72 * 3600, 'kept for 72 hours');const out = await t.ctx.s.db.many("SELECT envelope FROM service_outbox WHERE envelope->>'event_type' = 'connect.hook.received'");assert.strictEqual(out.length, 1);envelope = out[0].envelope;@@@ -88,7 +110,9 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';assert.strictEqual(got.headers['x-hub-signature-256'], 'sha256=abc');assert.strictEqual(got.headers['x-openvibe-hook'], hook.id);assert.strictEqual(got.headers['x-openvibe-event-id'], envelope.event_id, 'the event\'s own id, whichever way it came');++ assert.strictEqual(got.headers['x-openvibe-received-at'], envelope.payload.received_at, 'the original arrival time');assert.strictEqual(got.headers.cookie, undefined);++ assert.ok(await settled(envelope.payload.body_ref), 'the delivery settles');const again = await deliver(envelope);assert.strictEqual(again.status, 200);assert.deepStrictEqual(again.json(), { gone: true }, 'delivered once');@@@ -97,6 +121,15 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';assert.deepStrictEqual([kept.n, kept.status], [0, 200], 'its body is dropped once delivered');const h = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex })).json().hooks[0];assert.deepStrictEqual([h.received, h.delivered, h.last_status], [1, 1, 200]);++ assert.strictEqual(h.deliveries[0].status, 200, 'the owner sees the delivery');++ assert.strictEqual(h.deliveries[0].method, 'POST');++ assert.strictEqual(h.deliveries[0].path, '/webhooks/github');++ const log = await t.get(`/api/v1/shares/${made.share.id}/hooks/${hook.id}/deliveries`, { as: alex });++ assert.strictEqual(log.status, 200, log.text);++ assert.strictEqual(log.json().deliveries[0].status, 200);++ assert.strictEqual(log.json().deliveries[0].event_id, envelope.event_id);++ const page = await t.get(`/shares/${made.share.id}`, { as: alex });++ assert.match(page.text, /Delivery log/);c.stop();});@@@ -116,7 +149,7 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';assert.strictEqual(left.n, 0);});-- await check('the sweep removes webhooks past their 7 days', async () => {++ await check('the sweep removes webhooks past their 72 hours', async () => {const h2 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/x' } })).json().hook;const ref = (await t.get(`/h/${h2.id}`, { method: 'POST', body: 'old' })).json().id;await t.ctx.s.db.exec("UPDATE hook_requests SET expires_at = now() - interval '1 minute' WHERE id = $1", [ref]);@@@ -124,6 +157,29 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';assert.strictEqual((await t.ctx.s.db.maybe('SELECT id FROM hook_requests WHERE id = $1', [ref])), null);});++ await check('a body written before sealing is sealed by the sweep and still reaches the app', async () => {++ const h2 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/x/legacy' } })).json().hook;++ const ref = 'hb_legacy0123456789ab';++ await t.ctx.s.db.exec(`INSERT INTO hook_requests (id, hook_id, method, headers, body, received_at, expires_at)++ VALUES ($1, $2, 'POST', '[]'::jsonb, $3::bytea, now(), now() + interval '72 hours')`, [ref, h2.id, Buffer.from('plain at rest')]);++ await t.ctx.hooks.sweep();++ const row = await t.ctx.s.db.one('SELECT body FROM hook_requests WHERE id = $1', [ref]);++ assert.ok(!Buffer.from(row.body).toString().includes('plain at rest'), 'the sweep sealed it');++ assert.strictEqual((await t.ctx.hooks.request(ref)).body.toString(), 'plain at rest');++ await t.get(`/api/v1/shares/${made.share.id}/hooks/${h2.id}`, { as: alex, method: 'DELETE' });++ });++++ await check('a share holds at most 1,000 waiting webhooks', async () => {++ const h2 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/x/full' } })).json().hook;++ await t.ctx.s.db.exec(`INSERT INTO hook_requests (id, hook_id, method, headers, body, received_at, expires_at, event_id)++ SELECT 'hb_q' || g, $1, 'POST', '[]'::jsonb, ''::bytea, now(), now() + interval '72 hours', 'evt_q' || g++ FROM generate_series(1, 1000) g`, [h2.id]);++ const r = await t.get(`/h/${h2.id}`, { method: 'POST', body: '{}' });++ assert.strictEqual(r.status, 429, r.text);++ assert.strictEqual(r.json().error, 'hook.queue_full');++ await t.get(`/api/v1/shares/${made.share.id}/hooks/${h2.id}`, { as: alex, method: 'DELETE' });++ });++await check('a webhook that waited goes to the app as soon as its computer connects; Events\' retry then finds it settled', async () => {const h3 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/webhooks/waited' } })).json().hook;const w = await t.get(`/h/${h3.id}`, { method: 'POST', body: '{"waited":true}', headers: { 'content-type': 'application/json' } });@@@ -137,6 +193,7 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';assert.ok(got, 'handed over on connect');assert.strictEqual(got.body, '{"waited":true}');assert.strictEqual(got.headers['x-openvibe-event-id'], env.event_id, 'the event\'s own id');++ assert.ok(await settled(w.json().id), 'the handover settles');const late = await deliver(env);assert.deepStrictEqual(late.json(), { gone: true }, 'Events\' delivery finds it settled');assert.strictEqual(app.seen.filter((x) => x.url === '/webhooks/waited').length, 1, 'once');@@@ -157,6 +214,22 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';c.stop();});++ await check('every waiting webhook replays in order when its computer connects, past one batch', async () => {++ const h = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/webhooks/order' } })).json().hook;++ for (let i = 0; i < 105; i++) {++ const r = await t.get(`/h/${h.id}`, { method: 'POST', body: JSON.stringify({ i }), headers: { 'content-type': 'application/json' } });++ assert.strictEqual(r.status, 202, r.text);++ }++ const before = app.seen.length;++ const c = t.client({ key: made.key, port: app.port });++ await c.ready();++ const got = () => app.seen.slice(before).filter((x) => x.url === '/webhooks/order');++ for (let i = 0; i < 500 && got().length < 105; i++) await new Promise((r) => setTimeout(r, 20));++ assert.strictEqual(got().length, 105, 'every waiting webhook is handed over');++ assert.deepStrictEqual(got().map((x) => JSON.parse(x.body).i), Array.from({ length: 105 }, (_, i) => i), 'oldest first');++ c.stop();++ });++await check('the page: make one with a plain form', async () => {const r = await t.get(`/shares/${made.share.id}/hooks`, { as: alex, method: 'POST', form: { path: '/hooks/stripe', label: 'Stripe' } });assert.strictEqual(r.status, 201, r.text);
@@ -4,6 +4,11 @@ What changed in OpenVibe Connect, newest first.
@@ -4,6 +4,11 @@ What changed in OpenVibe Connect, newest first.
## Unreleased — 2026-11-03
## Unreleased — 2026-11-03
+- Durable webhooks, hardened (CON-15): a share holds at most 1,000 waiting webhooks for 72 hours, every body is
+ sealed at rest with a key derived from the session secret, and the app gets `x-openvibe-received-at` so it keeps the
+ original arrival time. The owner gets a delivery log (page and `GET /api/v1/shares/:id/hooks/:hook/deliveries`).
+ Events' retries now span the same 72 hours. Migration `0012_hook_deliveries`.
+
- MCP OAuth 2.1 (CON-16): a share's address works as a connector in Claude or any MCP client. The client is sent to
- MCP OAuth 2.1 (CON-16): a share's address works as a connector in Claude or any MCP client. The client is sent to
Connect to sign the person in with OpenVibe: protected-resource metadata on the share, then registration, authorize
Connect to sign the person in with OpenVibe: protected-resource metadata on the share, then registration, authorize
with PKCE S256 and consent, then tokens bound to that share and rotated on refresh. The owner sees and signs out each
with PKCE S256 and consent, then tokens bound to that share and rotated on refresh. The owner sees and signs out each
@@ -42,25 +42,29 @@ instead" lets the computer go and shows a new key once (ADR-057 decision 4, CON-
@@ -42,25 +42,29 @@ instead" lets the computer go and shows a new key once (ADR-057 decision 4, CON-
### Webhooks that wait for your computer
### Webhooks that wait for your computer
On a share's page, "Make a webhook address" gives you `https://connect.openvibe.website/h/hk_…` and a path on your app
On a share's page, "Make a webhook address" gives you `https://connect.openvibe.website/h/hk_…` and a path on your app
-(`/webhooks/github`). Give that address to GitHub, Stripe or anything else that sends webhooks. Each webhook is kept for
-up to 7 days and handed to your app, at that path, as soon as your computer is connected, so a laptop that sleeps
-misses nothing (CON-15, ADR-057 decision 7):
-
-- The address takes POST, PUT and PATCH, up to 1 MB and 120 a minute, and answers `202` at once.
-- Connect keeps the method, the provider's headers (`content-type`, `user-agent`, `x-…`, `stripe-signature`,
- `svix-…`, `webhook-…`; never cookies, authorization or forwarding headers) and the body. It publishes only a
- pointer, `connect.hook.received`, to OpenVibe.Events, in the same transaction.
+(`/webhooks/github`). Give that address to GitHub, Stripe or anything else that sends webhooks. Each webhook is sealed,
+kept for up to 72 hours and handed to your app, at that path, as soon as your computer is connected, so a laptop that
+sleeps misses nothing (CON-15, ADR-057 decision 7):
+
+- The address takes POST, PUT and PATCH, up to 1 MB and 120 a minute, and answers `202` at once. A share holds at most
+ 1,000 waiting webhooks; after that the address answers `429` and the provider retries.
+- Connect keeps the method, the provider's headers minus hop-by-hop, credential and forwarding headers, and the body.
+ The body is sealed at rest with a key derived from the service's session secret and is never readable from the
+ database alone. It publishes only a pointer, `connect.hook.received`, to OpenVibe.Events, in the same transaction.
- The moment the computer connects, Connect hands it every webhook that waited, oldest first. Events also delivers the
- The moment the computer connects, Connect hands it every webhook that waited, oldest first. Events also delivers the
- pointer back to Connect's own subscription, retried for about seven days, which finds it settled. Connect hands the
- webhook to your app with `x-openvibe-hook` and `x-openvibe-event-id` (dedupe on it). Your app's answer below 500 settles it, and
- its body is dropped. While the computer is offline, or the app answers 5xx, Connect says 503 and Events tries again.
+ pointer back to Connect's own subscription, retried for about 72 hours, which finds it settled. Connect hands the
+ webhook to your app with `x-openvibe-hook`, `x-openvibe-event-id` (dedupe on it) and `x-openvibe-received-at` (when
+ the provider's webhook first arrived). Your app's answer below 500 settles it, and its body is dropped. While the
+ computer is offline, or the app answers 5xx, Connect says 503 and Events tries again.
+- The share's page and the API's delivery log show what arrived, what the app answered and when (never a body).
- Nobody signs in to send a webhook, so check the provider's own signature in your app.
- Nobody signs in to send a webhook, so check the provider's own signature in your app.
| Call | What |
| Call | What |
|---|---|
|---|---|
-| `GET /api/v1/shares/:id/hooks` | its webhook addresses, with `received`, `delivered` and `waiting` |
+| `GET /api/v1/shares/:id/hooks` | its webhook addresses, with `received`, `delivered`, `waiting` and the last deliveries |
| `POST /api/v1/shares/:id/hooks` `{ path, label? }` | a new address → `201 { hook }` (at most 10 a share) |
| `POST /api/v1/shares/:id/hooks` `{ path, label? }` | a new address → `201 { hook }` (at most 10 a share) |
| `DELETE /api/v1/shares/:id/hooks/:hook` | remove it; webhooks still waiting are dropped |
| `DELETE /api/v1/shares/:id/hooks/:hook` | remove it; webhooks still waiting are dropped |
+| `GET /api/v1/shares/:id/hooks/:hook/deliveries` | the delivery log: method, path, status, error and when |
## What your app sees
## What your app sees
@@ -187,7 +191,8 @@ browser ──https──▶ Cloudflare ──▶ nginx ──▶ Connect (5240)
@@ -187,7 +191,8 @@ browser ──https──▶ Cloudflare ──▶ nginx ──▶ Connect (5240)
no bodies; a restart forgets them.
no bodies; a restart forgets them.
- `server/auth/node-auth.js` — a computer as OpenVibe Node: its node token, and the single-use tunnel tokens (CON-4).
- `server/auth/node-auth.js` — a computer as OpenVibe Node: its node token, and the single-use tunnel tokens (CON-4).
- `server/machines.js` — the owner's computers, read from Network with the owner's own token.
- `server/machines.js` — the owner's computers, read from Network with the owner's own token.
-- `server/hooks.js`, `server/http/hooks.js` — webhook addresses, the kept webhooks, and Events' deliveries (CON-15).
+- `server/hooks.js`, `server/http/hooks.js` — webhook addresses, the sealed kept webhooks, the 1,000-deep queue, the
+delivery log and Events' deliveries (CON-15).
- `public/ov-connect.js` — the client.
- `public/ov-connect.js` — the client.
The v0 transport is a WebSocket on 443 through Cloudflare (ADR-057's fallback path), so nothing needs a UDP port or a
The v0 transport is a WebSocket on 443 through Cloudflare (ADR-057's fallback path), so nothing needs a UDP port or a
@@ -3,7 +3,7 @@
@@ -3,7 +3,7 @@
"version": "0.1.0",
"version": "0.1.0",
"stage": "product",
"stage": "product",
"code": true,
"code": true,
- "plan": "Website Connect v0 (plan 2026-10-28 §5.9b, ADR-057 reviewed 2026-11-02): share one port on your computer at https://<name>--<user>.openvibe.website behind OpenVibe sign-in",
+ "plan": "Website Connect v0 (plan 2026-10-28 \u00a75.9b, ADR-057 reviewed 2026-11-02): share one port on your computer at https://<name>--<user>.openvibe.website behind OpenVibe sign-in",
"runtime": "node 22, express 4, ws 8, PostgreSQL 18 (openvibe-sdk/db, async)",
"runtime": "node 22, express 4, ws 8, PostgreSQL 18 (openvibe-sdk/db, async)",
"port": "5240",
"port": "5240",
"domain": "connect.openvibe.website",
"domain": "connect.openvibe.website",
@@ -15,11 +15,13 @@
@@ -15,11 +15,13 @@
"openvibe-sdk v0.91.0",
"openvibe-sdk v0.91.0",
"openvibe-shared v4.115.1"
"openvibe-shared v4.115.1"
],
],
- "summary": "OpenVibe Connect puts one port on a person's computer on the web at its own HTTPS address, behind OpenVibe sign-in. A one-file client (Node 22, no dependencies) opens one outgoing WebSocket through Cloudflare and carries requests to localhost: HTTP with streaming both ways, server-sent events and WebSocket upgrades. Every visitor signs in and must be allowed by the share's policy (only the owner, named people, or anyone signed in); the share host keeps its own host-only signed cookie. Nothing that passes through is stored.",
+ "summary": "OpenVibe Connect puts one port on a person's computer on the web at its own HTTPS address, behind OpenVibe sign-in. A one-file client (Node 22, no dependencies) opens one outgoing WebSocket through Cloudflare and carries requests to localhost: HTTP with streaming both ways, server-sent events and WebSocket upgrades. Every visitor signs in and must be allowed by the share's policy (only the owner, named people, or anyone signed in); the share host keeps its own host-only signed cookie. Nothing that passes through a share is stored; a webhook addressed to one is sealed and kept for up to 72 hours until its computer takes it.",
"works": [
"works": [
"Shares at <name>--<user>.openvibe.website, never colliding with Space sites; five per person; a closed share keeps its address",
"Shares at <name>--<user>.openvibe.website, never colliding with Space sites; five per person; a closed share keeps its address",
"The client: hello with a hashed-at-rest key, streams both ways with backpressure, SSE, WebSocket upgrades as a byte pipe, reconnect, takeover, stop on a bad key",
"The client: hello with a hashed-at-rest key, streams both ways with backpressure, SSE, WebSocket upgrades as a byte pipe, reconnect, takeover, stop on a bad key",
"The gate: /grant with OpenVibe sign-in, one-time codes, __Host-ovc bound to the share and its policy version",
"The gate: /grant with OpenVibe sign-in, one-time codes, __Host-ovc bound to the share and its policy version",
+ "Durable webhooks (CON-15): an address on each share keeps up to 1,000 webhooks for 72 hours, sealed at rest, replays them in order with x-openvibe-received-at and a delivery log the owner can read",
+ "The OpenVibe Node plugin (CON-4): a share served by the owner's paired computer, no key, with `openvibe-node share <port>` and the share in `openvibe-node status`",
"What the app gets: localhost Host, X-Forwarded-Host, X-OpenVibe-User/Subject; Set-Cookie Domain removed; no CDN caching",
"What the app gets: localhost Host, X-Forwarded-Host, X-OpenVibe-User/Subject; Set-Cookie Domain removed; no CDN caching",
"Pages (front door, your shares, a share's page), the API for a person's token, account export/deletion",
"Pages (front door, your shares, a share's page), the API for a person's token, account export/deletion",
"Checks on PGlite and PostgreSQL, including a share end to end through the real client"
"Checks on PGlite and PostgreSQL, including a share end to end through the real client"
@@ -29,9 +31,7 @@
@@ -29,9 +31,7 @@
"TCP/UDP shares (CON-11/12)",
"TCP/UDP shares (CON-11/12)",
"Custom domains (CON-7)",
"Custom domains (CON-7)",
"The QUIC edge (CON-3)",
"The QUIC edge (CON-3)",
- "The private mesh (CON-13)",
- "Durable webhooks (CON-15)",
- "The OpenVibe Node plugin (CON-4)"
+ "The private mesh (CON-13)"
],
],
"capabilitiesRegistered": false,
"capabilitiesRegistered": false,
"serviceManifestRegistered": true,
"serviceManifestRegistered": true,
new file mode 100644
new file mode 100644
@@ -0,0 +1,26 @@
@@ -0,0 +1,26 @@
+-- phase: expand
+-- Arrival order: a share's waiting webhooks replay oldest first, including two that arrive in the same millisecond.
+ALTER TABLE hook_requests ADD COLUMN IF NOT EXISTS seq bigserial;
+CREATE INDEX IF NOT EXISTS hook_requests_seq ON hook_requests (hook_id, seq);
+
+-- The owner's delivery log (CON-15): what arrived at a webhook address, what the computer's app answered (or why it
+-- could not), and when. It never keeps a body or a provider header: the log says that a delivery happened, not what
+-- the provider said. Rows are removed 72 hours after they are made, with the webhooks they describe.
+CREATE TABLE IF NOT EXISTS hook_deliveries (
+ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+ share_id text NOT NULL REFERENCES shares(id) ON DELETE CASCADE,
+ hook_id text NOT NULL REFERENCES share_hooks(id) ON DELETE CASCADE,
+ owner_subject text NOT NULL, -- usr_…: the person's account export names these rows
+ request_id text, -- hb_… of the kept webhook; gone once it is delivered
+ event_id text, -- evt_… the app also sees in x-openvibe-event-id
+ method text NOT NULL,
+ path text NOT NULL, -- the path on the computer's app
+ status integer, -- the app's answer; null when it did not answer
+ error text, -- why a delivery failed, at most 200 characters
+ received_at timestamptz, -- when the provider's webhook first arrived
+ delivered_at timestamptz, -- null for a failed attempt
+ created_at timestamptz NOT NULL DEFAULT now()
+);
+CREATE INDEX IF NOT EXISTS hook_deliveries_share ON hook_deliveries (share_id, created_at DESC);
+CREATE INDEX IF NOT EXISTS hook_deliveries_hook ON hook_deliveries (hook_id, created_at DESC);
+CREATE INDEX IF NOT EXISTS hook_deliveries_owner ON hook_deliveries (owner_subject, created_at DESC);
@@ -146,6 +146,12 @@
@@ -146,6 +146,12 @@
.cn-keyuse { display: grid; gap: 2px; }
.cn-keyuse { display: grid; gap: 2px; }
.cn-hooks .cn-keyuse-list { list-style: none; margin: 0; padding: 0; display: flex; flex-wrap: wrap; gap: 2px 12px; font-size: 13px; }
.cn-hooks .cn-keyuse-list { list-style: none; margin: 0; padding: 0; display: flex; flex-wrap: wrap; gap: 2px 12px; font-size: 13px; }
.cn-hooks .cn-keyuse-list li { display: inline; padding: 0; border: 0; border-radius: 0; background: none; color: var(--ovk-muted); }
.cn-hooks .cn-keyuse-list li { display: inline; padding: 0; border: 0; border-radius: 0; background: none; color: var(--ovk-muted); }
+.cn-hooks .cn-delivery-log { flex-basis: 100%; font-size: 13px; color: var(--ovk-muted); }
+.cn-hooks .cn-delivery-log summary { cursor: pointer; }
+.cn-hooks .cn-delivery-log ul { list-style: none; margin: 6px 0 0; padding: 0; display: grid; gap: 3px; }
+.cn-hooks .cn-delivery-log li { padding: 0; border: 0; border-radius: 0; background: none; justify-content: flex-start; }
+.cn-hooks .cn-delivery-log b { color: var(--ovk-text); font-weight: 600; }
+
.cn-muted { color: var(--ovk-muted); }
.cn-muted { color: var(--ovk-muted); }
.cn-hook-add { display: flex; flex-wrap: wrap; align-items: end; gap: 10px 14px; }
.cn-hook-add { display: flex; flex-wrap: wrap; align-items: end; gap: 10px 14px; }
.cn-hook-add .cn-field { flex: 1 1 200px; }
.cn-hook-add .cn-field { flex: 1 1 200px; }
@@ -104,7 +104,7 @@ async function createApp(opts = {}) {
@@ -104,7 +104,7 @@ async function createApp(opts = {}) {
table: 'service_outbox', eventTypes: ['connect.hook.received'],
table: 'service_outbox', eventTypes: ['connect.hook.received'],
validate: (env) => contracts.validate('events.event-envelope@1', env),
validate: (env) => contracts.validate('events.event-envelope@1', env),
});
});
- const hooks = createHooks(s, { config, outbox, now: s.now });
+ const hooks = createHooks(s, { config, outbox, now: s.now, log });
const ctx = { config, s, keys, sso, principal, shares, tunnels, gate, proxy, apiKeys, oauth, notifier, nodeAuth, machines, hooks, outbox, accountData, accountSend, log };
const ctx = { config, s, keys, sso, principal, shares, tunnels, gate, proxy, apiKeys, oauth, notifier, nodeAuth, machines, hooks, outbox, accountData, accountSend, log };
ctx.probe = createProbe({ shares, proxy, config, log, now: s.now });
ctx.probe = createProbe({ shares, proxy, config, log, now: s.now });
@@ -113,10 +113,16 @@ async function createApp(opts = {}) {
@@ -113,10 +113,16 @@ async function createApp(opts = {}) {
/** A share's computer is connected: its waiting webhooks go to its app, one at a time, oldest first. → delivered count */
/** A share's computer is connected: its waiting webhooks go to its app, one at a time, oldest first. → delivered count */
ctx.deliverWaiting = async (share) => {
ctx.deliverWaiting = async (share) => {
let n = 0;
let n = 0;
- for (const q of await hooks.waiting(share.id)) {
- const out = await proxy.deliverHook(share, { method: q.method, path: q.path, headers: q.headers, body: q.body, hookId: q.hookId, eventId: q.eventId || q.ref });
+ for (const ref of await hooks.pending(share.id)) {
+ const q = await hooks.request(ref);
+ if (!q) continue;
+ const out = await proxy.deliverHook(share, { method: q.method, path: q.path, headers: q.headers, body: q.body, hookId: q.hookId, eventId: q.eventId || q.ref, receivedAt: q.receivedAt });
if (out.offline) break;
if (out.offline) break;
- if (out.status && out.status < 500 && await hooks.settle(q.ref, out.status)) n++;
+ if (!out.status || out.status >= 500) {
+ await hooks.fail(q.ref, out.status || null, out.error || `the app answered ${out.status}`);
+ continue;
+ }
+ if (await hooks.settle(q.ref, out.status)) n++;
}
}
if (n) log.log(`[OpenVibe.Connect] ${n} waiting webhook(s) handed to ${share.host}`);
if (n) log.log(`[OpenVibe.Connect] ${n} waiting webhook(s) handed to ${share.host}`);
return n;
return n;
@@ -128,8 +134,8 @@ async function createApp(opts = {}) {
@@ -128,8 +134,8 @@ async function createApp(opts = {}) {
try { due = await shares.expireDue(); } catch (err) { log.warn('[OpenVibe.Connect] expiry sweep failed:', err.message); return []; }
try { due = await shares.expireDue(); } catch (err) { log.warn('[OpenVibe.Connect] expiry sweep failed:', err.message); return []; }
due.forEach((r) => { tunnels.close(r.id, 4003, 'the share ended'); proxy.forget(r.host); proxy.inspector.forget(r.id); });
due.forEach((r) => { tunnels.close(r.id, 4003, 'the share ended'); proxy.forget(r.host); proxy.inspector.forget(r.id); });
if (due.length) log.log(`[OpenVibe.Connect] ${due.length} share(s) ended on time`);
if (due.length) log.log(`[OpenVibe.Connect] ${due.length} share(s) ended on time`);
- // Webhooks kept past their 7 days go, delivered or not (CON-15).
- try { const gone = await hooks.sweep(); if (gone) log.log(`[OpenVibe.Connect] ${gone} webhook(s) past 7 days removed`); } catch (err) { log.warn('[OpenVibe.Connect] webhook sweep failed:', err.message); }
+ // Webhooks and delivery-log rows past their 72 hours go, delivered or not (CON-15).
+ try { const gone = await hooks.sweep(); if (gone) log.log(`[OpenVibe.Connect] ${gone} webhook(s) past 72 hours removed`); } catch (err) { log.warn('[OpenVibe.Connect] webhook sweep failed:', err.message); }
return due;
return due;
};
};
if (opts.expirySweep !== false) {
if (opts.expirySweep !== false) {
@@ -455,10 +455,11 @@ function createProxy({ config, shares, tunnels, gate, apiKeys = null, oauth = nu
@@ -455,10 +455,11 @@ function createProxy({ config, shares, tunnels, gate, apiKeys = null, oauth = nu
/**
/**
* One stored webhook to the share's computer (CON-15): the request as it arrived, at the hook's path, with
* One stored webhook to the share's computer (CON-15): the request as it arrived, at the hook's path, with
- * x-openvibe-hook and x-openvibe-event-id so the app can tell and dedupe. The answer's body is read and dropped.
+ * x-openvibe-hook, x-openvibe-event-id and x-openvibe-received-at (the original arrival time) so the app can tell,
+ * dedupe and preserve order. The answer's body is read and dropped.
* → { status } when the app answered, { offline: true } with no computer, { error } when it broke on the way.
* → { status } when the app answered, { offline: true } with no computer, { error } when it broke on the way.
*/
*/
- function deliverHook(share, { method, path, headers = [], body, hookId, eventId }) {
+ function deliverHook(share, { method, path, headers = [], body, hookId, eventId, receivedAt }) {
const tunnel = tunnels.get(share.id);
const tunnel = tunnels.get(share.id);
if (!tunnel) return Promise.resolve({ offline: true });
if (!tunnel) return Promise.resolve({ offline: true });
const port = (tunnel.clientInfo && tunnel.clientInfo.port) || share.local_port;
const port = (tunnel.clientInfo && tunnel.clientInfo.port) || share.local_port;
@@ -478,7 +479,9 @@ function createProxy({ config, shares, tunnels, gate, apiKeys = null, oauth = nu
@@ -478,7 +479,9 @@ function createProxy({ config, shares, tunnels, gate, apiKeys = null, oauth = nu
timer.unref();
timer.unref();
const out = (Array.isArray(headers) ? headers : []).filter((h) => Array.isArray(h) && h.length === 2 && !/^(host|content-length|x-openvibe-)/i.test(String(h[0])));
const out = (Array.isArray(headers) ? headers : []).filter((h) => Array.isArray(h) && h.length === 2 && !/^(host|content-length|x-openvibe-)/i.test(String(h[0])));
out.push(['host', `localhost:${port}`], ['x-forwarded-host', share.host], ['x-forwarded-proto', config.shares.scheme],
out.push(['host', `localhost:${port}`], ['x-forwarded-host', share.host], ['x-forwarded-proto', config.shares.scheme],
- ['x-openvibe-hook', String(hookId || '')], ['x-openvibe-event-id', String(eventId || '')], ['content-length', String(body ? body.length : 0)]);
+ ['x-openvibe-hook', String(hookId || '')], ['x-openvibe-event-id', String(eventId || '')],
+ ['x-openvibe-received-at', receivedAt ? new Date(receivedAt).toISOString() : new Date().toISOString()],
+ ['content-length', String(body ? body.length : 0)]);
tunnel.send({ t: 'req', id, method, path, headers: out });
tunnel.send({ t: 'req', id, method, path, headers: out });
if (body && body.length) {
if (body && body.length) {
rec.in += body.length;
rec.in += body.length;
@@ -3,15 +3,16 @@
@@ -3,15 +3,16 @@
/**
/**
* Hooks that wait for your computer (CON-15, ADR-057 decision 7). A hook is an address a provider sends webhooks to,
* Hooks that wait for your computer (CON-15, ADR-057 decision 7). A hook is an address a provider sends webhooks to,
* https://connect.openvibe.website/h/<hk_…>, bound to one share and a path on the computer's app. A webhook is kept
* https://connect.openvibe.website/h/<hk_…>, bound to one share and a path on the computer's app. A webhook is kept
- * (7 days) and a pointer to it is published as connect.hook.received through the outbox, in one transaction. Events
- * delivers it to Connect's own subscription (retried for up to seven days), and Connect hands it to the share's
- * computer when it is connected. The computer may be asleep for days and still get every webhook, in order of arrival
- * or not, each once it answers.
+ * (72 hours) and a pointer to it is published as connect.hook.received through the outbox, in one transaction. Events
+ * delivers it to Connect's own subscription (retried for up to 72 hours), and Connect hands it to the share's
+ * computer when it is connected. The computer may be asleep for days and still get every webhook, in order of arrival,
+ * each once it answers. A share's queue holds at most 1,000 undelivered webhooks, and each body is sealed at rest.
*
*
* create(owner, share, { path, label }) → hook list(shareId) → hooks remove(owner, id)
* create(owner, share, { path, label }) → hook list(shareId) → hooks remove(owner, id)
* get(id) receive(hook, { method, headers, body }) → { ref }
* get(id) receive(hook, { method, headers, body }) → { ref }
- * request(ref) → the stored request with its hook settle(ref, status) sweep() → rows removed
- * waiting(shareId) → its undelivered requests, oldest first (handed over at once when its computer connects)
+ * request(ref) → the stored request with its hook settle(ref, status) fail(ref, status, error)
+ * pending(shareId) → its undelivered refs, oldest first (handed over one at a time when its computer connects)
+ * deliveries(shareId, hookId?) → the owner's delivery log sweep() → rows removed
*/
*/
const crypto = require('crypto');
const crypto = require('crypto');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
@@ -20,24 +21,61 @@ const HOOK_ID = /^hk_[0-9A-HJKMNP-TV-Z]{26}$/;
@@ -20,24 +21,61 @@ const HOOK_ID = /^hk_[0-9A-HJKMNP-TV-Z]{26}$/;
const REF_RE = /^hb_[A-Za-z0-9_-]{16,64}$/;
const REF_RE = /^hb_[A-Za-z0-9_-]{16,64}$/;
const PATH_RE = /^\/[\x21-\x7e]{0,499}$/;
const PATH_RE = /^\/[\x21-\x7e]{0,499}$/;
const METHODS = ['POST', 'PUT', 'PATCH'];
const METHODS = ['POST', 'PUT', 'PATCH'];
-const KEEP_MS = 7 * 86400_000;
+const KEEP_MS = 72 * 3600_000;
const MAX_BODY = 1024 * 1024;
const MAX_BODY = 1024 * 1024;
+const MAX_WAITING = 1000;
const PER_SHARE = 10;
const PER_SHARE = 10;
-// The provider headers a webhook keeps: what apps check signatures and events with. Never cookies, authorization,
-// forwarding or Cloudflare headers, and never Connect's own.
-const DROP = /^(x-forwarded-|x-real-ip$|x-openvibe-|cf-|x-amz-cf-)/;
-const KEEP = /^(content-type|content-encoding|user-agent|stripe-signature|svix-.+|webhook-.+|x-.+)$/;
+// The request headers a webhook keeps: everything but the hop-by-hop headers, forwarding and Cloudflare headers, and
+// credentials (a provider's cookie or Authorization header is not the app's). Signatures and event names are kept.
+const DROP = new Set(['connection', 'keep-alive', 'proxy-authenticate', 'proxy-authorization', 'te', 'trailer', 'transfer-encoding',
+ 'upgrade', 'host', 'content-length', 'cookie', 'cookie2', 'authorization', 'forwarded', 'x-real-ip', 'cdn-loop', 'true-client-ip']);
+const DROP_RE = /^(x-forwarded-|x-openvibe-|cf-|x-amz-cf-)/;
+
+// A sealed body starts with this magic; a row written before sealing has no magic and is read as it is until the next
+// sweep seals it.
+const SEAL_MAGIC = Buffer.from('OVC1');
+const SEAL_IV = 12;
+const SEAL_TAG = 16;
class HookError extends Error {
class HookError extends Error {
constructor(status, code, message) { super(message); this.status = status; this.code = code; }
constructor(status, code, message) { super(message); this.status = status; this.code = code; }
}
}
+/** The key that seals a webhook body: derived from the session secret, never stored. */
+function bodyKey(secret) {
+ if (!secret) throw new Error('CONNECT_SESSION_SECRET is not set: a webhook body cannot be sealed');
+ return Buffer.from(crypto.hkdfSync('sha256', Buffer.from(String(secret)), Buffer.from('openvibe-connect'), 'hook-body-v1', 32));
+}
+
+function sealBody(key, hookId, ref, body) {
+ const iv = crypto.randomBytes(SEAL_IV);
+ const cipher = crypto.createCipheriv('aes-256-gcm', key, iv);
+ cipher.setAAD(Buffer.from(`connect.hook|${hookId}|${ref}`));
+ const sealed = Buffer.concat([cipher.update(body), cipher.final()]);
+ return Buffer.concat([SEAL_MAGIC, iv, cipher.getAuthTag(), sealed]);
+}
+
+/** → { body, sealed }. A body written before sealing has no magic and is returned as it is. */
+function openBody(key, hookId, ref, blob) {
+ const b = Buffer.from(blob || []);
+ if (b.length < SEAL_MAGIC.length + SEAL_IV + SEAL_TAG || !b.subarray(0, SEAL_MAGIC.length).equals(SEAL_MAGIC)) {
+ return { body: b, sealed: false };
+ }
+ const iv = b.subarray(SEAL_MAGIC.length, SEAL_MAGIC.length + SEAL_IV);
+ const tag = b.subarray(SEAL_MAGIC.length + SEAL_IV, SEAL_MAGIC.length + SEAL_IV + SEAL_TAG);
+ const sealed = b.subarray(SEAL_MAGIC.length + SEAL_IV + SEAL_TAG);
+ const decipher = crypto.createDecipheriv('aes-256-gcm', key, iv);
+ decipher.setAAD(Buffer.from(`connect.hook|${hookId}|${ref}`));
+ decipher.setAuthTag(tag);
+ return { body: Buffer.concat([decipher.update(sealed), decipher.final()]), sealed: true };
+}
+
/** The headers a stored webhook keeps: at most 40, each at most 2 KB. → [[name, value]] */
/** The headers a stored webhook keeps: at most 40, each at most 2 KB. → [[name, value]] */
function keepHeaders(headers) {
function keepHeaders(headers) {
const out = [];
const out = [];
for (const [k, v] of Object.entries(headers || {})) {
for (const [k, v] of Object.entries(headers || {})) {
const name = String(k).toLowerCase();
const name = String(k).toLowerCase();
- if (DROP.test(name) || !KEEP.test(name)) continue;
+ if (DROP.has(name) || DROP_RE.test(name)) continue;
const value = Array.isArray(v) ? v.join(', ') : String(v);
const value = Array.isArray(v) ? v.join(', ') : String(v);
if (value.length > 2048) continue;
if (value.length > 2048) continue;
out.push([name, value]);
out.push([name, value]);
@@ -46,15 +84,22 @@ function keepHeaders(headers) {
@@ -46,15 +84,22 @@ function keepHeaders(headers) {
return out;
return out;
}
}
-function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
+function createHooks(s, { config, outbox = null, now = () => Date.now(), log = console } = {}) {
const { db } = s;
const { db } = s;
const base = config.baseUrl;
const base = config.baseUrl;
+ const key = bodyKey(config.shares && config.shares.sessionSecret);
const iso = (v) => (v ? new Date(v).toISOString() : null);
const iso = (v) => (v ? new Date(v).toISOString() : null);
- const view = (r) => r && ({
+ const view = (r, recent) => r && ({
id: r.id, share_id: r.share_id, url: `${base}/h/${r.id}`, local_path: r.local_path, label: r.label || null, status: r.status,
id: r.id, share_id: r.share_id, url: `${base}/h/${r.id}`, local_path: r.local_path, label: r.label || null, status: r.status,
created_at: iso(r.created_at), received: Number(r.received || 0), delivered: Number(r.delivered || 0),
created_at: iso(r.created_at), received: Number(r.received || 0), delivered: Number(r.delivered || 0),
last_received_at: iso(r.last_received_at), last_delivered_at: iso(r.last_delivered_at), last_status: r.last_status == null ? null : Number(r.last_status),
last_received_at: iso(r.last_received_at), last_delivered_at: iso(r.last_delivered_at), last_status: r.last_status == null ? null : Number(r.last_status),
waiting: r.waiting == null ? undefined : Number(r.waiting),
waiting: r.waiting == null ? undefined : Number(r.waiting),
+ deliveries: recent === undefined ? undefined : recent,
+ });
+ const delivery = (d) => ({
+ id: Number(d.id), hook_id: d.hook_id, request_id: d.request_id || null, event_id: d.event_id || null,
+ method: d.method, path: d.path, status: d.status == null ? null : Number(d.status), error: d.error || null,
+ received_at: iso(d.received_at), delivered_at: iso(d.delivered_at), created_at: iso(d.created_at),
});
});
async function get(id) {
async function get(id) {
@@ -65,7 +110,15 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
@@ -65,7 +110,15 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
async function list(shareId) {
async function list(shareId) {
const rows = await db.many(`SELECT h.*, (SELECT count(*)::int FROM hook_requests r WHERE r.hook_id = h.id AND r.delivered_at IS NULL AND r.expires_at > now()) AS waiting
const rows = await db.many(`SELECT h.*, (SELECT count(*)::int FROM hook_requests r WHERE r.hook_id = h.id AND r.delivered_at IS NULL AND r.expires_at > now()) AS waiting
FROM share_hooks h WHERE h.share_id = $1 AND h.status = 'active' ORDER BY h.created_at`, [shareId]);
FROM share_hooks h WHERE h.share_id = $1 AND h.status = 'active' ORDER BY h.created_at`, [shareId]);
- return rows.map(view);
+ const recent = await db.many(`SELECT id, hook_id, request_id, event_id, method, path, status, error, received_at, delivered_at, created_at
+ FROM hook_deliveries WHERE share_id = $1 ORDER BY created_at DESC, id DESC LIMIT 100`, [shareId]);
+ const byHook = new Map();
+ for (const d of recent) {
+ const arr = byHook.get(d.hook_id) || [];
+ if (arr.length < 5) arr.push(delivery(d));
+ byHook.set(d.hook_id, arr);
+ }
+ return rows.map((r) => view(r, byHook.get(r.id) || []));
}
}
async function create(owner, share, input = {}) {
async function create(owner, share, input = {}) {
@@ -89,7 +142,7 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
@@ -89,7 +142,7 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
return view({ ...hook, status: 'removed' });
return view({ ...hook, status: 'removed' });
}
}
- /** A webhook arrived: kept, and its pointer published, together. → { ref } */
+ /** A webhook arrived: sealed, queued under the share's cap, and its pointer published, together. → { ref } */
async function receive(hook, { method, headers, body }) {
async function receive(hook, { method, headers, body }) {
if (!METHODS.includes(method)) throw new HookError(405, 'hook.method', 'A webhook address takes POST, PUT or PATCH.');
if (!METHODS.includes(method)) throw new HookError(405, 'hook.method', 'A webhook address takes POST, PUT or PATCH.');
const buf = Buffer.isBuffer(body) ? body : Buffer.from(body == null ? '' : String(body));
const buf = Buffer.isBuffer(body) ? body : Buffer.from(body == null ? '' : String(body));
@@ -101,9 +154,15 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
@@ -101,9 +154,15 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
// The event's id is chosen here and kept with the request: the app sees the same x-openvibe-event-id however
// The event's id is chosen here and kept with the request: the app sees the same x-openvibe-event-id however
// the webhook reaches it (handed over on connect, or Events' retry).
// the webhook reaches it (handed over on connect, or Events' retry).
const eventId = ids.newId('event');
const eventId = ids.newId('event');
+ const sealed = sealBody(key, hook.id, ref, buf);
await s.tx(async (t) => {
await s.tx(async (t) => {
+ // One row lock per share makes the queue cap exact while two providers post at once.
+ await t.exec('SELECT id FROM shares WHERE id = $1 FOR UPDATE', [hook.share_id]);
+ const q = await t.one(`SELECT count(*)::int AS n FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_id
+ WHERE h.share_id = $1 AND r.delivered_at IS NULL AND r.expires_at > $2`, [hook.share_id, at.toISOString()]);
+ if (q.n >= MAX_WAITING) throw new HookError(429, 'hook.queue_full', `This share is holding ${MAX_WAITING} webhooks already; they go to the computer first.`);
await t.exec('INSERT INTO hook_requests (id, hook_id, method, headers, body, received_at, expires_at, event_id) VALUES ($1, $2, $3, $4::jsonb, $5, $6, $7, $8)',
await t.exec('INSERT INTO hook_requests (id, hook_id, method, headers, body, received_at, expires_at, event_id) VALUES ($1, $2, $3, $4::jsonb, $5, $6, $7, $8)',
- [ref, hook.id, method, JSON.stringify(kept), buf, at.toISOString(), new Date(at.getTime() + KEEP_MS).toISOString(), eventId]);
+ [ref, hook.id, method, JSON.stringify(kept), sealed, at.toISOString(), new Date(at.getTime() + KEEP_MS).toISOString(), eventId]);
await t.exec('UPDATE share_hooks SET received = received + 1, last_received_at = $2 WHERE id = $1', [hook.id, at.toISOString()]);
await t.exec('UPDATE share_hooks SET received = received + 1, last_received_at = $2 WHERE id = $1', [hook.id, at.toISOString()]);
if (outbox) {
if (outbox) {
await outbox.emitIn(t, {
await outbox.emitIn(t, {
@@ -118,40 +177,91 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
@@ -118,40 +177,91 @@ function createHooks(s, { config, outbox = null, now = () => Date.now() }) {
return { ref };
return { ref };
}
}
- /** The stored request a delivery points at, with its hook; null when it was delivered, removed or expired. */
+ /** The stored request a delivery points at, with its hook and its original arrival time; null when it is gone. */
async function request(ref) {
async function request(ref) {
if (!REF_RE.test(String(ref || ''))) return null;
if (!REF_RE.test(String(ref || ''))) return null;
const r = await db.maybe(`SELECT r.*, h.share_id, h.local_path, h.status AS hook_status FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_id
const r = await db.maybe(`SELECT r.*, h.share_id, h.local_path, h.status AS hook_status FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_id
WHERE r.id = $1`, [ref]);
WHERE r.id = $1`, [ref]);
if (!r || r.hook_status !== 'active' || r.delivered_at || new Date(r.expires_at).getTime() <= now()) return null;
if (!r || r.hook_status !== 'active' || r.delivered_at || new Date(r.expires_at).getTime() <= now()) return null;
- return { ref: r.id, hookId: r.hook_id, shareId: r.share_id, path: r.local_path, method: r.method, headers: Array.isArray(r.headers) ? r.headers : [], body: r.body, eventId: r.event_id || null };
+ let body;
+ try { body = openBody(key, r.hook_id, r.id, r.body).body; }
+ catch (err) {
+ log.warn('[OpenVibe.Connect] a webhook body cannot be opened:', err && err.message);
+ return null;
+ }
+ return { ref: r.id, hookId: r.hook_id, shareId: r.share_id, path: r.local_path, method: r.method, headers: Array.isArray(r.headers) ? r.headers : [], body, eventId: r.event_id || null, receivedAt: iso(r.received_at) };
}
}
- /** A share's webhooks still waiting for its computer, oldest first (at most `limit`). */
- async function waiting(shareId, limit = 100) {
+ /**
+ * A share's webhooks still waiting for its computer, oldest first, up to the share's 1,000 cap. It returns refs,
+ * not bodies: the caller opens one at a time, so a full queue never sits in memory at once.
+ */
+ async function pending(shareId, limit = MAX_WAITING) {
const rows = await db.many(`SELECT r.id FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_id
const rows = await db.many(`SELECT r.id FROM hook_requests r JOIN share_hooks h ON h.id = r.hook_id
WHERE h.share_id = $1 AND h.status = 'active' AND r.delivered_at IS NULL AND r.expires_at > $2
WHERE h.share_id = $1 AND h.status = 'active' AND r.delivered_at IS NULL AND r.expires_at > $2
- ORDER BY r.received_at LIMIT $3`, [shareId, new Date(now()).toISOString(), limit]);
- const out = [];
- for (const r of rows) { const q = await request(r.id); if (q) out.push(q); }
- return out;
+ ORDER BY r.seq LIMIT $3`, [shareId, new Date(now()).toISOString(), Math.min(Number(limit) || MAX_WAITING, MAX_WAITING)]);
+ return rows.map((r) => r.id);
}
}
- /** The computer's app answered: the request is delivered (its body is dropped) and the hook counts it. */
+ /** The computer's app answered: the request is delivered (its body is dropped), logged and counted. */
async function settle(ref, status) {
async function settle(ref, status) {
+ let settled = false;
+ await s.tx(async (t) => {
+ const at = new Date(now()).toISOString();
+ const r = await t.maybe('UPDATE hook_requests SET delivered_at = $2, status = $3, body = \'\'::bytea WHERE id = $1 AND delivered_at IS NULL RETURNING hook_id, method, event_id, received_at', [ref, at, status]);
+ if (!r) return;
+ const h = await t.maybe('SELECT share_id, owner_subject, local_path FROM share_hooks WHERE id = $1', [r.hook_id]);
+ if (!h) return;
+ await t.exec(`INSERT INTO hook_deliveries (share_id, hook_id, owner_subject, request_id, event_id, method, path, status, received_at, delivered_at, created_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $9, $9)`,
+ [h.share_id, r.hook_id, h.owner_subject, ref, r.event_id || null, r.method, h.local_path, status, r.received_at || at]);
+ await t.exec('UPDATE share_hooks SET delivered = delivered + 1, last_delivered_at = $2, last_status = $3 WHERE id = $1', [r.hook_id, at, status]);
+ settled = true;
+ });
+ return settled;
+ }
+
+ /** The computer's app refused a webhook: the request stays queued for another try, and the owner sees why. */
+ async function fail(ref, status, error) {
+ const r = await db.maybe('SELECT hook_id, method, event_id, received_at FROM hook_requests WHERE id = $1', [ref]);
+ if (!r) return false;
+ const h = await db.maybe('SELECT share_id, owner_subject, local_path FROM share_hooks WHERE id = $1', [r.hook_id]);
+ if (!h) return false;
const at = new Date(now()).toISOString();
const at = new Date(now()).toISOString();
- const r = await db.maybe("UPDATE hook_requests SET delivered_at = $2, status = $3, body = ''::bytea WHERE id = $1 AND delivered_at IS NULL RETURNING hook_id", [ref, at, status]);
- if (r) await db.exec('UPDATE share_hooks SET delivered = delivered + 1, last_delivered_at = $2, last_status = $3 WHERE id = $1', [r.hook_id, at, status]);
- return Boolean(r);
+ const why = String(error || '').slice(0, 200) || `the app answered ${status || 'nothing'}`;
+ await db.exec(`INSERT INTO hook_deliveries (share_id, hook_id, owner_subject, request_id, event_id, method, path, status, error, received_at, created_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $10)`,
+ [h.share_id, r.hook_id, h.owner_subject, ref, r.event_id || null, r.method, h.local_path, status || null, why, r.received_at || at]);
+ await db.exec('UPDATE share_hooks SET last_status = $2 WHERE id = $1', [r.hook_id, status || null]);
+ return true;
+ }
+
+ /** The owner's delivery log, newest first: what arrived, what the app answered, and when. */
+ async function deliveries(shareId, hookId = null, limit = 50) {
+ const rows = hookId
+ ? await db.many('SELECT * FROM hook_deliveries WHERE share_id = $1 AND hook_id = $2 ORDER BY created_at DESC, id DESC LIMIT $3', [shareId, hookId, Math.max(1, Math.min(Math.trunc(Number(limit)) || 50, 200))])
+ : await db.many('SELECT * FROM hook_deliveries WHERE share_id = $1 ORDER BY created_at DESC, id DESC LIMIT $2', [shareId, Math.max(1, Math.min(Math.trunc(Number(limit)) || 50, 200))]);
+ return rows.map(delivery);
+ }
+
+ /** Bodies written before sealing are sealed now, so nothing readable is left at rest. → sealed count */
+ async function sealLegacy() {
+ const rows = await db.many("SELECT id, hook_id, body FROM hook_requests WHERE body <> ''::bytea AND encode(substr(body, 1, 4), 'hex') <> '4f564331'");
+ for (const r of rows) {
+ await db.exec('UPDATE hook_requests SET body = $2 WHERE id = $1', [r.id, sealBody(key, r.hook_id, r.id, Buffer.from(r.body))]);
+ }
+ return rows.length;
}
}
- /** Requests older than 7 days go, delivered or not. */
+ /** Requests and delivery-log rows older than 72 hours go, delivered or not. → removed request count */
async function sweep() {
async function sweep() {
+ await sealLegacy();
+ await db.exec('DELETE FROM hook_deliveries WHERE created_at <= $1', [new Date(now() - KEEP_MS).toISOString()]);
const r = await db.many('DELETE FROM hook_requests WHERE expires_at <= $1 RETURNING id', [new Date(now()).toISOString()]);
const r = await db.many('DELETE FROM hook_requests WHERE expires_at <= $1 RETURNING id', [new Date(now()).toISOString()]);
return r.length;
return r.length;
}
}
- return { get, list, create, remove, receive, request, waiting, settle, sweep, KEEP_MS, MAX_BODY };
+ return { get, list, create, remove, receive, request, pending, settle, fail, deliveries, sealLegacy, sweep, KEEP_MS, MAX_BODY, MAX_WAITING };
}
}
-module.exports = { createHooks, HookError, keepHeaders, HOOK_ID, METHODS, MAX_BODY, KEEP_MS };
+module.exports = { createHooks, HookError, keepHeaders, bodyKey, HOOK_ID, METHODS, MAX_BODY, MAX_WAITING, KEEP_MS };
@@ -15,9 +15,10 @@
@@ -15,9 +15,10 @@
* PUT /shares/:id/node { node: 'nod_…' } one of your computers serves it, as OpenVibe Node (CON-4): the key
* PUT /shares/:id/node { node: 'nod_…' } one of your computers serves it, as OpenVibe Node (CON-4): the key
* stops working → { share }; { node: null } lets the computer go → { share, key } (a new key)
* stops working → { share }; { node: null } lets the computer go → { share, key } (a new key)
* GET /machines your computers that can serve a share (Network's list of your machines)
* GET /machines your computers that can serve a share (Network's list of your machines)
- * GET /shares/:id/hooks its webhook addresses (CON-15): each kept webhook waits up to 7 days for the computer
+ * GET /shares/:id/hooks its webhook addresses (CON-15): each kept webhook waits up to 72 hours for the computer
* POST /shares/:id/hooks { path, label? } → 201 { hook }: https://connect.openvibe.website/h/<id> → that path
* POST /shares/:id/hooks { path, label? } → 201 { hook }: https://connect.openvibe.website/h/<id> → that path
* DELETE /shares/:id/hooks/:hook removes it; webhooks still waiting are dropped
* DELETE /shares/:id/hooks/:hook removes it; webhooks still waiting are dropped
+ * GET /shares/:id/hooks/:hook/deliveries the owner's log: what the app answered, and when
* GET /shares/:id/visits who opened it in the last 7 days
* GET /shares/:id/visits who opened it in the last 7 days
* GET /shares/:id/requests the last 100 requests through it, newest first (?after=<seq>: only newer ones);
* GET /shares/:id/requests the last 100 requests through it, newest first (?after=<seq>: only newer ones);
* memory only, query values shown as "…" (edge/inspector.js)
* memory only, query values shown as "…" (edge/inspector.js)
@@ -137,6 +138,13 @@ function createApi(ctx) {
@@ -137,6 +138,13 @@ function createApi(ctx) {
if (!hook || hook.share_id !== share.id) return fail(res, req, 404, 'hook.not_found', 'No such webhook address.');
if (!hook || hook.share_id !== share.id) return fail(res, req, 404, 'hook.not_found', 'No such webhook address.');
return res.json({ hook: await hooks.remove(owner(req), hook.id) });
return res.json({ hook: await hooks.remove(owner(req), hook.id) });
}));
}));
+ r.get('/shares/:id/hooks/:hook/deliveries', guard(async (req, res) => {
+ const share = await mine(req, res);
+ if (!share) return undefined;
+ const hook = await hooks.get(req.params.hook);
+ if (!hook || hook.share_id !== share.id) return fail(res, req, 404, 'hook.not_found', 'No such webhook address.');
+ return res.json({ deliveries: await hooks.deliveries(share.id, hook.id, req.query.limit) });
+ }));
// API keys (CON-16 v0): a program passes the gate with one. The key is in the answer that makes it, never again.
// API keys (CON-16 v0): a program passes the gate with one. The key is in the answer that makes it, never again.
r.get('/shares/:id/api-keys', guard(async (req, res) => {
r.get('/shares/:id/api-keys', guard(async (req, res) => {
@@ -3,11 +3,11 @@
@@ -3,11 +3,11 @@
/**
/**
* Hooks that wait for your computer (CON-15): the address providers send to, and Events' deliveries back.
* Hooks that wait for your computer (CON-15): the address providers send to, and Events' deliveries back.
*
*
- * POST|PUT|PATCH /h/:id a provider's webhook (1 MB): kept 7 days, its pointer published → 202
+ * POST|PUT|PATCH /h/:id a provider's webhook (1 MB): sealed, kept 72 hours, its pointer published → 202
* POST /internal/hook-deliveries Events, signed v2, loopback only (nginx answers 404 for /internal/): the
* POST /internal/hook-deliveries Events, signed v2, loopback only (nginx answers 404 for /internal/): the
* webhook goes to the share's computer. 200 once its app answered (any status
* webhook goes to the share's computer. 200 once its app answered (any status
* below 500) or when the webhook is gone; 503 while the computer is offline or
* below 500) or when the webhook is gone; 503 while the computer is offline or
- * its app fails, so Events tries again (up to seven days, server/index.js).
+ * its app fails, so Events tries again (up to 72 hours, server/index.js).
*/
*/
const express = require('express');
const express = require('express');
const { parseDelivery } = require('openvibe-sdk/events');
const { parseDelivery } = require('openvibe-sdk/events');
@@ -39,9 +39,12 @@ function createHookRoutes(ctx) {
@@ -39,9 +39,12 @@ function createHookRoutes(ctx) {
if (tooMany(hook.id)) return answer(res.set('Retry-After', '60'), 429, { error: 'rate_limited', detail: `At most ${PER_HOOK_MINUTE} webhooks a minute to one address.` });
if (tooMany(hook.id)) return answer(res.set('Retry-After', '60'), 429, { error: 'rate_limited', detail: `At most ${PER_HOOK_MINUTE} webhooks a minute to one address.` });
try {
try {
const { ref } = await hooks.receive(hook, { method: req.method, headers: req.headers, body: Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0) });
const { ref } = await hooks.receive(hook, { method: req.method, headers: req.headers, body: Buffer.isBuffer(req.body) ? req.body : Buffer.alloc(0) });
- return answer(res, 202, { received: true, id: ref, delivered_when: 'the computer is connected (kept up to 7 days)' });
+ return answer(res, 202, { received: true, id: ref, delivered_when: 'the computer is connected (kept up to 72 hours)' });
} catch (err) {
} catch (err) {
- if (err instanceof HookError) return answer(res, err.status, { error: err.code, detail: err.message });
+ if (err instanceof HookError) {
+ if (err.code === 'hook.queue_full') res.set('Retry-After', '300');
+ return answer(res, err.status, { error: err.code, detail: err.message });
+ }
log.warn('[OpenVibe.Connect] webhook not kept:', err && err.message);
log.warn('[OpenVibe.Connect] webhook not kept:', err && err.message);
return answer(res.set('Retry-After', '30'), 503, { error: 'unavailable', detail: 'Try again shortly.' });
return answer(res.set('Retry-After', '30'), 503, { error: 'unavailable', detail: 'Try again shortly.' });
}
}
@@ -61,9 +64,12 @@ function createHookRoutes(ctx) {
@@ -61,9 +64,12 @@ function createHookRoutes(ctx) {
if (!stored) return res.status(200).json({ gone: true });
if (!stored) return res.status(200).json({ gone: true });
const share = await shares.get(stored.shareId);
const share = await shares.get(stored.shareId);
if (!share || share.status !== 'active') return res.status(200).json({ gone: true });
if (!share || share.status !== 'active') return res.status(200).json({ gone: true });
- const out = await proxy.deliverHook(share, { method: stored.method, path: stored.path, headers: stored.headers, body: stored.body, hookId: stored.hookId, eventId: e.event_id });
+ const out = await proxy.deliverHook(share, { method: stored.method, path: stored.path, headers: stored.headers, body: stored.body, hookId: stored.hookId, eventId: e.event_id, receivedAt: stored.receivedAt });
if (out.offline) return res.status(503).set('Retry-After', '300').json({ waiting: 'the computer is not connected' });
if (out.offline) return res.status(503).set('Retry-After', '300').json({ waiting: 'the computer is not connected' });
- if (!out.status || out.status >= 500) return res.status(503).json({ retry: out.error || `the app answered ${out.status}` });
+ if (!out.status || out.status >= 500) {
+ await hooks.fail(stored.ref, out.status || null, out.error || `the app answered ${out.status}`);
+ return res.status(503).json({ retry: out.error || `the app answered ${out.status}` });
+ }
await hooks.settle(stored.ref, out.status);
await hooks.settle(stored.ref, out.status);
return res.status(200).json({ delivered: true, status: out.status });
return res.status(200).json({ delivered: true, status: out.status });
});
});
@@ -18,6 +18,9 @@ const TABLES = [
@@ -18,6 +18,9 @@ const TABLES = [
// the account is taken off it.
// the account is taken off it.
{ table: 'share_reports', subject: 'reporter_subject', value: (usr) => usr, file: 'reports.json', erase: { anonymize: {} },
{ table: 'share_reports', subject: 'reporter_subject', value: (usr) => usr, file: 'reports.json', erase: { anonymize: {} },
columns: ['id', 'host', 'reason', 'details', 'status', 'created_at'] },
columns: ['id', 'host', 'reason', 'details', 'status', 'created_at'] },
+ // The delivery log of a person's webhook addresses (CON-15): method, path and the app's answer, never a body.
+ { table: 'hook_deliveries', subject: 'owner_subject', value: (usr) => usr, file: 'webhook-deliveries.json', erase: 'delete',
+ columns: ['id', 'share_id', 'hook_id', 'request_id', 'event_id', 'method', 'path', 'status', 'error', 'received_at', 'delivered_at', 'created_at'] },
// Webhook addresses on a person's shares (CON-15). Their waiting webhooks (hook_requests) go with them, and with
// Webhook addresses on a person's shares (CON-15). Their waiting webhooks (hook_requests) go with them, and with
// the shares (ON DELETE CASCADE); the export names the addresses and their counts, never a webhook's body.
// the shares (ON DELETE CASCADE); the export names the addresses and their counts, never a webhook's body.
{ table: 'share_hooks', subject: 'owner_subject', value: (usr) => usr, file: 'webhooks.json', erase: 'delete',
{ table: 'share_hooks', subject: 'owner_subject', value: (usr) => usr, file: 'webhooks.json', erase: 'delete',
@@ -41,7 +44,7 @@ function create({ db, log = console } = {}) {
@@ -41,7 +44,7 @@ function create({ db, log = console } = {}) {
db,
db,
service: 'connect',
service: 'connect',
tables: TABLES,
tables: TABLES,
- note: 'Your shares on OpenVibe Connect, their webhook addresses, and your visits to other people\'s shares. Connect never stores what passed through a share; a webhook to one of your addresses is kept only until your computer takes it, at most 7 days.',
+ note: 'Your shares on OpenVibe Connect, their webhook addresses, and your visits to other people\'s shares. Connect never stores what passed through a share; a webhook to one of your addresses is kept only until your computer takes it, at most 72 hours.',
log,
log,
});
});
}
}
@@ -61,7 +61,7 @@ async function start() {
@@ -61,7 +61,7 @@ async function start() {
log: ctx.log || console,
log: ctx.log || console,
});
});
// Hooks (CON-15): the outbox relay publishes webhook pointers, and Connect's own subscription brings them back to
// Hooks (CON-15): the outbox relay publishes webhook pointers, and Connect's own subscription brings them back to
- // /internal/hook-deliveries, retried for about seven days while a computer is offline. Off in drill mode and
+ // /internal/hook-deliveries, retried for about 72 hours while a computer is offline. Off in drill mode and
// without the events URL, the delivery secret and the client secret.
// without the events URL, the delivery secret and the client secret.
const hookSubs = config.drill ? null : ensureHookSubscription({ config, log: ctx.log || console });
const hookSubs = config.drill ? null : ensureHookSubscription({ config, log: ctx.log || console });
if (!config.drill && ctx.outbox) ctx.outbox.start();
if (!config.drill && ctx.outbox) ctx.outbox.start();
@@ -70,10 +70,10 @@ async function start() {
@@ -70,10 +70,10 @@ async function start() {
return { server, ctx };
return { server, ctx };
}
}
-// About seven days of retries (Events allows 20 attempts, each wait at most a day): 5 s, 30 s, 2 min, 10 min, 30 min,
-// 1, 2, 3, 4, 6, 8 hours, four of 12 hours and four of a day.
+// About 72 hours of retries (Events allows 20 attempts): 5 s, 30 s, 2 min, 10 min, 30 min, 1, 2, 3, 4, 6, 8 hours,
+// four of 4 hours, three of 6 hours and two of 8 hours.
const HOOK_BACKOFF_MS = [5e3, 30e3, 120e3, 600e3, 1800e3, 3600e3, 7200e3, 10800e3, 14400e3, 21600e3, 28800e3,
const HOOK_BACKOFF_MS = [5e3, 30e3, 120e3, 600e3, 1800e3, 3600e3, 7200e3, 10800e3, 14400e3, 21600e3, 28800e3,
- 43200e3, 43200e3, 43200e3, 43200e3, 86400e3, 86400e3, 86400e3, 86400e3];
+ 14400e3, 14400e3, 14400e3, 14400e3, 21600e3, 21600e3, 21600e3, 28800e3, 28800e3];
/** Connect's subscription to its own connect.hook.received, made once (409 means it is there); tried again every 10 minutes until it is. */
/** Connect's subscription to its own connect.hook.received, made once (409 means it is there); tried again every 10 minutes until it is. */
function ensureHookSubscription({ config, log = console, fetchImpl = globalThis.fetch }) {
function ensureHookSubscription({ config, log = console, fetchImpl = globalThis.fetch }) {
@@ -264,12 +264,17 @@ function nodeBlock(share, machines, now) {
@@ -264,12 +264,17 @@ function nodeBlock(share, machines, now) {
* was handed over, and how many wait; the form for another. Plain forms, no script.
* was handed over, and how many wait; the form for another. Plain forms, no script.
*/
*/
function hooksBlock(share, hooks, now) {
function hooksBlock(share, hooks, now) {
+ const log = (h) => {
+ const rows = Array.isArray(h.deliveries) ? h.deliveries.slice(0, 5) : [];
+ if (!rows.length) return '';
+ return `<details class="cn-delivery-log"><summary>Delivery log</summary><ul>${rows.map((d) => `<li><b>${d.status == null ? 'failed' : esc(d.status)}</b> ${esc(d.method)} ${esc(d.path)} · ${esc(ago(d.delivered_at || d.created_at, now))}${d.error ? ` · ${esc(d.error)}` : ''}</li>`).join('')}</ul></details>`;
+ };
const rows = hooks.length ? `<ul class="cn-hooks">${hooks.map((h, i) => `<li><div class="cn-hook-main"><div class="cn-copy"><code id="cn-hook-${i}">${esc(h.url)}</code><button type="button" class="cn-btn cn-btn-quiet" data-ovk-copy="#cn-hook-${i}" hidden>${kit.icon('copy')} Copy</button></div>
const rows = hooks.length ? `<ul class="cn-hooks">${hooks.map((h, i) => `<li><div class="cn-hook-main"><div class="cn-copy"><code id="cn-hook-${i}">${esc(h.url)}</code><button type="button" class="cn-btn cn-btn-quiet" data-ovk-copy="#cn-hook-${i}" hidden>${kit.icon('copy')} Copy</button></div>
-<small>→ localhost:${esc(share.local_port)}${esc(h.local_path)}${h.label ? ` · ${esc(h.label)}` : ''} · ${esc(h.received)} received, ${esc(h.delivered)} handed over${h.waiting ? `, <b>${esc(h.waiting)} waiting for your computer</b>` : ''}${h.last_received_at ? ` · last ${esc(ago(h.last_received_at, now))}` : ''}${h.last_status ? ` · your app answered ${esc(h.last_status)}` : ''}</small></div>
+<small>→ localhost:${esc(share.local_port)}${esc(h.local_path)}${h.label ? ` · ${esc(h.label)}` : ''} · ${esc(h.received)} received, ${esc(h.delivered)} handed over${h.waiting ? `, <b>${esc(h.waiting)} waiting for your computer</b>` : ''}${h.last_received_at ? ` · last ${esc(ago(h.last_received_at, now))}` : ''}${h.last_status ? ` · your app answered ${esc(h.last_status)}` : ''}</small>${log(h)}</div>
<form method="post" action="/shares/${esc(share.id)}/hooks/${esc(h.id)}/remove"><button class="cn-btn cn-btn-quiet" type="submit">${kit.icon('close')} Remove</button></form></li>`).join('')}</ul>`
<form method="post" action="/shares/${esc(share.id)}/hooks/${esc(h.id)}/remove"><button class="cn-btn cn-btn-quiet" type="submit">${kit.icon('close')} Remove</button></form></li>`).join('')}</ul>`
: '<p class="cn-empty">No webhook addresses yet.</p>';
: '<p class="cn-empty">No webhook addresses yet.</p>';
return `<section class="cn-hooks-sec" aria-labelledby="cn-hooks-h"><h2 id="cn-hooks-h">Webhooks that wait for your computer</h2>
return `<section class="cn-hooks-sec" aria-labelledby="cn-hooks-h"><h2 id="cn-hooks-h">Webhooks that wait for your computer</h2>
-<p>Give a service (GitHub, Stripe, anything that sends webhooks) an address here instead of your share's. Each webhook is kept for up to 7 days and handed to your app, at the path you pick, as soon as your computer is connected: a laptop that sleeps misses nothing. Nobody has to sign in to send one, so check the provider's signature in your app.</p>
+<p>Give a service (GitHub, Stripe, anything that sends webhooks) an address here instead of your share's. Each webhook is sealed, kept for up to 72 hours, and handed to your app, at the path you pick, as soon as your computer is connected: a laptop that sleeps misses nothing. A share holds at most 1,000 waiting webhooks; the log below says what your app answered. Nobody has to sign in to send one, so check the provider's signature in your app.</p>
${rows}
${rows}
<form class="cn-form cn-hook-add" method="post" action="/shares/${esc(share.id)}/hooks">
<form class="cn-form cn-hook-add" method="post" action="/shares/${esc(share.id)}/hooks">
<label class="cn-field"><span>Path on your app</span><input name="path" maxlength="500" value="/webhooks" placeholder="/webhooks/github" autocapitalize="none" autocomplete="off" spellcheck="false"></label>
<label class="cn-field"><span>Path on your app</span><input name="path" maxlength="500" value="/webhooks" placeholder="/webhooks/github" autocapitalize="none" autocomplete="off" spellcheck="false"></label>
@@ -2,12 +2,13 @@
@@ -2,12 +2,13 @@
/**
/**
* Hooks that wait for your computer (CON-15): a webhook address on a share, a provider's webhook kept with only the
* Hooks that wait for your computer (CON-15): a webhook address on a share, a provider's webhook kept with only the
* provider's headers and its pointer in the outbox (one transaction), Events' signed delivery handed to the computer
* provider's headers and its pointer in the outbox (one transaction), Events' signed delivery handed to the computer
- * once it is connected (503 while it is not, so Events retries), once only, and the sweep after 7 days.
+ * once it is connected (503 while it is not, so Events retries), once only, and the sweep after 72 hours.
* node test/hooks.test.js
* node test/hooks.test.js
*/
*/
const assert = require('assert');
const assert = require('assert');
const contracts = require('openvibe-contracts');
const contracts = require('openvibe-contracts');
const { signDeliveryHeaders } = require('openvibe-sdk/events');
const { signDeliveryHeaders } = require('openvibe-sdk/events');
+const { keepHeaders } = require('../server/hooks');
const { boot, startLocalApp } = require('./helpers/boot');
const { boot, startLocalApp } = require('./helpers/boot');
const { check, done } = require('./helpers/check');
const { check, done } = require('./helpers/check');
@@ -22,6 +23,15 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -22,6 +23,15 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
let hook;
let hook;
let envelope;
let envelope;
+ const settled = async (ref) => {
+ for (let i = 0; i < 200; i++) {
+ const row = await t.ctx.s.db.maybe('SELECT delivered_at FROM hook_requests WHERE id = $1', [ref]);
+ if (!row || row.delivered_at) return true;
+ await new Promise((r) => setTimeout(r, 10));
+ }
+ return false;
+ };
+
const deliver = async (env, secret = SECRET) => {
const deliver = async (env, secret = SECRET) => {
const raw = Buffer.from(JSON.stringify({ event: env }));
const raw = Buffer.from(JSON.stringify({ event: env }));
const headers = { 'content-type': 'application/json', ...signDeliveryHeaders(raw, secret), 'x-openvibe-subscription-id': 'sub_x', 'x-openvibe-delivery-attempt': '1' };
const headers = { 'content-type': 'application/json', ...signDeliveryHeaders(raw, secret), 'x-openvibe-subscription-id': 'sub_x', 'x-openvibe-delivery-attempt': '1' };
@@ -44,18 +54,30 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -44,18 +54,30 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
assert.ok(page.text.includes(hook.url));
assert.ok(page.text.includes(hook.url));
});
});
- await check('a provider\'s webhook: 202, kept with its headers (never cookies or forwarding), its pointer in the outbox', async () => {
+ await check('a provider\'s webhook: 202, sealed, kept with its headers minus hop-by-hop and credentials, its pointer in the outbox', async () => {
const body = JSON.stringify({ ref: 'refs/heads/main', commits: 3 });
const body = JSON.stringify({ ref: 'refs/heads/main', commits: 3 });
- const r = await t.get(`/h/${hook.id}`, { method: 'POST', body, headers: { 'content-type': 'application/json', 'x-github-event': 'push', 'x-hub-signature-256': 'sha256=abc', cookie: 'session=secret', 'x-forwarded-for': '203.0.113.9', authorization: 'Bearer nope' } });
+ const r = await t.get(`/h/${hook.id}`, { method: 'POST', body, headers: {
+ 'content-type': 'application/json', 'x-github-event': 'push', 'x-hub-signature-256': 'sha256=abc',
+ cookie: 'session=secret', 'x-forwarded-for': '203.0.113.9', authorization: 'Bearer nope', 'x-openvibe-user': 'mallory',
+ } });
assert.strictEqual(r.status, 202, r.text);
assert.strictEqual(r.status, 202, r.text);
const ref = r.json().id;
const ref = r.json().id;
assert.match(ref, /^hb_/);
assert.match(ref, /^hb_/);
- const row = await t.ctx.s.db.one('SELECT method, headers, body FROM hook_requests WHERE id = $1', [ref]);
+ const row = await t.ctx.s.db.one('SELECT method, headers, body, received_at, expires_at FROM hook_requests WHERE id = $1', [ref]);
assert.strictEqual(row.method, 'POST');
assert.strictEqual(row.method, 'POST');
- assert.strictEqual(Buffer.from(row.body).toString(), body);
+ const kept = t.ctx.hooks.request(ref);
+ assert.strictEqual((await kept).body.toString(), body, 'the body is there for the computer, not for the database');
+ assert.ok(!Buffer.from(row.body).toString().includes('refs/heads/main'), 'the body is sealed at rest');
+ assert.match(Buffer.from(row.body).subarray(0, 4).toString(), /^OVC1$/);
const names = row.headers.map(([k]) => k);
const names = row.headers.map(([k]) => k);
assert.ok(names.includes('x-github-event') && names.includes('x-hub-signature-256') && names.includes('content-type'), names.join(','));
assert.ok(names.includes('x-github-event') && names.includes('x-hub-signature-256') && names.includes('content-type'), names.join(','));
- assert.ok(!names.includes('cookie') && !names.includes('x-forwarded-for') && !names.includes('authorization'), names.join(','));
+ for (const gone of ['cookie', 'x-forwarded-for', 'authorization', 'x-openvibe-user']) {
+ assert.ok(!names.includes(gone), `${gone} must not be kept: ${names.join(',')}`);
+ }
+ const hop = keepHeaders({ connection: 'keep-alive', te: 'trailers', 'proxy-authorization': 'Basic nope', upgrade: 'h2c', 'transfer-encoding': 'chunked', 'x-github-event': 'push' }).map(([k]) => k);
+ assert.deepStrictEqual(hop, ['x-github-event'], 'hop-by-hop headers are dropped');
+ const lifetime = await t.ctx.s.db.one('SELECT extract(epoch FROM expires_at - received_at)::int AS s FROM hook_requests WHERE id = $1', [ref]);
+ assert.strictEqual(lifetime.s, 72 * 3600, 'kept for 72 hours');
const out = await t.ctx.s.db.many("SELECT envelope FROM service_outbox WHERE envelope->>'event_type' = 'connect.hook.received'");
const out = await t.ctx.s.db.many("SELECT envelope FROM service_outbox WHERE envelope->>'event_type' = 'connect.hook.received'");
assert.strictEqual(out.length, 1);
assert.strictEqual(out.length, 1);
envelope = out[0].envelope;
envelope = out[0].envelope;
@@ -88,7 +110,9 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -88,7 +110,9 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
assert.strictEqual(got.headers['x-hub-signature-256'], 'sha256=abc');
assert.strictEqual(got.headers['x-hub-signature-256'], 'sha256=abc');
assert.strictEqual(got.headers['x-openvibe-hook'], hook.id);
assert.strictEqual(got.headers['x-openvibe-hook'], hook.id);
assert.strictEqual(got.headers['x-openvibe-event-id'], envelope.event_id, 'the event\'s own id, whichever way it came');
assert.strictEqual(got.headers['x-openvibe-event-id'], envelope.event_id, 'the event\'s own id, whichever way it came');
+ assert.strictEqual(got.headers['x-openvibe-received-at'], envelope.payload.received_at, 'the original arrival time');
assert.strictEqual(got.headers.cookie, undefined);
assert.strictEqual(got.headers.cookie, undefined);
+ assert.ok(await settled(envelope.payload.body_ref), 'the delivery settles');
const again = await deliver(envelope);
const again = await deliver(envelope);
assert.strictEqual(again.status, 200);
assert.strictEqual(again.status, 200);
assert.deepStrictEqual(again.json(), { gone: true }, 'delivered once');
assert.deepStrictEqual(again.json(), { gone: true }, 'delivered once');
@@ -97,6 +121,15 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -97,6 +121,15 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
assert.deepStrictEqual([kept.n, kept.status], [0, 200], 'its body is dropped once delivered');
assert.deepStrictEqual([kept.n, kept.status], [0, 200], 'its body is dropped once delivered');
const h = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex })).json().hooks[0];
const h = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex })).json().hooks[0];
assert.deepStrictEqual([h.received, h.delivered, h.last_status], [1, 1, 200]);
assert.deepStrictEqual([h.received, h.delivered, h.last_status], [1, 1, 200]);
+ assert.strictEqual(h.deliveries[0].status, 200, 'the owner sees the delivery');
+ assert.strictEqual(h.deliveries[0].method, 'POST');
+ assert.strictEqual(h.deliveries[0].path, '/webhooks/github');
+ const log = await t.get(`/api/v1/shares/${made.share.id}/hooks/${hook.id}/deliveries`, { as: alex });
+ assert.strictEqual(log.status, 200, log.text);
+ assert.strictEqual(log.json().deliveries[0].status, 200);
+ assert.strictEqual(log.json().deliveries[0].event_id, envelope.event_id);
+ const page = await t.get(`/shares/${made.share.id}`, { as: alex });
+ assert.match(page.text, /Delivery log/);
c.stop();
c.stop();
});
});
@@ -116,7 +149,7 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -116,7 +149,7 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
assert.strictEqual(left.n, 0);
assert.strictEqual(left.n, 0);
});
});
- await check('the sweep removes webhooks past their 7 days', async () => {
+ await check('the sweep removes webhooks past their 72 hours', async () => {
const h2 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/x' } })).json().hook;
const h2 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/x' } })).json().hook;
const ref = (await t.get(`/h/${h2.id}`, { method: 'POST', body: 'old' })).json().id;
const ref = (await t.get(`/h/${h2.id}`, { method: 'POST', body: 'old' })).json().id;
await t.ctx.s.db.exec("UPDATE hook_requests SET expires_at = now() - interval '1 minute' WHERE id = $1", [ref]);
await t.ctx.s.db.exec("UPDATE hook_requests SET expires_at = now() - interval '1 minute' WHERE id = $1", [ref]);
@@ -124,6 +157,29 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -124,6 +157,29 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
assert.strictEqual((await t.ctx.s.db.maybe('SELECT id FROM hook_requests WHERE id = $1', [ref])), null);
assert.strictEqual((await t.ctx.s.db.maybe('SELECT id FROM hook_requests WHERE id = $1', [ref])), null);
});
});
+ await check('a body written before sealing is sealed by the sweep and still reaches the app', async () => {
+ const h2 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/x/legacy' } })).json().hook;
+ const ref = 'hb_legacy0123456789ab';
+ await t.ctx.s.db.exec(`INSERT INTO hook_requests (id, hook_id, method, headers, body, received_at, expires_at)
+ VALUES ($1, $2, 'POST', '[]'::jsonb, $3::bytea, now(), now() + interval '72 hours')`, [ref, h2.id, Buffer.from('plain at rest')]);
+ await t.ctx.hooks.sweep();
+ const row = await t.ctx.s.db.one('SELECT body FROM hook_requests WHERE id = $1', [ref]);
+ assert.ok(!Buffer.from(row.body).toString().includes('plain at rest'), 'the sweep sealed it');
+ assert.strictEqual((await t.ctx.hooks.request(ref)).body.toString(), 'plain at rest');
+ await t.get(`/api/v1/shares/${made.share.id}/hooks/${h2.id}`, { as: alex, method: 'DELETE' });
+ });
+
+ await check('a share holds at most 1,000 waiting webhooks', async () => {
+ const h2 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/x/full' } })).json().hook;
+ await t.ctx.s.db.exec(`INSERT INTO hook_requests (id, hook_id, method, headers, body, received_at, expires_at, event_id)
+ SELECT 'hb_q' || g, $1, 'POST', '[]'::jsonb, ''::bytea, now(), now() + interval '72 hours', 'evt_q' || g
+ FROM generate_series(1, 1000) g`, [h2.id]);
+ const r = await t.get(`/h/${h2.id}`, { method: 'POST', body: '{}' });
+ assert.strictEqual(r.status, 429, r.text);
+ assert.strictEqual(r.json().error, 'hook.queue_full');
+ await t.get(`/api/v1/shares/${made.share.id}/hooks/${h2.id}`, { as: alex, method: 'DELETE' });
+ });
+
await check('a webhook that waited goes to the app as soon as its computer connects; Events\' retry then finds it settled', async () => {
await check('a webhook that waited goes to the app as soon as its computer connects; Events\' retry then finds it settled', async () => {
const h3 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/webhooks/waited' } })).json().hook;
const h3 = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/webhooks/waited' } })).json().hook;
const w = await t.get(`/h/${h3.id}`, { method: 'POST', body: '{"waited":true}', headers: { 'content-type': 'application/json' } });
const w = await t.get(`/h/${h3.id}`, { method: 'POST', body: '{"waited":true}', headers: { 'content-type': 'application/json' } });
@@ -137,6 +193,7 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -137,6 +193,7 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
assert.ok(got, 'handed over on connect');
assert.ok(got, 'handed over on connect');
assert.strictEqual(got.body, '{"waited":true}');
assert.strictEqual(got.body, '{"waited":true}');
assert.strictEqual(got.headers['x-openvibe-event-id'], env.event_id, 'the event\'s own id');
assert.strictEqual(got.headers['x-openvibe-event-id'], env.event_id, 'the event\'s own id');
+ assert.ok(await settled(w.json().id), 'the handover settles');
const late = await deliver(env);
const late = await deliver(env);
assert.deepStrictEqual(late.json(), { gone: true }, 'Events\' delivery finds it settled');
assert.deepStrictEqual(late.json(), { gone: true }, 'Events\' delivery finds it settled');
assert.strictEqual(app.seen.filter((x) => x.url === '/webhooks/waited').length, 1, 'once');
assert.strictEqual(app.seen.filter((x) => x.url === '/webhooks/waited').length, 1, 'once');
@@ -157,6 +214,22 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
@@ -157,6 +214,22 @@ const SECRET = 'hook-delivery-secret-0123456789abcdef';
c.stop();
c.stop();
});
});
+ await check('every waiting webhook replays in order when its computer connects, past one batch', async () => {
+ const h = (await t.get(`/api/v1/shares/${made.share.id}/hooks`, { as: alex, json: { path: '/webhooks/order' } })).json().hook;
+ for (let i = 0; i < 105; i++) {
+ const r = await t.get(`/h/${h.id}`, { method: 'POST', body: JSON.stringify({ i }), headers: { 'content-type': 'application/json' } });
+ assert.strictEqual(r.status, 202, r.text);
+ }
+ const before = app.seen.length;
+ const c = t.client({ key: made.key, port: app.port });
+ await c.ready();
+ const got = () => app.seen.slice(before).filter((x) => x.url === '/webhooks/order');
+ for (let i = 0; i < 500 && got().length < 105; i++) await new Promise((r) => setTimeout(r, 20));
+ assert.strictEqual(got().length, 105, 'every waiting webhook is handed over');
+ assert.deepStrictEqual(got().map((x) => JSON.parse(x.body).i), Array.from({ length: 105 }, (_, i) => i), 'oldest first');
+ c.stop();
+ });
+
await check('the page: make one with a plain form', async () => {
await check('the page: make one with a plain form', async () => {
const r = await t.get(`/shares/${made.share.id}/hooks`, { as: alex, method: 'POST', form: { path: '/hooks/stripe', label: 'Stripe' } });
const r = await t.get(`/shares/${made.share.id}/hooks`, { as: alex, method: 'POST', form: { path: '/hooks/stripe', label: 'Stripe' } });
assert.strictEqual(r.status, 201, r.text);
assert.strictEqual(r.status, 201, r.text);