openvibers/openvibe.ai Commits
Memory M3: import ChatGPT, Claude and Gemini exports into the store (#309)
Changes
nnew file mode 100644@@@ -0,0 +1,47 @@++# Memory import (M3): ChatGPT, Claude and Gemini exports++++The Memory page in Codes (`/memory?tab=import`) walks a person through bringing what their cloud assistants++already remember into OpenVibe Memory. The parsing and the redaction live in OpenVibe.AI, beside the store M2++reads; Codes only forwards the person's token and the bytes, then renders what comes back.++++## The flow++++1. **Preview** — `POST /api/v1/memory/import/preview` (person token; a service token is refused).++ The body is a `.zip` export, one `multipart/form-data` form from the page, or JSON `{ provider, text }` for a++ pasted answer. AI parses it, redacts every candidate with `openvibe-sdk/redact`, and returns the candidates.++ Nothing is written to `memory_items`. The candidates (already redacted) are sealed into one `memory_imports`++ row with `AI_MEMORY_KEY` (AAD `memory-import|<owner>|<id>`) and expire after two hours.++2. **Keep** — `POST /api/v1/memory/import/:id/commit` with `{ keep: ["k1", "k3"] }`.++ Only those keys go through the store's normal batch import (which redacts again and dedupes), and the draft is++ deleted. Keys that are not ticked are deleted with it. `DELETE /api/v1/memory/import/:id` throws the draft away.++3. **Walkthroughs** — `GET /api/v1/memory/import/providers` returns the three cards (where to request the export,++ what arrives, how long) and the extraction prompt. It contains no note.++++## What each parser reads++++| Provider | File | What it gives |++|---|---|---|++| ChatGPT | `conversations.json` | the current branch of each conversation (`mapping`, `current_node`); only the person's sentences that announce a durable memory become candidates |++| Claude | `memories.json` | every saved memory, confidence 0.9 (no marker needed) |++| Claude | `conversations.json` | `chat_messages[]`, `sender: "human"`, content blocks |++| Gemini | `Takeout/My Activity/Gemini Apps/My Activity.json` | the activity record's prompt title (`Said …`, `Prompted …`) |++| Gemini | `Gemini/conversations/*.json` | `messages[]` with `role: "user"` |++| Any | pasted answer | bullets, with an optional `[preference]` / `[fact]` / `[project rule]` / `[procedure]` / `[decision]` / `[lesson]` label |++++An export with no candidate is still a successful preview: it says so and points at the extraction prompt, which++is the method Claude's own memory import uses. Every candidate keeps its source and date, so the note can be++removed with the rest of that source later.++++## Limits and safety++++- at most 25 MB uploaded, 5,000 zip entries, 60 MB of JSON, 200 JSON entries, 200 candidates per preview;++- only stored (0) and deflate (8) entries, basic ZIP64; encrypted entries and other methods are refused;++- the archive is parsed in memory and never extracted to disk;++- no secret is returned in a preview: redaction runs before the candidate leaves the module, and again on commit;++- no route logs a note's text; a preview is sealed and owned, and belongs to the person's account deletion.++++## Tests++++`test/memory-import.test.js` builds its own exports with `test/fixtures/zip.js` (no real exports in the repo):++redaction before the preview, no note stored by a preview, the sealed draft, the per-item commit, Claude++`memories.json`, a Gemini Takeout path, a pasted answer, service-token refusal, owner isolation and expiry.nnew file mode 100644@@@ -0,0 +1,16 @@++-- phase: expand++-- OpenVibe Memory M3: one sealed draft per import preview. The person drops a ChatGPT, Claude or Google Takeout++-- export, sees every candidate, and keeps or removes each; nothing enters memory_items until they choose.++--++-- payload_enc is the whole draft (summary and candidates) sealed with AI_MEMORY_KEY, AAD++-- `memory-import|<owner>|<id>`, so the table holds no plaintext. A draft lives for two hours and is deleted on++-- commit, on discard, on expiry and with the person's account (server/account-data.js).++CREATE TABLE IF NOT EXISTS memory_imports (++ id text COLLATE "C" PRIMARY KEY,++ owner text COLLATE "C" NOT NULL,++ provider text COLLATE "C" NOT NULL,++ payload_enc text COLLATE "C" NOT NULL,++ created_at text COLLATE "C" NOT NULL,++ expires_at text COLLATE "C" NOT NULL++);++CREATE INDEX IF NOT EXISTS memory_imports_owner_expires ON memory_imports (owner, expires_at);@@@ -40,6 +40,9 @@ const TABLES = [{ table: 'memory_items', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },{ table: 'memory_profiles', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },{ table: 'memory_reads', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },++ // Memory M3: import previews are drafts, never part of the person's export; they go with the account like the++ // notes they would become. The payload is sealed, and the row is never read by the account export.++ { table: 'memory_imports', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },{ table: 'free_allowance_usage', subject: 'subject', file: 'free-allowance.json', erase: { keep: ledger } },// D11 owner settings: the revisions are the service's configuration audit trail, kept like the catalogues.{ table: 'config_snapshots', subject: 'created_by', file: null, erase: { keep: 'the configuration history is the service\'s; created_by is the operator audit trail' } },@@@ -24,7 +24,7 @@ const pkg = require('../package.json');const { adminRoutes } = require('openvibe-shared/config');--function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, community = null }) {++function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, memoryImport = null, community = null }) {const app = express();// /.well-known/agent.json (openvibe-shared/agent-layer, SEO-2): what this site serves, for agents.require('openvibe-shared/agent-layer').mount(app, { descriptor: require('./agent-descriptor') });@@@ -147,7 +147,7 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys// The Anthropic-compatible endpoint (POST /v1/messages, GET /v1/models): runs of ai.messages (Codes C3).app.use(require('./api/messages').messagesRouter({ runs, auth, config, registry, log }));app.use(require('./api/attribution-quotas').attributionQuotasRouter({ db, quotas, auth, clock, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));-- if (memory) app.use(require('./memory').memoryRouter({ memory, auth, config, clock, log, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));++ if (memory) app.use(require('./memory').memoryRouter({ memory, memoryImport, auth, config, clock, log, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));if (credentials) app.use(require('./credentials').credentialsRouter({ credentials, auth, registry, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));// The budget of a Codes run token (Codes C3): Codes registers it when it mints the token.if (runBudgets) app.use(require('./api/run-budgets').runBudgetsRouter({ runBudgets, auth, log, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));@@@ -125,9 +125,12 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowconst auth = createAuth({ config, log });// The person's memory (Memory M2a): sealed item text, the profile cache and the agent-read audit.const memory = require('./memory').createMemory({ db, config, clock, log, registry, pool });-- // The store's own maintenance (Memory M2c): owner by owner, with a per-run cap. It is off in test mode and off-- // while a provider is in a drill (server/memory/jobs.js).-- const memoryJobs = require('./memory/jobs').createMemoryJobs({ memory, config, clock, log, registry });++ // The cloud import (Memory M3): parse an export into a sealed, two-hour preview. No note goes to the store++ // until the person chooses it.++ const memoryImport = require('./memory/import').createMemoryImport({ memory, db, config, clock, log });++ // The store's own maintenance (Memory M2c/M3): owner by owner, with a per-run cap, and the expired import++ // drafts. It is off in test mode and off while a provider is in a drill (server/memory/jobs.js).++ const memoryJobs = require('./memory/jobs').createMemoryJobs({ memory, memoryImport, config, clock, log, registry });// The Events consumer side: a signed bench.rating.changed webhook, subscribed at boot (server/bench/events.js).const accountData = accountDataLib.create({ db, log, memory });const accountSend = env.OV_OAUTH_CLIENT_SECRET ? createNetworkSender({@@@ -135,7 +138,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowclientSecret: env.OV_OAUTH_CLIENT_SECRET, fetch: fetchImpl,}) : null;const benchEvents = createBenchEvents({ bench, accountData, accountSend, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });-- const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, community });++ const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, memoryImport, community });let jwks = null;let keyLoaded = Promise.resolve(null);if (!keys.pinned && config.nodeEnv !== 'test') {nnew file mode 100644@@@ -0,0 +1,187 @@++'use strict';++/**++ * The memory import (OpenVibe Memory M3): a guided, preview-first import of a cloud assistant's export.++ *++ * The flow, and why it is in two steps:++ * 1. POST /api/v1/memory/import/preview parse, redact, and hold the candidates in one sealed, two-hour draft.++ * Nothing enters the memory store yet: the person sees every candidate and unticks what they do not want.++ * 2. POST /api/v1/memory/import/:id/commit take the ticked keys and import them, through the same store path++ * M2 reads (the batch import), then delete the draft. Left-out candidates are deleted with it.++ *++ * The draft is a row in memory_imports whose payload is sealed with the memory key and bound to its owner and id++ * (AAD `memory-import|<owner>|<id>`). The table holds no plaintext, the routes never log text, and an expired draft++ * is deleted on the next preview. A service token cannot preview or commit: this is the person's own memory.++ */++++const { ids } = require('openvibe-contracts');++const { AiError, iso } = require('../../util');++const { readZip, ZipError } = require('./zip');++const { PROVIDERS, EXTRACTION_PROMPT, ImportError, normalizeProvider, parseExport, providerName } = require('./providers');++++const IMPORT_TTL_MS = 2 * 60 * 60 * 1000;++const MAX_BYTES = 25 * 1024 * 1024;++const MAX_PARTS = 20;++const MAX_KEEP = 500;++const KEEP_RE = /^k[1-9][0-9]{0,5}$/;++++function aadFor(owner, id) { return `memory-import|${owner}|${id}`; }++++/** The boundary and the one file / few fields of our own multipart form. Binary-safe through latin1. */++function parseMultipart(buf, contentType) {++ const m = /boundary=(?:"([^"]+)"|([^;\s]+))/i.exec(String(contentType || ''));++ if (!m) throw new ImportError('That upload was not a valid form. Try again.');++ const boundary = '--' + (m[1] || m[2]);++ const text = buf.toString('latin1');++ const fields = {};++ let file = null;++ let parts = 0;++ for (const chunk of text.split(boundary).slice(1)) {++ if (chunk.startsWith('--')) break;++ parts++;++ if (parts > MAX_PARTS) throw new ImportError('That form had too many parts.');++ const part = chunk.replace(/^\r\n/, '');++ const split = part.indexOf('\r\n\r\n');++ if (split < 0) continue;++ const head = part.slice(0, split);++ let body = part.slice(split + 4);++ if (body.endsWith('\r\n')) body = body.slice(0, -2);++ const name = /name="([^"]+)"/i.exec(head);++ if (!name) continue;++ const filename = /filename="([^"]*)"/i.exec(head);++ if (filename && filename[1]) file = { name: filename[1], data: Buffer.from(body, 'latin1') };++ else fields[name[1]] = Buffer.from(body, 'latin1').toString('utf8');++ }++ return { fields, file };++}++++function createMemoryImport({ memory, db, config, clock = { now: () => Date.now() }, log = console }) {++ const q = (sql) => db.prepare(sql);++ const expire = async (owner) => {++ await q('DELETE FROM memory_imports WHERE owner = ? AND expires_at < ?').run(owner, iso(clock.now()));++ };++++ /** The request's bytes, as a file or as pasted text, with the provider named in the form or by its shape. */++ function readInput(req) {++ const asked = normalizeProvider((req.query && req.query.provider) || null);++ const type = String(req.get('content-type') || '').toLowerCase();++ let provider = asked;++ let filename = '';++ let text = null;++ let files = [];++ let body = req.body;++ if (Buffer.isBuffer(body)) {++ if (body.length > MAX_BYTES) throw new ImportError('That upload is larger than 25 MB. Export only the assistant’s data, or paste the extraction answer instead.');++ if (type.includes('multipart/form-data')) {++ const parsed = parseMultipart(body, type);++ if (parsed.fields.provider) provider = normalizeProvider(parsed.fields.provider) || provider;++ if (parsed.fields.text) text = parsed.fields.text;++ if (parsed.file) { filename = parsed.file.name; files = readZip(parsed.file.data); }++ } else if (type.includes('text/plain')) {++ text = body.toString('utf8');++ } else {++ files = readZip(body);++ }++ } else if (body && typeof body === 'object') {++ if (body.provider) provider = normalizeProvider(body.provider) || provider;++ if (typeof body.text === 'string') text = body.text;++ } else {++ throw new ImportError('Drop an export .zip or paste the answer to the extraction prompt.');++ }++ if (!files.length && (text == null || !String(text).trim())) {++ throw new ImportError('Drop an export .zip or paste the answer to the extraction prompt.');++ }++ return { provider, filename, files, text };++ }++++ /** Parse and redact. Nothing is written here; the draft below is the only row. */++ async function preview(req, owner) {++ memory.assertConfigured();++ let parsed;++ try {++ const input = readInput(req);++ parsed = parseExport(input);++ } catch (err) {++ if (err instanceof ZipError) throw new AiError(422, 'import.bad_zip', err.message);++ if (err instanceof ImportError) throw new AiError(err.status || 422, err.code, err.message);++ throw err;++ }++ const now = iso(clock.now());++ await expire(owner);++ if (!parsed.items.length) {++ return { import: { id: null, provider: parsed.provider, summary: parsed.summary, items: [], expires_at: null } };++ }++ const id = `mimp_${ids.ulid()}`;++ const items = parsed.items.map((item, i) => ({ ...item, key: `k${i + 1}` }));++ const expires = new Date(clock.now() + IMPORT_TTL_MS).toISOString();++ const payload = { provider: parsed.provider, summary: parsed.summary, items };++ await q('INSERT INTO memory_imports (id, owner, provider, payload_enc, created_at, expires_at) VALUES (?, ?, ?, ?, ?, ?)')++ .run(id, owner, parsed.provider, memory.seal(JSON.stringify(payload), aadFor(owner, id)), now, expires);++ return {++ import: {++ id, provider: parsed.provider, provider_name: providerName(parsed.provider),++ summary: parsed.summary, items, created_at: now, expires_at: expires,++ },++ };++ }++++ async function draft(owner, id) {++ if (!/^mimp_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(id || ''))) throw new AiError(404, 'import.not_found', 'That preview is no longer here.');++ const row = await q('SELECT * FROM memory_imports WHERE owner = ? AND id = ?').get(owner, id);++ if (!row) throw new AiError(404, 'import.not_found', 'That preview is no longer here.');++ if (String(row.expires_at) < iso(clock.now())) {++ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);++ throw new AiError(410, 'import.expired', 'That preview has expired. Drop the export again.');++ }++ let payload;++ try { payload = JSON.parse(memory.open(row.payload_enc, aadFor(owner, id))); }++ catch (err) { throw err instanceof AiError ? err : new AiError(500, 'memory.unreadable', 'the preview does not open'); }++ return { row, payload };++ }++++ /** Keep the ticked candidates. The store's own batch path redacts again and dedupes; the draft is then gone. */++ async function commit(req, owner, id) {++ memory.assertConfigured();++ const { payload } = await draft(owner, id);++ const body = req.body && typeof req.body === 'object' ? req.body : {};++ const raw = body.keep == null ? [] : Array.isArray(body.keep) ? body.keep : [body.keep];++ if (raw.length > MAX_KEEP) throw new AiError(422, 'import.too_many', `Keep at most ${MAX_KEEP} notes at a time.`);++ const keys = new Set();++ for (const key of raw) {++ if (!KEEP_RE.test(String(key))) throw new AiError(422, 'import.invalid', 'one of the chosen notes is not from this preview');++ keys.add(String(key));++ }++ const selected = (payload.items || []).filter((item) => keys.has(item.key));++ const leftOut = Math.max(0, (payload.items || []).length - selected.length);++ if (!selected.length) {++ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);++ return { kept: 0, created: 0, deduped: 0, left_out: leftOut };++ }++ const out = await memory.batch({ body: { items: selected }, principal: req.principal }, owner);++ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);++ log.log(`[memory] imported ${out.created + out.deduped} note(s) from ${payload.provider}; ${leftOut} left out`);++ return { kept: out.created + out.deduped, created: out.created, deduped: out.deduped, left_out: leftOut };++ }++++ async function discard(owner, id) {++ const { payload } = await draft(owner, id);++ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);++ return { discarded: (payload.items || []).length };++ }++++ /** The maintenance loop's sweep: expired drafts go without waiting for the person to come back. */++ async function expireDrafts(limit = 100) {++ const n = Math.max(1, Math.min(500, Number(limit) || 100));++ const rows = await q(`SELECT id FROM memory_imports WHERE expires_at < ? ORDER BY expires_at LIMIT ${n}`).all(iso(clock.now()));++ for (const row of rows) await q('DELETE FROM memory_imports WHERE id = ?').run(row.id);++ return rows.length;++ }++++ /** The walkthrough copy and the extraction prompt. No note ever appears in it. */++ function providers() {++ return { providers: PROVIDERS, prompt: EXTRACTION_PROMPT };++ }++++ return { preview, commit, discard, providers, expire, expireDrafts, aadFor };++}++++module.exports = { createMemoryImport, parseMultipart, IMPORT_TTL_MS, MAX_BYTES };nnew file mode 100644@@@ -0,0 +1,515 @@++'use strict';++/**++ * The cloud import adapters (OpenVibe Memory M3): ChatGPT, Claude and Gemini exports, and the answer to the++ * extraction prompt. This file turns an export into *candidate* memory items; it never stores anything and it++ * never returns raw conversation text. Only a sentence that looks like a memory survives, already redacted.++ *++ * The formats, read from the vendors' own exports:++ * ChatGPT a zip with conversations.json: an array of { title, create_time, mapping }, where a message has++ * `author.role` and `content.parts`.++ * Claude a zip with conversations.json ({ chat_messages[] } with `sender` and content blocks) and, on plans++ * that have memory, memories.json ({ content, created_at }). The saved memories are the strongest signal.++ * Gemini Google Takeout: Takeout/My Activity/Gemini Apps/My Activity.json (activity records with a `title`++ * holding the prompt), and, on some accounts, Gemini/conversations/*.json.++ *++ * The extraction prompt is the method Claude's own memory import uses: ask the assistant to list what it remembers,++ * then paste the answer back. The answer is parsed as bullets, with an optional [preference] / [fact] / … label.++ *++ * Redaction runs before a candidate is returned, so a secret never reaches the preview, the browser or the store.++ * The store redacts again in its batch import; two passes are deliberate.++ */++++const { redact, onlySecret } = require('openvibe-sdk/redact');++++const MAX_ITEMS = 200;++const MAX_ITEM_CHARS = 600;++const MAX_MESSAGES = 200000;++++const EXTRACTION_PROMPT = [++ 'List everything you remember about me as short bullet points.',++ 'For each bullet, give only the memory in one sentence, then a label in square brackets: [preference], [fact],',++ '[project rule], [procedure], [decision] or [lesson]. Include saved memories, custom instructions and things you',++ 'have learned about how I work. Do not guess. Do not include passwords, keys, tokens or anything else secret.',++].join(' ');++++const PROVIDERS = [++ {++ id: 'chatgpt',++ name: 'ChatGPT',++ what: 'Your chats. ChatGPT does not put its saved-memory list in the data export, so paste the extraction prompt too if you want that list.',++ url: 'https://chatgpt.com/#settings/DataControls',++ export_label: 'Settings → Data controls → Export data',++ steps: [++ 'Open Settings → Data controls → Export data and choose Export.',++ 'OpenAI emails you a download link; the file is a .zip.',++ 'Drop the .zip here. We read conversations.json and skip the HTML copy.',++ ],++ arrives: 'A .zip with conversations.json, chat.html, user.json and a few small JSON files.',++ time: 'Usually a few minutes. A very large history can take up to 24 hours; the link is emailed to you.',++ prompt_where: 'Start a new chat in ChatGPT and paste the prompt below, then paste its answer back here.',++ },++ {++ id: 'claude-ai',++ name: 'Claude',++ what: 'Claude’s own memory list, when your plan exports it, plus the chats in the same export.',++ url: 'https://claude.ai/settings/data-privacy-controls',++ export_label: 'Settings → Privacy → Export data',++ steps: [++ 'Open Settings → Privacy → Export data (Claude emails you a link when it is ready).',++ 'Download the .zip and drop it here.',++ 'We read memories.json first, then conversations.json for things you asked Claude to remember.',++ ],++ arrives: 'A .zip with conversations.json, memories.json (on plans with memory), projects.json and users.json.',++ time: 'Usually minutes, sometimes a few hours. The link is valid for a limited time.',++ prompt_where: 'Start a new chat in Claude and paste the prompt below, then paste its answer back here.',++ },++ {++ id: 'gemini-apps',++ name: 'Gemini',++ what: 'Your Gemini Apps activity from Google Takeout. The separate “Gemini” Takeout option is Gems, not chats.',++ url: 'https://takeout.google.com',++ export_label: 'Takeout → My Activity → Gemini Apps',++ steps: [++ 'In Google Takeout choose Deselect all, then select My Activity.',++ 'Open All activity data included, deselect all, and tick only Gemini Apps (called Bard on older accounts).',++ 'Export once as a .zip, download it and drop it here.',++ ],++ arrives: 'Takeout/My Activity/Gemini Apps/My Activity.json, and sometimes Gemini/conversations/*.json.',++ time: 'A few minutes to several hours; Google emails you when it is ready. The download link expires in 7 days.',++ prompt_where: 'Start a new chat in Gemini and paste the prompt below, then paste its answer back here.',++ },++];++++const PROVIDER_BY_ID = new Map(PROVIDERS.map((p) => [p.id, p]));++const ALIASES = { chatgpt: 'chatgpt', openai: 'chatgpt', claude: 'claude-ai', 'claude-ai': 'claude-ai', claudeai: 'claude-ai', gemini: 'gemini-apps', bard: 'gemini-apps', 'gemini-apps': 'gemini-apps', takeout: 'gemini-apps', paste: 'paste', other: 'paste' };++const SOURCE_NAMES = { chatgpt: 'ChatGPT', 'claude-ai': 'Claude', 'gemini-apps': 'Gemini', paste: 'Pasted memory' };++++class ImportError extends Error {++ constructor(message, code = 'import.invalid') {++ super(message);++ this.code = code;++ this.status = 422;++ }++}++++function normalizeProvider(value, fallback = null) {++ const key = String(value == null ? '' : value).trim().toLowerCase();++ return ALIASES[key] || fallback;++}++++function providerName(id) {++ return SOURCE_NAMES[id] || String(id || 'memory');++}++++/** An ISO date-time from a vendor timestamp (seconds as a number, milliseconds, or a string). */++function when(value, fallback) {++ if (value == null || value === '') return fallback || null;++ let d;++ if (typeof value === 'number') d = new Date(value < 1e12 ? value * 1000 : value);++ else d = new Date(value);++ return Number.isFinite(d.getTime()) ? d.toISOString() : (fallback || null);++}++++function clip(text, max = MAX_ITEM_CHARS) {++ const s = String(text == null ? '' : text).replace(/\s+/g, ' ').trim();++ return s.length > max ? s.slice(0, max - 1).trimEnd() + '…' : s;++}++++const MARKERS = [++ /\bremem(?:ber|bered)\b/i,++ /\bdon'?t forget\b/i,++ /\bkeep (?:this )?in mind\b/i,++ /\bmy name is\b/i,++ /\bcall me\b/i,++ /\bi(?:'m| am) (?:a|an)\b/i,++ /\bi (?:live|work|grew up) (?:in|at|on|for)\b/i,++ /\bi (?:prefer|like|love|hate|avoid|use|always|never|usually)\b/i,++ /\bmy (?:favourite|favorite|preferred|pronouns?|timezone|birthday|team|company|role|job|editor|setup|stack|repo|project)\b/i,++ /\bwe (?:decided|agreed|chose|use|always|never)\b/i,++ /\b(?:in|for) this (?:project|repo|repository|codebase|app)\b/i,++ /\bplease (?:always|never|use|keep|remember)\b/i,++ /\bfrom now on\b/i,++ /\b(?:always|never) (?:use|do|run|write|name|call|put)\b/i,++];++++const KIND_LABELS = {++ preference: 'preference', preferred: 'preference', fact: 'fact', 'project rule': 'project_rule',++ 'project-rule': 'project_rule', project_rule: 'project_rule', procedure: 'procedure', process: 'procedure',++ skill: 'procedure', decision: 'decision', lesson: 'lesson',++};++++function guessKind(text) {++ const s = String(text || '');++ if (/\b(?:decid(?:ed|e)|decision|agreed|settled on|chose)\b/i.test(s)) return 'decision';++ if (/\b(?:learned|learnt|lesson|mistake|realized|realised)\b/i.test(s)) return 'lesson';++ if (/\b(?:project|repo|repository|codebase|our stack|this app)\b/i.test(s)++ && /\b(?:use|rule|convention|always|never|keep|must)\b/i.test(s)) return 'project_rule';++ if (/\b(?:run|install|command|step|workflow|process|procedure)\b/i.test(s)++ && /\b(?:to|then|first|before|after|use)\b/i.test(s)) return 'procedure';++ if (/\b(?:prefer|favourite|favorite|like|love|hate|rather|always|never|avoid)\b/i.test(s)) return 'preference';++ return 'fact';++}++++/** A stored-memory candidate, after redaction. Null means it was blank or nothing but a secret. */++function candidate(text, details) {++ const raw = clip(text);++ if (raw.length < 8) return null;++ const clean = clip(redact(raw).text);++ if (!clean || onlySecret(clean)) return null;++ return {++ text: clean,++ kind: KIND_LABELS[String(details.label || '').toLowerCase()] || details.kind || guessKind(clean),++ scope: details.scope || 'user',++ source: details.source,++ source_ref: details.source_ref || null,++ seen_at: details.seen_at || null,++ confidence: details.confidence == null ? 0.55 : details.confidence,++ };++}++++function splitSentences(text) {++ return String(text || '')++ .split(/(?<=[.!?])\s+|\n+/)++ .map((s) => s.trim())++ .filter(Boolean);++}++++function stripRemember(text) {++ return String(text || '')++ .replace(/^\s*(?:please\s+)?(?:do\s+)?remember(?:\s+that)?[:,]?\s+/i, '')++ .replace(/^\s*(?:please\s+)?don'?t forget(?:\s+that)?[:,]?\s+/i, '')++ .replace(/^\s*keep in mind(?:\s+that)?[:,]?\s+/i, '')++ .replace(/^[-*•\d.)\s]+/, '')++ .trim();++}++++/** The user's own sentences that announce a durable memory; assistant text is never a candidate by itself. */++function candidatesFromText(text, details) {++ const out = [];++ for (const sentence of splitSentences(text)) {++ if (sentence.length < 12 || sentence.length > 1200) continue;++ if (!MARKERS.some((re) => re.test(sentence))) continue;++ const item = candidate(stripRemember(sentence), details);++ if (item) out.push(item);++ }++ return out;++}++++/* ── ChatGPT ─────────────────────────────────────────────────────────────────────────────────────────────── */++++function chatgptText(message) {++ const content = message && message.content;++ if (!content || typeof content !== 'object') return '';++ if (!Array.isArray(content.parts)) return '';++ return content.parts.map((part) => {++ if (typeof part === 'string') return part;++ if (part && typeof part === 'object' && typeof part.text === 'string') return part.text;++ return '';++ }).filter(Boolean).join('\n');++}++++/** The current branch of a conversation, root → leaf. A mapping is a tree; only the branch the person sees. */++function chatgptMessages(conversation) {++ const mapping = conversation && conversation.mapping;++ if (!mapping || typeof mapping !== 'object') return [];++ const chain = [];++ let id = conversation.current_node;++ const seen = new Set();++ while (id && mapping[id] && !seen.has(id)) {++ seen.add(id);++ chain.push(mapping[id]);++ id = mapping[id].parent;++ }++ if (!chain.length) chain.push(...Object.values(mapping));++ chain.reverse();++ const out = [];++ for (const node of chain) {++ const message = node && node.message;++ if (!message) continue;++ const role = String((message.author && message.author.role) || '');++ const text = chatgptText(message);++ if (!text) continue;++ out.push({ role, text, at: when(message.create_time) });++ }++ return out;++}++++function parseChatgpt(conversations, file, summary) {++ const out = [];++ for (const conversation of Array.isArray(conversations) ? conversations : []) {++ if (!conversation || typeof conversation !== 'object') continue;++ const title = clip(conversation.title || 'ChatGPT conversation', 300);++ const messages = chatgptMessages(conversation);++ summary.conversations++;++ summary.messages += messages.length;++ const at = when(conversation.create_time, messages.find((m) => m.at)?.at);++ for (const message of messages) {++ if (message.role !== 'user') continue;++ out.push(...candidatesFromText(message.text, {++ source: 'chatgpt', source_ref: { path: file, heading: title }, seen_at: message.at || at,++ }));++ }++ }++ return out;++}++++/* ── Claude ──────────────────────────────────────────────────────────────────────────────────────────────── */++++function claudeText(message) {++ if (Array.isArray(message && message.content)) {++ return message.content.map((block) => (block && typeof block.text === 'string' ? block.text : '')).filter(Boolean).join('\n');++ }++ return String((message && message.text) || '');++}++++function parseClaude(conversations, file, summary) {++ const out = [];++ for (const conversation of Array.isArray(conversations) ? conversations : []) {++ if (!conversation || typeof conversation !== 'object') continue;++ const title = clip(conversation.name || conversation.title || 'Claude conversation', 300);++ const messages = Array.isArray(conversation.chat_messages) ? conversation.chat_messages++ : Array.isArray(conversation.messages) ? conversation.messages : [];++ summary.conversations++;++ summary.messages += messages.length;++ const at = when(conversation.created_at, messages.find((m) => m.created_at)?.created_at);++ for (const message of messages) {++ const sender = String((message && (message.sender || message.role)) || '');++ if (sender !== 'human' && sender !== 'user') continue;++ out.push(...candidatesFromText(claudeText(message), {++ source: 'claude-ai', source_ref: { path: file, heading: title }, seen_at: when(message.created_at, at),++ }));++ }++ }++ return out;++}++++/** Claude's saved memory list: every entry is already a memory, so no marker is needed. */++function parseMemories(list, file, source) {++ const out = [];++ const entries = Array.isArray(list) ? list : (list && Array.isArray(list.memories) ? list.memories : []);++ for (const entry of entries) {++ if (!entry || typeof entry !== 'object') continue;++ const text = entry.content || entry.memory || entry.text || entry.value;++ if (typeof text !== 'string') continue;++ const item = candidate(text, {++ source, source_ref: { path: file, heading: 'Saved memory' },++ seen_at: when(entry.created_at || entry.updated_at), confidence: 0.9,++ });++ if (item) out.push(item);++ }++ return out;++}++++/* ── Gemini ─────────────────────────────────────────────────────────────────────────────────────────────── */++++function geminiPrompts(record) {++ const raw = String((record && (record.title || record.header)) || '').trim();++ if (!raw) return [];++ // My Activity titles read "Said …", "You said …", "Prompted …", "Asked …".++ const stripped = raw.replace(/^(?:you\s+)?(?:said|prompted|asked|searched for)\s*[:,]?\s*/i, '').replace(/^["“]|["”]$/g, '').trim();++ return stripped && stripped.length >= 8 ? [stripped] : [];++}++++function parseGeminiActivity(records, file, summary) {++ const out = [];++ const list = Array.isArray(records) ? records : (records && Array.isArray(records.items) ? records.items : []);++ for (const record of list) {++ if (!record || typeof record !== 'object') continue;++ const prompts = geminiPrompts(record);++ summary.conversations++;++ summary.messages += prompts.length;++ for (const prompt of prompts) {++ out.push(...candidatesFromText(prompt, {++ source: 'gemini-apps', source_ref: { path: file, heading: 'Gemini Apps activity' },++ seen_at: when(record.time || record.timestamp),++ }));++ }++ }++ return out;++}++++function geminiMessages(conversation) {++ const messages = Array.isArray(conversation && conversation.messages) ? conversation.messages++ : Array.isArray(conversation && conversation.turns) ? conversation.turns : [];++ return messages.map((message) => ({++ role: String((message && (message.role || message.author)) || ''),++ text: String((message && (message.text || message.content)) || ''),++ at: when(message && (message.createTime || message.created_at || message.time)),++ }));++}++++function parseGeminiConversations(conversations, file, summary) {++ const out = [];++ for (const conversation of Array.isArray(conversations) ? conversations : []) {++ if (!conversation || typeof conversation !== 'object') continue;++ const title = clip(conversation.title || 'Gemini conversation', 300);++ const messages = geminiMessages(conversation);++ summary.conversations++;++ summary.messages += messages.length;++ const at = when(conversation.createdTime || conversation.created_at, messages.find((m) => m.at)?.at);++ for (const message of messages) {++ if (message.role !== 'user' && message.role !== 'human') continue;++ out.push(...candidatesFromText(message.text, {++ source: 'gemini-apps', source_ref: { path: file, heading: title }, seen_at: message.at || at,++ }));++ }++ }++ return out;++}++++/* ── the extraction prompt's paste-back ─────────────────────────────────────────────────────────────────── */++++function labelFrom(text) {++ const bracket = /\[([a-z _-]{3,30})\]/i.exec(text);++ if (bracket) return { label: bracket[1], text: text.replace(bracket[0], ' ') };++ const paren = /\(([a-z _-]{3,30})\)\s*$/i.exec(text);++ if (paren && KIND_LABELS[paren[1].toLowerCase()]) return { label: paren[1], text: text.replace(paren[0], ' ') };++ const colon = /^\s*([a-z _-]{3,30})\s*:\s*/i.exec(text);++ if (colon && KIND_LABELS[colon[1].toLowerCase()]) return { label: colon[1], text: text.slice(colon[0].length) };++ return { label: null, text };++}++++function parsePaste(text, provider) {++ const source = provider === 'chatgpt' || provider === 'claude-ai' || provider === 'gemini-apps' ? provider : 'paste';++ const out = [];++ const raw = String(text == null ? '' : text).trim();++ if (!raw) return out;++ let parsed = null;++ if ((raw.startsWith('[') && raw.endsWith(']')) || (raw.startsWith('{') && raw.endsWith('}'))) {++ try { parsed = JSON.parse(raw); } catch { parsed = null; }++ }++ if (parsed) {++ const list = Array.isArray(parsed) ? parsed : (Array.isArray(parsed.items) ? parsed.items : []);++ for (const entry of list) {++ if (!entry || typeof entry !== 'object') continue;++ const body = entry.text || entry.content || entry.memory || entry.value;++ if (typeof body !== 'string') continue;++ const item = candidate(body, {++ source, source_ref: { path: 'pasted text', heading: providerName(source) }, label: entry.kind, confidence: 0.8,++ });++ if (item) out.push(item);++ }++ return out;++ }++ let fenced = false;++ for (const line of raw.split(/\r?\n/)) {++ if (/^\s*```/.test(line)) { fenced = !fenced; continue; }++ if (fenced) continue;++ if (/^\s*#{1,6}\s/.test(line)) continue;++ const bare = line.replace(/^\s*(?:[-*•]|\d+[.)])\s+/, '').replace(/^\*\*([^*]{2,40})\*\*\s*:?\s*/, '$1: ').trim();++ if (bare.length < 8) continue;++ const labelled = labelFrom(bare);++ // A paste is prose, not a transcript: a bullet without a label is kept when it announces a memory or starts++ // with the person's own voice ("I …", "My …", "We …"). Ordinary chatter is not a candidate.++ if (!labelled.label && !MARKERS.some((re) => re.test(labelled.text)) && !/^(?:i|my|we|our)\b/i.test(labelled.text)) continue;++ const item = candidate(labelled.text, {++ source, source_ref: { path: 'pasted text', heading: providerName(source) }, label: labelled.label, confidence: 0.8,++ });++ if (item) out.push(item);++ }++ return out;++}++++/* ── the export as a whole ────────────────────────────────────────────────────────────────────────────── */++++function readJson(file) {++ try { return JSON.parse(file.data.toString('utf8')); }++ catch { throw new ImportError(`The file ${file.name} is not valid JSON.`); }++}++++function baseName(name) { return String(name || '').split('/').pop(); }++++function detectProvider(files, provider) {++ const explicit = normalizeProvider(provider);++ if (explicit) return explicit;++ const names = files.map((f) => f.name.toLowerCase());++ if (names.some((n) => /(^|\/)my activity\.json$/.test(n) || n.includes('gemini apps/') || n.includes('/bard/'))) return 'gemini-apps';++ for (const file of files) {++ if (!/(^|\/)conversations\.json$/i.test(file.name)) continue;++ const doc = readJson(file);++ if (Array.isArray(doc) && doc.some((c) => c && c.chat_messages)) return 'claude-ai';++ if (Array.isArray(doc) && doc.some((c) => c && c.mapping)) return 'chatgpt';++ }++ if (names.some((n) => /memories\.json$/.test(n))) return 'claude-ai';++ return null;++}++++/**++ * Parse one export. `files` is what readZip returned, or `text` is the paste. No bytes are written anywhere.++ * The result is `{ provider, summary, items }`; items are redacted candidates without keys (the importer adds them).++ */++function parseExport({ provider, files = [], text = null, filename = '' } = {}) {++ const summary = {++ files: [...new Set(files.map((f) => f.name))].slice(0, 20),++ conversations: 0, messages: 0, found: 0, duplicates: 0, secrets_removed: 0,++ date_from: null, date_to: null, note: '',++ };++ let chosen = normalizeProvider(provider);++ if (text != null && !files.length) {++ chosen = chosen || 'paste';++ const items = parsePaste(text, chosen);++ summary.found = items.length;++ return { provider: chosen, summary, items };++ }++ if (!files.length) throw new ImportError('That file is empty, or it is not a zip archive.');++ chosen = detectProvider(files, chosen);++ if (!chosen) throw new ImportError('This zip is not a ChatGPT, Claude or Gemini export. Choose the assistant, or drop the .zip the export email links to.');++ const seen = new Set();++ const items = [];++ const add = (list) => {++ for (const item of list) {++ const key = item.text.toLowerCase().replace(/[^a-z0-9]+/g, ' ').trim();++ if (!key) continue;++ if (seen.has(key)) { summary.duplicates++; continue; }++ seen.add(key);++ items.push(item);++ }++ };++ const fileNames = files.map((f) => f.name.toLowerCase());++ const conversations = files.filter((f) => /(^|\/)conversations\.json$/i.test(f.name));++ const memories = files.filter((f) => /(^|\/)memories\.json$/i.test(f.name));++ if (chosen === 'chatgpt' || chosen === 'claude-ai') {++ for (const file of memories) add(parseMemories(readJson(file), baseName(file.name), chosen));++ for (const file of conversations) {++ const doc = readJson(file);++ if (!Array.isArray(doc)) throw new ImportError(`The file ${baseName(file.name)} is not the conversations list this import reads.`);++ add(chosen === 'chatgpt' ? parseChatgpt(doc, baseName(file.name), summary) : parseClaude(doc, baseName(file.name), summary));++ }++ if (!conversations.length && !memories.length) throw new ImportError('This zip has no conversations.json or memories.json in it.');++ } else if (chosen === 'gemini-apps') {++ const activity = files.filter((f) => /(^|\/)my ?activity\.json$/i.test(f.name)++ || (f.name.toLowerCase().includes('gemini apps/') && /\.json$/i.test(f.name)));++ for (const file of activity) add(parseGeminiActivity(readJson(file), baseName(file.name), summary));++ const gem = files.filter((f) => /(^|\/)gemini\/conversations\/[^/]+\.json$/i.test(f.name));++ for (const file of gem) {++ const doc = readJson(file);++ add(parseGeminiConversations(Array.isArray(doc) ? doc : [doc], baseName(file.name), summary));++ }++ if (!activity.length && !gem.length) {++ throw new ImportError('This Takeout zip has no Gemini Apps activity JSON. In Takeout, choose My Activity → Gemini Apps, not “Gemini”.');++ }++ } else if (chosen === 'paste') {++ throw new ImportError('Paste the answer to the extraction prompt, or drop an export zip.');++ }++ // Keep the preview readable: the newest candidates first, at most 200. The rest stay in the person's own file.++ items.sort((a, b) => String(b.seen_at || '').localeCompare(String(a.seen_at || '')));++ for (const item of items) {++ if (!item.seen_at) continue;++ if (!summary.date_from || item.seen_at < summary.date_from) summary.date_from = item.seen_at;++ if (!summary.date_to || item.seen_at > summary.date_to) summary.date_to = item.seen_at;++ }++ summary.found = items.length;++ if (items.length > MAX_ITEMS) summary.note = `Showing the ${MAX_ITEMS} newest of ${items.length} possible notes; the rest stay in your file.`;++ const limited = items.slice(0, MAX_ITEMS);++ summary.secrets_removed = limited.filter((i) => i.text.includes('[redacted]')).length;++ summary.kept = limited.length;++ summary.skipped = Math.max(0, items.length - limited.length);++ summary.filename = filename ? baseName(filename) : null;++ if (!limited.length) summary.note = 'No sentences in this export looked like a memory. The extraction prompt usually finds more.';++ return { provider: chosen, summary, items: limited };++}++++module.exports = {++ PROVIDERS, PROVIDER_BY_ID, EXTRACTION_PROMPT, MAX_ITEMS, ImportError,++ normalizeProvider, providerName, parseExport, parsePaste, guessKind, detectProvider,++};nnew file mode 100644@@@ -0,0 +1,144 @@++'use strict';++/**++ * A small, read-only ZIP reader for the memory import (Memory M3).++ *++ * A person's data export arrives as a .zip from ChatGPT, Claude or Google Takeout. We never write it to disk: the++ * bytes are parsed in memory, only the JSON entries are inflated, and every limit is checked before anything is++ * decompressed. No archive library is added to the service for this; the format we need is the central directory,++ * stored (0) and deflate (8) entries, and the basic ZIP64 fields used by exports over 4 GB.++ *++ * Limits: at most 5,000 entries, at most 60 MB of JSON, and at most 60 MB for any one entry. A broken archive is a++ * 422 to the person, never a stack trace: the messages here name the problem, never the archive's content.++ */++++const zlib = require('zlib');++++const EOCD = 0x06054b50;++const ZIP64_EOCD = 0x06064b50;++const ZIP64_LOCATOR = 0x07064b50;++const CENTRAL = 0x02014b50;++const LOCAL = 0x04034b50;++const MAX_ENTRIES = 5000;++const MAX_TOTAL = 60 * 1024 * 1024;++const MAX_ENTRY = 60 * 1024 * 1024;++const MAX_JSON_ENTRIES = 200;++++class ZipError extends Error {++ constructor(message) {++ super(message);++ this.code = 'import.bad_zip';++ }++}++++/** The offset of the end-of-central-directory record, or -1. It lives in the last 65,557 bytes. */++function eocdOffset(buf) {++ const from = Math.max(0, buf.length - 0xffff - 22);++ for (let i = buf.length - 22; i >= from; i--) if (buf.readUInt32LE(i) === EOCD) return i;++ return -1;++}++++/** The central directory's offset and entry count, including the ZIP64 form. */++function directory(buf) {++ const eocd = eocdOffset(buf);++ if (eocd < 0) throw new ZipError('that file is not a zip archive');++ let count = buf.readUInt16LE(eocd + 10);++ let offset = buf.readUInt32LE(eocd + 16);++ if (count === 0xffff || offset === 0xffffffff) {++ const locator = eocd - 20;++ if (locator < 0 || buf.readUInt32LE(locator) !== ZIP64_LOCATOR) throw new ZipError('that zip archive uses an unsupported ZIP64 layout');++ const z64 = Number(buf.readBigUInt64LE(locator + 8));++ if (z64 < 0 || z64 + 56 > buf.length || buf.readUInt32LE(z64) !== ZIP64_EOCD) throw new ZipError('that zip archive has a broken ZIP64 directory');++ count = Number(buf.readBigUInt64LE(z64 + 32));++ offset = Number(buf.readBigUInt64LE(z64 + 48));++ }++ if (count > MAX_ENTRIES) throw new ZipError(`that zip archive has more than ${MAX_ENTRIES} files`);++ if (offset < 0 || offset >= buf.length) throw new ZipError('that zip archive has a broken directory');++ return { count, offset };++}++++/** The ZIP64 extra field, when a central entry says a size or offset lives there. */++function zip64Extra(extra, wants) {++ let at = 0;++ while (at + 4 <= extra.length) {++ const id = extra.readUInt16LE(at);++ const size = extra.readUInt16LE(at + 2);++ const body = extra.subarray(at + 4, at + 4 + size);++ if (id === 0x0001) {++ let p = 0;++ const take = () => {++ if (p + 8 > body.length) throw new ZipError('that zip archive has a broken ZIP64 field');++ const n = Number(body.readBigUInt64LE(p));++ p += 8;++ return n;++ };++ return {++ size: wants.size ? take() : null,++ compressed: wants.compressed ? take() : null,++ offset: wants.offset ? take() : null,++ };++ }++ at += 4 + size;++ }++ throw new ZipError('that zip archive has a broken ZIP64 field');++}++++/**++ * Read one archive. Returns an array of `{ name, data }` for the JSON files only; a directory and any other file++ * are skipped, because a memory export is JSON and nothing here needs to inflate a picture.++ */++function readZip(buf) {++ if (!Buffer.isBuffer(buf) || buf.length < 22) throw new ZipError('that file is not a zip archive');++ const dir = directory(buf);++ const out = [];++ let total = 0;++ let actual = 0;++ let at = dir.offset;++ for (let n = 0; n < dir.count; n++) {++ if (at + 46 > buf.length || buf.readUInt32LE(at) !== CENTRAL) throw new ZipError('that zip archive has a broken directory');++ const flags = buf.readUInt16LE(at + 8);++ const method = buf.readUInt16LE(at + 10);++ let compressed = buf.readUInt32LE(at + 20);++ let size = buf.readUInt32LE(at + 24);++ const nameLen = buf.readUInt16LE(at + 28);++ const extraLen = buf.readUInt16LE(at + 30);++ const commentLen = buf.readUInt16LE(at + 32);++ let local = buf.readUInt32LE(at + 42);++ const name = buf.subarray(at + 46, at + 46 + nameLen).toString('utf8').replace(/\\/g, '/');++ const extra = buf.subarray(at + 46 + nameLen, at + 46 + nameLen + extraLen);++ if (compressed === 0xffffffff || size === 0xffffffff || local === 0xffffffff) {++ const z = zip64Extra(extra, { size: size === 0xffffffff, compressed: compressed === 0xffffffff, offset: local === 0xffffffff });++ if (z.compressed != null) compressed = z.compressed;++ if (z.size != null) size = z.size;++ if (z.offset != null) local = z.offset;++ }++ at += 46 + nameLen + extraLen + commentLen;++ if (name.endsWith('/')) continue;++ if (!/\.json$/i.test(name)) continue;++ if (out.length >= MAX_JSON_ENTRIES) throw new ZipError(`that zip archive has more than ${MAX_JSON_ENTRIES} JSON files`);++ if (flags & 0x1) throw new ZipError('that zip archive is encrypted');++ if (method !== 0 && method !== 8) throw new ZipError('that zip archive uses a compression this service does not read');++ if (size > MAX_ENTRY) throw new ZipError('a file in that zip archive is larger than 60 MB');++ total += size;++ if (total > MAX_TOTAL) throw new ZipError('that zip archive unpacks to more than 60 MB of JSON');++ if (local + 30 > buf.length || buf.readUInt32LE(local) !== LOCAL) throw new ZipError('that zip archive has a broken entry');++ const lName = buf.readUInt16LE(local + 26);++ const lExtra = buf.readUInt16LE(local + 28);++ const start = local + 30 + lName + lExtra;++ if (start + compressed > buf.length) throw new ZipError('that zip archive ends in the middle of a file');++ const raw = buf.subarray(start, start + compressed);++ let data;++ if (method === 0) {++ data = Buffer.from(raw);++ } else {++ try { data = zlib.inflateRawSync(raw, { maxOutputLength: Math.min(MAX_ENTRY, size || MAX_ENTRY) + 1 }); }++ catch { throw new ZipError('a file in that zip archive did not unpack'); }++ if (data.length > MAX_ENTRY) throw new ZipError('a file in that zip archive is larger than 60 MB');++ }++ actual += data.length;++ if (actual > MAX_TOTAL) throw new ZipError('that zip archive unpacks to more than 60 MB of JSON');++ out.push({ name, data });++ }++ return out;++}++++module.exports = { readZip, ZipError, MAX_ENTRY, MAX_TOTAL, MAX_JSON_ENTRIES };@@@ -22,6 +22,10 @@* at most once an hour per owner and scope* GET /api/v1/memory/search?q=&scopes= for agents (ai.memory.read + on_behalf_of): active items only, in the* granted scopes, at most 8, audited in memory_reads with ids and never text++ * GET /api/v1/memory/import/providers the cloud walkthroughs and the extraction prompt (M3, person token only)++ * POST /api/v1/memory/import/preview parse and redact an export into a sealed two-hour draft (M3)++ * POST /api/v1/memory/import/:id/commit keep the ticked candidates and delete the draft (M3)++ * DELETE /api/v1/memory/import/:id throw the draft away (M3)** Every SQL query that reads or changes a memory table is scoped by `owner = $…` (test/memory.test.js greps for it).* The account export decrypts the person's own items for their own export; deletion removes the rows.@@@ -888,8 +892,14 @@ function createMemory({ db, config, clock = { now: () => Date.now() }, log = con* names the person with `on_behalf_of`, but it may only search (audited) and import suggestions. Every other route is* person-only: a service gets 403 `capability.denied`.*/--function memoryRouter({ memory, auth, config, clock = { now: () => Date.now() }, log = console, sendError }) {++function memoryRouter({ memory, memoryImport = null, auth, config, clock = { now: () => Date.now() }, log = console, sendError }) {const r = express.Router();++ // Memory M3: an export zip is raw bytes (or one multipart form) and can be a few MB. The same route also takes++ // pasted text as JSON, which the app-level JSON parser has already read.++ const importBody = express.raw({++ type: ['multipart/form-data', 'application/zip', 'application/x-zip-compressed', 'application/octet-stream', 'text/plain'],++ limit: '25mb',++ });function guard(check) {return async function memoryGuard(req, res, next) {const ctx = req.ov;@@@ -947,6 +957,14 @@ function memoryRouter({ memory, auth, config, clock = { now: () => Date.now() },// browsing routes: a service has no business in the person's audit.r.get('/api/v1/memory/reads', guard(READ_CAP), personOnly, run(async (req) => await memory.reads(req, ownerOf(req))));r.get('/api/v1/memory/search', guard(READ_CAP), run(async (req) => await memory.search(req, ownerOf(req))));++ // M3: the guided cloud import. Person-only: a service token cannot see a person's export, hold a draft or keep++ // what it finds. The preview stores only the sealed draft; the commit goes through the same batch as M2a.++ if (memoryImport) {++ r.get('/api/v1/memory/import/providers', guard(READ_CAP), personOnly, run(async () => memoryImport.providers()));++ r.post('/api/v1/memory/import/preview', guard(WRITE_CAP), personOnly, importBody, run(async (req) => await memoryImport.preview(req, ownerOf(req))));++ r.post('/api/v1/memory/import/:id/commit', guard(WRITE_CAP), personOnly, run(async (req) => await memoryImport.commit(req, ownerOf(req), req.params.id)));++ r.delete('/api/v1/memory/import/:id', guard(WRITE_CAP), personOnly, run(async (req) => await memoryImport.discard(ownerOf(req), req.params.id)));++ }// M2c: undo a merge, and settle a conflict. Both are the person's own: a service may only search and suggest.r.post('/api/v1/memory/items/:id/split', guard(WRITE_CAP), personOnly, run(async (req) => await memory.split(req, ownerOf(req), req.params.id)));r.post('/api/v1/memory/items/:id/settle', guard(WRITE_CAP), personOnly, run(async (req) => await memory.settle(req, ownerOf(req), req.params.id)));@@@ -13,10 +13,10 @@* the log carries counts and error codes only, never text.*/--const DEFAULT_CAPS = { owners: 10, scopes: 4, embeddings: 100, modelCalls: 20, profiles: 10 };++const DEFAULT_CAPS = { owners: 10, scopes: 4, embeddings: 100, modelCalls: 20, profiles: 10, drafts: 100 };const DEFAULT_INTERVAL_MS = 15 * 60 * 1000;--function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now() }, log = console, registry = null }) {++function createMemoryJobs({ memory, memoryImport = null, config = {}, clock = { now: () => Date.now() }, log = console, registry = null }) {const settings = config.memoryJobs || {};const caps = {owners: Math.max(1, Number(settings.maxOwners) || DEFAULT_CAPS.owners),@@@ -24,6 +24,7 @@ function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now()embeddings: Math.max(0, Number(settings.maxEmbeddings) || DEFAULT_CAPS.embeddings),modelCalls: Math.max(0, Number(settings.maxModelCalls) || DEFAULT_CAPS.modelCalls),profiles: Math.max(0, Number(settings.maxProfiles) || DEFAULT_CAPS.profiles),++ drafts: Math.max(0, Number(settings.maxDrafts) || DEFAULT_CAPS.drafts),};let timer = null;let running = false;@@@ -42,9 +43,11 @@ function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now()if (running) return { skipped: 'running' };if (settings.enabled === false) return { skipped: 'disabled' };running = true;-- const out = { owners: 0, scopes: 0, embeddings: 0, merged: 0, conflicts: 0, calls: 0, profiles: 0, error: null };++ const out = { owners: 0, scopes: 0, embeddings: 0, merged: 0, conflicts: 0, calls: 0, profiles: 0, drafts: 0, error: null };try {if (await drillActive()) return { skipped: 'drill' };++ // M3: a preview that expired is deleted whether or not its person comes back.++ if (memoryImport && caps.drafts) out.drafts = await memoryImport.expireDrafts(caps.drafts);const owners = await memory.ownersForJobs(caps.owners);for (const owner of owners) {const who = owner.owner;@@@ -157,6 +157,39 @@ route('get', '/memory/search', {parameters: [query('q', str()), query('scopes', str(), 'Comma-separated scopes; default user'), query('purpose', str(), 'The task or run the person sees'), query('on_behalf_of', str())],responses: { 200: json('At most 8 active items', memoryResult), 403: problem, 503: problem, ...errors([401]) },});++// ── Memory M3: the guided cloud import (person token only) ─────────────────────────────────++route('get', '/memory/import/providers', {++ operationId: 'getMemoryImportProviders', summary: 'The walkthroughs and the extraction prompt for the cloud import', tags: ['memory'], security: memorySecurity,++ description: 'The person’s own token only. Where to request each export (ChatGPT, Claude, Gemini), what arrives, how long it takes, and the prompt to paste into any assistant. The answer contains no note.',++ responses: { 200: json('The providers and the prompt', { type: 'object', properties: { providers: arrayOf({ type: 'object' }), prompt: str() } }), 403: problem, 503: problem, ...errors([401]) },++});++route('post', '/memory/import/preview', {++ operationId: 'previewMemoryImport', summary: 'Parse and redact an export into a two-hour draft', tags: ['memory'], security: memorySecurity,++ description: 'The person’s own token only. Send a .zip export, one multipart form (the Codes page forwards it untouched), a text/plain paste, or JSON { provider, text }. Nothing enters memory_items: the candidates are sealed into memory_imports with AI_MEMORY_KEY and expire after two hours. The answer is the draft id and the redacted candidates.',++ requestBody: {++ required: true,++ content: {++ 'application/zip': { schema: { type: 'string', format: 'binary' } },++ 'multipart/form-data': { schema: { type: 'object', properties: { provider: str({ enum: ['chatgpt', 'claude-ai', 'gemini-apps', 'other'] }), file: { type: 'string', format: 'binary' }, text: str() } } },++ 'text/plain': { schema: { type: 'string' } },++ 'application/json': { schema: { type: 'object', properties: { provider: str(), text: str() } } },++ },++ },++ responses: { 200: json('The sealed draft and its candidates', ref('MemoryResult')), 413: problem, 422: problem, 503: problem, ...errors([401, 403]) },++});++route('post', '/memory/import/{id}/commit', {++ operationId: 'commitMemoryImport', summary: 'Keep the ticked candidates and delete the draft', tags: ['memory'], security: memorySecurity,++ description: 'The person’s own token only. keep names the preview keys (k1, k2, …) to store. They go through the normal batch import (redaction again, dedupe), and the draft is deleted; the keys not sent are not stored. An expired draft answers 410.',++ parameters: [path('id', str())],++ requestBody: body({ type: 'object', properties: { keep: arrayOf(str()) } }),++ responses: { 200: json('What was kept and left out', ref('MemoryResult')), 404: problem, 410: problem, 422: problem, 503: problem, ...errors([401, 403]) },++});++route('delete', '/memory/import/{id}', {++ operationId: 'discardMemoryImport', summary: 'Throw a preview draft away', tags: ['memory'], security: memorySecurity,++ description: 'The person’s own token only. The draft and its candidates are deleted; nothing was ever stored in memory_items.',++ parameters: [path('id', str())],++ responses: { 200: json('How many candidates were discarded', ref('MemoryResult')), 404: problem, 410: problem, 503: problem, ...errors([401, 403]) },++});// ── Runs ──────────────────────────────────────────────────────────────────────────────────────route('post', '/runs', {nnew file mode 100644@@@ -0,0 +1,63 @@++'use strict';++/**++ * A tiny ZIP writer for the memory-import tests. The fixtures are synthetic (no real exports in the repo); this++ * makes one in memory so a test can hand the service the same bytes a browser would upload.++ *++ * Standard local headers, deflate entries, a central directory and an end record. The reader does not verify the++ * CRC, so the writer leaves it zero; the sizes and offsets are real.++ */++const zlib = require('zlib');++++function makeZip(files) {++ const locals = [];++ const central = [];++ let offset = 0;++ for (const [name, content] of Object.entries(files)) {++ const body = Buffer.isBuffer(content) ? content : Buffer.from(String(content), 'utf8');++ const data = zlib.deflateRawSync(body);++ const nameBuf = Buffer.from(name, 'utf8');++ const local = Buffer.alloc(30);++ local.writeUInt32LE(0x04034b50, 0);++ local.writeUInt16LE(20, 4);++ local.writeUInt16LE(0x800, 6);++ local.writeUInt16LE(8, 8);++ local.writeUInt32LE(0, 14);++ local.writeUInt32LE(data.length, 18);++ local.writeUInt32LE(body.length, 22);++ local.writeUInt16LE(nameBuf.length, 26);++ local.writeUInt16LE(0, 28);++ locals.push(local, nameBuf, data);++++ const cent = Buffer.alloc(46);++ cent.writeUInt32LE(0x02014b50, 0);++ cent.writeUInt16LE(20, 4);++ cent.writeUInt16LE(20, 6);++ cent.writeUInt16LE(0x800, 8);++ cent.writeUInt16LE(8, 10);++ cent.writeUInt32LE(0, 16);++ cent.writeUInt32LE(data.length, 20);++ cent.writeUInt32LE(body.length, 24);++ cent.writeUInt16LE(nameBuf.length, 28);++ cent.writeUInt16LE(0, 30);++ cent.writeUInt16LE(0, 32);++ cent.writeUInt16LE(0, 34);++ cent.writeUInt16LE(0, 36);++ cent.writeUInt32LE(0, 38);++ cent.writeUInt32LE(offset, 42);++ central.push(cent, nameBuf);++ offset += local.length + nameBuf.length + data.length;++ }++ const directory = Buffer.concat(central);++ const end = Buffer.alloc(22);++ end.writeUInt32LE(0x06054b50, 0);++ end.writeUInt16LE(0, 4);++ end.writeUInt16LE(0, 6);++ end.writeUInt16LE(Object.keys(files).length, 8);++ end.writeUInt16LE(Object.keys(files).length, 10);++ end.writeUInt32LE(directory.length, 12);++ end.writeUInt32LE(offset, 16);++ end.writeUInt16LE(0, 20);++ return Buffer.concat([...locals, directory, end]);++}++++module.exports = { makeZip };nnew file mode 100644@@@ -0,0 +1,254 @@++'use strict';++/**++ * Memory M3: the guided cloud import. Synthetic exports only (test/fixtures/zip.js builds them in memory): the++ * preview redacts and stores no note, the draft is sealed, the commit keeps only the ticked cells, and a service++ * token can neither preview nor commit. No test reaches the network.++ */++const assert = require('assert');++const crypto = require('crypto');++const { boot, request, suite, privateKey, ISSUER } = require('./helpers');++const { makeZip } = require('./fixtures/zip');++++const t = suite('memory-import');++const KEY = 'b'.repeat(64);++const A = 'usr_01JZ0000000000000000000AAA';++const B = 'usr_01JZ0000000000000000000BBB';++const SECRET = `ghp_${'A'.repeat(36)}`;++const b64 = (o) => Buffer.from(JSON.stringify(o)).toString('base64url');++++function userToken(subject, expiresIn = 3600) {++ const now = Math.floor(Date.now() / 1000);++ const input = `${b64({ alg: 'RS256', typ: 'JWT' })}.${b64({ iss: ISSUER, sub: subject, subject_id: subject, aud: ['openvibe.network'], iat: now, exp: now + expiresIn, jti: `tok_${crypto.randomBytes(8).toString('hex')}` })}`;++ return `${input}.${crypto.sign('RSA-SHA256', Buffer.from(input), privateKey).toString('base64url')}`;++}++++/** A raw-body request: an export zip goes up as application/zip, a paste as JSON through the normal helper. */++async function upload(base, path, tok, body, contentType = 'application/zip') {++ const res = await fetch(base + path, { method: 'POST', headers: { Authorization: `Bearer ${tok}`, 'Content-Type': contentType }, body });++ const text = await res.text();++ let json = null;++ try { json = text ? JSON.parse(text) : null; } catch { json = null; }++ return { status: res.status, body: json, text };++}++++/** One multipart form, exactly what the Codes page forwards to this route. */++function multipartBody(fields, file) {++ const boundary = '----ai-memory-test';++ const parts = [];++ for (const [name, value] of Object.entries(fields || {})) {++ parts.push(`--${boundary}\r\nContent-Disposition: form-data; name="${name}"\r\n\r\n${value}\r\n`);++ }++ if (file) {++ parts.push(`--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="${file.name}"\r\nContent-Type: application/zip\r\n\r\n`);++ parts.push(file.data);++ parts.push('\r\n');++ }++ parts.push(`--${boundary}--\r\n`);++ return { boundary, body: Buffer.concat(parts.map((x) => Buffer.isBuffer(x) ? x : Buffer.from(x, 'utf8'))) };++}++++const chatgptExport = () => makeZip({++ 'conversations.json': JSON.stringify([{++ title: 'How I like to work', create_time: 1700000000, current_node: 'b',++ mapping: {++ a: { parent: null, message: { author: { role: 'user' }, content: { parts: [`Please remember that I prefer short answers and my deploy token is ${SECRET}.`] }, create_time: 1700000001 } },++ b: { parent: 'a', message: { author: { role: 'assistant' }, content: { parts: ['Noted.'] }, create_time: 1700000002 } },++ },++ }, {++ title: 'Lunch', create_time: 1700000100, current_node: 'x',++ mapping: { x: { parent: null, message: { author: { role: 'user' }, content: { parts: ['What should I eat?'] }, create_time: 1700000101 } } },++ }]),++ 'user.json': JSON.stringify({ id: 'user-1' }),++});++++const claudeExport = () => makeZip({++ 'memories.json': JSON.stringify([++ { content: 'I work as a staff engineer at a small company. I prefer British spelling.', created_at: '2026-01-10T08:00:00Z' },++ ]),++ 'conversations.json': JSON.stringify([{++ name: 'Deploy notes', created_at: '2026-01-11T09:00:00Z',++ chat_messages: [++ { sender: 'human', content: [{ type: 'text', text: 'Remember that I always deploy on Thursdays.' }], created_at: '2026-01-11T09:01:00Z' },++ { sender: 'assistant', content: [{ type: 'text', text: 'I will remember that.' }], created_at: '2026-01-11T09:02:00Z' },++ ],++ }]),++ 'projects.json': JSON.stringify([{ name: 'Demo' }]),++});++++const geminiExport = () => makeZip({++ 'Takeout/My Activity/Gemini Apps/My Activity.json': JSON.stringify([++ { title: 'Said "Remember that I like my answers with sources."', time: '2026-03-05T10:30:00Z' },++ { title: 'What is the weather?', time: '2026-03-05T11:00:00Z' },++ ]),++});++++t.test('the walkthroughs are person-only and carry the three providers and the prompt', async () => {++ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });++ try {++ const r = await request(h.base, 'GET', '/api/v1/memory/import/providers', { tok: userToken(A) });++ assert.equal(r.status, 200);++ assert.deepStrictEqual(r.body.providers.map((p) => p.id), ['chatgpt', 'claude-ai', 'gemini-apps']);++ assert.ok(r.body.providers.every((p) => p.url && p.steps.length && p.time && p.arrives), 'each walkthrough is complete');++ assert.match(r.body.prompt, /remember about me/i);++ const service = await request(h.base, 'GET', '/api/v1/memory/import/providers', { tok: require('./helpers').token('codes', ['ai.memory.read']) });++ assert.equal(service.status, 403);++ } finally { await h.stop(); }++});++++t.test('the multipart form Codes forwards is parsed here: a zip with its provider, and a pasted answer', async () => {++ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });++ try {++ const who = userToken(A);++ const zip = multipartBody({ provider: 'chatgpt' }, { name: 'export.zip', data: chatgptExport() });++ const file = await upload(h.base, '/api/v1/memory/import/preview', who, zip.body, `multipart/form-data; boundary=${zip.boundary}`);++ assert.equal(file.status, 200);++ assert.equal(file.body.import.provider, 'chatgpt');++ assert.equal(file.body.import.items.length, 1);++++ const paste = multipartBody({ provider: 'claude-ai', text: '- I prefer British spelling. [preference]' });++ const text = await upload(h.base, '/api/v1/memory/import/preview', who, paste.body, `multipart/form-data; boundary=${paste.boundary}`);++ assert.equal(text.status, 200);++ assert.equal(text.body.import.items.length, 1);++ assert.equal(text.body.import.items[0].source, 'claude-ai');++ } finally { await h.stop(); }++});++++t.test('a ChatGPT zip previews redacted candidates, stores no note, and seals the draft', async () => {++ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });++ try {++ const who = userToken(A);++ const before = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_items WHERE owner = ?').get(`user:${A}`);++ assert.equal(Number(before.n), 0);++ const out = await upload(h.base, '/api/v1/memory/import/preview', who, chatgptExport());++ assert.equal(out.status, 200);++ const draft = out.body.import;++ assert.match(draft.id, /^mimp_/);++ assert.equal(draft.provider, 'chatgpt');++ assert.equal(draft.items.length, 1, 'only the sentence that looks like a memory');++ assert.ok(!draft.items[0].text.includes('ghp_'), 'the token is gone before the preview');++ assert.ok(draft.items[0].text.includes('[redacted]'));++ assert.equal(draft.items[0].source, 'chatgpt');++ assert.equal(draft.items[0].source_ref.heading, 'How I like to work');++ assert.equal(draft.summary.conversations, 2);++ assert.equal(draft.summary.secrets_removed, 1);++ const notes = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_items WHERE owner = ?').get(`user:${A}`);++ assert.equal(Number(notes.n), 0, 'a preview is not an import');++ const row = await h.db.prepare('SELECT * FROM memory_imports WHERE owner = ?').get(`user:${A}`);++ assert.match(row.payload_enc, /^m1:/, 'the draft is sealed');++ assert.ok(!row.payload_enc.includes('short answers'), 'no plaintext draft');++ } finally { await h.stop(); }++});++++t.test('the commit keeps only the ticked cells, and the left-out ones are not stored', async () => {++ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });++ try {++ const who = userToken(A);++ const zip = makeZip({++ 'conversations.json': JSON.stringify([{++ title: 'Two memories', create_time: 1700000000, current_node: 'b',++ mapping: {++ a: { parent: null, message: { author: { role: 'user' }, content: { parts: ['Remember that I prefer tabs. Remember that I use vim every day.'] }, create_time: 1700000001 } },++ b: { parent: 'a', message: { author: { role: 'assistant' }, content: { parts: ['ok'] } } },++ },++ }]),++ });++ const preview = await upload(h.base, '/api/v1/memory/import/preview', who, zip);++ assert.equal(preview.status, 200);++ const { id, items } = preview.body.import;++ assert.equal(items.length, 2);++ const commit = await request(h.base, 'POST', `/api/v1/memory/import/${id}/commit`, { tok: who, body: { keep: [items[0].key] } });++ assert.deepStrictEqual([commit.status, commit.body.kept, commit.body.left_out], [200, 1, 1]);++ const rows = await h.db.prepare('SELECT source, status FROM memory_items WHERE owner = ?').all(`user:${A}`);++ assert.equal(rows.length, 1);++ assert.equal(rows[0].source, 'chatgpt');++ assert.equal(rows[0].status, 'active', 'the person chose it, so it is not a suggestion');++ const drafts = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_imports WHERE owner = ?').get(`user:${A}`);++ assert.equal(Number(drafts.n), 0, 'the draft is gone');++ } finally { await h.stop(); }++});++++t.test('Claude saved memories come first, and a Gemini Takeout activity file is found on its own', async () => {++ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });++ try {++ const who = userToken(A);++ const claude = await upload(h.base, '/api/v1/memory/import/preview', who, claudeExport());++ assert.equal(claude.status, 200);++ assert.equal(claude.body.import.provider, 'claude-ai');++ assert.equal(claude.body.import.items.length, 2);++ assert.ok(claude.body.import.items.some((i) => i.confidence === 0.9 && /staff engineer/.test(i.text)), 'memories.json is read directly');++ assert.ok(claude.body.import.items.some((i) => /Thursdays/.test(i.text)), 'and the conversation marker too');++++ const gemini = await upload(h.base, '/api/v1/memory/import/preview', who, geminiExport());++ assert.equal(gemini.status, 200);++ assert.equal(gemini.body.import.provider, 'gemini-apps');++ assert.equal(gemini.body.import.items.length, 1);++ assert.match(gemini.body.import.items[0].text, /answers with sources/);++ assert.equal(gemini.body.import.items[0].source, 'gemini-apps');++ } finally { await h.stop(); }++});++++t.test('a pasted extraction answer is parsed by its bullets and labels', async () => {++ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });++ try {++ const who = userToken(A);++ const out = await request(h.base, 'POST', '/api/v1/memory/import/preview', {++ tok: who,++ body: { provider: 'claude-ai', text: '- I prefer long walks. [preference]\n- Project rule: always run the tests before shipping. [project rule]\n- **Preference:** I prefer evening work.\n- Just some chatter' },++ });++ assert.equal(out.status, 200);++ assert.equal(out.body.import.items.length, 3);++ assert.deepStrictEqual(out.body.import.items.map((i) => i.kind).sort(), ['preference', 'preference', 'project_rule']);++ assert.ok(!out.body.import.items.some((i) => /chatter/.test(i.text)), 'unlabelled chatter is not a memory');++ const commit = await request(h.base, 'POST', `/api/v1/memory/import/${out.body.import.id}/commit`, { tok: who, body: { keep: out.body.import.items.map((i) => i.key) } });++ assert.equal(commit.body.kept, 3);++ } finally { await h.stop(); }++});++++t.test('a service token cannot preview or commit, another person cannot open the draft, and expiry is a 410', async () => {++ let now = Date.now();++ const h = await boot({ env: { AI_MEMORY_KEY: KEY }, clock: { now: () => now } });++ try {++ const who = userToken(A, 5 * 3600);++ const service = require('./helpers').token('codes', ['ai.memory.write']);++ const denied = await upload(h.base, '/api/v1/memory/import/preview', service, chatgptExport());++ assert.equal(denied.status, 403);++ const preview = await upload(h.base, '/api/v1/memory/import/preview', who, chatgptExport());++ const id = preview.body.import.id;++ const other = await request(h.base, 'POST', `/api/v1/memory/import/${id}/commit`, { tok: userToken(B), body: { keep: ['k1'] } });++ assert.deepStrictEqual([other.status, other.body.code], [404, 'import.not_found']);++ now += 3 * 60 * 60 * 1000;++ const late = await request(h.base, 'POST', `/api/v1/memory/import/${id}/commit`, { tok: who, body: { keep: ['k1'] } });++ assert.deepStrictEqual([late.status, late.body.code], [410, 'import.expired']);++ } finally { await h.stop(); }++});++++t.test('the maintenance loop sweeps a draft that expired while its person was away', async () => {++ let now = Date.now();++ const h = await boot({ env: { AI_MEMORY_KEY: KEY, AI_MEMORY_JOBS: 'true' }, clock: { now: () => now } });++ try {++ const preview = await upload(h.base, '/api/v1/memory/import/preview', userToken(A), chatgptExport());++ assert.equal(preview.status, 200);++ now += 3 * 60 * 60 * 1000;++ const run = await h.memoryJobs.runOnce();++ assert.equal(run.drafts, 1, 'the sweep counted the draft');++ const rows = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_imports WHERE owner = ?').get(`user:${A}`);++ assert.equal(Number(rows.n), 0, 'the expired draft is gone');++ } finally { await h.stop(); }++});++++t.test('a file that is not a zip is a 422, never a 500, and an unsupported zip says what is missing', async () => {++ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });++ try {++ const who = userToken(A);++ const bad = await upload(h.base, '/api/v1/memory/import/preview', who, Buffer.from('not a zip at all'));++ assert.deepStrictEqual([bad.status, bad.body.code], [422, 'import.bad_zip']);++ const empty = await upload(h.base, '/api/v1/memory/import/preview', who, makeZip({ 'notes.txt': 'hello' }));++ assert.deepStrictEqual([empty.status, empty.body.code], [422, 'import.invalid']);++ const noMemory = await upload(h.base, '/api/v1/memory/import/preview', who, makeZip({ 'conversations.json': JSON.stringify([{ title: 'x', mapping: {} }]) }));++ assert.equal(noMemory.status, 200, 'a valid export with no candidate is a preview that says so');++ assert.equal(noMemory.body.import.items.length, 0);++ assert.equal(noMemory.body.import.id, null);++ } finally { await h.stop(); }++});++++t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,47 @@
@@ -0,0 +1,47 @@
+# Memory import (M3): ChatGPT, Claude and Gemini exports
+
+The Memory page in Codes (`/memory?tab=import`) walks a person through bringing what their cloud assistants
+already remember into OpenVibe Memory. The parsing and the redaction live in OpenVibe.AI, beside the store M2
+reads; Codes only forwards the person's token and the bytes, then renders what comes back.
+
+## The flow
+
+1. **Preview** — `POST /api/v1/memory/import/preview` (person token; a service token is refused).
+ The body is a `.zip` export, one `multipart/form-data` form from the page, or JSON `{ provider, text }` for a
+ pasted answer. AI parses it, redacts every candidate with `openvibe-sdk/redact`, and returns the candidates.
+ Nothing is written to `memory_items`. The candidates (already redacted) are sealed into one `memory_imports`
+ row with `AI_MEMORY_KEY` (AAD `memory-import|<owner>|<id>`) and expire after two hours.
+2. **Keep** — `POST /api/v1/memory/import/:id/commit` with `{ keep: ["k1", "k3"] }`.
+ Only those keys go through the store's normal batch import (which redacts again and dedupes), and the draft is
+ deleted. Keys that are not ticked are deleted with it. `DELETE /api/v1/memory/import/:id` throws the draft away.
+3. **Walkthroughs** — `GET /api/v1/memory/import/providers` returns the three cards (where to request the export,
+ what arrives, how long) and the extraction prompt. It contains no note.
+
+## What each parser reads
+
+| Provider | File | What it gives |
+|---|---|---|
+| ChatGPT | `conversations.json` | the current branch of each conversation (`mapping`, `current_node`); only the person's sentences that announce a durable memory become candidates |
+| Claude | `memories.json` | every saved memory, confidence 0.9 (no marker needed) |
+| Claude | `conversations.json` | `chat_messages[]`, `sender: "human"`, content blocks |
+| Gemini | `Takeout/My Activity/Gemini Apps/My Activity.json` | the activity record's prompt title (`Said …`, `Prompted …`) |
+| Gemini | `Gemini/conversations/*.json` | `messages[]` with `role: "user"` |
+| Any | pasted answer | bullets, with an optional `[preference]` / `[fact]` / `[project rule]` / `[procedure]` / `[decision]` / `[lesson]` label |
+
+An export with no candidate is still a successful preview: it says so and points at the extraction prompt, which
+is the method Claude's own memory import uses. Every candidate keeps its source and date, so the note can be
+removed with the rest of that source later.
+
+## Limits and safety
+
+- at most 25 MB uploaded, 5,000 zip entries, 60 MB of JSON, 200 JSON entries, 200 candidates per preview;
+- only stored (0) and deflate (8) entries, basic ZIP64; encrypted entries and other methods are refused;
+- the archive is parsed in memory and never extracted to disk;
+- no secret is returned in a preview: redaction runs before the candidate leaves the module, and again on commit;
+- no route logs a note's text; a preview is sealed and owned, and belongs to the person's account deletion.
+
+## Tests
+
+`test/memory-import.test.js` builds its own exports with `test/fixtures/zip.js` (no real exports in the repo):
+redaction before the preview, no note stored by a preview, the sealed draft, the per-item commit, Claude
+`memories.json`, a Gemini Takeout path, a pasted answer, service-token refusal, owner isolation and expiry.
new file mode 100644
new file mode 100644
@@ -0,0 +1,16 @@
@@ -0,0 +1,16 @@
+-- phase: expand
+-- OpenVibe Memory M3: one sealed draft per import preview. The person drops a ChatGPT, Claude or Google Takeout
+-- export, sees every candidate, and keeps or removes each; nothing enters memory_items until they choose.
+--
+-- payload_enc is the whole draft (summary and candidates) sealed with AI_MEMORY_KEY, AAD
+-- `memory-import|<owner>|<id>`, so the table holds no plaintext. A draft lives for two hours and is deleted on
+-- commit, on discard, on expiry and with the person's account (server/account-data.js).
+CREATE TABLE IF NOT EXISTS memory_imports (
+ id text COLLATE "C" PRIMARY KEY,
+ owner text COLLATE "C" NOT NULL,
+ provider text COLLATE "C" NOT NULL,
+ payload_enc text COLLATE "C" NOT NULL,
+ created_at text COLLATE "C" NOT NULL,
+ expires_at text COLLATE "C" NOT NULL
+);
+CREATE INDEX IF NOT EXISTS memory_imports_owner_expires ON memory_imports (owner, expires_at);
@@ -40,6 +40,9 @@ const TABLES = [
@@ -40,6 +40,9 @@ const TABLES = [
{ table: 'memory_items', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },
{ table: 'memory_items', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },
{ table: 'memory_profiles', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },
{ table: 'memory_profiles', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },
{ table: 'memory_reads', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },
{ table: 'memory_reads', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },
+ // Memory M3: import previews are drafts, never part of the person's export; they go with the account like the
+ // notes they would become. The payload is sealed, and the row is never read by the account export.
+ { table: 'memory_imports', subject: 'owner', value: (usr) => `user:${usr}`, file: null, erase: 'delete' },
{ table: 'free_allowance_usage', subject: 'subject', file: 'free-allowance.json', erase: { keep: ledger } },
{ table: 'free_allowance_usage', subject: 'subject', file: 'free-allowance.json', erase: { keep: ledger } },
// D11 owner settings: the revisions are the service's configuration audit trail, kept like the catalogues.
// D11 owner settings: the revisions are the service's configuration audit trail, kept like the catalogues.
{ table: 'config_snapshots', subject: 'created_by', file: null, erase: { keep: 'the configuration history is the service\'s; created_by is the operator audit trail' } },
{ table: 'config_snapshots', subject: 'created_by', file: null, erase: { keep: 'the configuration history is the service\'s; created_by is the operator audit trail' } },
@@ -24,7 +24,7 @@ const pkg = require('../package.json');
@@ -24,7 +24,7 @@ const pkg = require('../package.json');
const { adminRoutes } = require('openvibe-shared/config');
const { adminRoutes } = require('openvibe-shared/config');
-function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, community = null }) {
+function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, memoryImport = null, community = null }) {
const app = express();
const app = express();
// /.well-known/agent.json (openvibe-shared/agent-layer, SEO-2): what this site serves, for agents.
// /.well-known/agent.json (openvibe-shared/agent-layer, SEO-2): what this site serves, for agents.
require('openvibe-shared/agent-layer').mount(app, { descriptor: require('./agent-descriptor') });
require('openvibe-shared/agent-layer').mount(app, { descriptor: require('./agent-descriptor') });
@@ -147,7 +147,7 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
@@ -147,7 +147,7 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
// The Anthropic-compatible endpoint (POST /v1/messages, GET /v1/models): runs of ai.messages (Codes C3).
// The Anthropic-compatible endpoint (POST /v1/messages, GET /v1/models): runs of ai.messages (Codes C3).
app.use(require('./api/messages').messagesRouter({ runs, auth, config, registry, log }));
app.use(require('./api/messages').messagesRouter({ runs, auth, config, registry, log }));
app.use(require('./api/attribution-quotas').attributionQuotasRouter({ db, quotas, auth, clock, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
app.use(require('./api/attribution-quotas').attributionQuotasRouter({ db, quotas, auth, clock, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
- if (memory) app.use(require('./memory').memoryRouter({ memory, auth, config, clock, log, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
+ if (memory) app.use(require('./memory').memoryRouter({ memory, memoryImport, auth, config, clock, log, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
if (credentials) app.use(require('./credentials').credentialsRouter({ credentials, auth, registry, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
if (credentials) app.use(require('./credentials').credentialsRouter({ credentials, auth, registry, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
// The budget of a Codes run token (Codes C3): Codes registers it when it mints the token.
// The budget of a Codes run token (Codes C3): Codes registers it when it mints the token.
if (runBudgets) app.use(require('./api/run-budgets').runBudgetsRouter({ runBudgets, auth, log, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
if (runBudgets) app.use(require('./api/run-budgets').runBudgetsRouter({ runBudgets, auth, log, sendError: (res, err, ctx) => sendError(res, err, ctx, log) }));
@@ -125,9 +125,12 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -125,9 +125,12 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
const auth = createAuth({ config, log });
const auth = createAuth({ config, log });
// The person's memory (Memory M2a): sealed item text, the profile cache and the agent-read audit.
// The person's memory (Memory M2a): sealed item text, the profile cache and the agent-read audit.
const memory = require('./memory').createMemory({ db, config, clock, log, registry, pool });
const memory = require('./memory').createMemory({ db, config, clock, log, registry, pool });
- // The store's own maintenance (Memory M2c): owner by owner, with a per-run cap. It is off in test mode and off
- // while a provider is in a drill (server/memory/jobs.js).
- const memoryJobs = require('./memory/jobs').createMemoryJobs({ memory, config, clock, log, registry });
+ // The cloud import (Memory M3): parse an export into a sealed, two-hour preview. No note goes to the store
+ // until the person chooses it.
+ const memoryImport = require('./memory/import').createMemoryImport({ memory, db, config, clock, log });
+ // The store's own maintenance (Memory M2c/M3): owner by owner, with a per-run cap, and the expired import
+ // drafts. It is off in test mode and off while a provider is in a drill (server/memory/jobs.js).
+ const memoryJobs = require('./memory/jobs').createMemoryJobs({ memory, memoryImport, config, clock, log, registry });
// The Events consumer side: a signed bench.rating.changed webhook, subscribed at boot (server/bench/events.js).
// The Events consumer side: a signed bench.rating.changed webhook, subscribed at boot (server/bench/events.js).
const accountData = accountDataLib.create({ db, log, memory });
const accountData = accountDataLib.create({ db, log, memory });
const accountSend = env.OV_OAUTH_CLIENT_SECRET ? createNetworkSender({
const accountSend = env.OV_OAUTH_CLIENT_SECRET ? createNetworkSender({
@@ -135,7 +138,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -135,7 +138,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
clientSecret: env.OV_OAUTH_CLIENT_SECRET, fetch: fetchImpl,
clientSecret: env.OV_OAUTH_CLIENT_SECRET, fetch: fetchImpl,
}) : null;
}) : null;
const benchEvents = createBenchEvents({ bench, accountData, accountSend, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });
const benchEvents = createBenchEvents({ bench, accountData, accountSend, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });
- const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, community });
+ const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, memoryImport, community });
let jwks = null;
let jwks = null;
let keyLoaded = Promise.resolve(null);
let keyLoaded = Promise.resolve(null);
if (!keys.pinned && config.nodeEnv !== 'test') {
if (!keys.pinned && config.nodeEnv !== 'test') {
new file mode 100644
new file mode 100644
@@ -0,0 +1,187 @@
@@ -0,0 +1,187 @@
+'use strict';
+/**
+ * The memory import (OpenVibe Memory M3): a guided, preview-first import of a cloud assistant's export.
+ *
+ * The flow, and why it is in two steps:
+ * 1. POST /api/v1/memory/import/preview parse, redact, and hold the candidates in one sealed, two-hour draft.
+ * Nothing enters the memory store yet: the person sees every candidate and unticks what they do not want.
+ * 2. POST /api/v1/memory/import/:id/commit take the ticked keys and import them, through the same store path
+ * M2 reads (the batch import), then delete the draft. Left-out candidates are deleted with it.
+ *
+ * The draft is a row in memory_imports whose payload is sealed with the memory key and bound to its owner and id
+ * (AAD `memory-import|<owner>|<id>`). The table holds no plaintext, the routes never log text, and an expired draft
+ * is deleted on the next preview. A service token cannot preview or commit: this is the person's own memory.
+ */
+
+const { ids } = require('openvibe-contracts');
+const { AiError, iso } = require('../../util');
+const { readZip, ZipError } = require('./zip');
+const { PROVIDERS, EXTRACTION_PROMPT, ImportError, normalizeProvider, parseExport, providerName } = require('./providers');
+
+const IMPORT_TTL_MS = 2 * 60 * 60 * 1000;
+const MAX_BYTES = 25 * 1024 * 1024;
+const MAX_PARTS = 20;
+const MAX_KEEP = 500;
+const KEEP_RE = /^k[1-9][0-9]{0,5}$/;
+
+function aadFor(owner, id) { return `memory-import|${owner}|${id}`; }
+
+/** The boundary and the one file / few fields of our own multipart form. Binary-safe through latin1. */
+function parseMultipart(buf, contentType) {
+ const m = /boundary=(?:"([^"]+)"|([^;\s]+))/i.exec(String(contentType || ''));
+ if (!m) throw new ImportError('That upload was not a valid form. Try again.');
+ const boundary = '--' + (m[1] || m[2]);
+ const text = buf.toString('latin1');
+ const fields = {};
+ let file = null;
+ let parts = 0;
+ for (const chunk of text.split(boundary).slice(1)) {
+ if (chunk.startsWith('--')) break;
+ parts++;
+ if (parts > MAX_PARTS) throw new ImportError('That form had too many parts.');
+ const part = chunk.replace(/^\r\n/, '');
+ const split = part.indexOf('\r\n\r\n');
+ if (split < 0) continue;
+ const head = part.slice(0, split);
+ let body = part.slice(split + 4);
+ if (body.endsWith('\r\n')) body = body.slice(0, -2);
+ const name = /name="([^"]+)"/i.exec(head);
+ if (!name) continue;
+ const filename = /filename="([^"]*)"/i.exec(head);
+ if (filename && filename[1]) file = { name: filename[1], data: Buffer.from(body, 'latin1') };
+ else fields[name[1]] = Buffer.from(body, 'latin1').toString('utf8');
+ }
+ return { fields, file };
+}
+
+function createMemoryImport({ memory, db, config, clock = { now: () => Date.now() }, log = console }) {
+ const q = (sql) => db.prepare(sql);
+ const expire = async (owner) => {
+ await q('DELETE FROM memory_imports WHERE owner = ? AND expires_at < ?').run(owner, iso(clock.now()));
+ };
+
+ /** The request's bytes, as a file or as pasted text, with the provider named in the form or by its shape. */
+ function readInput(req) {
+ const asked = normalizeProvider((req.query && req.query.provider) || null);
+ const type = String(req.get('content-type') || '').toLowerCase();
+ let provider = asked;
+ let filename = '';
+ let text = null;
+ let files = [];
+ let body = req.body;
+ if (Buffer.isBuffer(body)) {
+ if (body.length > MAX_BYTES) throw new ImportError('That upload is larger than 25 MB. Export only the assistant’s data, or paste the extraction answer instead.');
+ if (type.includes('multipart/form-data')) {
+ const parsed = parseMultipart(body, type);
+ if (parsed.fields.provider) provider = normalizeProvider(parsed.fields.provider) || provider;
+ if (parsed.fields.text) text = parsed.fields.text;
+ if (parsed.file) { filename = parsed.file.name; files = readZip(parsed.file.data); }
+ } else if (type.includes('text/plain')) {
+ text = body.toString('utf8');
+ } else {
+ files = readZip(body);
+ }
+ } else if (body && typeof body === 'object') {
+ if (body.provider) provider = normalizeProvider(body.provider) || provider;
+ if (typeof body.text === 'string') text = body.text;
+ } else {
+ throw new ImportError('Drop an export .zip or paste the answer to the extraction prompt.');
+ }
+ if (!files.length && (text == null || !String(text).trim())) {
+ throw new ImportError('Drop an export .zip or paste the answer to the extraction prompt.');
+ }
+ return { provider, filename, files, text };
+ }
+
+ /** Parse and redact. Nothing is written here; the draft below is the only row. */
+ async function preview(req, owner) {
+ memory.assertConfigured();
+ let parsed;
+ try {
+ const input = readInput(req);
+ parsed = parseExport(input);
+ } catch (err) {
+ if (err instanceof ZipError) throw new AiError(422, 'import.bad_zip', err.message);
+ if (err instanceof ImportError) throw new AiError(err.status || 422, err.code, err.message);
+ throw err;
+ }
+ const now = iso(clock.now());
+ await expire(owner);
+ if (!parsed.items.length) {
+ return { import: { id: null, provider: parsed.provider, summary: parsed.summary, items: [], expires_at: null } };
+ }
+ const id = `mimp_${ids.ulid()}`;
+ const items = parsed.items.map((item, i) => ({ ...item, key: `k${i + 1}` }));
+ const expires = new Date(clock.now() + IMPORT_TTL_MS).toISOString();
+ const payload = { provider: parsed.provider, summary: parsed.summary, items };
+ await q('INSERT INTO memory_imports (id, owner, provider, payload_enc, created_at, expires_at) VALUES (?, ?, ?, ?, ?, ?)')
+ .run(id, owner, parsed.provider, memory.seal(JSON.stringify(payload), aadFor(owner, id)), now, expires);
+ return {
+ import: {
+ id, provider: parsed.provider, provider_name: providerName(parsed.provider),
+ summary: parsed.summary, items, created_at: now, expires_at: expires,
+ },
+ };
+ }
+
+ async function draft(owner, id) {
+ if (!/^mimp_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(id || ''))) throw new AiError(404, 'import.not_found', 'That preview is no longer here.');
+ const row = await q('SELECT * FROM memory_imports WHERE owner = ? AND id = ?').get(owner, id);
+ if (!row) throw new AiError(404, 'import.not_found', 'That preview is no longer here.');
+ if (String(row.expires_at) < iso(clock.now())) {
+ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);
+ throw new AiError(410, 'import.expired', 'That preview has expired. Drop the export again.');
+ }
+ let payload;
+ try { payload = JSON.parse(memory.open(row.payload_enc, aadFor(owner, id))); }
+ catch (err) { throw err instanceof AiError ? err : new AiError(500, 'memory.unreadable', 'the preview does not open'); }
+ return { row, payload };
+ }
+
+ /** Keep the ticked candidates. The store's own batch path redacts again and dedupes; the draft is then gone. */
+ async function commit(req, owner, id) {
+ memory.assertConfigured();
+ const { payload } = await draft(owner, id);
+ const body = req.body && typeof req.body === 'object' ? req.body : {};
+ const raw = body.keep == null ? [] : Array.isArray(body.keep) ? body.keep : [body.keep];
+ if (raw.length > MAX_KEEP) throw new AiError(422, 'import.too_many', `Keep at most ${MAX_KEEP} notes at a time.`);
+ const keys = new Set();
+ for (const key of raw) {
+ if (!KEEP_RE.test(String(key))) throw new AiError(422, 'import.invalid', 'one of the chosen notes is not from this preview');
+ keys.add(String(key));
+ }
+ const selected = (payload.items || []).filter((item) => keys.has(item.key));
+ const leftOut = Math.max(0, (payload.items || []).length - selected.length);
+ if (!selected.length) {
+ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);
+ return { kept: 0, created: 0, deduped: 0, left_out: leftOut };
+ }
+ const out = await memory.batch({ body: { items: selected }, principal: req.principal }, owner);
+ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);
+ log.log(`[memory] imported ${out.created + out.deduped} note(s) from ${payload.provider}; ${leftOut} left out`);
+ return { kept: out.created + out.deduped, created: out.created, deduped: out.deduped, left_out: leftOut };
+ }
+
+ async function discard(owner, id) {
+ const { payload } = await draft(owner, id);
+ await q('DELETE FROM memory_imports WHERE owner = ? AND id = ?').run(owner, id);
+ return { discarded: (payload.items || []).length };
+ }
+
+ /** The maintenance loop's sweep: expired drafts go without waiting for the person to come back. */
+ async function expireDrafts(limit = 100) {
+ const n = Math.max(1, Math.min(500, Number(limit) || 100));
+ const rows = await q(`SELECT id FROM memory_imports WHERE expires_at < ? ORDER BY expires_at LIMIT ${n}`).all(iso(clock.now()));
+ for (const row of rows) await q('DELETE FROM memory_imports WHERE id = ?').run(row.id);
+ return rows.length;
+ }
+
+ /** The walkthrough copy and the extraction prompt. No note ever appears in it. */
+ function providers() {
+ return { providers: PROVIDERS, prompt: EXTRACTION_PROMPT };
+ }
+
+ return { preview, commit, discard, providers, expire, expireDrafts, aadFor };
+}
+
+module.exports = { createMemoryImport, parseMultipart, IMPORT_TTL_MS, MAX_BYTES };
new file mode 100644
new file mode 100644
@@ -0,0 +1,515 @@
@@ -0,0 +1,515 @@
+'use strict';
+/**
+ * The cloud import adapters (OpenVibe Memory M3): ChatGPT, Claude and Gemini exports, and the answer to the
+ * extraction prompt. This file turns an export into *candidate* memory items; it never stores anything and it
+ * never returns raw conversation text. Only a sentence that looks like a memory survives, already redacted.
+ *
+ * The formats, read from the vendors' own exports:
+ * ChatGPT a zip with conversations.json: an array of { title, create_time, mapping }, where a message has
+ * `author.role` and `content.parts`.
+ * Claude a zip with conversations.json ({ chat_messages[] } with `sender` and content blocks) and, on plans
+ * that have memory, memories.json ({ content, created_at }). The saved memories are the strongest signal.
+ * Gemini Google Takeout: Takeout/My Activity/Gemini Apps/My Activity.json (activity records with a `title`
+ * holding the prompt), and, on some accounts, Gemini/conversations/*.json.
+ *
+ * The extraction prompt is the method Claude's own memory import uses: ask the assistant to list what it remembers,
+ * then paste the answer back. The answer is parsed as bullets, with an optional [preference] / [fact] / … label.
+ *
+ * Redaction runs before a candidate is returned, so a secret never reaches the preview, the browser or the store.
+ * The store redacts again in its batch import; two passes are deliberate.
+ */
+
+const { redact, onlySecret } = require('openvibe-sdk/redact');
+
+const MAX_ITEMS = 200;
+const MAX_ITEM_CHARS = 600;
+const MAX_MESSAGES = 200000;
+
+const EXTRACTION_PROMPT = [
+ 'List everything you remember about me as short bullet points.',
+ 'For each bullet, give only the memory in one sentence, then a label in square brackets: [preference], [fact],',
+ '[project rule], [procedure], [decision] or [lesson]. Include saved memories, custom instructions and things you',
+ 'have learned about how I work. Do not guess. Do not include passwords, keys, tokens or anything else secret.',
+].join(' ');
+
+const PROVIDERS = [
+ {
+ id: 'chatgpt',
+ name: 'ChatGPT',
+ what: 'Your chats. ChatGPT does not put its saved-memory list in the data export, so paste the extraction prompt too if you want that list.',
+ url: 'https://chatgpt.com/#settings/DataControls',
+ export_label: 'Settings → Data controls → Export data',
+ steps: [
+ 'Open Settings → Data controls → Export data and choose Export.',
+ 'OpenAI emails you a download link; the file is a .zip.',
+ 'Drop the .zip here. We read conversations.json and skip the HTML copy.',
+ ],
+ arrives: 'A .zip with conversations.json, chat.html, user.json and a few small JSON files.',
+ time: 'Usually a few minutes. A very large history can take up to 24 hours; the link is emailed to you.',
+ prompt_where: 'Start a new chat in ChatGPT and paste the prompt below, then paste its answer back here.',
+ },
+ {
+ id: 'claude-ai',
+ name: 'Claude',
+ what: 'Claude’s own memory list, when your plan exports it, plus the chats in the same export.',
+ url: 'https://claude.ai/settings/data-privacy-controls',
+ export_label: 'Settings → Privacy → Export data',
+ steps: [
+ 'Open Settings → Privacy → Export data (Claude emails you a link when it is ready).',
+ 'Download the .zip and drop it here.',
+ 'We read memories.json first, then conversations.json for things you asked Claude to remember.',
+ ],
+ arrives: 'A .zip with conversations.json, memories.json (on plans with memory), projects.json and users.json.',
+ time: 'Usually minutes, sometimes a few hours. The link is valid for a limited time.',
+ prompt_where: 'Start a new chat in Claude and paste the prompt below, then paste its answer back here.',
+ },
+ {
+ id: 'gemini-apps',
+ name: 'Gemini',
+ what: 'Your Gemini Apps activity from Google Takeout. The separate “Gemini” Takeout option is Gems, not chats.',
+ url: 'https://takeout.google.com',
+ export_label: 'Takeout → My Activity → Gemini Apps',
+ steps: [
+ 'In Google Takeout choose Deselect all, then select My Activity.',
+ 'Open All activity data included, deselect all, and tick only Gemini Apps (called Bard on older accounts).',
+ 'Export once as a .zip, download it and drop it here.',
+ ],
+ arrives: 'Takeout/My Activity/Gemini Apps/My Activity.json, and sometimes Gemini/conversations/*.json.',
+ time: 'A few minutes to several hours; Google emails you when it is ready. The download link expires in 7 days.',
+ prompt_where: 'Start a new chat in Gemini and paste the prompt below, then paste its answer back here.',
+ },
+];
+
+const PROVIDER_BY_ID = new Map(PROVIDERS.map((p) => [p.id, p]));
+const ALIASES = { chatgpt: 'chatgpt', openai: 'chatgpt', claude: 'claude-ai', 'claude-ai': 'claude-ai', claudeai: 'claude-ai', gemini: 'gemini-apps', bard: 'gemini-apps', 'gemini-apps': 'gemini-apps', takeout: 'gemini-apps', paste: 'paste', other: 'paste' };
+const SOURCE_NAMES = { chatgpt: 'ChatGPT', 'claude-ai': 'Claude', 'gemini-apps': 'Gemini', paste: 'Pasted memory' };
+
+class ImportError extends Error {
+ constructor(message, code = 'import.invalid') {
+ super(message);
+ this.code = code;
+ this.status = 422;
+ }
+}
+
+function normalizeProvider(value, fallback = null) {
+ const key = String(value == null ? '' : value).trim().toLowerCase();
+ return ALIASES[key] || fallback;
+}
+
+function providerName(id) {
+ return SOURCE_NAMES[id] || String(id || 'memory');
+}
+
+/** An ISO date-time from a vendor timestamp (seconds as a number, milliseconds, or a string). */
+function when(value, fallback) {
+ if (value == null || value === '') return fallback || null;
+ let d;
+ if (typeof value === 'number') d = new Date(value < 1e12 ? value * 1000 : value);
+ else d = new Date(value);
+ return Number.isFinite(d.getTime()) ? d.toISOString() : (fallback || null);
+}
+
+function clip(text, max = MAX_ITEM_CHARS) {
+ const s = String(text == null ? '' : text).replace(/\s+/g, ' ').trim();
+ return s.length > max ? s.slice(0, max - 1).trimEnd() + '…' : s;
+}
+
+const MARKERS = [
+ /\bremem(?:ber|bered)\b/i,
+ /\bdon'?t forget\b/i,
+ /\bkeep (?:this )?in mind\b/i,
+ /\bmy name is\b/i,
+ /\bcall me\b/i,
+ /\bi(?:'m| am) (?:a|an)\b/i,
+ /\bi (?:live|work|grew up) (?:in|at|on|for)\b/i,
+ /\bi (?:prefer|like|love|hate|avoid|use|always|never|usually)\b/i,
+ /\bmy (?:favourite|favorite|preferred|pronouns?|timezone|birthday|team|company|role|job|editor|setup|stack|repo|project)\b/i,
+ /\bwe (?:decided|agreed|chose|use|always|never)\b/i,
+ /\b(?:in|for) this (?:project|repo|repository|codebase|app)\b/i,
+ /\bplease (?:always|never|use|keep|remember)\b/i,
+ /\bfrom now on\b/i,
+ /\b(?:always|never) (?:use|do|run|write|name|call|put)\b/i,
+];
+
+const KIND_LABELS = {
+ preference: 'preference', preferred: 'preference', fact: 'fact', 'project rule': 'project_rule',
+ 'project-rule': 'project_rule', project_rule: 'project_rule', procedure: 'procedure', process: 'procedure',
+ skill: 'procedure', decision: 'decision', lesson: 'lesson',
+};
+
+function guessKind(text) {
+ const s = String(text || '');
+ if (/\b(?:decid(?:ed|e)|decision|agreed|settled on|chose)\b/i.test(s)) return 'decision';
+ if (/\b(?:learned|learnt|lesson|mistake|realized|realised)\b/i.test(s)) return 'lesson';
+ if (/\b(?:project|repo|repository|codebase|our stack|this app)\b/i.test(s)
+ && /\b(?:use|rule|convention|always|never|keep|must)\b/i.test(s)) return 'project_rule';
+ if (/\b(?:run|install|command|step|workflow|process|procedure)\b/i.test(s)
+ && /\b(?:to|then|first|before|after|use)\b/i.test(s)) return 'procedure';
+ if (/\b(?:prefer|favourite|favorite|like|love|hate|rather|always|never|avoid)\b/i.test(s)) return 'preference';
+ return 'fact';
+}
+
+/** A stored-memory candidate, after redaction. Null means it was blank or nothing but a secret. */
+function candidate(text, details) {
+ const raw = clip(text);
+ if (raw.length < 8) return null;
+ const clean = clip(redact(raw).text);
+ if (!clean || onlySecret(clean)) return null;
+ return {
+ text: clean,
+ kind: KIND_LABELS[String(details.label || '').toLowerCase()] || details.kind || guessKind(clean),
+ scope: details.scope || 'user',
+ source: details.source,
+ source_ref: details.source_ref || null,
+ seen_at: details.seen_at || null,
+ confidence: details.confidence == null ? 0.55 : details.confidence,
+ };
+}
+
+function splitSentences(text) {
+ return String(text || '')
+ .split(/(?<=[.!?])\s+|\n+/)
+ .map((s) => s.trim())
+ .filter(Boolean);
+}
+
+function stripRemember(text) {
+ return String(text || '')
+ .replace(/^\s*(?:please\s+)?(?:do\s+)?remember(?:\s+that)?[:,]?\s+/i, '')
+ .replace(/^\s*(?:please\s+)?don'?t forget(?:\s+that)?[:,]?\s+/i, '')
+ .replace(/^\s*keep in mind(?:\s+that)?[:,]?\s+/i, '')
+ .replace(/^[-*•\d.)\s]+/, '')
+ .trim();
+}
+
+/** The user's own sentences that announce a durable memory; assistant text is never a candidate by itself. */
+function candidatesFromText(text, details) {
+ const out = [];
+ for (const sentence of splitSentences(text)) {
+ if (sentence.length < 12 || sentence.length > 1200) continue;
+ if (!MARKERS.some((re) => re.test(sentence))) continue;
+ const item = candidate(stripRemember(sentence), details);
+ if (item) out.push(item);
+ }
+ return out;
+}
+
+/* ── ChatGPT ─────────────────────────────────────────────────────────────────────────────────────────────── */
+
+function chatgptText(message) {
+ const content = message && message.content;
+ if (!content || typeof content !== 'object') return '';
+ if (!Array.isArray(content.parts)) return '';
+ return content.parts.map((part) => {
+ if (typeof part === 'string') return part;
+ if (part && typeof part === 'object' && typeof part.text === 'string') return part.text;
+ return '';
+ }).filter(Boolean).join('\n');
+}
+
+/** The current branch of a conversation, root → leaf. A mapping is a tree; only the branch the person sees. */
+function chatgptMessages(conversation) {
+ const mapping = conversation && conversation.mapping;
+ if (!mapping || typeof mapping !== 'object') return [];
+ const chain = [];
+ let id = conversation.current_node;
+ const seen = new Set();
+ while (id && mapping[id] && !seen.has(id)) {
+ seen.add(id);
+ chain.push(mapping[id]);
+ id = mapping[id].parent;
+ }
+ if (!chain.length) chain.push(...Object.values(mapping));
+ chain.reverse();
+ const out = [];
+ for (const node of chain) {
+ const message = node && node.message;
+ if (!message) continue;
+ const role = String((message.author && message.author.role) || '');
+ const text = chatgptText(message);
+ if (!text) continue;
+ out.push({ role, text, at: when(message.create_time) });
+ }
+ return out;
+}
+
+function parseChatgpt(conversations, file, summary) {
+ const out = [];
+ for (const conversation of Array.isArray(conversations) ? conversations : []) {
+ if (!conversation || typeof conversation !== 'object') continue;
+ const title = clip(conversation.title || 'ChatGPT conversation', 300);
+ const messages = chatgptMessages(conversation);
+ summary.conversations++;
+ summary.messages += messages.length;
+ const at = when(conversation.create_time, messages.find((m) => m.at)?.at);
+ for (const message of messages) {
+ if (message.role !== 'user') continue;
+ out.push(...candidatesFromText(message.text, {
+ source: 'chatgpt', source_ref: { path: file, heading: title }, seen_at: message.at || at,
+ }));
+ }
+ }
+ return out;
+}
+
+/* ── Claude ──────────────────────────────────────────────────────────────────────────────────────────────── */
+
+function claudeText(message) {
+ if (Array.isArray(message && message.content)) {
+ return message.content.map((block) => (block && typeof block.text === 'string' ? block.text : '')).filter(Boolean).join('\n');
+ }
+ return String((message && message.text) || '');
+}
+
+function parseClaude(conversations, file, summary) {
+ const out = [];
+ for (const conversation of Array.isArray(conversations) ? conversations : []) {
+ if (!conversation || typeof conversation !== 'object') continue;
+ const title = clip(conversation.name || conversation.title || 'Claude conversation', 300);
+ const messages = Array.isArray(conversation.chat_messages) ? conversation.chat_messages
+ : Array.isArray(conversation.messages) ? conversation.messages : [];
+ summary.conversations++;
+ summary.messages += messages.length;
+ const at = when(conversation.created_at, messages.find((m) => m.created_at)?.created_at);
+ for (const message of messages) {
+ const sender = String((message && (message.sender || message.role)) || '');
+ if (sender !== 'human' && sender !== 'user') continue;
+ out.push(...candidatesFromText(claudeText(message), {
+ source: 'claude-ai', source_ref: { path: file, heading: title }, seen_at: when(message.created_at, at),
+ }));
+ }
+ }
+ return out;
+}
+
+/** Claude's saved memory list: every entry is already a memory, so no marker is needed. */
+function parseMemories(list, file, source) {
+ const out = [];
+ const entries = Array.isArray(list) ? list : (list && Array.isArray(list.memories) ? list.memories : []);
+ for (const entry of entries) {
+ if (!entry || typeof entry !== 'object') continue;
+ const text = entry.content || entry.memory || entry.text || entry.value;
+ if (typeof text !== 'string') continue;
+ const item = candidate(text, {
+ source, source_ref: { path: file, heading: 'Saved memory' },
+ seen_at: when(entry.created_at || entry.updated_at), confidence: 0.9,
+ });
+ if (item) out.push(item);
+ }
+ return out;
+}
+
+/* ── Gemini ─────────────────────────────────────────────────────────────────────────────────────────────── */
+
+function geminiPrompts(record) {
+ const raw = String((record && (record.title || record.header)) || '').trim();
+ if (!raw) return [];
+ // My Activity titles read "Said …", "You said …", "Prompted …", "Asked …".
+ const stripped = raw.replace(/^(?:you\s+)?(?:said|prompted|asked|searched for)\s*[:,]?\s*/i, '').replace(/^["“]|["”]$/g, '').trim();
+ return stripped && stripped.length >= 8 ? [stripped] : [];
+}
+
+function parseGeminiActivity(records, file, summary) {
+ const out = [];
+ const list = Array.isArray(records) ? records : (records && Array.isArray(records.items) ? records.items : []);
+ for (const record of list) {
+ if (!record || typeof record !== 'object') continue;
+ const prompts = geminiPrompts(record);
+ summary.conversations++;
+ summary.messages += prompts.length;
+ for (const prompt of prompts) {
+ out.push(...candidatesFromText(prompt, {
+ source: 'gemini-apps', source_ref: { path: file, heading: 'Gemini Apps activity' },
+ seen_at: when(record.time || record.timestamp),
+ }));
+ }
+ }
+ return out;
+}
+
+function geminiMessages(conversation) {
+ const messages = Array.isArray(conversation && conversation.messages) ? conversation.messages
+ : Array.isArray(conversation && conversation.turns) ? conversation.turns : [];
+ return messages.map((message) => ({
+ role: String((message && (message.role || message.author)) || ''),
+ text: String((message && (message.text || message.content)) || ''),
+ at: when(message && (message.createTime || message.created_at || message.time)),
+ }));
+}
+
+function parseGeminiConversations(conversations, file, summary) {
+ const out = [];
+ for (const conversation of Array.isArray(conversations) ? conversations : []) {
+ if (!conversation || typeof conversation !== 'object') continue;
+ const title = clip(conversation.title || 'Gemini conversation', 300);
+ const messages = geminiMessages(conversation);
+ summary.conversations++;
+ summary.messages += messages.length;
+ const at = when(conversation.createdTime || conversation.created_at, messages.find((m) => m.at)?.at);
+ for (const message of messages) {
+ if (message.role !== 'user' && message.role !== 'human') continue;
+ out.push(...candidatesFromText(message.text, {
+ source: 'gemini-apps', source_ref: { path: file, heading: title }, seen_at: message.at || at,
+ }));
+ }
+ }
+ return out;
+}
+
+/* ── the extraction prompt's paste-back ─────────────────────────────────────────────────────────────────── */
+
+function labelFrom(text) {
+ const bracket = /\[([a-z _-]{3,30})\]/i.exec(text);
+ if (bracket) return { label: bracket[1], text: text.replace(bracket[0], ' ') };
+ const paren = /\(([a-z _-]{3,30})\)\s*$/i.exec(text);
+ if (paren && KIND_LABELS[paren[1].toLowerCase()]) return { label: paren[1], text: text.replace(paren[0], ' ') };
+ const colon = /^\s*([a-z _-]{3,30})\s*:\s*/i.exec(text);
+ if (colon && KIND_LABELS[colon[1].toLowerCase()]) return { label: colon[1], text: text.slice(colon[0].length) };
+ return { label: null, text };
+}
+
+function parsePaste(text, provider) {
+ const source = provider === 'chatgpt' || provider === 'claude-ai' || provider === 'gemini-apps' ? provider : 'paste';
+ const out = [];
+ const raw = String(text == null ? '' : text).trim();
+ if (!raw) return out;
+ let parsed = null;
+ if ((raw.startsWith('[') && raw.endsWith(']')) || (raw.startsWith('{') && raw.endsWith('}'))) {
+ try { parsed = JSON.parse(raw); } catch { parsed = null; }
+ }
+ if (parsed) {
+ const list = Array.isArray(parsed) ? parsed : (Array.isArray(parsed.items) ? parsed.items : []);
+ for (const entry of list) {
+ if (!entry || typeof entry !== 'object') continue;
+ const body = entry.text || entry.content || entry.memory || entry.value;
+ if (typeof body !== 'string') continue;
+ const item = candidate(body, {
+ source, source_ref: { path: 'pasted text', heading: providerName(source) }, label: entry.kind, confidence: 0.8,
+ });
+ if (item) out.push(item);
+ }
+ return out;
+ }
+ let fenced = false;
+ for (const line of raw.split(/\r?\n/)) {
+ if (/^\s*```/.test(line)) { fenced = !fenced; continue; }
+ if (fenced) continue;
+ if (/^\s*#{1,6}\s/.test(line)) continue;
+ const bare = line.replace(/^\s*(?:[-*•]|\d+[.)])\s+/, '').replace(/^\*\*([^*]{2,40})\*\*\s*:?\s*/, '$1: ').trim();
+ if (bare.length < 8) continue;
+ const labelled = labelFrom(bare);
+ // A paste is prose, not a transcript: a bullet without a label is kept when it announces a memory or starts
+ // with the person's own voice ("I …", "My …", "We …"). Ordinary chatter is not a candidate.
+ if (!labelled.label && !MARKERS.some((re) => re.test(labelled.text)) && !/^(?:i|my|we|our)\b/i.test(labelled.text)) continue;
+ const item = candidate(labelled.text, {
+ source, source_ref: { path: 'pasted text', heading: providerName(source) }, label: labelled.label, confidence: 0.8,
+ });
+ if (item) out.push(item);
+ }
+ return out;
+}
+
+/* ── the export as a whole ────────────────────────────────────────────────────────────────────────────── */
+
+function readJson(file) {
+ try { return JSON.parse(file.data.toString('utf8')); }
+ catch { throw new ImportError(`The file ${file.name} is not valid JSON.`); }
+}
+
+function baseName(name) { return String(name || '').split('/').pop(); }
+
+function detectProvider(files, provider) {
+ const explicit = normalizeProvider(provider);
+ if (explicit) return explicit;
+ const names = files.map((f) => f.name.toLowerCase());
+ if (names.some((n) => /(^|\/)my activity\.json$/.test(n) || n.includes('gemini apps/') || n.includes('/bard/'))) return 'gemini-apps';
+ for (const file of files) {
+ if (!/(^|\/)conversations\.json$/i.test(file.name)) continue;
+ const doc = readJson(file);
+ if (Array.isArray(doc) && doc.some((c) => c && c.chat_messages)) return 'claude-ai';
+ if (Array.isArray(doc) && doc.some((c) => c && c.mapping)) return 'chatgpt';
+ }
+ if (names.some((n) => /memories\.json$/.test(n))) return 'claude-ai';
+ return null;
+}
+
+/**
+ * Parse one export. `files` is what readZip returned, or `text` is the paste. No bytes are written anywhere.
+ * The result is `{ provider, summary, items }`; items are redacted candidates without keys (the importer adds them).
+ */
+function parseExport({ provider, files = [], text = null, filename = '' } = {}) {
+ const summary = {
+ files: [...new Set(files.map((f) => f.name))].slice(0, 20),
+ conversations: 0, messages: 0, found: 0, duplicates: 0, secrets_removed: 0,
+ date_from: null, date_to: null, note: '',
+ };
+ let chosen = normalizeProvider(provider);
+ if (text != null && !files.length) {
+ chosen = chosen || 'paste';
+ const items = parsePaste(text, chosen);
+ summary.found = items.length;
+ return { provider: chosen, summary, items };
+ }
+ if (!files.length) throw new ImportError('That file is empty, or it is not a zip archive.');
+ chosen = detectProvider(files, chosen);
+ if (!chosen) throw new ImportError('This zip is not a ChatGPT, Claude or Gemini export. Choose the assistant, or drop the .zip the export email links to.');
+ const seen = new Set();
+ const items = [];
+ const add = (list) => {
+ for (const item of list) {
+ const key = item.text.toLowerCase().replace(/[^a-z0-9]+/g, ' ').trim();
+ if (!key) continue;
+ if (seen.has(key)) { summary.duplicates++; continue; }
+ seen.add(key);
+ items.push(item);
+ }
+ };
+ const fileNames = files.map((f) => f.name.toLowerCase());
+ const conversations = files.filter((f) => /(^|\/)conversations\.json$/i.test(f.name));
+ const memories = files.filter((f) => /(^|\/)memories\.json$/i.test(f.name));
+ if (chosen === 'chatgpt' || chosen === 'claude-ai') {
+ for (const file of memories) add(parseMemories(readJson(file), baseName(file.name), chosen));
+ for (const file of conversations) {
+ const doc = readJson(file);
+ if (!Array.isArray(doc)) throw new ImportError(`The file ${baseName(file.name)} is not the conversations list this import reads.`);
+ add(chosen === 'chatgpt' ? parseChatgpt(doc, baseName(file.name), summary) : parseClaude(doc, baseName(file.name), summary));
+ }
+ if (!conversations.length && !memories.length) throw new ImportError('This zip has no conversations.json or memories.json in it.');
+ } else if (chosen === 'gemini-apps') {
+ const activity = files.filter((f) => /(^|\/)my ?activity\.json$/i.test(f.name)
+ || (f.name.toLowerCase().includes('gemini apps/') && /\.json$/i.test(f.name)));
+ for (const file of activity) add(parseGeminiActivity(readJson(file), baseName(file.name), summary));
+ const gem = files.filter((f) => /(^|\/)gemini\/conversations\/[^/]+\.json$/i.test(f.name));
+ for (const file of gem) {
+ const doc = readJson(file);
+ add(parseGeminiConversations(Array.isArray(doc) ? doc : [doc], baseName(file.name), summary));
+ }
+ if (!activity.length && !gem.length) {
+ throw new ImportError('This Takeout zip has no Gemini Apps activity JSON. In Takeout, choose My Activity → Gemini Apps, not “Gemini”.');
+ }
+ } else if (chosen === 'paste') {
+ throw new ImportError('Paste the answer to the extraction prompt, or drop an export zip.');
+ }
+ // Keep the preview readable: the newest candidates first, at most 200. The rest stay in the person's own file.
+ items.sort((a, b) => String(b.seen_at || '').localeCompare(String(a.seen_at || '')));
+ for (const item of items) {
+ if (!item.seen_at) continue;
+ if (!summary.date_from || item.seen_at < summary.date_from) summary.date_from = item.seen_at;
+ if (!summary.date_to || item.seen_at > summary.date_to) summary.date_to = item.seen_at;
+ }
+ summary.found = items.length;
+ if (items.length > MAX_ITEMS) summary.note = `Showing the ${MAX_ITEMS} newest of ${items.length} possible notes; the rest stay in your file.`;
+ const limited = items.slice(0, MAX_ITEMS);
+ summary.secrets_removed = limited.filter((i) => i.text.includes('[redacted]')).length;
+ summary.kept = limited.length;
+ summary.skipped = Math.max(0, items.length - limited.length);
+ summary.filename = filename ? baseName(filename) : null;
+ if (!limited.length) summary.note = 'No sentences in this export looked like a memory. The extraction prompt usually finds more.';
+ return { provider: chosen, summary, items: limited };
+}
+
+module.exports = {
+ PROVIDERS, PROVIDER_BY_ID, EXTRACTION_PROMPT, MAX_ITEMS, ImportError,
+ normalizeProvider, providerName, parseExport, parsePaste, guessKind, detectProvider,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,144 @@
@@ -0,0 +1,144 @@
+'use strict';
+/**
+ * A small, read-only ZIP reader for the memory import (Memory M3).
+ *
+ * A person's data export arrives as a .zip from ChatGPT, Claude or Google Takeout. We never write it to disk: the
+ * bytes are parsed in memory, only the JSON entries are inflated, and every limit is checked before anything is
+ * decompressed. No archive library is added to the service for this; the format we need is the central directory,
+ * stored (0) and deflate (8) entries, and the basic ZIP64 fields used by exports over 4 GB.
+ *
+ * Limits: at most 5,000 entries, at most 60 MB of JSON, and at most 60 MB for any one entry. A broken archive is a
+ * 422 to the person, never a stack trace: the messages here name the problem, never the archive's content.
+ */
+
+const zlib = require('zlib');
+
+const EOCD = 0x06054b50;
+const ZIP64_EOCD = 0x06064b50;
+const ZIP64_LOCATOR = 0x07064b50;
+const CENTRAL = 0x02014b50;
+const LOCAL = 0x04034b50;
+const MAX_ENTRIES = 5000;
+const MAX_TOTAL = 60 * 1024 * 1024;
+const MAX_ENTRY = 60 * 1024 * 1024;
+const MAX_JSON_ENTRIES = 200;
+
+class ZipError extends Error {
+ constructor(message) {
+ super(message);
+ this.code = 'import.bad_zip';
+ }
+}
+
+/** The offset of the end-of-central-directory record, or -1. It lives in the last 65,557 bytes. */
+function eocdOffset(buf) {
+ const from = Math.max(0, buf.length - 0xffff - 22);
+ for (let i = buf.length - 22; i >= from; i--) if (buf.readUInt32LE(i) === EOCD) return i;
+ return -1;
+}
+
+/** The central directory's offset and entry count, including the ZIP64 form. */
+function directory(buf) {
+ const eocd = eocdOffset(buf);
+ if (eocd < 0) throw new ZipError('that file is not a zip archive');
+ let count = buf.readUInt16LE(eocd + 10);
+ let offset = buf.readUInt32LE(eocd + 16);
+ if (count === 0xffff || offset === 0xffffffff) {
+ const locator = eocd - 20;
+ if (locator < 0 || buf.readUInt32LE(locator) !== ZIP64_LOCATOR) throw new ZipError('that zip archive uses an unsupported ZIP64 layout');
+ const z64 = Number(buf.readBigUInt64LE(locator + 8));
+ if (z64 < 0 || z64 + 56 > buf.length || buf.readUInt32LE(z64) !== ZIP64_EOCD) throw new ZipError('that zip archive has a broken ZIP64 directory');
+ count = Number(buf.readBigUInt64LE(z64 + 32));
+ offset = Number(buf.readBigUInt64LE(z64 + 48));
+ }
+ if (count > MAX_ENTRIES) throw new ZipError(`that zip archive has more than ${MAX_ENTRIES} files`);
+ if (offset < 0 || offset >= buf.length) throw new ZipError('that zip archive has a broken directory');
+ return { count, offset };
+}
+
+/** The ZIP64 extra field, when a central entry says a size or offset lives there. */
+function zip64Extra(extra, wants) {
+ let at = 0;
+ while (at + 4 <= extra.length) {
+ const id = extra.readUInt16LE(at);
+ const size = extra.readUInt16LE(at + 2);
+ const body = extra.subarray(at + 4, at + 4 + size);
+ if (id === 0x0001) {
+ let p = 0;
+ const take = () => {
+ if (p + 8 > body.length) throw new ZipError('that zip archive has a broken ZIP64 field');
+ const n = Number(body.readBigUInt64LE(p));
+ p += 8;
+ return n;
+ };
+ return {
+ size: wants.size ? take() : null,
+ compressed: wants.compressed ? take() : null,
+ offset: wants.offset ? take() : null,
+ };
+ }
+ at += 4 + size;
+ }
+ throw new ZipError('that zip archive has a broken ZIP64 field');
+}
+
+/**
+ * Read one archive. Returns an array of `{ name, data }` for the JSON files only; a directory and any other file
+ * are skipped, because a memory export is JSON and nothing here needs to inflate a picture.
+ */
+function readZip(buf) {
+ if (!Buffer.isBuffer(buf) || buf.length < 22) throw new ZipError('that file is not a zip archive');
+ const dir = directory(buf);
+ const out = [];
+ let total = 0;
+ let actual = 0;
+ let at = dir.offset;
+ for (let n = 0; n < dir.count; n++) {
+ if (at + 46 > buf.length || buf.readUInt32LE(at) !== CENTRAL) throw new ZipError('that zip archive has a broken directory');
+ const flags = buf.readUInt16LE(at + 8);
+ const method = buf.readUInt16LE(at + 10);
+ let compressed = buf.readUInt32LE(at + 20);
+ let size = buf.readUInt32LE(at + 24);
+ const nameLen = buf.readUInt16LE(at + 28);
+ const extraLen = buf.readUInt16LE(at + 30);
+ const commentLen = buf.readUInt16LE(at + 32);
+ let local = buf.readUInt32LE(at + 42);
+ const name = buf.subarray(at + 46, at + 46 + nameLen).toString('utf8').replace(/\\/g, '/');
+ const extra = buf.subarray(at + 46 + nameLen, at + 46 + nameLen + extraLen);
+ if (compressed === 0xffffffff || size === 0xffffffff || local === 0xffffffff) {
+ const z = zip64Extra(extra, { size: size === 0xffffffff, compressed: compressed === 0xffffffff, offset: local === 0xffffffff });
+ if (z.compressed != null) compressed = z.compressed;
+ if (z.size != null) size = z.size;
+ if (z.offset != null) local = z.offset;
+ }
+ at += 46 + nameLen + extraLen + commentLen;
+ if (name.endsWith('/')) continue;
+ if (!/\.json$/i.test(name)) continue;
+ if (out.length >= MAX_JSON_ENTRIES) throw new ZipError(`that zip archive has more than ${MAX_JSON_ENTRIES} JSON files`);
+ if (flags & 0x1) throw new ZipError('that zip archive is encrypted');
+ if (method !== 0 && method !== 8) throw new ZipError('that zip archive uses a compression this service does not read');
+ if (size > MAX_ENTRY) throw new ZipError('a file in that zip archive is larger than 60 MB');
+ total += size;
+ if (total > MAX_TOTAL) throw new ZipError('that zip archive unpacks to more than 60 MB of JSON');
+ if (local + 30 > buf.length || buf.readUInt32LE(local) !== LOCAL) throw new ZipError('that zip archive has a broken entry');
+ const lName = buf.readUInt16LE(local + 26);
+ const lExtra = buf.readUInt16LE(local + 28);
+ const start = local + 30 + lName + lExtra;
+ if (start + compressed > buf.length) throw new ZipError('that zip archive ends in the middle of a file');
+ const raw = buf.subarray(start, start + compressed);
+ let data;
+ if (method === 0) {
+ data = Buffer.from(raw);
+ } else {
+ try { data = zlib.inflateRawSync(raw, { maxOutputLength: Math.min(MAX_ENTRY, size || MAX_ENTRY) + 1 }); }
+ catch { throw new ZipError('a file in that zip archive did not unpack'); }
+ if (data.length > MAX_ENTRY) throw new ZipError('a file in that zip archive is larger than 60 MB');
+ }
+ actual += data.length;
+ if (actual > MAX_TOTAL) throw new ZipError('that zip archive unpacks to more than 60 MB of JSON');
+ out.push({ name, data });
+ }
+ return out;
+}
+
+module.exports = { readZip, ZipError, MAX_ENTRY, MAX_TOTAL, MAX_JSON_ENTRIES };
@@ -22,6 +22,10 @@
@@ -22,6 +22,10 @@
* at most once an hour per owner and scope
* at most once an hour per owner and scope
* GET /api/v1/memory/search?q=&scopes= for agents (ai.memory.read + on_behalf_of): active items only, in the
* GET /api/v1/memory/search?q=&scopes= for agents (ai.memory.read + on_behalf_of): active items only, in the
* granted scopes, at most 8, audited in memory_reads with ids and never text
* granted scopes, at most 8, audited in memory_reads with ids and never text
+ * GET /api/v1/memory/import/providers the cloud walkthroughs and the extraction prompt (M3, person token only)
+ * POST /api/v1/memory/import/preview parse and redact an export into a sealed two-hour draft (M3)
+ * POST /api/v1/memory/import/:id/commit keep the ticked candidates and delete the draft (M3)
+ * DELETE /api/v1/memory/import/:id throw the draft away (M3)
*
*
* Every SQL query that reads or changes a memory table is scoped by `owner = $…` (test/memory.test.js greps for it).
* Every SQL query that reads or changes a memory table is scoped by `owner = $…` (test/memory.test.js greps for it).
* The account export decrypts the person's own items for their own export; deletion removes the rows.
* The account export decrypts the person's own items for their own export; deletion removes the rows.
@@ -888,8 +892,14 @@ function createMemory({ db, config, clock = { now: () => Date.now() }, log = con
@@ -888,8 +892,14 @@ function createMemory({ db, config, clock = { now: () => Date.now() }, log = con
* names the person with `on_behalf_of`, but it may only search (audited) and import suggestions. Every other route is
* names the person with `on_behalf_of`, but it may only search (audited) and import suggestions. Every other route is
* person-only: a service gets 403 `capability.denied`.
* person-only: a service gets 403 `capability.denied`.
*/
*/
-function memoryRouter({ memory, auth, config, clock = { now: () => Date.now() }, log = console, sendError }) {
+function memoryRouter({ memory, memoryImport = null, auth, config, clock = { now: () => Date.now() }, log = console, sendError }) {
const r = express.Router();
const r = express.Router();
+ // Memory M3: an export zip is raw bytes (or one multipart form) and can be a few MB. The same route also takes
+ // pasted text as JSON, which the app-level JSON parser has already read.
+ const importBody = express.raw({
+ type: ['multipart/form-data', 'application/zip', 'application/x-zip-compressed', 'application/octet-stream', 'text/plain'],
+ limit: '25mb',
+ });
function guard(check) {
function guard(check) {
return async function memoryGuard(req, res, next) {
return async function memoryGuard(req, res, next) {
const ctx = req.ov;
const ctx = req.ov;
@@ -947,6 +957,14 @@ function memoryRouter({ memory, auth, config, clock = { now: () => Date.now() },
@@ -947,6 +957,14 @@ function memoryRouter({ memory, auth, config, clock = { now: () => Date.now() },
// browsing routes: a service has no business in the person's audit.
// browsing routes: a service has no business in the person's audit.
r.get('/api/v1/memory/reads', guard(READ_CAP), personOnly, run(async (req) => await memory.reads(req, ownerOf(req))));
r.get('/api/v1/memory/reads', guard(READ_CAP), personOnly, run(async (req) => await memory.reads(req, ownerOf(req))));
r.get('/api/v1/memory/search', guard(READ_CAP), run(async (req) => await memory.search(req, ownerOf(req))));
r.get('/api/v1/memory/search', guard(READ_CAP), run(async (req) => await memory.search(req, ownerOf(req))));
+ // M3: the guided cloud import. Person-only: a service token cannot see a person's export, hold a draft or keep
+ // what it finds. The preview stores only the sealed draft; the commit goes through the same batch as M2a.
+ if (memoryImport) {
+ r.get('/api/v1/memory/import/providers', guard(READ_CAP), personOnly, run(async () => memoryImport.providers()));
+ r.post('/api/v1/memory/import/preview', guard(WRITE_CAP), personOnly, importBody, run(async (req) => await memoryImport.preview(req, ownerOf(req))));
+ r.post('/api/v1/memory/import/:id/commit', guard(WRITE_CAP), personOnly, run(async (req) => await memoryImport.commit(req, ownerOf(req), req.params.id)));
+ r.delete('/api/v1/memory/import/:id', guard(WRITE_CAP), personOnly, run(async (req) => await memoryImport.discard(ownerOf(req), req.params.id)));
+ }
// M2c: undo a merge, and settle a conflict. Both are the person's own: a service may only search and suggest.
// M2c: undo a merge, and settle a conflict. Both are the person's own: a service may only search and suggest.
r.post('/api/v1/memory/items/:id/split', guard(WRITE_CAP), personOnly, run(async (req) => await memory.split(req, ownerOf(req), req.params.id)));
r.post('/api/v1/memory/items/:id/split', guard(WRITE_CAP), personOnly, run(async (req) => await memory.split(req, ownerOf(req), req.params.id)));
r.post('/api/v1/memory/items/:id/settle', guard(WRITE_CAP), personOnly, run(async (req) => await memory.settle(req, ownerOf(req), req.params.id)));
r.post('/api/v1/memory/items/:id/settle', guard(WRITE_CAP), personOnly, run(async (req) => await memory.settle(req, ownerOf(req), req.params.id)));
@@ -13,10 +13,10 @@
@@ -13,10 +13,10 @@
* the log carries counts and error codes only, never text.
* the log carries counts and error codes only, never text.
*/
*/
-const DEFAULT_CAPS = { owners: 10, scopes: 4, embeddings: 100, modelCalls: 20, profiles: 10 };
+const DEFAULT_CAPS = { owners: 10, scopes: 4, embeddings: 100, modelCalls: 20, profiles: 10, drafts: 100 };
const DEFAULT_INTERVAL_MS = 15 * 60 * 1000;
const DEFAULT_INTERVAL_MS = 15 * 60 * 1000;
-function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now() }, log = console, registry = null }) {
+function createMemoryJobs({ memory, memoryImport = null, config = {}, clock = { now: () => Date.now() }, log = console, registry = null }) {
const settings = config.memoryJobs || {};
const settings = config.memoryJobs || {};
const caps = {
const caps = {
owners: Math.max(1, Number(settings.maxOwners) || DEFAULT_CAPS.owners),
owners: Math.max(1, Number(settings.maxOwners) || DEFAULT_CAPS.owners),
@@ -24,6 +24,7 @@ function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now()
@@ -24,6 +24,7 @@ function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now()
embeddings: Math.max(0, Number(settings.maxEmbeddings) || DEFAULT_CAPS.embeddings),
embeddings: Math.max(0, Number(settings.maxEmbeddings) || DEFAULT_CAPS.embeddings),
modelCalls: Math.max(0, Number(settings.maxModelCalls) || DEFAULT_CAPS.modelCalls),
modelCalls: Math.max(0, Number(settings.maxModelCalls) || DEFAULT_CAPS.modelCalls),
profiles: Math.max(0, Number(settings.maxProfiles) || DEFAULT_CAPS.profiles),
profiles: Math.max(0, Number(settings.maxProfiles) || DEFAULT_CAPS.profiles),
+ drafts: Math.max(0, Number(settings.maxDrafts) || DEFAULT_CAPS.drafts),
};
};
let timer = null;
let timer = null;
let running = false;
let running = false;
@@ -42,9 +43,11 @@ function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now()
@@ -42,9 +43,11 @@ function createMemoryJobs({ memory, config = {}, clock = { now: () => Date.now()
if (running) return { skipped: 'running' };
if (running) return { skipped: 'running' };
if (settings.enabled === false) return { skipped: 'disabled' };
if (settings.enabled === false) return { skipped: 'disabled' };
running = true;
running = true;
- const out = { owners: 0, scopes: 0, embeddings: 0, merged: 0, conflicts: 0, calls: 0, profiles: 0, error: null };
+ const out = { owners: 0, scopes: 0, embeddings: 0, merged: 0, conflicts: 0, calls: 0, profiles: 0, drafts: 0, error: null };
try {
try {
if (await drillActive()) return { skipped: 'drill' };
if (await drillActive()) return { skipped: 'drill' };
+ // M3: a preview that expired is deleted whether or not its person comes back.
+ if (memoryImport && caps.drafts) out.drafts = await memoryImport.expireDrafts(caps.drafts);
const owners = await memory.ownersForJobs(caps.owners);
const owners = await memory.ownersForJobs(caps.owners);
for (const owner of owners) {
for (const owner of owners) {
const who = owner.owner;
const who = owner.owner;
@@ -157,6 +157,39 @@ route('get', '/memory/search', {
@@ -157,6 +157,39 @@ route('get', '/memory/search', {
parameters: [query('q', str()), query('scopes', str(), 'Comma-separated scopes; default user'), query('purpose', str(), 'The task or run the person sees'), query('on_behalf_of', str())],
parameters: [query('q', str()), query('scopes', str(), 'Comma-separated scopes; default user'), query('purpose', str(), 'The task or run the person sees'), query('on_behalf_of', str())],
responses: { 200: json('At most 8 active items', memoryResult), 403: problem, 503: problem, ...errors([401]) },
responses: { 200: json('At most 8 active items', memoryResult), 403: problem, 503: problem, ...errors([401]) },
});
});
+// ── Memory M3: the guided cloud import (person token only) ─────────────────────────────────
+route('get', '/memory/import/providers', {
+ operationId: 'getMemoryImportProviders', summary: 'The walkthroughs and the extraction prompt for the cloud import', tags: ['memory'], security: memorySecurity,
+ description: 'The person’s own token only. Where to request each export (ChatGPT, Claude, Gemini), what arrives, how long it takes, and the prompt to paste into any assistant. The answer contains no note.',
+ responses: { 200: json('The providers and the prompt', { type: 'object', properties: { providers: arrayOf({ type: 'object' }), prompt: str() } }), 403: problem, 503: problem, ...errors([401]) },
+});
+route('post', '/memory/import/preview', {
+ operationId: 'previewMemoryImport', summary: 'Parse and redact an export into a two-hour draft', tags: ['memory'], security: memorySecurity,
+ description: 'The person’s own token only. Send a .zip export, one multipart form (the Codes page forwards it untouched), a text/plain paste, or JSON { provider, text }. Nothing enters memory_items: the candidates are sealed into memory_imports with AI_MEMORY_KEY and expire after two hours. The answer is the draft id and the redacted candidates.',
+ requestBody: {
+ required: true,
+ content: {
+ 'application/zip': { schema: { type: 'string', format: 'binary' } },
+ 'multipart/form-data': { schema: { type: 'object', properties: { provider: str({ enum: ['chatgpt', 'claude-ai', 'gemini-apps', 'other'] }), file: { type: 'string', format: 'binary' }, text: str() } } },
+ 'text/plain': { schema: { type: 'string' } },
+ 'application/json': { schema: { type: 'object', properties: { provider: str(), text: str() } } },
+ },
+ },
+ responses: { 200: json('The sealed draft and its candidates', ref('MemoryResult')), 413: problem, 422: problem, 503: problem, ...errors([401, 403]) },
+});
+route('post', '/memory/import/{id}/commit', {
+ operationId: 'commitMemoryImport', summary: 'Keep the ticked candidates and delete the draft', tags: ['memory'], security: memorySecurity,
+ description: 'The person’s own token only. keep names the preview keys (k1, k2, …) to store. They go through the normal batch import (redaction again, dedupe), and the draft is deleted; the keys not sent are not stored. An expired draft answers 410.',
+ parameters: [path('id', str())],
+ requestBody: body({ type: 'object', properties: { keep: arrayOf(str()) } }),
+ responses: { 200: json('What was kept and left out', ref('MemoryResult')), 404: problem, 410: problem, 422: problem, 503: problem, ...errors([401, 403]) },
+});
+route('delete', '/memory/import/{id}', {
+ operationId: 'discardMemoryImport', summary: 'Throw a preview draft away', tags: ['memory'], security: memorySecurity,
+ description: 'The person’s own token only. The draft and its candidates are deleted; nothing was ever stored in memory_items.',
+ parameters: [path('id', str())],
+ responses: { 200: json('How many candidates were discarded', ref('MemoryResult')), 404: problem, 410: problem, 503: problem, ...errors([401, 403]) },
+});
// ── Runs ──────────────────────────────────────────────────────────────────────────────────────
// ── Runs ──────────────────────────────────────────────────────────────────────────────────────
route('post', '/runs', {
route('post', '/runs', {
new file mode 100644
new file mode 100644
@@ -0,0 +1,63 @@
@@ -0,0 +1,63 @@
+'use strict';
+/**
+ * A tiny ZIP writer for the memory-import tests. The fixtures are synthetic (no real exports in the repo); this
+ * makes one in memory so a test can hand the service the same bytes a browser would upload.
+ *
+ * Standard local headers, deflate entries, a central directory and an end record. The reader does not verify the
+ * CRC, so the writer leaves it zero; the sizes and offsets are real.
+ */
+const zlib = require('zlib');
+
+function makeZip(files) {
+ const locals = [];
+ const central = [];
+ let offset = 0;
+ for (const [name, content] of Object.entries(files)) {
+ const body = Buffer.isBuffer(content) ? content : Buffer.from(String(content), 'utf8');
+ const data = zlib.deflateRawSync(body);
+ const nameBuf = Buffer.from(name, 'utf8');
+ const local = Buffer.alloc(30);
+ local.writeUInt32LE(0x04034b50, 0);
+ local.writeUInt16LE(20, 4);
+ local.writeUInt16LE(0x800, 6);
+ local.writeUInt16LE(8, 8);
+ local.writeUInt32LE(0, 14);
+ local.writeUInt32LE(data.length, 18);
+ local.writeUInt32LE(body.length, 22);
+ local.writeUInt16LE(nameBuf.length, 26);
+ local.writeUInt16LE(0, 28);
+ locals.push(local, nameBuf, data);
+
+ const cent = Buffer.alloc(46);
+ cent.writeUInt32LE(0x02014b50, 0);
+ cent.writeUInt16LE(20, 4);
+ cent.writeUInt16LE(20, 6);
+ cent.writeUInt16LE(0x800, 8);
+ cent.writeUInt16LE(8, 10);
+ cent.writeUInt32LE(0, 16);
+ cent.writeUInt32LE(data.length, 20);
+ cent.writeUInt32LE(body.length, 24);
+ cent.writeUInt16LE(nameBuf.length, 28);
+ cent.writeUInt16LE(0, 30);
+ cent.writeUInt16LE(0, 32);
+ cent.writeUInt16LE(0, 34);
+ cent.writeUInt16LE(0, 36);
+ cent.writeUInt32LE(0, 38);
+ cent.writeUInt32LE(offset, 42);
+ central.push(cent, nameBuf);
+ offset += local.length + nameBuf.length + data.length;
+ }
+ const directory = Buffer.concat(central);
+ const end = Buffer.alloc(22);
+ end.writeUInt32LE(0x06054b50, 0);
+ end.writeUInt16LE(0, 4);
+ end.writeUInt16LE(0, 6);
+ end.writeUInt16LE(Object.keys(files).length, 8);
+ end.writeUInt16LE(Object.keys(files).length, 10);
+ end.writeUInt32LE(directory.length, 12);
+ end.writeUInt32LE(offset, 16);
+ end.writeUInt16LE(0, 20);
+ return Buffer.concat([...locals, directory, end]);
+}
+
+module.exports = { makeZip };
new file mode 100644
new file mode 100644
@@ -0,0 +1,254 @@
@@ -0,0 +1,254 @@
+'use strict';
+/**
+ * Memory M3: the guided cloud import. Synthetic exports only (test/fixtures/zip.js builds them in memory): the
+ * preview redacts and stores no note, the draft is sealed, the commit keeps only the ticked cells, and a service
+ * token can neither preview nor commit. No test reaches the network.
+ */
+const assert = require('assert');
+const crypto = require('crypto');
+const { boot, request, suite, privateKey, ISSUER } = require('./helpers');
+const { makeZip } = require('./fixtures/zip');
+
+const t = suite('memory-import');
+const KEY = 'b'.repeat(64);
+const A = 'usr_01JZ0000000000000000000AAA';
+const B = 'usr_01JZ0000000000000000000BBB';
+const SECRET = `ghp_${'A'.repeat(36)}`;
+const b64 = (o) => Buffer.from(JSON.stringify(o)).toString('base64url');
+
+function userToken(subject, expiresIn = 3600) {
+ const now = Math.floor(Date.now() / 1000);
+ const input = `${b64({ alg: 'RS256', typ: 'JWT' })}.${b64({ iss: ISSUER, sub: subject, subject_id: subject, aud: ['openvibe.network'], iat: now, exp: now + expiresIn, jti: `tok_${crypto.randomBytes(8).toString('hex')}` })}`;
+ return `${input}.${crypto.sign('RSA-SHA256', Buffer.from(input), privateKey).toString('base64url')}`;
+}
+
+/** A raw-body request: an export zip goes up as application/zip, a paste as JSON through the normal helper. */
+async function upload(base, path, tok, body, contentType = 'application/zip') {
+ const res = await fetch(base + path, { method: 'POST', headers: { Authorization: `Bearer ${tok}`, 'Content-Type': contentType }, body });
+ const text = await res.text();
+ let json = null;
+ try { json = text ? JSON.parse(text) : null; } catch { json = null; }
+ return { status: res.status, body: json, text };
+}
+
+/** One multipart form, exactly what the Codes page forwards to this route. */
+function multipartBody(fields, file) {
+ const boundary = '----ai-memory-test';
+ const parts = [];
+ for (const [name, value] of Object.entries(fields || {})) {
+ parts.push(`--${boundary}\r\nContent-Disposition: form-data; name="${name}"\r\n\r\n${value}\r\n`);
+ }
+ if (file) {
+ parts.push(`--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="${file.name}"\r\nContent-Type: application/zip\r\n\r\n`);
+ parts.push(file.data);
+ parts.push('\r\n');
+ }
+ parts.push(`--${boundary}--\r\n`);
+ return { boundary, body: Buffer.concat(parts.map((x) => Buffer.isBuffer(x) ? x : Buffer.from(x, 'utf8'))) };
+}
+
+const chatgptExport = () => makeZip({
+ 'conversations.json': JSON.stringify([{
+ title: 'How I like to work', create_time: 1700000000, current_node: 'b',
+ mapping: {
+ a: { parent: null, message: { author: { role: 'user' }, content: { parts: [`Please remember that I prefer short answers and my deploy token is ${SECRET}.`] }, create_time: 1700000001 } },
+ b: { parent: 'a', message: { author: { role: 'assistant' }, content: { parts: ['Noted.'] }, create_time: 1700000002 } },
+ },
+ }, {
+ title: 'Lunch', create_time: 1700000100, current_node: 'x',
+ mapping: { x: { parent: null, message: { author: { role: 'user' }, content: { parts: ['What should I eat?'] }, create_time: 1700000101 } } },
+ }]),
+ 'user.json': JSON.stringify({ id: 'user-1' }),
+});
+
+const claudeExport = () => makeZip({
+ 'memories.json': JSON.stringify([
+ { content: 'I work as a staff engineer at a small company. I prefer British spelling.', created_at: '2026-01-10T08:00:00Z' },
+ ]),
+ 'conversations.json': JSON.stringify([{
+ name: 'Deploy notes', created_at: '2026-01-11T09:00:00Z',
+ chat_messages: [
+ { sender: 'human', content: [{ type: 'text', text: 'Remember that I always deploy on Thursdays.' }], created_at: '2026-01-11T09:01:00Z' },
+ { sender: 'assistant', content: [{ type: 'text', text: 'I will remember that.' }], created_at: '2026-01-11T09:02:00Z' },
+ ],
+ }]),
+ 'projects.json': JSON.stringify([{ name: 'Demo' }]),
+});
+
+const geminiExport = () => makeZip({
+ 'Takeout/My Activity/Gemini Apps/My Activity.json': JSON.stringify([
+ { title: 'Said "Remember that I like my answers with sources."', time: '2026-03-05T10:30:00Z' },
+ { title: 'What is the weather?', time: '2026-03-05T11:00:00Z' },
+ ]),
+});
+
+t.test('the walkthroughs are person-only and carry the three providers and the prompt', async () => {
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });
+ try {
+ const r = await request(h.base, 'GET', '/api/v1/memory/import/providers', { tok: userToken(A) });
+ assert.equal(r.status, 200);
+ assert.deepStrictEqual(r.body.providers.map((p) => p.id), ['chatgpt', 'claude-ai', 'gemini-apps']);
+ assert.ok(r.body.providers.every((p) => p.url && p.steps.length && p.time && p.arrives), 'each walkthrough is complete');
+ assert.match(r.body.prompt, /remember about me/i);
+ const service = await request(h.base, 'GET', '/api/v1/memory/import/providers', { tok: require('./helpers').token('codes', ['ai.memory.read']) });
+ assert.equal(service.status, 403);
+ } finally { await h.stop(); }
+});
+
+t.test('the multipart form Codes forwards is parsed here: a zip with its provider, and a pasted answer', async () => {
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });
+ try {
+ const who = userToken(A);
+ const zip = multipartBody({ provider: 'chatgpt' }, { name: 'export.zip', data: chatgptExport() });
+ const file = await upload(h.base, '/api/v1/memory/import/preview', who, zip.body, `multipart/form-data; boundary=${zip.boundary}`);
+ assert.equal(file.status, 200);
+ assert.equal(file.body.import.provider, 'chatgpt');
+ assert.equal(file.body.import.items.length, 1);
+
+ const paste = multipartBody({ provider: 'claude-ai', text: '- I prefer British spelling. [preference]' });
+ const text = await upload(h.base, '/api/v1/memory/import/preview', who, paste.body, `multipart/form-data; boundary=${paste.boundary}`);
+ assert.equal(text.status, 200);
+ assert.equal(text.body.import.items.length, 1);
+ assert.equal(text.body.import.items[0].source, 'claude-ai');
+ } finally { await h.stop(); }
+});
+
+t.test('a ChatGPT zip previews redacted candidates, stores no note, and seals the draft', async () => {
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });
+ try {
+ const who = userToken(A);
+ const before = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_items WHERE owner = ?').get(`user:${A}`);
+ assert.equal(Number(before.n), 0);
+ const out = await upload(h.base, '/api/v1/memory/import/preview', who, chatgptExport());
+ assert.equal(out.status, 200);
+ const draft = out.body.import;
+ assert.match(draft.id, /^mimp_/);
+ assert.equal(draft.provider, 'chatgpt');
+ assert.equal(draft.items.length, 1, 'only the sentence that looks like a memory');
+ assert.ok(!draft.items[0].text.includes('ghp_'), 'the token is gone before the preview');
+ assert.ok(draft.items[0].text.includes('[redacted]'));
+ assert.equal(draft.items[0].source, 'chatgpt');
+ assert.equal(draft.items[0].source_ref.heading, 'How I like to work');
+ assert.equal(draft.summary.conversations, 2);
+ assert.equal(draft.summary.secrets_removed, 1);
+ const notes = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_items WHERE owner = ?').get(`user:${A}`);
+ assert.equal(Number(notes.n), 0, 'a preview is not an import');
+ const row = await h.db.prepare('SELECT * FROM memory_imports WHERE owner = ?').get(`user:${A}`);
+ assert.match(row.payload_enc, /^m1:/, 'the draft is sealed');
+ assert.ok(!row.payload_enc.includes('short answers'), 'no plaintext draft');
+ } finally { await h.stop(); }
+});
+
+t.test('the commit keeps only the ticked cells, and the left-out ones are not stored', async () => {
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });
+ try {
+ const who = userToken(A);
+ const zip = makeZip({
+ 'conversations.json': JSON.stringify([{
+ title: 'Two memories', create_time: 1700000000, current_node: 'b',
+ mapping: {
+ a: { parent: null, message: { author: { role: 'user' }, content: { parts: ['Remember that I prefer tabs. Remember that I use vim every day.'] }, create_time: 1700000001 } },
+ b: { parent: 'a', message: { author: { role: 'assistant' }, content: { parts: ['ok'] } } },
+ },
+ }]),
+ });
+ const preview = await upload(h.base, '/api/v1/memory/import/preview', who, zip);
+ assert.equal(preview.status, 200);
+ const { id, items } = preview.body.import;
+ assert.equal(items.length, 2);
+ const commit = await request(h.base, 'POST', `/api/v1/memory/import/${id}/commit`, { tok: who, body: { keep: [items[0].key] } });
+ assert.deepStrictEqual([commit.status, commit.body.kept, commit.body.left_out], [200, 1, 1]);
+ const rows = await h.db.prepare('SELECT source, status FROM memory_items WHERE owner = ?').all(`user:${A}`);
+ assert.equal(rows.length, 1);
+ assert.equal(rows[0].source, 'chatgpt');
+ assert.equal(rows[0].status, 'active', 'the person chose it, so it is not a suggestion');
+ const drafts = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_imports WHERE owner = ?').get(`user:${A}`);
+ assert.equal(Number(drafts.n), 0, 'the draft is gone');
+ } finally { await h.stop(); }
+});
+
+t.test('Claude saved memories come first, and a Gemini Takeout activity file is found on its own', async () => {
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });
+ try {
+ const who = userToken(A);
+ const claude = await upload(h.base, '/api/v1/memory/import/preview', who, claudeExport());
+ assert.equal(claude.status, 200);
+ assert.equal(claude.body.import.provider, 'claude-ai');
+ assert.equal(claude.body.import.items.length, 2);
+ assert.ok(claude.body.import.items.some((i) => i.confidence === 0.9 && /staff engineer/.test(i.text)), 'memories.json is read directly');
+ assert.ok(claude.body.import.items.some((i) => /Thursdays/.test(i.text)), 'and the conversation marker too');
+
+ const gemini = await upload(h.base, '/api/v1/memory/import/preview', who, geminiExport());
+ assert.equal(gemini.status, 200);
+ assert.equal(gemini.body.import.provider, 'gemini-apps');
+ assert.equal(gemini.body.import.items.length, 1);
+ assert.match(gemini.body.import.items[0].text, /answers with sources/);
+ assert.equal(gemini.body.import.items[0].source, 'gemini-apps');
+ } finally { await h.stop(); }
+});
+
+t.test('a pasted extraction answer is parsed by its bullets and labels', async () => {
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });
+ try {
+ const who = userToken(A);
+ const out = await request(h.base, 'POST', '/api/v1/memory/import/preview', {
+ tok: who,
+ body: { provider: 'claude-ai', text: '- I prefer long walks. [preference]\n- Project rule: always run the tests before shipping. [project rule]\n- **Preference:** I prefer evening work.\n- Just some chatter' },
+ });
+ assert.equal(out.status, 200);
+ assert.equal(out.body.import.items.length, 3);
+ assert.deepStrictEqual(out.body.import.items.map((i) => i.kind).sort(), ['preference', 'preference', 'project_rule']);
+ assert.ok(!out.body.import.items.some((i) => /chatter/.test(i.text)), 'unlabelled chatter is not a memory');
+ const commit = await request(h.base, 'POST', `/api/v1/memory/import/${out.body.import.id}/commit`, { tok: who, body: { keep: out.body.import.items.map((i) => i.key) } });
+ assert.equal(commit.body.kept, 3);
+ } finally { await h.stop(); }
+});
+
+t.test('a service token cannot preview or commit, another person cannot open the draft, and expiry is a 410', async () => {
+ let now = Date.now();
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY }, clock: { now: () => now } });
+ try {
+ const who = userToken(A, 5 * 3600);
+ const service = require('./helpers').token('codes', ['ai.memory.write']);
+ const denied = await upload(h.base, '/api/v1/memory/import/preview', service, chatgptExport());
+ assert.equal(denied.status, 403);
+ const preview = await upload(h.base, '/api/v1/memory/import/preview', who, chatgptExport());
+ const id = preview.body.import.id;
+ const other = await request(h.base, 'POST', `/api/v1/memory/import/${id}/commit`, { tok: userToken(B), body: { keep: ['k1'] } });
+ assert.deepStrictEqual([other.status, other.body.code], [404, 'import.not_found']);
+ now += 3 * 60 * 60 * 1000;
+ const late = await request(h.base, 'POST', `/api/v1/memory/import/${id}/commit`, { tok: who, body: { keep: ['k1'] } });
+ assert.deepStrictEqual([late.status, late.body.code], [410, 'import.expired']);
+ } finally { await h.stop(); }
+});
+
+t.test('the maintenance loop sweeps a draft that expired while its person was away', async () => {
+ let now = Date.now();
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY, AI_MEMORY_JOBS: 'true' }, clock: { now: () => now } });
+ try {
+ const preview = await upload(h.base, '/api/v1/memory/import/preview', userToken(A), chatgptExport());
+ assert.equal(preview.status, 200);
+ now += 3 * 60 * 60 * 1000;
+ const run = await h.memoryJobs.runOnce();
+ assert.equal(run.drafts, 1, 'the sweep counted the draft');
+ const rows = await h.db.prepare('SELECT COUNT(*) AS n FROM memory_imports WHERE owner = ?').get(`user:${A}`);
+ assert.equal(Number(rows.n), 0, 'the expired draft is gone');
+ } finally { await h.stop(); }
+});
+
+t.test('a file that is not a zip is a 422, never a 500, and an unsupported zip says what is missing', async () => {
+ const h = await boot({ env: { AI_MEMORY_KEY: KEY } });
+ try {
+ const who = userToken(A);
+ const bad = await upload(h.base, '/api/v1/memory/import/preview', who, Buffer.from('not a zip at all'));
+ assert.deepStrictEqual([bad.status, bad.body.code], [422, 'import.bad_zip']);
+ const empty = await upload(h.base, '/api/v1/memory/import/preview', who, makeZip({ 'notes.txt': 'hello' }));
+ assert.deepStrictEqual([empty.status, empty.body.code], [422, 'import.invalid']);
+ const noMemory = await upload(h.base, '/api/v1/memory/import/preview', who, makeZip({ 'conversations.json': JSON.stringify([{ title: 'x', mapping: {} }]) }));
+ assert.equal(noMemory.status, 200, 'a valid export with no candidate is a preview that says so');
+ assert.equal(noMemory.body.import.items.length, 0);
+ assert.equal(noMemory.body.import.id, null);
+ } finally { await h.stop(); }
+});
+
+t.run();