openvibers/openvibe.ai Commits
OpenVibe GPUs: the GPU provisioner (cheapest fit, the monthly cap, scale to zero, failover) (#304)
Changes
nnew file mode 100644@@@ -0,0 +1,109 @@++# OpenVibe GPUs (the GPU provisioner)++++The design (§2 "Our own GPUs", the Exchange remake) asks for a provisioner that rents a GPU from a++marketplace, serves a model on it, registers the endpoint as an offer for the router and scales the++whole thing to zero when nobody is using it - under a hard monthly cap the owner sets.++++This is that provisioner. It lives in AI (server/gpu/) and it is **off until the owner has set a++marketplace API key and a monthly cap**.++++## The owner settings (the namespace `ai.gpus`)++++| Key | Class | What it is |++| --- | --- | --- |++| `vast.origin` / `vast.key` | internal / secret (bound to the origin) | the owner's Vast.ai API key, sealed to `https://console.vast.ai` |++| `runpod.origin` / `runpod.key` | internal / secret (bound to the origin) | the owner's RunPod API key, sealed to `https://api.runpod.io` |++| `monthly_cap` | money (`x-danger`, `nano_usd`) | the hard monthly cap. It needs a fresh sign-in to change |++| `idle_minutes` | internal | how long an unused instance stays up (default 10) |++| `max_runtime_minutes` | internal | how long one instance may run at most (default 720) |++| `warm_window` | internal | an optional UTC window ("19:00-23:00") in which idle instances stay up |++| `region` | internal | the region the offers must be in (default US) |++++The owner changes them the way every other AI setting changes: through Network's admin panel, which++proxies to AI's `/internal/config/ai.gpus`. The keys are write-only (the panel never shows a value++again), the origins are written by AI, and the cap is a money key.++++Without a key **or** without a cap the provisioner does nothing: no search, no instance, no money.++++## What it does++++1. **Cheapest fit.** It searches the marketplace for the offers that fit the model (VRAM, disk,++ reliability, region) and takes the cheapest one. The order is deterministic: the same offers in++ the same order always give the same instance.++2. **Serve.** It rents the offer with the model's image and command (vLLM or llama.cpp with the++ chosen weights) and waits for the endpoint to answer.++3. **Register.** The endpoint becomes a normal provider (`kind: openai`, `provider_access:++ OWN_DEPLOYMENT`) with the model on it, so the router, the Exchange catalogue, the health checks,++ the breaker and the failover all work as they do for every other offer. Nothing new was written++ for the router.++4. **Health.** The endpoint is checked (`/v1/models`). After two failed checks the instance is dead:++ the offer leaves the pool (the router fails over), the instance is given back and a new one is++ rented.++5. **Scale to zero.** An instance that served nobody for `idle_minutes` is given back (unless it is++ inside the warm window). An instance that has run for `max_runtime_minutes` is given back too.++6. **Cold start.** The time from the launch to the first answered request is measured and shown on the++ console page (the design asks for the cold start to be said up front).++++## Orphans: an instance we paid for is never lost++++The row is written **before** the marketplace is called, and the instance is given the label of that++row (`openvibe-<row id>`). Every pass:++++- the marketplace instances that carry our label are listed (Vast: `GET /api/v1/instances`, RunPod:++ `GET /v2/pods`);++- one whose label is a live row of ours is left alone - and **linked** when the row does not know its++ marketplace id yet (a crash between the marketplace's answer and the row's update costs nothing: the++ instance is found by its label and adopted);++- one that no live row of ours owns is an **orphan**: it is given back and the fact is logged++ (`gpu.orphan.destroyed` on the audit log).++++A destroy the marketplace refuses leaves the instance running and still being paid for. The try is++counted on the row, the next pass tries again, and after **three** tries the operator is alerted: the++audit row `gpu.orphan.destroy_failed`, the log, and the gauge `ai_gpu_destroy_failures` that the++Host rule `OpenVibeAiGpuInstanceNotDestroyed` pages on++([OpenVibe.Host #303](https://github.com/OpenVibers/OpenVibe.Host/pull/303)).++++## The money++++The money is integer nano-USD in BigInt, never a float (the same rule AI's Credits follow). Every++instance holds a **reservation** - the budget it may cost - before it starts, and the reservation++is part of the month's committed spend. A new instance is only started when its reservation still++fits under the cap; if it does not, the launch is refused and the reason is returned. The instance is++given back when its reservation has run out, so the money stops there. The worst case (the++committed spend) therefore never passes the cap.++++Vast and RunPod bill **per second**, so the money is charged for the seconds an instance really ran++(the ledger rounds up to the nano, so it never charges less than the GPU was paid for). The console page++shows the price per hour and the price per second next to it.++++The ledger is `gpu_spend` (one row per settled instance); the instances, their reservations, their spent++money, their destroy failures and their measured cold start are in `gpu_instances` (migration++`0033_gpu.sql`).++++## The console page++++`GET /console/gpus` (the "GPUs" entry in the console) shows whether the provisioner is on and why not,++the spend this month, the committed worst case and the cap, the models with their weights and their++state, the measured cold start (from the launch to the first answered request), and every instance++with its price (the hour and the second), endpoint, cold start and money. "Start" and "Stop" write a request row;++the reconciler (one implementation, the same one the service runs) does the work.++++## The tests++++`test/gpu.test.js` runs the provisioner against the recorded marketplace answers++(`test/fixtures/gpu/vast.json`) and a fake clock:++++- it is off without a key and without a cap,++- the cheapest offer that fits is rented, registered as an offer and health-checked,++- the monthly cap is never passed: the launch is refused and says why,++- an idle instance scales to zero and the money is settled,++- an instance that dies is replaced and the router gets the new endpoint,++- an instance the marketplace accepted before the row knew its id is recovered, not thrown away,++- an instance with our label and no row of ours is found and given back,++- a destroy that keeps failing is tried again every pass and the operator is alerted.++++## What the owner has to check before the first launch++++The weights URL, the image and the container command are data in `server/gpu/catalog.js` - they are++what the first launch really runs, and the console page shows them. The owner checks them (and++corrects them there) before the first rented instance starts.nnew file mode 100644@@@ -0,0 +1,70 @@++-- phase: expand++-- OpenVibe GPUs (the GPU provisioner): the marketplace instances AI rents, the money they cost and the++-- owner's requests to start or stop them. Money is integer nano-USD in bigint (AI's D1 rule: never a++-- float). Additive only: nothing existing is changed.++CREATE TABLE IF NOT EXISTS gpu_instances (++ id text COLLATE "C" PRIMARY KEY,++ model_key text COLLATE "C" NOT NULL,++ provider_key text COLLATE "C" NOT NULL,++ marketplace text COLLATE "C" NOT NULL,++ offer_id text COLLATE "C",++ remote_id text COLLATE "C",++ status text COLLATE "C" NOT NULL CHECK (status IN ('starting','running','stopping','destroyed','dead','refused','orphan')),++ endpoint text COLLATE "C",++ month text COLLATE "C" NOT NULL,++ gpu_name text COLLATE "C",++ gpu_ram_mb integer,++ num_gpus integer,++ disk_gb integer,++ reliability double precision,++ region text COLLATE "C",++ price_per_hour text COLLATE "C", -- the marketplace's own decimal price, as it quoted it++ hourly_nano bigint NOT NULL DEFAULT 0, -- that price as exact nano-USD per hour++ reserved_nano bigint NOT NULL DEFAULT 0, -- the budget held for this instance (never above the cap)++ spent_nano bigint NOT NULL DEFAULT 0, -- the money this instance really cost++ max_runtime_minutes integer NOT NULL,++ started_at timestamptz,++ charged_at timestamptz, -- the moment the money was last put on the ledger++ ready_at timestamptz,++ last_used_at timestamptz,++ last_health_at timestamptz,++ health_failures integer NOT NULL DEFAULT 0,++ destroy_failures integer NOT NULL DEFAULT 0, -- how many times in a row giving it back failed++ label text COLLATE "C", -- the label we gave the marketplace instance++ cold_start_ms bigint, -- from the launch to the first answered request++ destroyed_at timestamptz,++ error text COLLATE "C",++ created_at timestamptz NOT NULL,++ updated_at timestamptz NOT NULL++);++CREATE INDEX IF NOT EXISTS gpu_instances_model ON gpu_instances (model_key, status);++CREATE INDEX IF NOT EXISTS gpu_instances_month ON gpu_instances (month);++++-- The money the GPU instances really cost. One row per settled instance (and per refund); the month's++-- spend is their sum, so the ledger and the cap are the same numbers.++CREATE TABLE IF NOT EXISTS gpu_spend (++ id bigserial PRIMARY KEY,++ at timestamptz NOT NULL,++ month text COLLATE "C" NOT NULL,++ kind text COLLATE "C" NOT NULL CHECK (kind IN ('charge','credit')),++ nano_usd bigint NOT NULL,++ instance_id text COLLATE "C",++ detail text COLLATE "C",++ actor text COLLATE "C",++ created_at timestamptz NOT NULL++);++CREATE INDEX IF NOT EXISTS gpu_spend_month ON gpu_spend (month);++++-- What the owner asked for: start or stop a model's GPU. The reconciler fulfils it (the console and the++-- admin API only write the request; one implementation acts on it).++CREATE TABLE IF NOT EXISTS gpu_requests (++ id bigserial PRIMARY KEY,++ model_key text COLLATE "C" NOT NULL,++ action text COLLATE "C" NOT NULL CHECK (action IN ('launch','stop')),++ at timestamptz NOT NULL,++ actor text COLLATE "C",++ created_at timestamptz NOT NULL,++ done_at timestamptz,++ error text COLLATE "C"++);++CREATE INDEX IF NOT EXISTS gpu_requests_model ON gpu_requests (model_key, done_at);@@@ -24,7 +24,7 @@ const pkg = require('../package.json');const { adminRoutes } = require('openvibe-shared/config');--function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, memoryImport = null, community = null, presets = null, receipts = null, balance = null }) {++function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, memoryImport = null, community = null, presets = null, receipts = null, balance = null, gpus = null }) {const app = express();// /.well-known/agent.json (openvibe-shared/agent-layer, SEO-2): what this site serves, for agents.require('openvibe-shared/agent-layer').mount(app, { descriptor: require('./agent-descriptor') });@@@ -36,7 +36,7 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys// GET /metrics answers direct loopback callers only (Track O).const metrics = instrument(app, { service: 'ai', release: release.release });app.use(require('openvibe-sdk/service').telemetryMiddleware); // request telemetry: a no-op until server/index.js calls telemetry.init()-- registerAiGauges(metrics.registry, { runs, registry, pool, balance });++ registerAiGauges(metrics.registry, { runs, registry, pool, balance, gpus });app.use(http.middleware());// Browser reads of the two public GETs (/stats, /release.json) only: an exact origin from// AI_CORS_ORIGINS is echoed back, never credentials, and OPTIONS answers their preflight.@@@ -125,10 +125,10 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keysif (benchEvents) app.use(benchEvents.router);// D11 owner settings (openvibe-shared/config): Network's admin proxy, acting for the owner, reads and writes-- // ai.exchange and ai.provider_keys. Loopback only (the /internal guard above), guarded by ai.config.manage; the++ // ai.exchange, ai.provider_keys and ai.gpus (the GPU provisioner's marketplace keys and its monthly cap). Loopback only (the /internal guard above), guarded by ai.config.manage; the// actor is Network's X-OV-Subject (usr_…), and a secret or x-danger change needs X-OV-Fresh: 1.if (settings) {-- const configRoutes = adminRoutes([settings.exchange, settings.providerKeys.proxy], {++ const configRoutes = adminRoutes([settings.exchange, settings.providerKeys.proxy, settings.gpus.proxy], {basePath: '/internal/config',requireAdmin: auth.requireCap(CAPS.configManage),actor: (req) => {@@@ -115,6 +115,9 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, econst creditsOn = Boolean(config.credits && config.credits.enabled);const routingOn = Boolean(config.routingEvidence && config.routingEvidence.enabled);const exchangeOn = creditsOn || routingOn;++ // The GPU provisioner's state (the same module the service runs): the page shows the instances, the++ // spend this month and the cap. The actions only write a request; the reconciler fulfils it.++ const gpus = require('../gpu/provisioner').createGpuProvisioner({ db, registry, settings, clock, fetchImpl, log });const r = express.Router();const scoped = ['/console', '/auth'];const baseOrigin = (() => { try { return new URL(config.baseUrl).origin; } catch { return null; } })();@@@ -540,6 +543,19 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e}));}));++ // ── OpenVibe GPUs: the rented instances, the spend this month and the cap ──++ // The page is read-only; the two actions ask for a change (a request row) and the reconciler does it.++ r.get('/console/gpus', read, wrap(async (req, res) => {++ await send(res, 200, pages.gpus({++ ...common(req), canManage: req.staff.caps.includes(CAPS.providerManage), notice: notice(req),++ state: await gpus.state(),++ }));++ }));++ r.post('/console/gpus/:model/:action', needs(CAPS.providerManage), wrap(async (req, res) => {++ await gpus.request(String(req.params.model || ''), String(req.params.action || ''), req.staff.subject);++ res.redirect(303, '/console/gpus?done=gpu_requested');++ }));++// ── AI Exchange (P7): Credits settle queue and routing evidence ──// Read-only except "mark resolved", which acknowledges one review row (a 409/422 needs a person) and// deliberately does not retry it. Both sections are absent unless their flag is on.@@@ -122,6 +122,7 @@ const NAV = [['/console', 'Overview', 'overview'], ['/console/providers', 'Providers', 'providers'], ['/console/routes', 'Routes', 'routes'],['/console/templates', 'Templates', 'templates'], ['/console/workflows', 'Workflows', 'workflows'], ['/console/runs', 'Runs', 'runs'],['/console/quotas', 'Quotas', 'quotas'], ['/console/usage', 'Usage', 'usage'], ['/console/cache', 'Cache', 'cache'], ['/console/audit', 'Audit', 'audit'],++ ['/console/gpus', 'GPUs', 'gpus'],];// The console's pages are noindex; the one public page drawn with this layout (/stats) passes robots, a description and@@@ -558,6 +559,74 @@ ${routing ? routingSection({ routing }) : ''}`,});}++/** The GPU money as a dollar string: nano-USD in, exact dollars out (never a float). */++function usdNano(nano) {++ const n = BigInt(nano == null ? 0 : nano);++ const abs = n < 0n ? -n : n;++ const whole = abs / 1000000000n;++ const frac = (abs % 1000000000n).toString().padStart(9, '0').replace(/0+$/, '');++ return (n < 0n ? '-' : '') + '$' + whole.toString() + (frac ? '.' + frac : '');++}++++/** A measured cold start: from the launch to the first answered request. */++function coldStart(ms) {++ const n = Math.max(0, Math.round(Number(ms) || 0));++ if (n < 1000) return n + ' ms';++ return (n / 1000).toFixed(1) + ' s';++}++++/** The price of one second of a GPU, from its hourly nano-USD rate (Vast and RunPod bill per second). */++function usdPerSecond(hourlyNano) {++ const n = BigInt(hourlyNano == null ? 0 : hourlyNano);++ return usdNano(n / 3600n);++}++++function gpuModelRow(m, canManage, csrf) {++ return html`<tr>++ <td><code>${m.key}</code><br><span class="muted">${m.display_name}</span><br><span class="muted">${clip(m.weights_url, 90)}</span></td>++ <td>${m.marketplace}</td><td>${m.runtime}</td>++ <td>${m.instance ? m.instance.gpu_name + ' · $' + m.instance.price_per_hour + '/h' : '—'}</td>++ <td>${m.instance ? badge(m.instance.status) : html`<span class="muted">stopped</span>`}</td>++ <td>${m.cold_start_ms == null ? html`<span class="muted">not measured yet</span>` : coldStart(m.cold_start_ms)}</td>++ <td>${canManage ? html`<form class="inline" method="post" action="/console/gpus/${enc(m.key)}/${m.instance ? 'stop' : 'launch'}">${csrfField(csrf)}<button type="submit">${m.instance ? 'Stop' : 'Start'}</button></form>` : ''}</td>++</tr>`;++}++++function gpuInstanceRow(i) {++ return html`<tr>++ <td><code>${i.model_key}</code></td><td>${i.marketplace}</td><td>${i.gpu_name}</td><td>$${i.price_per_hour}/h · ${usdPerSecond(i.hourly_nano)}/s</td>++ <td>${badge(i.status)}</td><td>${i.endpoint ? html`<span class="muted">${clip(i.endpoint, 60)}</span>` : '—'}</td>++ <td>${when(i.started_at)}</td><td>${i.cold_start_ms == null ? '—' : coldStart(i.cold_start_ms)}</td><td>${usdNano(i.reserved_nano)}</td><td>${usdNano(i.spent_nano)}</td>++</tr>`;++}++++/**++ * The GPU page: the instances AI rents, what they cost this month and the cap they may never pass.++ * The actions only ask (a request row); the reconciler does the work, so a click is never a second++ * implementation of the provisioner.++ */++function gpus({ staff, csrf, state, canManage, notice, error }) {++ const st = state || {};++ const models = st.models || [];++ const instances = st.instances || [];++ const mk = st.marketplaces || {};++ return layout({++ title: 'OpenVibe GPUs', section: 'gpus', staff, csrf, notice, error, body: html`<h1>OpenVibe GPUs</h1>++<p class="muted">The rented GPUs: what is running, what it cost this month, and the cap it may never pass. The provisioner is ${st.enabled ? 'on' : 'off'}${st.enabled ? '' : ' (' + st.reason + ')'}: it starts nothing until a marketplace API key and a monthly cap are set in the owner settings.</p>++<div class="grid">++ <div class="card"><div class="k">Status</div><div class="v">${st.enabled ? 'on' : 'off'}</div><div class="muted">${st.reason || 'a key and a cap are set'}</div></div>++ <div class="card"><div class="k">Spent this month</div><div class="v">${usdNano(st.spent_nano)}</div><div class="muted">${st.month}</div></div>++ <div class="card"><div class="k">Committed</div><div class="v">${usdNano(st.committed_nano)}</div><div class="muted">the worst case: what the running instances may still cost</div></div>++ <div class="card"><div class="k">Monthly cap</div><div class="v">${st.cap_nano ? usdNano(st.cap_nano) : '—'}</div><div class="muted">never passed</div></div>++</div>++<p class="muted">Idle ${n(st.idle_minutes)} minute(s) before an instance scales to zero · max runtime ${n(st.max_runtime_minutes)} minute(s) · warm window ${st.warm_window || 'none'} · region ${st.region} · Vast.ai key ${mk.vast ? 'set' : 'not set'} · RunPod key ${mk.runpod ? 'set' : 'not set'}</p>++<p class="muted">Vast and RunPod bill per second, so the ledger charges the seconds an instance really ran: the price per second is next to the hourly price. The cold start is measured from the launch to the first answered request.</p>++${table('Models', ['Model', 'Marketplace', 'Runtime', 'GPU', 'Status', 'Cold start', 'Actions'], models.map((m) => gpuModelRow(m, canManage, csrf)), 'no model is known.')}++${table('Instances', ['Model', 'Marketplace', 'GPU', 'Price / hour', 'Status', 'Endpoint', 'Started', 'Cold start', 'Reserved', 'Spent'], instances.map((i) => gpuInstanceRow(i)), 'no instance has been rented.')}++`,++ });++}++function cache({ staff, csrf, stats, canManage, notice, error }) {const entries = stats.reduce((a, s) => a + s.entries, 0);const hits = stats.reduce((a, s) => a + (s.hits || 0), 0);@@@ -665,4 +734,4 @@ function allowance(cards) {/** The price and latency cells, as plain HTML strings for the public /stats page (server/home.js renderStats). */const statsFormat = { ms, rate, price: (cards) => render(price(cards)), allowance: (cards) => render(allowance(cards)) };--module.exports = { CSP, html, esc, safeUrl, statsFormat, pages: { signIn, message, overview, providers, providerNew, definitions, definition, runs, run, runReceipt, quotas, usage, cache, audit, exchange } };++module.exports = { CSP, html, esc, safeUrl, statsFormat, pages: { signIn, message, overview, providers, providerNew, definitions, definition, runs, run, runReceipt, quotas, usage, cache, audit, exchange, gpus } };nnew file mode 100644@@@ -0,0 +1,97 @@++'use strict';++/**++ * The models the GPU provisioner knows how to serve, and what each one needs from a GPU.++ *++ * The first two are the design's starting point: a DeepSeek R1 32B distill at Q4 on one RTX 3090 or++ * 4090 - once with the refusal training removed (abliterated) and once plain. A 32B Q4_K_M quant is++ * about 20 GB of weights, so it fits one 24 GB card.++ *++ * The weights URL, the image and the command are data, not code: they are what the owner's first++ * launch really runs, so they are shown on the console's GPU page and in the audit row. The owner++ * checks them before the first launch and corrects them here.++ *++ * vram_mb the VRAM the weights and the KV cache need (the marketplace's filter)++ * disk_gb the disk the image and the weights need++ * gpu_names the cards that fit the model++ * throughput the tokens an hour the card is expected to serve (a planning estimate; it prices the++ * offer for the router)++ */++const MODELS = Object.freeze([++ {++ key: 'deepseek-r1-32b-abliterated',++ display_name: 'DeepSeek R1 32B distill (abliterated), Q4',++ provider_key: 'gpu.deepseek-r1-32b-abliterated',++ model_key: 'deepseek-r1-32b-abliterated',++ marketplace: 'vast',++ context_window: 8192,++ throughput_tokens_per_hour: 180000,++ requirements: { vram_mb: 22000, disk_gb: 60, gpu_names: ['RTX 3090', 'RTX 4090'], num_gpus: 1, min_reliability: 0.99 },++ runtime: {++ kind: 'llama.cpp',++ image: 'ghcr.io/ggml-org/llama.cpp:server-cuda',++ port: 8000,++ weights: {++ repo: 'mradermacher/DeepSeek-R1-Distill-Llama-32B-abliterated-GGUF',++ file: 'DeepSeek-R1-Distill-Llama-32B-abliterated-Q4_K_M.gguf',++ },++ // The container's command: fetch the weights, then serve them with the OpenAI API.++ command: 'mkdir -p /models && curl -fL --retry 3 -o /models/{file} {weights_url} '++ + '&& exec /app/llama-server --model /models/{file} --host 0.0.0.0 --port {port} '++ + '--alias {model} --ctx-size {ctx} --n-gpu-layers 999',++ },++ },++ {++ key: 'deepseek-r1-32b',++ display_name: 'DeepSeek R1 32B distill, Q4',++ provider_key: 'gpu.deepseek-r1-32b',++ model_key: 'deepseek-r1-32b',++ marketplace: 'vast',++ context_window: 8192,++ throughput_tokens_per_hour: 180000,++ requirements: { vram_mb: 22000, disk_gb: 60, gpu_names: ['RTX 3090', 'RTX 4090'], num_gpus: 1, min_reliability: 0.99 },++ runtime: {++ kind: 'llama.cpp',++ image: 'ghcr.io/ggml-org/llama.cpp:server-cuda',++ port: 8000,++ weights: {++ repo: 'bartowski/DeepSeek-R1-Distill-Llama-32B-GGUF',++ file: 'DeepSeek-R1-Distill-Llama-32B-Q4_K_M.gguf',++ },++ command: 'mkdir -p /models && curl -fL --retry 3 -o /models/{file} {weights_url} '++ + '&& exec /app/llama-server --model /models/{file} --host 0.0.0.0 --port {port} '++ + '--alias {model} --ctx-size {ctx} --n-gpu-layers 999',++ },++ },++]);++++const byKey = new Map(MODELS.map((m) => [m.key, m]));++++/** The model a key names, or null. */++function find(key) {++ return byKey.get(String(key || '')) || null;++}++++/** Every model the provisioner knows. */++function list() {++ return [...MODELS];++}++++/** The URL the weights come from (a public Hugging Face file). */++function weightsUrl(spec) {++ const w = (spec && spec.runtime && spec.runtime.weights) || {};++ if (!w.repo || !w.file) return '';++ return 'https://huggingface.co/' + w.repo + '/resolve/main/' + w.file;++}++++/** The container command of a model, with its placeholders filled in. */++function commandOf(spec) {++ const r = (spec && spec.runtime) || {};++ return String(r.command || '')++ .replace(/\{file\}/g, (r.weights && r.weights.file) || '')++ .replace(/\{weights_url\}/g, weightsUrl(spec))++ .replace(/\{port\}/g, String(r.port || 8000))++ .replace(/\{model\}/g, (spec && spec.model_key) || '')++ .replace(/\{ctx\}/g, String((spec && spec.context_window) || 8192));++}++++module.exports = { MODELS, find, list, weightsUrl, commandOf };nnew file mode 100644@@@ -0,0 +1,28 @@++'use strict';++/**++ * One HTTP call to a GPU marketplace. The API key never leaves this process: it goes in the++ * Authorization header, never in a URL, and a failure never carries the key back.++ */++const { AiError } = require('../util');++++async function apiJson(fetchImpl, url, { method = 'GET', body = null, key = null, label = 'the marketplace' } = {}) {++ const headers = { Accept: 'application/json' };++ if (body != null) headers['Content-Type'] = 'application/json';++ if (key) headers.Authorization = 'Bearer ' + key;++ let res;++ try {++ res = await fetchImpl(url, { method, headers, body: body != null ? JSON.stringify(body) : undefined });++ } catch (err) {++ throw new AiError(502, 'gpu.marketplace_unreachable', label + ' did not answer: ' + String(err && err.message).slice(0, 200));++ }++ const text = await res.text().catch(() => '');++ let json = null;++ try { json = text ? JSON.parse(text) : null; } catch { json = null; }++ if (!res.ok) {++ const detail = (json && (json.detail || json.message || json.error)) || String(res.status);++ throw new AiError(502, 'gpu.marketplace_error', label + ' refused: ' + String(detail).slice(0, 200) + ' (' + res.status + ')');++ }++ return json;++}++++module.exports = { apiJson };nnew file mode 100644@@@ -0,0 +1,62 @@++'use strict';++/**++ * The money of the GPU provisioner: integer nano-USD in BigInt, never a float (AI's D1 rule; the same++ * rule server/credits.js follows for Credits). Nothing here multiplies or divides a float:++ *++ * nanoCount() an amount the owner set (the monthly cap): an integer count of nano-USD.++ * nanoUsd() a marketplace's own decimal price ("0.1348" dollars an hour) -> exact nano-USD.++ * nanoForSeconds() what an instance at a known hourly rate costs for a known number of seconds,++ * rounded up to the nano: the ledger never charges less than the GPU was paid for.++ * usdText() nano-USD as a dollar string for a page (exact; a page never does arithmetic on it).++ */++const { AiError } = require('../util');++const { parseCostUsd } = require('../credits');++++const NANO_PER_USD = 1000000000n;++const NANO_PER_MILLION = 1000000000000n;++const SECONDS_PER_HOUR = 3600n;++++/** An integer count of nano-USD (the settings' money unit). null when absent; never a float. */++function nanoCount(value, field = 'amount') {++ if (value == null || value === '') return null;++ if (typeof value === 'bigint') return value;++ const raw = String(value).trim();++ if (!/^[0-9]+$/.test(raw)) throw new AiError(422, 'ai.invalid', field + ' must be an integer number of nano-USD');++ return BigInt(raw);++}++++/** A plain decimal dollar amount (a marketplace's price) -> exact nano-USD. */++function nanoUsd(value, field = 'price') {++ if (value == null || value === '') return null;++ return parseCostUsd(value, field);++}++++/** What an instance at hourlyNano costs for seconds: rounded up to the nano, never less. */++function nanoForSeconds(hourlyNano, seconds) {++ const s = BigInt(Math.max(0, Math.ceil(Number(seconds) || 0)));++ if (s === 0n) return 0n;++ return (BigInt(hourlyNano) * s + SECONDS_PER_HOUR - 1n) / SECONDS_PER_HOUR;++}++++/** nano-USD as a dollar string (exact; for a page only). */++function usdText(nano) {++ const n = BigInt(nano == null ? 0 : nano);++ const abs = n < 0n ? -n : n;++ const whole = abs / NANO_PER_USD;++ const frac = (abs % NANO_PER_USD).toString().padStart(9, '0').replace(/0+$/, '');++ return (n < 0n ? '-' : '') + '$' + whole.toString() + (frac ? '.' + frac : '');++}++++/**++ * The price of one million tokens on a GPU that costs hourlyNano an hour and serves tokensPerHour++ * tokens: the hourly rate spread over the tokens it can produce. It is an estimate for the router's++ * price comparison (the authoritative money is the GPU ledger), computed from the exact nano rate so++ * the two never drift apart.++ */++function usdPerMillionTokens(hourlyNano, tokensPerHour) {++ const t = BigInt(Math.max(1, Math.trunc(Number(tokensPerHour) || 0)));++ const nano = (BigInt(hourlyNano) * NANO_PER_MILLION + t - 1n) / t;++ return Number(nano) / 1e9;++}++++module.exports = { NANO_PER_USD, NANO_PER_MILLION, nanoCount, nanoUsd, nanoForSeconds, usdText, usdPerMillionTokens };nnew file mode 100644@@@ -0,0 +1,797 @@++'use strict';++/**++ * The GPU provisioner (the design's "OpenVibe GPUs").++ *++ * It rents the cheapest marketplace GPU that fits a model, serves the model's weights with vLLM or++ * llama.cpp, registers the endpoint as an offer for the router, health-checks it, scales it to zero++ * when it is idle, and never spends past the owner's monthly cap.++ *++ * The rules it keeps, in the order they matter:++ *++ * 1. It is off until the owner has set a marketplace API key AND a monthly cap. No key or no cap++ * means no instance, no search and no money.++ * 2. It never passes the monthly cap. Every instance holds a reservation (the budget it may cost)++ * before it starts, the reservation is part of the month's committed spend, and a new instance++ * is only started when the reservation still fits under the cap. When an instance's reservation++ * has run out the instance is given back, so the money stops there.++ * 3. The money is integer nano-USD in BigInt (server/gpu/money.js). Nothing here is a float.++ * 4. It picks the cheapest offer that fits (VRAM, disk, reliability, region) and the choice is++ * deterministic: the same offers in the same order always give the same instance.++ */++const { randomUUID } = require('node:crypto');++const { AiError } = require('../util');++const money = require('./money');++const catalog = require('./catalog');++const { vastAdapter } = require('./vast');++const { runpodAdapter } = require('./runpod');++++const HEALTH_INTERVAL_MS = 30_000;++const HEALTH_FAILURES = 2;++const DEFAULT_IDLE_MINUTES = 10;++const DEFAULT_MAX_RUNTIME_MINUTES = 720;++const DEFAULT_REGION = 'US';++const LIVE = ['starting', 'running', 'stopping'];++const DEAD_STATES = ['exited', 'unknown', 'offline', 'dead', 'terminated', 'stopped', 'error'];++// Every instance we rent carries our row id as its label (an openvibe- prefix), so the sweep can find++// it again and give it back when no live row of ours owns it.++const LABEL_PREFIX = 'openvibe-';++const LABEL_RE = /^openvibe-[0-9a-f]{12}$/;++// Three failed destroys in a row: the instance is still running and still being paid for, so the++// operator is alerted (the gauge ai_gpu_destroy_failures and the rule in OpenVibe.Host's prometheus rules).++const ALERT_AFTER = 3;++// A row whose rental never came back is given up on after this long: its reservation is free again.++const STALE_ROW_MS = 10 * 60_000;++const PRICE_SOURCE = { vast: 'https://vast.ai/pricing', runpod: 'https://www.runpod.io/pricing' };++++function clampInt(value, min, max, fallback) {++ if (value == null || value === '') return fallback;++ const n = Number(value);++ if (!Number.isFinite(n)) return fallback;++ return Math.max(min, Math.min(max, Math.trunc(n)));++}++++/** A database value as milliseconds, or null. */++function toMs(value) {++ if (value == null) return null;++ if (value instanceof Date) return value.getTime();++ const t = Date.parse(value);++ return Number.isFinite(t) ? t : null;++}++++/** A database value as exact nano-USD. A value that is not an integer is loud, never rounded. */++function toNano(value) {++ if (value == null) return 0n;++ if (typeof value === 'bigint') return value;++ const s = String(value).trim();++ if (/^-?[0-9]+$/.test(s)) return BigInt(s);++ throw new AiError(500, 'gpu.ledger_corrupt', 'the GPU ledger holds a value that is not an integer number of nano-USD');++}++++/**++ * The provisioner.++ *++ * db the database handle (open vibe sdk db)++ * registry the provider registry: the instance is registered there as an offer for the router++ * settings AI's settings; settings.gpus.values() is the owner's GPU settings++ * clock { now(): number } - a fake clock in the tests++ * fetchImpl the HTTP client the marketplace calls go through (a fake in the tests)++ */++function createGpuProvisioner({++ db,++ registry,++ settings,++ clock = { now: () => Date.now() },++ fetchImpl = globalThis.fetch,++ env = process.env,++ log = console,++ models = catalog.list(),++} = {}) {++ const now = () => clock.now();++ const iso = (ms) => new Date(ms).toISOString();++ const monthOf = (ms) => new Date(ms).toISOString().slice(0, 7);++++ // ── the owner's settings ──────────────────────────────────────────────++++ /** The owner's GPU settings, with the money exact and the defaults applied. */++ function config() {++ const v = (settings && settings.gpus && typeof settings.gpus.values === 'function') ? settings.gpus.values() : {};++ let cap = null;++ try {++ cap = money.nanoCount(v.monthly_cap, 'the monthly GPU cap');++ } catch {++ cap = null;++ }++ const vastKey = String(v.vast_key || '');++ const runpodKey = String(v.runpod_key || '');++ let reason = '';++ if (!vastKey && !runpodKey) reason = 'no Vast.ai or RunPod API key';++ else if (cap == null || cap <= 0n) reason = 'no monthly cap';++ return {++ vast_key: vastKey,++ runpod_key: runpodKey,++ cap_nano: cap,++ idle_minutes: clampInt(v.idle_minutes, 1, 1440, DEFAULT_IDLE_MINUTES),++ max_runtime_minutes: clampInt(v.max_runtime_minutes, 1, 10080, DEFAULT_MAX_RUNTIME_MINUTES),++ warm_window: String(v.warm_window || '').trim(),++ region: String(v.region || DEFAULT_REGION).trim().toUpperCase() || DEFAULT_REGION,++ enabled: Boolean(vastKey || runpodKey) && cap != null && cap > 0n,++ reason,++ };++ }++++ /** The adapter of a marketplace, or null when its key is not set. */++ function adapterFor(marketplace, cfg) {++ if (marketplace === 'runpod') {++ if (!cfg.runpod_key) return null;++ return runpodAdapter({ fetchImpl, key: cfg.runpod_key });++ }++ if (!cfg.vast_key) return null;++ return vastAdapter({ fetchImpl, key: cfg.vast_key });++ }++++ /** Is the moment inside the owner's warm window (UTC)? Then an idle instance stays up. */++ function inWarmWindow(window, at) {++ const m = /^([0-2]\d):([0-5]\d)-([0-2]\d):([0-5]\d)$/.exec(String(window || '').trim());++ if (!m) return false;++ const minutes = (h, mi) => Number(h) * 60 + Number(mi);++ const start = minutes(m[1], m[2]);++ const end = minutes(m[3], m[4]);++ if (start === end) return false;++ const d = new Date(at);++ const cur = d.getUTCHours() * 60 + d.getUTCMinutes();++ if (start < end) return cur >= start && cur < end;++ return cur >= start || cur < end;++ }++++ // ── the money ─────────────────────────────────────────────────────────++++ /** The month's money: what was really spent, and what is committed (the worst case). */++ async function moneySummary() {++ const month = monthOf(now());++ const cfg = config();++ const ledger = await db.prepare('SELECT COALESCE(SUM(nano_usd), 0) AS total FROM gpu_spend WHERE month = ?').get(month);++ const settled = toNano(ledger && ledger.total);++ const rows = await db.prepare('SELECT * FROM gpu_instances WHERE month = ? AND status IN (?, ?, ?)').all(month, 'starting', 'running', 'stopping');++ let reserved = 0n;++ let running = 0n;++ for (const row of rows) {++ reserved += toNano(row.reserved_nano);++ const started = toMs(row.started_at);++ if (started != null) running += money.nanoForSeconds(toNano(row.hourly_nano), (now() - started) / 1000);++ }++ return {++ month,++ cap_nano: cfg.cap_nano,++ spent_nano: settled + running,++ committed_nano: settled + reserved,++ reserved_nano: reserved,++ settled_nano: settled,++ };++ }++++ /** Write the money an instance has cost since the last charge. */++ async function charge(row, until, reason) {++ const hourly = toNano(row.hourly_nano);++ const from = toMs(row.charged_at) || toMs(row.started_at) || until;++ const seconds = Math.max(0, (until - from) / 1000);++ if (seconds <= 0) return 0n;++ const amount = money.nanoForSeconds(hourly, seconds);++ if (amount <= 0n) return 0n;++ await db.prepare('INSERT INTO gpu_spend (at, month, kind, nano_usd, instance_id, detail, actor, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)')++ .run(iso(until), monthOf(until), 'charge', amount.toString(), row.id, reason || 'the GPU ran', 'service:ai', iso(now()));++ await db.prepare('UPDATE gpu_instances SET charged_at = ?, spent_nano = spent_nano + ?, updated_at = ? WHERE id = ?')++ .run(iso(until), amount.toString(), iso(now()), row.id);++ return amount;++ }++++ // ── the instances ─────────────────────────────────────────────────────++++ async function liveInstance(modelKey) {++ const row = await db.prepare('SELECT * FROM gpu_instances WHERE model_key = ? AND status IN (?, ?, ?) ORDER BY created_at DESC LIMIT 1')++ .get(String(modelKey), 'starting', 'running', 'stopping');++ return row || null;++ }++++ async function instanceRows() {++ const rows = await db.prepare('SELECT * FROM gpu_instances ORDER BY created_at DESC').all();++ return rows || [];++ }++++ /** The cheapest offer that fits. The same offers in the same order always give the same one. */++ function cheapest(offers) {++ const usable = (offers || []).filter((o) => o && o.price_per_hour != null && Number(o.price_per_hour) > 0);++ usable.sort((a, b) => {++ const pa = Number(a.price_per_hour);++ const pb = Number(b.price_per_hour);++ if (pa !== pb) return pa - pb;++ return String(a.id) < String(b.id) ? -1 : String(a.id) > String(b.id) ? 1 : 0;++ });++ return usable[0] || null;++ }++++ /** Ask for a model to start or stop. The reconciler does it; nothing here acts directly. */++ async function request(modelKey, action, actor = 'service:ai') {++ if (!catalog.find(modelKey) && !models.some((m) => m.key === modelKey)) throw new AiError(404, 'ai.not_found', 'no such GPU model');++ if (action !== 'launch' && action !== 'stop') throw new AiError(422, 'ai.invalid', 'the action must be launch or stop');++ await db.prepare('INSERT INTO gpu_requests (model_key, action, at, actor, created_at) VALUES (?, ?, ?, ?, ?)')++ .run(String(modelKey), action, iso(now()), String(actor || '').slice(0, 200), iso(now()));++ return { ok: true, model_key: modelKey, action };++ }++++ /** Is the model wanted? It is wanted when the owner asked for it or when it was used recently. */++ async function wanted(spec, cfg) {++ const pending = await db.prepare("SELECT COUNT(*) AS n FROM gpu_requests WHERE model_key = ? AND action = 'launch' AND done_at IS NULL")++ .get(spec.key);++ if (pending && Number(pending.n) > 0) return true;++ const used = await db.prepare('SELECT MAX(created_at) AS last FROM runs WHERE provider_key = ?').get(spec.provider_key);++ const last = toMs(used && used.last);++ if (last != null && now() - last <= cfg.idle_minutes * 60_000) return true;++ return false;++ }++++ /**++ * Start an instance for a model: the cheapest offer that fits, inside the cap.++ * Returns { ok, reason } - it never throws for the owner's reasons (off, no offer, cap).++ */++ async function launch(modelKey, { actor = 'service:ai' } = {}) {++ const cfg = config();++ if (!cfg.enabled) return { ok: false, code: 'gpu.off', reason: cfg.reason };++ const spec = models.find((m) => m.key === modelKey) || catalog.find(modelKey);++ if (!spec) throw new AiError(404, 'ai.not_found', 'no such GPU model');++ const existing = await liveInstance(spec.key);++ if (existing) return { ok: true, instance: describe(existing), note: 'already running' };++ const adapter = adapterFor(spec.marketplace, cfg);++ if (!adapter) return { ok: false, code: 'gpu.no_key', reason: 'no ' + spec.marketplace + ' API key' };++++ // 1. the cheapest offer that fits the model++ let offers;++ try {++ offers = await adapter.searchOffers({++ gpuNames: spec.requirements.gpu_names,++ vramMb: spec.requirements.vram_mb,++ diskGb: spec.requirements.disk_gb,++ numGpus: spec.requirements.num_gpus,++ minReliability: spec.requirements.min_reliability,++ region: cfg.region,++ maxPricePerHour: 10,++ limit: 50,++ });++ } catch (err) {++ return { ok: false, code: 'gpu.search_failed', reason: String(err && err.message).slice(0, 200) };++ }++ const offer = cheapest(offers);++ if (!offer) {++ return { ok: false, code: 'gpu.no_offer', reason: 'no ' + spec.marketplace + ' offer fits ' + spec.display_name };++ }++++ // 2. the cap: the reservation has to fit under what is left of it++ const hourly = money.nanoUsd(offer.price_per_hour, "the offer's price");++ if (hourly == null || hourly <= 0n) {++ return { ok: false, code: 'gpu.bad_price', reason: "the offer has no usable price" };++ }++ const sum = await moneySummary();++ const cap = cfg.cap_nano;++ if (sum.committed_nano >= cap) {++ return { ok: false, code: 'gpu.cap', reason: 'the monthly cap is reached (' + money.usdText(sum.committed_nano) + ' of ' + money.usdText(cap) + ')' };++ }++ const affordableHours = (cap - sum.committed_nano) / hourly;++ if (affordableHours < 1n) {++ return { ok: false, code: 'gpu.cap', reason: 'the monthly cap leaves less than an hour of GPU time (' + money.usdText(cap - sum.committed_nano) + ' left)' };++ }++ const maxMinutes = Math.min(cfg.max_runtime_minutes, Number(affordableHours) * 60);++ const reserved = hourly * BigInt(Math.ceil(maxMinutes / 60));++ if (sum.committed_nano + reserved > cap) {++ return { ok: false, code: 'gpu.cap', reason: 'the monthly cap leaves no room for ' + spec.display_name };++ }++++ // 3. the row comes first. The marketplace is told the label of a row that already exists and++ // holds the reservation, so an instance we paid for is never an orphan: the sweep finds it by++ // the label and gives it back when no live row of ours owns it.++ const id = LABEL_PREFIX + randomUUID().replace(/-/g, '').slice(0, 12);++ const t = now();++ await db.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,++ gpu_name, gpu_ram_mb, num_gpus, disk_gb, reliability, region, price_per_hour, hourly_nano, reserved_nano, spent_nano,++ max_runtime_minutes, started_at, charged_at, destroy_failures, label, cold_start_ms, health_failures, created_at, updated_at)++ VALUES (@id, @model_key, @provider_key, @marketplace, @offer_id, NULL, 'starting', NULL, @month,++ @gpu_name, @gpu_ram_mb, @num_gpus, @disk_gb, @reliability, @region, @price_per_hour, @hourly_nano, @reserved_nano, 0,++ @max_runtime_minutes, @started_at, @charged_at, 0, @label, NULL, 0, @created_at, @updated_at)`)++ .run({++ id, model_key: spec.key, provider_key: spec.provider_key, marketplace: spec.marketplace,++ offer_id: String(offer.id), month: monthOf(t),++ gpu_name: String(offer.gpu_name || ''), gpu_ram_mb: Number(offer.gpu_ram_mb) || 0,++ num_gpus: Number(offer.num_gpus) || 1, disk_gb: Number(offer.disk_gb) || 0,++ reliability: Number(offer.reliability) || 0, region: String(offer.region || ''),++ price_per_hour: String(offer.price_per_hour), hourly_nano: hourly.toString(), reserved_nano: reserved.toString(),++ max_runtime_minutes: maxMinutes, started_at: iso(t), charged_at: iso(t), label: id,++ created_at: iso(t), updated_at: iso(t),++ });++ await audit(actor, 'gpu.instance.requested', 'gpu_instance', id, {++ model: spec.key, marketplace: spec.marketplace, offer_id: String(offer.id), hourly_nano: hourly.toString(),++ reserved_nano: reserved.toString(), max_runtime_minutes: maxMinutes, label: id,++ });++++ // 4. rent it. The marketplace is given the label of the row above, so the instance can always be++ // found again. If the rental fails the row says so and the reservation is free again.++ let created;++ try {++ created = await adapter.createInstance(offer, spec, {++ command: catalog.commandOf(spec),++ weights_url: catalog.weightsUrl(spec),++ label: id,++ });++ } catch (err) {++ const reason = String(err && err.message).slice(0, 200);++ await db.prepare("UPDATE gpu_instances SET status = 'refused', error = ?, updated_at = ? WHERE id = ?")++ .run(reason, iso(now()), id);++ await audit(actor, 'gpu.instance.refused', 'gpu_instance', id, { reason });++ return { ok: false, code: 'gpu.create_failed', reason };++ }++ await db.prepare('UPDATE gpu_instances SET remote_id = ?, updated_at = ? WHERE id = ?')++ .run(String(created.remote_id || ''), iso(now()), id);++ await audit(actor, 'gpu.instance.create', 'gpu_instance', id, { remote_id: String(created.remote_id || '') });++ const row = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);++ return { ok: true, instance: describe(row), offer, reserved_nano: reserved, hourly_nano: hourly };++ }++++ /** The offer leaves the pool: the router fails over to the other providers. */++ async function disableProvider(providerKey, reason) {++ try {++ await registry.upsertProvider({ key: providerKey, status: 'disabled' }, { actor: 'service:ai', origin: 'gpu' });++ } catch (err) {++ log.warn && log.warn('[gpu] could not disable the offer ' + providerKey + ': ' + String(err && err.message));++ }++ }++++ /** Register the instance's endpoint as an offer for the router. */++ async function register(spec, row) {++ const base = String(row.endpoint || '').replace(/\/+$/, '') + '/v1';++ if (!/^https?:\/\/\S+$/i.test(base)) throw new AiError(502, 'gpu.no_endpoint', 'the instance has no usable endpoint');++ const price = money.usdPerMillionTokens(toNano(row.hourly_nano), spec.throughput_tokens_per_hour);++ const gpu = {++ marketplace: row.marketplace,++ offer_id: row.offer_id,++ instance_id: row.id,++ gpu_name: row.gpu_name,++ price_per_hour: row.price_per_hour,++ region: row.region,++ runtime: spec.runtime.kind,++ weights: catalog.weightsUrl(spec),++ };++ await registry.upsertProvider({++ key: spec.provider_key,++ display_name: spec.display_name,++ kind: 'openai',++ status: 'active',++ base_url: base,++ auth_mode: 'none',++ secret_ref: null,++ default_model: spec.model_key,++ capabilities: ['chat'],++ timeout_ms: 300000,++ priority: 40,++ billing_profile: 'metered',++ provider_access: 'OWN_DEPLOYMENT',++ metadata: { gpu },++ }, { actor: 'service:ai', origin: 'gpu' });++ await registry.upsertModel({++ provider_key: spec.provider_key,++ model_key: spec.model_key,++ display_name: spec.display_name,++ type: 'chat',++ status: 'active',++ context_window: spec.context_window,++ cost: {++ in_per_mtok: price,++ out_per_mtok: price,++ cached_per_mtok: price,++ source: PRICE_SOURCE[row.marketplace] || 'urn:openvibe:ai:unverified:gpu',++ verified_at: iso(now()).slice(0, 10),++ },++ supports: { json: false, tools: false, streaming: true, vision: false },++ metadata: { gpu },++ }, { actor: 'service:ai' });++ return base;++ }++++ /** Is the endpoint answering? The OpenAI-compatible /models is the cheapest honest check. */++ async function healthy(endpoint) {++ if (!endpoint) return false;++ try {++ const res = await fetchImpl(String(endpoint).replace(/\/+$/, '') + '/v1/models', { method: 'GET', headers: { Accept: 'application/json' } });++ return Boolean(res && res.ok);++ } catch {++ return false;++ }++ }++++ /**++ * Give the instance back and settle what it cost.++ *++ * A destroy that fails leaves the instance running (and being paid for): the row keeps its place,++ * the try is counted, the next pass tries again and after three tries the operator is alerted++ * (the gauge ai_gpu_destroy_failures and the rule in OpenVibe.Host's prometheus rules).++ */++ async function destroy(row, reason, { actor = 'service:ai' } = {}) {++ const cfg = config();++ const adapter = adapterFor(row.marketplace, cfg);++ if (adapter && row.remote_id) {++ try {++ await adapter.destroyInstance(row.remote_id);++ } catch (err) {++ const failures = Number(row.destroy_failures || 0) + 1;++ await db.prepare('UPDATE gpu_instances SET destroy_failures = ?, updated_at = ? WHERE id = ?')++ .run(failures, iso(now()), row.id);++ log.warn && log.warn('[gpu] could not give instance ' + row.id + ' back (' + failures + ' tries): ' + String(err && err.message));++ if (failures >= ALERT_AFTER) {++ await audit(actor, 'gpu.instance.destroy_failed', 'gpu_instance', row.id, { reason, failures, marketplace: row.marketplace });++ log.error && log.error('[gpu] an instance could not be given back after ' + failures + ' tries: it is still running and still being paid for');++ }++ return { ok: false, failed: true, reason, failures };++ }++ }++ const t = now();++ // the freshest row: the money already on the ledger is not charged twice++ const fresh = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(row.id);++ await charge(fresh || row, t, reason);++ await db.prepare("UPDATE gpu_instances SET status = 'destroyed', destroyed_at = ?, endpoint = NULL, destroy_failures = 0, updated_at = ? WHERE id = ?")++ .run(iso(t), iso(t), row.id);++ await disableProvider(row.provider_key, reason);++ await audit(actor, 'gpu.instance.destroy', 'gpu_instance', row.id, { reason, model: row.model_key });++ return { ok: true, reason };++ }++++ /**++ * Every pass: the marketplace instances that carry our label, and what to do with them.++ *++ * - one whose label is a live row of ours: left alone, and linked when the row does not know its++ * marketplace id yet (a crash between the marketplace's answer and the row's update).++ * - one no live row of ours owns: an orphan. It is given back and the fact is logged.++ *++ * A destroy that fails is tried again on the next pass; after three tries the operator is alerted.++ */++ async function sweepOrphans(cfg) {++ const t = now();++ let seen = 0;++ for (const marketplace of ['vast', 'runpod']) {++ const adapter = adapterFor(marketplace, cfg);++ if (!adapter) continue;++ let instances = [];++ try {++ instances = await adapter.listInstances();++ } catch (err) {++ log.warn && log.warn('[gpu] could not list the ' + marketplace + ' instances: ' + String(err && err.message).slice(0, 200));++ continue;++ }++ for (const inst of instances) {++ const label = String(inst.label || '');++ if (!LABEL_RE.test(label)) continue; // not ours++ seen += 1;++ const row = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(label);++ if (row && LIVE.includes(row.status)) {++ // ours and alive: a crash after the marketplace accepted but before the row knew the++ // id is repaired here, not thrown away.++ if (!row.remote_id && inst.id) {++ await db.prepare('UPDATE gpu_instances SET remote_id = ?, updated_at = ? WHERE id = ?')++ .run(String(inst.id), iso(now()), row.id);++ await audit('service:ai', 'gpu.instance.recovered', 'gpu_instance', row.id, { marketplace, remote_id: String(inst.id) });++ log.log && log.log('[gpu] recovered the instance ' + inst.id + ' of the row ' + row.id + ' from the ' + marketplace + ' label');++ }++ continue;++ }++ await destroyOrphan({ marketplace, remote_id: String(inst.id), label }, row, cfg);++ }++ }++ // The rows whose rental never came back (a crash before the marketplace answered, or an++ // answer we never got): their reservation is free again, or the month would carry a budget++ // for an instance that does not exist.++ const stale = await db.prepare("SELECT * FROM gpu_instances WHERE status = 'starting' AND (remote_id IS NULL OR remote_id = '') AND started_at < ?").all(iso(t - STALE_ROW_MS));++ for (const row of stale || []) {++ await db.prepare("UPDATE gpu_instances SET status = 'refused', error = ?, updated_at = ? WHERE id = ?")++ .run('the rental never came back', iso(t), row.id);++ await audit('service:ai', 'gpu.instance.abandoned', 'gpu_instance', row.id, { marketplace: row.marketplace });++ log.warn && log.warn('[gpu] gave up on the row ' + row.id + ': the rental never came back, its reservation is free again');++ }++ return seen;++ }++++ /** An instance with our label that no live row owns: give it back, and keep the tries. */++ async function destroyOrphan(inst, row, cfg) {++ const adapter = adapterFor(inst.marketplace, cfg);++ const id = row ? row.id : inst.label;++ const t = now();++ if (!row) {++ // the row the orphan needs: its retries and its money need a home of their own++ await db.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,++ gpu_name, hourly_nano, reserved_nano, spent_nano, max_runtime_minutes, started_at, charged_at, destroy_failures, label, cold_start_ms, health_failures, created_at, updated_at)++ VALUES (@id, '', '', @marketplace, '', @remote_id, 'orphan', NULL, @month, '', 0, 0, 0, 0, @started_at, @charged_at, 0, @label, NULL, 0, @created_at, @updated_at)`)++ .run({++ id, marketplace: inst.marketplace, remote_id: inst.remote_id, month: monthOf(t),++ started_at: iso(t), charged_at: iso(t), label: inst.label, created_at: iso(t), updated_at: iso(t),++ });++ row = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);++ }++ // the orphan ran and was paid for: the seconds since the last charge are on the ledger++ await charge(row, t, 'the orphan ran');++ try {++ await adapter.destroyInstance(inst.remote_id);++ } catch (err) {++ const failures = Number(row.destroy_failures || 0) + 1;++ await db.prepare('UPDATE gpu_instances SET destroy_failures = ?, updated_at = ? WHERE id = ?')++ .run(failures, iso(t), id);++ log.warn && log.warn('[gpu] could not give the orphan ' + inst.remote_id + ' back (' + failures + ' tries): ' + String(err && err.message).slice(0, 200));++ if (failures >= ALERT_AFTER) {++ await audit('service:ai', 'gpu.orphan.destroy_failed', 'gpu_instance', id, { marketplace: inst.marketplace, remote_id: inst.remote_id, failures });++ log.error && log.error('[gpu] an orphan instance could not be given back after ' + failures + ' tries: it is still running and still being paid for');++ }++ return { ok: false, failed: true };++ }++ await db.prepare("UPDATE gpu_instances SET status = 'destroyed', destroyed_at = ?, destroy_failures = 0, updated_at = ? WHERE id = ?")++ .run(iso(t), iso(t), id);++ await audit('service:ai', 'gpu.orphan.destroyed', 'gpu_instance', id, { marketplace: inst.marketplace, remote_id: inst.remote_id });++ log.warn && log.warn('[gpu] gave back an orphan instance ' + inst.remote_id + ' (' + inst.marketplace + '): it carried our label but no live row owned it');++ return { ok: true };++ }++++ /** The destroy failures that are still open, per marketplace (the operator alert reads them). */++ async function destroyFailures() {++ const rows = await db.prepare('SELECT marketplace, MAX(destroy_failures) AS failures FROM gpu_instances WHERE destroy_failures > 0 GROUP BY marketplace').all();++ return (rows || []).map((r) => ({ labels: { marketplace: r.marketplace }, value: Number(r.failures) || 0 }));++ }++++ /** The instances with our label that no live row owns and that are still waiting to be given back. */++ async function orphanCount() {++ const rows = await db.prepare("SELECT marketplace, COUNT(*) AS n FROM gpu_instances WHERE status = 'orphan' GROUP BY marketplace").all();++ return (rows || []).map((r) => ({ labels: { marketplace: r.marketplace }, value: Number(r.n) || 0 }));++ }++++ /** One pass over the live instances: the reservation, the idle timer, the health. */++ async function reconcileInstance(row, cfg) {++ const spec = models.find((m) => m.key === row.model_key) || catalog.find(row.model_key);++ if (!spec) return { changed: false };++ const t = now();++ const started = toMs(row.started_at) || t;++++ // the money that ran while we were away++ await charge(row, t, 'the GPU ran');++++ // 1. the reservation ran out: the instance goes back, so the money stops there++ if (t - started >= Number(row.max_runtime_minutes) * 60_000) {++ await destroy({ ...row, charged_at: row.charged_at }, 'the reservation ran out');++ return { changed: true, action: 'destroyed', reason: 'the reservation ran out' };++ }++++ // 2. the idle timer: the offer scales to zero (unless it is inside the warm window). The clock runs++ // from the moment the instance was ready, not from the moment it was rented: an instance that is still++ // starting is never idle.++ if (row.status === 'running') {++ const lastUsed = toMs(row.last_used_at) || toMs(row.ready_at) || started;++ if (t - lastUsed >= cfg.idle_minutes * 60_000 && !inWarmWindow(cfg.warm_window, t)) {++ await destroy({ ...row, charged_at: row.charged_at }, 'idle');++ return { changed: true, action: 'scaled_to_zero', reason: 'idle for ' + cfg.idle_minutes + ' minutes' };++ }++ }++++ // 3. the health of the instance++ if (row.status === 'starting') {++ const adapter = adapterFor(row.marketplace, cfg);++ if (!adapter) return { changed: false };++ // a row without a marketplace id: the crash between the rental and the row's update. The++ // sweep finds the instance by the label and links it; until then there is nothing to poll.++ if (!row.remote_id) return { changed: false };++ let info = null;++ try {++ info = await adapter.showInstance(row.remote_id);++ } catch (err) {++ return { changed: false, error: String(err && err.message).slice(0, 200) };++ }++ const state = String((info && info.state) || 'unknown').toLowerCase();++ if (DEAD_STATES.includes(state)) {++ await destroy({ ...row, charged_at: row.charged_at }, 'the instance ' + state);++ await request(spec.key, 'launch', 'service:ai');++ return { changed: true, action: 'died', reason: 'the instance ' + state };++ }++ const endpoint = info && info.endpoint ? String(info.endpoint) : '';++ if (endpoint && !(await healthy(endpoint))) {++ // the endpoint is there but does not answer: the instance is dead, not slow++ const failures = Number(row.health_failures || 0) + 1;++ await db.prepare('UPDATE gpu_instances SET health_failures = ?, last_health_at = ?, updated_at = ? WHERE id = ?')++ .run(failures, iso(t), iso(t), row.id);++ if (failures >= HEALTH_FAILURES) {++ await destroy({ ...row, charged_at: row.charged_at }, 'the endpoint never answered');++ await request(spec.key, 'launch', 'service:ai');++ return { changed: true, action: 'died', reason: 'the endpoint never answered' };++ }++ return { changed: false, health_failures: failures };++ }++ if (endpoint && await healthy(endpoint)) {++ // the cold start: from the launch to the first answered request++ await db.prepare("UPDATE gpu_instances SET status = 'running', endpoint = ?, ready_at = ?, last_health_at = ?, health_failures = 0, cold_start_ms = ?, updated_at = ? WHERE id = ?")++ .run(endpoint, iso(t), iso(t), String(Math.max(0, t - started)), iso(t), row.id);++ await register(spec, { ...row, endpoint });++ return { changed: true, action: 'ready' };++ }++ return { changed: false };++ }++++ if (row.status === 'running') {++ const lastHealth = toMs(row.last_health_at) || 0;++ if (t - lastHealth < HEALTH_INTERVAL_MS) return { changed: false };++ const ok = await healthy(row.endpoint);++ if (ok) {++ await db.prepare('UPDATE gpu_instances SET health_failures = 0, last_health_at = ?, updated_at = ? WHERE id = ?')++ .run(iso(t), iso(t), row.id);++ return { changed: false };++ }++ const failures = Number(row.health_failures || 0) + 1;++ await db.prepare('UPDATE gpu_instances SET health_failures = ?, last_health_at = ?, updated_at = ? WHERE id = ?')++ .run(failures, iso(t), iso(t), row.id);++ if (failures < HEALTH_FAILURES) return { changed: false, health_failures: failures };++ // the instance died: the offer leaves the pool and a new one is rented++ await destroy({ ...row, charged_at: row.charged_at }, 'the endpoint stopped answering');++ await request(spec.key, 'launch', 'service:ai');++ return { changed: true, action: 'died', reason: 'the endpoint stopped answering' };++ }++ return { changed: false };++ }++++ /** The audit row. The registry's audit is used when it is there; the money rows are the record. */++ async function audit(actor, action, kind, key, metadata) {++ try {++ if (registry && typeof registry.audit === 'function') {++ await registry.audit(actor, action, kind, key, { metadata });++ return;++ }++ } catch (err) {++ log.warn && log.warn('[gpu] the audit row failed: ' + String(err && err.message));++ }++ }++++ /** A row as the console and the API show it. */++ function describe(row) {++ if (!row) return null;++ return {++ id: row.id,++ model_key: row.model_key,++ provider_key: row.provider_key,++ marketplace: row.marketplace,++ offer_id: row.offer_id,++ remote_id: row.remote_id,++ status: row.status,++ endpoint: row.endpoint,++ month: row.month,++ gpu_name: row.gpu_name,++ gpu_ram_mb: row.gpu_ram_mb,++ num_gpus: row.num_gpus,++ disk_gb: row.disk_gb,++ reliability: row.reliability,++ region: row.region,++ price_per_hour: row.price_per_hour,++ hourly_nano: String(toNano(row.hourly_nano)),++ reserved_nano: String(toNano(row.reserved_nano)),++ spent_nano: String(toNano(row.spent_nano)),++ max_runtime_minutes: row.max_runtime_minutes,++ label: row.label,++ cold_start_ms: row.cold_start_ms == null ? null : String(row.cold_start_ms),++ destroy_failures: Number(row.destroy_failures || 0),++ started_at: row.started_at,++ ready_at: row.ready_at,++ last_used_at: row.last_used_at,++ destroyed_at: row.destroyed_at,++ error: row.error,++ };++ }++++ // ── the reconciler ────────────────────────────────────────────────────++++ /** One pass: the owner's requests, the live instances, and the models that are wanted. */++ async function tick({ actor = 'service:ai' } = {}) {++ const cfg = config();++ if (!cfg.enabled) return { ok: false, reason: cfg.reason, launched: 0, stopped: 0, destroyed: 0 };++++ let launched = 0;++ let stopped = 0;++ let destroyed = 0;++++ // 0. the marketplace instances that carry our label: the ones no live row owns are given back++ await sweepOrphans(cfg);++++ // 1. the owner's requests++ const requests = await db.prepare('SELECT * FROM gpu_requests WHERE done_at IS NULL ORDER BY id').all();++ for (const req of requests || []) {++ try {++ if (req.action === 'launch') {++ const res = await launch(req.model_key, { actor: req.actor || actor });++ if (!res.ok) {++ await db.prepare('UPDATE gpu_requests SET done_at = ?, error = ? WHERE id = ?')++ .run(iso(now()), String(res.reason).slice(0, 300), req.id);++ continue;++ }++ launched += 1;++ } else {++ const row = await liveInstance(req.model_key);++ if (row) {++ await destroy(row, 'the owner stopped it', { actor: req.actor || actor });++ stopped += 1;++ }++ }++ await db.prepare('UPDATE gpu_requests SET done_at = ?, error = NULL WHERE id = ?').run(iso(now()), req.id);++ } catch (err) {++ await db.prepare('UPDATE gpu_requests SET done_at = ?, error = ? WHERE id = ?')++ .run(iso(now()), String(err && err.message).slice(0, 300), req.id);++ }++ }++++ // 2. the live instances++ const rows = await db.prepare('SELECT * FROM gpu_instances WHERE status IN (?, ?) ORDER BY created_at').all('starting', 'running');++ for (const row of rows || []) {++ const res = await reconcileInstance(row, cfg);++ if (res && (res.action === 'destroyed' || res.action === 'scaled_to_zero' || res.action === 'died')) destroyed += 1;++ }++++ // 3. the models that are wanted but have no instance++ for (const spec of models) {++ if (await liveInstance(spec.key)) continue;++ if (!await wanted(spec, cfg)) continue;++ const res = await launch(spec.key, { actor });++ if (res.ok) launched += 1;++ }++++ return { ok: true, launched, stopped, destroyed };++ }++++ /** The state the console page and the admin API show. */++ async function state() {++ const cfg = config();++ const sum = await moneySummary();++ const rows = await instanceRows();++ const list = models.map((spec) => {++ const live = rows.find((r) => r.model_key === spec.key && LIVE.includes(r.status)) || null;++ // the last measured cold start (the rows are newest first)++ const measured = rows.find((r) => r.model_key === spec.key && r.cold_start_ms != null) || null;++ return {++ key: spec.key,++ display_name: spec.display_name,++ marketplace: spec.marketplace,++ runtime: spec.runtime.kind,++ requirements: spec.requirements,++ weights_url: catalog.weightsUrl(spec),++ command: catalog.commandOf(spec),++ cold_start_ms: measured ? String(measured.cold_start_ms) : null,++ instance: describe(live),++ };++ });++ return {++ enabled: cfg.enabled,++ reason: cfg.reason,++ month: sum.month,++ cap_nano: cfg.cap_nano == null ? null : String(cfg.cap_nano),++ spent_nano: String(sum.spent_nano),++ committed_nano: String(sum.committed_nano),++ reserved_nano: String(sum.reserved_nano),++ idle_minutes: cfg.idle_minutes,++ max_runtime_minutes: cfg.max_runtime_minutes,++ warm_window: cfg.warm_window,++ region: cfg.region,++ marketplaces: { vast: Boolean(cfg.vast_key), runpod: Boolean(cfg.runpod_key) },++ models: list,++ instances: rows.map(describe),++ };++ }++++ return {++ state,++ tick,++ launch,++ request,++ config,++ moneySummary,++ sweepOrphans,++ destroyFailures,++ orphanCount,++ /** For the tests: the same thing the reconciler does with one instance. */++ reconcileInstance,++ inWarmWindow,++ };++}++++module.exports = { createGpuProvisioner, HEALTH_INTERVAL_MS, HEALTH_FAILURES };nnew file mode 100644@@@ -0,0 +1,142 @@++'use strict';++/**++ * RunPod: the marketplace for bigger cards (the H100 that fits a large model).++ *++ * The calls are the ones the RunPod v2 API documents:++ * GET /v2/gpu-types the catalogue of cards and their prices++ * POST /v2/pods rent a pod++ * GET /v2/pods/{id} the pod++ * DELETE /v2/pods/{id} give the pod back++ *++ * The API key goes in the Authorization header and nowhere else.++ */++const { apiJson } = require('./http');++++const BASE_URL = 'https://api.runpod.io';++++/** A RunPod adapter: searchOffers, createInstance, showInstance, destroyInstance. */++function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BASE_URL } = {}) {++ const auth = { key };++++ /** The endpoint of a running pod: its proxy URL, or the documented proxy address. */++ function endpointOf(pod, port) {++ const ports = (pod && pod.ports) || {};++ const p = ports[port] || ports[String(port)];++ if (p) {++ if (p.proxyUrl) return String(p.proxyUrl);++ if (p.publicPort && pod.ip) return 'https://' + pod.ip + ':' + p.publicPort;++ }++ if (pod && pod.id && port) return 'https://' + pod.id + '-' + port + '.proxy.runpod.io';++ return '';++ }++++ /** The card's price: the secure price when there is one, else the community price. */++ function priceOf(gpu) {++ const price = (gpu && gpu.price) || {};++ const secure = Number(price.secure);++ const community = Number(price.community);++ if (Number.isFinite(secure) && secure > 0) return String(secure);++ if (Number.isFinite(community) && community > 0) return String(community);++ return '0';++ }++++ return {++ id: 'runpod',++ label: 'RunPod',++ baseUrl,++++ async searchOffers(req) {++ const out = await apiJson(fetchImpl, baseUrl + '/v2/gpu-types', { ...auth, label: 'RunPod' });++ const gpus = (out && out.gpus) || [];++ const wanted = (req.gpuNames || []).map((n) => String(n).toLowerCase());++ const offers = [];++ for (const gpu of gpus) {++ const name = String(gpu.name || '');++ const lower = name.toLowerCase();++ if (wanted.length && !wanted.some((w) => lower.includes(w.toLowerCase()))) continue;++ if (Number(gpu.memory) < Number(req.vramMb)) continue;++ if (Number(gpu.maxCount || 0) < Number(req.numGpus || 1)) continue;++ const price = priceOf(gpu);++ if (Number(price) > Number(req.maxPricePerHour)) continue;++ for (const dc of (gpu.dataCenters || [])) {++ if (dc.availability !== undefined && !dc.availability) continue;++ offers.push({++ id: String(gpu.id) + ':' + String(dc.id),++ marketplace: 'runpod',++ gpu_type_id: String(gpu.id),++ data_center_id: String(dc.id),++ gpu_name: name,++ gpu_ram_mb: Number(gpu.memory) || 0,++ num_gpus: Number(req.numGpus || 1),++ disk_gb: Number(req.diskGb || 0),++ price_per_hour: price,++ reliability: 1,++ region: String(dc.name || dc.id || ''),++ });++ }++ }++ return offers;++ },++++ /**++ * The pods of this account that carry our name. Runpod lists them at GET /v2/pods; the sweep++ * reads them to find the ones no live row owns.++ */++ async listInstances() {++ const body = await apiJson(fetchImpl, baseUrl + '/v2/pods', { ...auth, label: 'RunPod' });++ const list = (body && (body.pods || body.items || body.data)) || [];++ return list.map((pod) => ({++ id: String(pod.id == null ? '' : pod.id),++ label: String(pod.name || ''),++ state: String(pod.status || 'unknown').toLowerCase(),++ price_per_hour: pod.costPerHour == null ? '' : String(pod.costPerHour),++ }));++ },++++ async createInstance(offer, spec, opts = {}) {++ const r = spec.runtime;++ const body = {++ // the name carries our row id: the sweep finds the pod again and gives it back++ // when no live row of ours owns it.++ name: String(opts.label || ('openvibe-' + spec.key)),++ image: r.image,++ gpu: { id: offer.gpu_type_id, count: spec.requirements.num_gpus },++ disk: { size: spec.requirements.disk_gb, type: 'volume' },++ ports: [String(r.port || 8000) + '/http'],++ env: [++ 'MODEL_FILE=' + ((r.weights && r.weights.file) || ''),++ 'MODEL_URL=' + (opts.weights_url || ''),++ 'MODEL_ALIAS=' + spec.model_key,++ ],++ start: true,++ };++ if (offer.data_center_id) body.dataCenterId = offer.data_center_id;++ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods', { method: 'POST', body, ...auth, label: 'RunPod' });++ const remoteId = String((out && out.id) || '');++ if (!remoteId) {++ const err = new Error('RunPod answered without a pod id');++ err.status = 502;++ throw err;++ }++ return { remote_id: remoteId, endpoint: '', state: String((out && out.status) || 'PROVISIONING').toLowerCase() };++ },++++ async showInstance(remoteId) {++ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { ...auth, label: 'RunPod' });++ const status = String((out && out.status) || 'unknown').toLowerCase();++ return {++ remote_id: remoteId,++ state: status,++ endpoint: endpointOf(out, 8000),++ raw: out,++ };++ },++++ async destroyInstance(remoteId) {++ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'RunPod' });++ return Boolean(out);++ },++ };++}++++module.exports = { runpodAdapter, BASE_URL };nnew file mode 100644@@@ -0,0 +1,151 @@++'use strict';++/**++ * Vast.ai: the marketplace for consumer GPUs (the RTX 3090/4090 that fit a 32B Q4 distill).++ *++ * The calls are the ones the Vast API documents:++ * POST /api/v0/bundles search the offers++ * POST /api/v0/asks/{id}/ rent an offer (the answer's new_contract is the instance id)++ * GET /api/v0/instances/{id} the instance (its actual_status and its ports)++ * DELETE /api/v0/instances/{id} give the instance back++ *++ * The API key goes in the Authorization header and nowhere else.++ */++const { apiJson } = require('./http');++++const BASE_URL = 'https://console.vast.ai';++++/** A Vast adapter: searchOffers, createInstance, showInstance, destroyInstance. */++function vastAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BASE_URL } = {}) {++ const auth = { key };++++ /** The endpoint of a running instance: its proxy URL, or the direct address. */++ function endpointOf(instance, port) {++ const ports = (instance && instance.ports) || {};++ const p = ports[port] || ports[String(port)];++ if (p && p.proxy) return String(p.proxy);++ const host = instance.ssh_node || instance.ssh_host || instance.public_ip || instance.ip;++ const pr = p && p.port ? p.port : port;++ if (host && pr) return 'http://' + host + ':' + pr;++ return '';++ }++++ return {++ id: 'vast',++ label: 'Vast.ai',++ baseUrl,++++ async searchOffers(req) {++ const body = {++ type: 'ondemand',++ verified: { eq: true },++ rentable: { eq: true },++ gpu_name: { in: req.gpuNames },++ reliability: { gte: req.minReliability },++ num_gpus: { gte: req.numGpus },++ gpu_ram: { gte: req.vramMb },++ disk_space: { gte: req.diskGb },++ dph_total: { lte: req.maxPricePerHour },++ geolocation: { in: [req.region] },++ limit: req.limit || 50,++ };++ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/bundles', { method: 'POST', body, ...auth, label: 'Vast.ai' });++ const offers = (out && out.offers) || [];++ return offers.map((o) => ({++ id: String(o.id),++ marketplace: 'vast',++ gpu_name: String(o.gpu_name || ''),++ gpu_ram_mb: Number(o.gpu_ram) || 0,++ num_gpus: Number(o.num_gpus) || 1,++ disk_gb: Number(o.disk_space) || 0,++ price_per_hour: String(o.dph_total == null ? '0' : o.dph_total),++ reliability: Number(o.reliability) || 0,++ region: String(o.geolocation || ''),++ }));++ },++++ /**++ * The instances of this account that carry our label. Vast lists them at GET /api/v1/instances++ * (25 per page, keyset pagination); the sweep reads them to find the ones no live row owns.++ */++ async listInstances() {++ const out = [];++ let after = null;++ for (let page = 0; page < 20; page++) {++ const url = baseUrl + '/api/v1/instances?limit=25&select_cols=id,label,actual_status,dph_total'++ + (after ? '&after_token=' + encodeURIComponent(after) : '');++ const body = await apiJson(fetchImpl, url, { ...auth, label: 'Vast.ai' });++ const list = (body && (body.instances || body.results || body.data)) || [];++ for (const inst of list) {++ out.push({++ id: String(inst.id == null ? '' : inst.id),++ label: String(inst.label || ''),++ state: String(inst.actual_status || inst.status || 'unknown').toLowerCase(),++ price_per_hour: inst.dph_total == null ? '' : String(inst.dph_total),++ });++ }++ after = body && body.next_token;++ if (!after) break;++ }++ return out;++ },++++ async createInstance(offer, spec, opts = {}) {++ const r = spec.runtime;++ const body = {++ image: r.image,++ args: opts.command || '',++ env: {++ MODEL_FILE: (r.weights && r.weights.file) || '',++ MODEL_URL: opts.weights_url || '',++ MODEL_ALIAS: spec.model_key,++ },++ disk: spec.requirements.disk_gb,++ runtype: 'args',++ // the label carries our row id: the sweep finds the instance again and gives it back++ // when no live row of ours owns it.++ label: String(opts.label || ('openvibe-' + spec.key)),++ };++ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/asks/' + offer.id + '/', { method: 'POST', body, ...auth, label: 'Vast.ai' });++ if (!out || out.success === false) {++ const err = new Error('Vast.ai did not rent the offer');++ err.status = 502;++ throw err;++ }++ const remoteId = String(out.new_contract || out.id || '');++ if (!remoteId) {++ const err = new Error('Vast.ai answered without an instance id');++ err.status = 502;++ throw err;++ }++ return { remote_id: remoteId, endpoint: '', state: 'starting' };++ },++++ async showInstance(remoteId) {++ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { ...auth, label: 'Vast.ai' });++ const status = String((out && (out.actual_status || out.status)) || 'unknown');++ return {++ remote_id: remoteId,++ state: status,++ endpoint: endpointOf(out, (opts_port(out))),++ raw: out,++ };++ },++++ async destroyInstance(remoteId) {++ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'Vast.ai' });++ return Boolean(out && out.success !== false);++ },++ };++}++++// The port a Vast instance serves on is in its ports map; the first one that is not SSH is the app.++function opts_port(instance) {++ const ports = (instance && instance.ports) || {};++ for (const key of Object.keys(ports)) {++ const n = Number(key);++ if (Number.isFinite(n) && n >= 8000) return n;++ }++ return 8000;++}++++module.exports = { vastAdapter, BASE_URL };@@@ -53,6 +53,10 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now// D11 owner settings: ai.exchange and ai.provider_keys on openvibe-shared/config. The provider-key store also// refreshes the synchronous vault cache, and its one-time migration runs before any provider is built.const settings = await require('./settings').createSettings({ db, config, registry, clock, env, log, onExchange: () => ensureCredits() });++ // The GPU provisioner (the design's "OpenVibe GPUs"): it is off until the owner has set a marketplace++ // API key and a monthly cap in the owner settings (ai.gpus). The reconciler rents, health-checks and++ // scales the instances to zero; it never spends past the cap.++ const gpus = require('./gpu/provisioner').createGpuProvisioner({ db, registry, settings, clock, fetchImpl, env, log });const vault = require('./vault').createVault({ settings, clock, log });await require('./vault').migrate({ db, config, settings, log }).catch((err) => log.warn(`[Vault] migration failed: ${err.message}`));await vault.load().catch((err) => log.warn(`[Vault] ${err.message}`));@@@ -146,7 +150,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowclientSecret: env.OV_OAUTH_CLIENT_SECRET, fetch: fetchImpl,}) : null;const benchEvents = createBenchEvents({ bench, accountData, accountSend, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });-- const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, memoryImport, community, presets, receipts, balance });++ const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, memoryImport, community, presets, receipts, balance, gpus });let jwks = null;let keyLoaded = Promise.resolve(null);if (!keys.pinned && config.nodeEnv !== 'test') {@@@ -167,6 +171,16 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now}, 60 * 60 * 1000);housekeeping.unref?.();++ // The GPU reconciler: once a minute it fulfils the owner's requests, health-checks the running++ // instances, scales the idle ones to zero and starts the models that are wanted. Never in test mode.++ let gpuReconciler = null;++ if (listen && config.nodeEnv !== 'test') {++ gpuReconciler = setInterval(() => {++ gpus.tick().catch((err) => log.warn('[gpu] the reconciler failed: ' + err.message));++ }, 60 * 1000);++ gpuReconciler.unref?.();++ }++let server = null;let benchSubscription = null;let accountSubscription = null;@@@ -202,6 +216,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowasync function close() {clearInterval(housekeeping);++ if (gpuReconciler) clearInterval(gpuReconciler);balance.stop();if (benchSubscription) benchSubscription.stop();if (accountSubscription) accountSubscription.stop();@@@ -222,7 +237,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.nowif (!givenDb) await db.close();}-- return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, bench, benchEvents, memory, memoryJobs, credits: creditsRef, settings, vault, community, close };++ return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, bench, benchEvents, memory, memoryJobs, credits: creditsRef, settings, vault, community, gpus, close };}if (require.main === module) {@@@ -88,7 +88,7 @@ function createAiReadiness({ db, registry, pool, keys, runs, release = null }) {}/** AI gauges on the openvibe-shared/metrics registry. */--function registerAiGauges(registry, { runs, registry: ai, pool, balance = null }) {++function registerAiGauges(registry, { runs, registry: ai, pool, balance = null, gpus = null }) {registry.gauge({name: 'ai_runs', help: 'Runs waiting for a slot (queued) and holding one (running)', labelNames: ['state'],collect: () => {@@@ -111,6 +111,17 @@ function registerAiGauges(registry, { runs, registry: ai, pool, balance = null }name: 'ai_provider_balance_usd', help: 'Provider account balance in USD, as last read by the hourly probe', labelNames: ['provider'],collect: () => (balance ? balance.values() : []).map(r => ({ labels: { provider: r.provider }, value: r.usd })),});++ // The GPU provisioner (server/gpu/): a marketplace instance that could not be given back is still running++ // and still being paid for. The operator alert fires at three tries in a row (OpenVibe.Host's prometheus++ // rules), so the gauge counts the tries that are still open.++ registry.gauge({++ name: 'ai_gpu_destroy_failures', help: 'Marketplace GPU instances OpenVibe.AI could not give back, in a row (the operator alert fires at 3)', labelNames: ['marketplace'],++ collect: async () => (gpus ? await gpus.destroyFailures() : []),++ });++ registry.gauge({++ name: 'ai_gpu_orphan_instances', help: 'Marketplace GPU instances with our label that no live row owns and that are still waiting to be given back', labelNames: ['marketplace'],++ collect: async () => (gpus ? await gpus.orphanCount() : []),++ });}module.exports = { createAiReadiness, registerAiGauges };@@@ -20,10 +20,15 @@ const { AiError } = require('./util');const EXCHANGE_NAMESPACE = 'ai.exchange';const PROVIDER_KEYS_NAMESPACE = 'ai.provider_keys';++const GPU_NAMESPACE = 'ai.gpus';const SERVICE = 'ai';const SELF = Object.freeze({ type: 'service', id: SERVICE });const GROUP = 'AI Exchange';const PROVIDER_GROUP = 'Provider keys';++const GPU_GROUP = 'OpenVibe GPUs';++// The marketplaces the GPU provisioner rents from. AI owns these origins and writes them into the++// namespace, so the owner's key is sealed to the marketplace it was entered for.++const GPU_ORIGINS = Object.freeze({ vast: 'https://console.vast.ai', runpod: 'https://api.runpod.io' });/** Bounds from server/config.js (load()), so the schema and the code agree. */const EXCHANGE_SCHEMA = Object.freeze({@@@ -108,6 +113,69 @@ function originOf(baseUrl) {} catch { return ''; }}++/**++ * The ai.gpus namespace: the owner's GPU settings.++ *++ * vast.origin / vast.key Vast.ai: the origin AI owns and the owner's write-only API key, sealed++ * to that origin (a key can never be sent to another host).++ * runpod.origin / runpod.key the same for RunPod.++ * monthly_cap the hard monthly cap in nano-USD. The provisioner never passes it: it++ * refuses to start an instance and says why. It is a money key, so it++ * needs a fresh sign-in to change.++ * idle_minutes how long an unused instance stays up before it scales to zero.++ * max_runtime_minutes how long one instance may run at most (its reservation).++ * warm_window an optional window (UTC, "19:00-23:00") in which idle instances stay up.++ * region the region the marketplace offers must be in.++ */++const GPU_SCHEMA = Object.freeze({++ $schema: 'https://json-schema.org/draft/2020-12/schema',++ title: 'OpenVibe GPUs',++ type: 'object',++ additionalProperties: false,++ properties: {++ 'vast.origin': {++ type: 'string', title: 'Vast.ai origin', 'x-group': GPU_GROUP,++ description: 'The origin of the Vast.ai API. AI writes it; the key below is sealed to it.',++ },++ 'vast.key': {++ type: 'string', title: 'Vast.ai API key', 'x-group': GPU_GROUP, 'x-bind': 'vast.origin',++ description: "The owner's write-only Vast.ai API key, sealed to the origin above.",++ },++ 'runpod.origin': {++ type: 'string', title: 'RunPod origin', 'x-group': GPU_GROUP,++ description: 'The origin of the RunPod API. AI writes it; the key below is sealed to it.',++ },++ 'runpod.key': {++ type: 'string', title: 'RunPod API key', 'x-group': GPU_GROUP, 'x-bind': 'runpod.origin',++ description: "The owner's write-only RunPod API key, sealed to the origin above.",++ },++ monthly_cap: {++ type: 'integer', title: 'Monthly GPU cap', 'x-group': GPU_GROUP, 'x-unit': 'nano_usd', 'x-danger': true,++ minimum: 0,++ description: 'The hard monthly cap for the rented GPUs, in nano-USD. The provisioner never spends past it: it refuses to start an instance and says why. Without a cap the provisioner stays off.',++ },++ idle_minutes: {++ type: 'integer', title: 'Idle minutes', 'x-group': GPU_GROUP,++ minimum: 1, maximum: 1440,++ description: 'How long an instance serves nobody before it scales to zero (the default is 10).',++ },++ max_runtime_minutes: {++ type: 'integer', title: 'Max runtime (minutes)', 'x-group': GPU_GROUP,++ minimum: 1, maximum: 10080,++ description: 'How long one instance may run at most. When the time is up the instance is given back, so the money never runs past the cap (the default is 720).',++ },++ warm_window: {++ type: 'string', title: 'Warm window (UTC)', 'x-group': GPU_GROUP,++ pattern: '^([0-2][0-9]):[0-5][0-9]-([0-2][0-9]):[0-5][0-9]$',++ description: 'An optional window in which idle instances stay up, as "19:00-23:00" (UTC). Empty means no warm window.',++ },++ region: {++ type: 'string', title: 'Region', 'x-group': GPU_GROUP,++ description: 'The region the marketplace offers must be in (the default is US).',++ },++ },++});++/** The namespace for the providers there are today: <p>.origin (internal) and <p>.key (secret, bound to it). */function providerKeysSchema(providers) {const properties = {};@@@ -251,6 +319,45 @@ function createProviderKeys({ db, registry, env = process.env, log = console, on};}++/**++ * The ai.gpus store: the owner's GPU settings. The marketplace origins are written once (AI owns them),++ * the keys and the cap are the owner's.++ */++async function createGpuSettings({ db, env = process.env, log = console, onChange }) {++ const store = await createConfigStore({++ db, service: SERVICE, namespace: GPU_NAMESPACE, schema: GPU_SCHEMA, classify, env, log, onChange,++ });++ const origins = {};++ for (const [name, origin] of Object.entries(GPU_ORIGINS)) origins[name + '.origin'] = origin;++ const changes = await store.changes(origins, { merge: true });++ if (changes.length) await store.apply(origins, { merge: true, actor: SELF, reason: 'the GPU marketplace origins' });++ if (store.sealed) await store.reseal().catch((err) => log.warn('[config] ' + GPU_NAMESPACE + ': reseal failed: ' + err.message));++++ /** The owner's GPU settings. The keys are the secrets; they never leave this process. */++ function values() {++ const v = store.get() || {};++ return {++ vast_origin: String(v['vast.origin'] || ''),++ vast_key: String(v['vast.key'] || ''),++ runpod_origin: String(v['runpod.origin'] || ''),++ runpod_key: String(v['runpod.key'] || ''),++ monthly_cap: v.monthly_cap == null ? null : String(v.monthly_cap),++ idle_minutes: v.idle_minutes == null ? null : Number(v.idle_minutes),++ max_runtime_minutes: v.max_runtime_minutes == null ? null : Number(v.max_runtime_minutes),++ warm_window: String(v.warm_window || ''),++ region: String(v.region || ''),++ };++ }++++ return {++ store,++ proxy: store,++ namespace: GPU_NAMESPACE,++ values,++ get sealed() { return Boolean(store.sealed); },++ };++}++/*** Create both namespaces (the DDL is migrations/0028_config.sql), apply ai.exchange to the live config object,* load the provider-key cache and reseal plaintext/old-KEK rows once when OV_CONFIG_KEKS is set.@@@ -270,12 +377,13 @@ async function createSettings({ db, config, registry, clock = { now: () => Date.applyExchange(config, exchange.get());const providerKeys = createProviderKeys({ db, registry, env, log, onChange });await providerKeys.refresh();++ const gpus = await createGpuSettings({ db, env, log, onChange });if (exchange.sealed) await exchange.reseal().catch((err) => log.warn(`[config] ${EXCHANGE_NAMESPACE}: reseal failed: ${err.message}`));if (providerKeys.sealed) await providerKeys.store.reseal().catch((err) => log.warn(`[config] ${PROVIDER_KEYS_NAMESPACE}: reseal failed: ${err.message}`));-- return { exchange, providerKeys, onChange };++ return { exchange, providerKeys, gpus, onChange };}module.exports = {-- EXCHANGE_NAMESPACE, PROVIDER_KEYS_NAMESPACE, SERVICE, SELF, EXCHANGE_SCHEMA,-- applyExchange, originOf, providerKeysSchema, createSettings,++ EXCHANGE_NAMESPACE, PROVIDER_KEYS_NAMESPACE, GPU_NAMESPACE, SERVICE, SELF, EXCHANGE_SCHEMA, GPU_SCHEMA, GPU_ORIGINS,++ applyExchange, originOf, providerKeysSchema, createGpuSettings, createSettings,};nnew file mode 100644@@@ -0,0 +1,70 @@++'use strict';++/**++ * The recorded marketplace: the answers Vast.ai gave when the offer search, the rental, the instance,++ * the list and the health check ran (test/fixtures/gpu/vast.json). The fake serves them, so no test++ * reaches the network and every run sees the same offers. The instance answers are the recorded ones++ * with the port the rental was given.++ *++ * api.dead the first endpoint stops answering (the instance died)++ * api.destroyFails every destroy is refused (the marketplace will not give the instance back)++ * api.extra the instances the account holds that carry our label (the orphans)++ */++const fs = require('fs');++const path = require('path');++++const RECORDED = JSON.parse(fs.readFileSync(path.join(__dirname, 'gpu', 'vast.json'), 'utf8'));++++/** A fake Vast.ai. */++function createGpuFetch() {++ const calls = { search: 0, create: [], show: 0, destroy: [], health: 0, list: 0 };++ const rented = new Map(); // the instances this account rented, id -> { id, label, state }++ let created = 0;++ const api = {++ calls,++ dead: false,++ destroyFails: false,++ extra: [],++ fetchImpl: async (url, opts = {}) => {++ const u = String(url);++ const method = String(opts.method || 'GET');++ const reply = (status, body) => ({ ok: status >= 200 && status < 300, status, text: async () => JSON.stringify(body) });++ if (u.includes('/api/v0/bundles')) {++ calls.search += 1;++ return reply(200, RECORDED.search);++ }++ if (u.includes('/api/v0/asks/')) {++ const body = JSON.parse(opts.body || '{}');++ calls.create.push({ url: u, body });++ created += 1;++ const id = 'inst-' + created;++ rented.set(id, { id, label: String(body.label || ''), state: 'running' });++ return reply(200, { success: true, new_contract: id });++ }++ if (u.includes('/api/v1/instances')) {++ calls.list += 1;++ return reply(200, { instances: [...rented.values(), ...api.extra] });++ }++ if (u.includes('/api/v0/instances/')) {++ if (method === 'DELETE') {++ calls.destroy.push(u);++ if (api.destroyFails) return reply(500, { detail: 'the marketplace refused to give the instance back' });++ rented.delete(u.split('/').pop());++ return reply(200, { success: true, msg: 'destroyed' });++ }++ calls.show += 1;++ const id = u.split('/').pop();++ const port = 12345 + Math.max(0, created - 1);++ return reply(200, { ...RECORDED.instance, id, ports: { 8000: { port, proxy: 'https://gw.vast.ai:' + port } } });++ }++ if (u.includes('/v1/models')) {++ calls.health += 1;++ if (api.dead && u.includes(':12345')) return reply(503, { error: 'the endpoint stopped answering' });++ return reply(200, { object: 'list', data: [{ id: 'deepseek-r1-32b' }] });++ }++ return reply(404, { detail: 'no recorded answer for ' + method + ' ' + u });++ },++ };++ return api;++}++++module.exports = { createGpuFetch, RECORDED };nnew file mode 100644@@@ -0,0 +1,52 @@++{++ "search": {++ "offers": [++ {++ "id": 222,++ "gpu_name": "RTX 3090",++ "gpu_ram": 24576,++ "num_gpus": 1,++ "disk_space": 120,++ "dph_total": "0.30",++ "reliability": 0.995,++ "geolocation": "US",++ "rentable": true,++ "verified": "verified"++ },++ {++ "id": 111,++ "gpu_name": "RTX 3090",++ "gpu_ram": 24576,++ "num_gpus": 1,++ "disk_space": 100,++ "dph_total": "0.1348",++ "reliability": 0.998,++ "geolocation": "US",++ "rentable": true,++ "verified": "verified"++ },++ {++ "id": 333,++ "gpu_name": "RTX 4090",++ "gpu_ram": 24576,++ "num_gpus": 1,++ "disk_space": 200,++ "dph_total": "0.20",++ "reliability": 0.999,++ "geolocation": "US",++ "rentable": true,++ "verified": "verified"++ }++ ]++ },++ "instance": {++ "actual_status": "running",++ "ssh_node": "ssh1.vast.ai",++ "ports": {++ "8000": {++ "port": 12345,++ "proxy": "https://gw.vast.ai:12345"++ }++ }++ }++}\\ No newline at end of filennew file mode 100644@@@ -0,0 +1,256 @@++'use strict';++/**++ * The GPU provisioner: the cheapest offer that fits the model, the monthly cap it never passes, the++ * idle instance that scales to zero and the instance that dies. The marketplace answers are recorded++ * (test/fixtures/gpu/vast.json) and the clock is a fake one, so no test reaches the network and no++ * test waits for real time.++ */++const assert = require('assert');++const { boot, suite } = require('./helpers');++const { createGpuFetch } = require('./fixtures/gpu-fetch');++++const T0 = Date.parse('2026-11-06T12:00:00Z');++const OWNER = { type: 'user', id: 'usr_' + 'A'.repeat(26) };++++/** The owner's GPU settings: the marketplace key (a secret) and the monthly cap (money). */++async function settings(h, { key = 'vast-test-key', cap = '50000000000', idle = null, maxRuntime = null } = {}) {++ const store = h.settings.gpus.store;++ const values = { monthly_cap: Number(cap) };++ if (idle != null) values.idle_minutes = idle;++ if (maxRuntime != null) values.max_runtime_minutes = maxRuntime;++ await store.apply(values, { merge: true, actor: OWNER, reason: 'the test' });++ if (key) await store.setSecret('vast.key', key, { actor: OWNER, reason: 'the test' });++}++++++/**++ * The row a crash between the marketplace's answer and the row's update leaves behind: the row is++ * there (it was written before the rental) but it does not know the marketplace id yet.++ */++async function crashedRow(h, id, at) {++ const iso = (ms) => new Date(ms).toISOString();++ await h.db.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,++ gpu_name, gpu_ram_mb, num_gpus, disk_gb, reliability, region, price_per_hour, hourly_nano, reserved_nano, spent_nano,++ max_runtime_minutes, started_at, charged_at, destroy_failures, label, cold_start_ms, health_failures, created_at, updated_at)++ VALUES (?, ?, ?, 'vast', '111', NULL, 'starting', NULL, ?, 'RTX 3090', 24576, 1, 100, 0.998, 'US', '0.1348', '134800000', '134800000', 0,++ 720, ?, ?, 0, ?, NULL, 0, ?, ?)`)++ .run(id, 'deepseek-r1-32b', 'gpu.deepseek-r1-32b', '2026-11', iso(at), iso(at), id, iso(at), iso(at));++}++++const t = suite('gpu');++++t.test('the provisioner is off without a key and without a cap', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ let res = await h.gpus.launch('deepseek-r1-32b');++ assert.strictEqual(res.ok, false);++ assert.strictEqual(res.code, 'gpu.off');++ assert.match(res.reason, /API key/);++ assert.strictEqual(api.calls.search, 0);++ // a key, but no cap: still off++ await settings(h, { cap: '0' });++ res = await h.gpus.launch('deepseek-r1-32b');++ assert.strictEqual(res.ok, false);++ assert.strictEqual(res.code, 'gpu.off');++ assert.strictEqual(api.calls.create.length, 0);++ const st = await h.gpus.state();++ assert.strictEqual(st.enabled, false);++ } finally { await h.stop(); }++});++++t.test('the cheapest offer that fits is rented, registered as an offer and health-checked', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '50000000000' }); // $50++ const res = await h.gpus.launch('deepseek-r1-32b');++ assert.ok(res.ok, res.reason);++ // offer 111 (0.1348/h) is the cheapest of the three that fit++ assert.strictEqual(api.calls.create.length, 1);++ assert.ok(api.calls.create[0].url.includes('/asks/111/'), api.calls.create[0].url);++ const st = await h.gpus.state();++ assert.strictEqual(st.instances.length, 1);++ assert.strictEqual(st.instances[0].offer_id, '111');++ assert.strictEqual(st.instances[0].hourly_nano, '134800000');++ assert.strictEqual(st.instances[0].status, 'starting');++ assert.ok(BigInt(st.committed_nano) <= BigInt(st.cap_nano), 'the reservation is inside the cap');++++ // the next pass finds the endpoint, checks it and registers the offer for the router++ now += 60_000;++ await h.gpus.tick();++ assert.ok(api.calls.health >= 1, 'the endpoint was health-checked');++ const provider = await h.registry.getProvider('gpu.deepseek-r1-32b');++ assert.ok(provider, 'the offer is registered');++ assert.strictEqual(provider.status, 'active');++ assert.strictEqual(provider.base_url, 'https://gw.vast.ai:12345/v1');++ assert.strictEqual(provider.provider_access, 'OWN_DEPLOYMENT');++ const model = await h.registry.getModel('gpu.deepseek-r1-32b', 'deepseek-r1-32b');++ assert.ok(model, 'the model is registered');++ assert.ok(model.cost.out_per_mtok > 0, 'the offer is priced');++ } finally { await h.stop(); }++});++++t.test('the monthly cap is never passed: the launch is refused and says why', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '1000000' }); // $0.001: less than an hour of a 0.1348/h GPU++ const res = await h.gpus.launch('deepseek-r1-32b');++ assert.strictEqual(res.ok, false);++ assert.strictEqual(res.code, 'gpu.cap');++ assert.match(res.reason, /cap/);++ assert.strictEqual(api.calls.create.length, 0, 'nothing was rented');++ const st = await h.gpus.state();++ assert.strictEqual(st.instances.length, 0);++ assert.strictEqual(st.spent_nano, '0');++ } finally { await h.stop(); }++});++++t.test('an idle instance scales to zero and the money is settled', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '50000000000', idle: 1 });++ await h.gpus.launch('deepseek-r1-32b');++ now += 60_000;++ await h.gpus.tick(); // the instance comes up++ assert.strictEqual((await h.gpus.state()).instances[0].status, 'running');++ now += 90_000; // it has served nobody for more than an idle minute++ await h.gpus.tick();++ assert.strictEqual(api.calls.destroy.length, 1, 'the instance was given back');++ const st = await h.gpus.state();++ assert.strictEqual(st.instances[0].status, 'destroyed');++ assert.ok(BigInt(st.spent_nano) > 0n, 'the money it cost is on the ledger');++ assert.ok(BigInt(st.committed_nano) <= BigInt(st.cap_nano), 'the money is inside the cap');++ const provider = await h.registry.getProvider('gpu.deepseek-r1-32b');++ assert.strictEqual(provider.status, 'disabled', 'the offer leaves the pool');++ } finally { await h.stop(); }++});++++t.test('an instance that dies is replaced and the router gets the new endpoint', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '50000000000' });++ await h.gpus.launch('deepseek-r1-32b');++ now += 60_000;++ await h.gpus.tick();++ assert.strictEqual((await h.gpus.state()).instances[0].status, 'running');++++ api.dead = true; // the endpoint stops answering++ now += 60_000;++ await h.gpus.tick(); // the first failed check++ now += 60_000;++ await h.gpus.tick(); // the second: the instance is dead++ assert.strictEqual(api.calls.destroy.length, 1, 'the dead instance was given back');++ assert.strictEqual((await h.registry.getProvider('gpu.deepseek-r1-32b')).status, 'disabled');++++ now += 60_000;++ await h.gpus.tick(); // a new instance is rented++ assert.strictEqual(api.calls.create.length, 2, 'a second instance was rented');++ now += 60_000;++ await h.gpus.tick(); // and comes up++ const provider = await h.registry.getProvider('gpu.deepseek-r1-32b');++ assert.strictEqual(provider.status, 'active');++ assert.strictEqual(provider.base_url, 'https://gw.vast.ai:12346/v1', 'the router got the new endpoint');++ const st = await h.gpus.state();++ assert.ok(BigInt(st.committed_nano) <= BigInt(st.cap_nano), 'the money is inside the cap');++ } finally { await h.stop(); }++});++++++t.test('an instance the marketplace accepted before the row knew its id is recovered, not thrown away', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '50000000000' });++ // the crash: the marketplace accepted the instance, the row never learned its id++ const id = 'openvibe-abc123def456';++ api.extra.push({ id: 'inst-crash', label: id, state: 'running' });++ await crashedRow(h, id, now);++ now += 60_000;++ await h.gpus.tick();++ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);++ assert.strictEqual(row.remote_id, 'inst-crash', 'the row knows its instance again');++ assert.strictEqual(row.status, 'running', 'and the instance serves');++ assert.strictEqual(api.calls.destroy.length, 0, 'a paid instance of ours is never thrown away');++ const audit = await h.registry.listAudit({ action: 'gpu.instance.recovered', targetId: id });++ assert.strictEqual(audit.length, 1, 'the recovery is on the audit log');++ } finally { await h.stop(); }++});++++t.test('an instance with our label and no row of ours is found and given back', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '50000000000' });++ const label = 'openvibe-111111111111';++ api.extra.push({ id: 'inst-orphan', label, state: 'running' });++ now += 60_000;++ await h.gpus.tick();++ assert.ok(api.calls.destroy.some((u) => u.includes('inst-orphan')), 'the orphan was given back');++ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(label);++ assert.strictEqual(row.status, 'destroyed');++ const audit = await h.registry.listAudit({ action: 'gpu.orphan.destroyed' });++ assert.strictEqual(audit.length, 1, 'the orphan is on the audit log');++ assert.deepStrictEqual(await h.gpus.orphanCount(), [], 'no orphan is left waiting');++ } finally { await h.stop(); }++});++++t.test('a destroy that keeps failing is tried again every pass and the operator is alerted', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '50000000000' });++ api.destroyFails = true;++ const label = 'openvibe-222222222222';++ api.extra.push({ id: 'inst-stuck', label, state: 'running' });++ for (let i = 0; i < 3; i++) {++ now += 60_000;++ await h.gpus.tick();++ }++ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(label);++ assert.strictEqual(Number(row.destroy_failures), 3, 'the tries are counted');++ assert.strictEqual(row.status, 'orphan', 'and the instance is still running');++ assert.strictEqual(api.calls.destroy.filter((u) => u.includes('inst-stuck')).length, 3, 'every pass tries again');++ const gauges = await h.gpus.destroyFailures();++ assert.strictEqual(gauges.length, 1);++ assert.strictEqual(gauges[0].labels.marketplace, 'vast');++ assert.strictEqual(gauges[0].value, 3, 'the gauge the operator alert reads');++ const audit = await h.registry.listAudit({ action: 'gpu.orphan.destroy_failed' });++ assert.strictEqual(audit.length, 1, 'the operator alert is on the audit log');++ } finally { await h.stop(); }++});++++t.test('a row whose rental never came back gives its reservation back', async () => {++ const api = createGpuFetch();++ let now = T0;++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });++ try {++ await settings(h, { cap: '50000000000' });++ // the crash before the marketplace answered: the row holds a reservation for an instance++ // that does not exist++ const id = 'openvibe-def456abc789';++ await crashedRow(h, id, now);++ const before = await h.gpus.moneySummary();++ assert.ok(BigInt(before.committed_nano) > 0n, 'the row holds its reservation');++ now += 11 * 60_000; // past the give-up time++ await h.gpus.tick();++ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);++ assert.strictEqual(row.status, 'refused');++ const after = await h.gpus.moneySummary();++ assert.strictEqual(BigInt(after.committed_nano), 0n, 'the reservation is free again');++ } finally { await h.stop(); }++});++++if (require.main === module) t.run();++module.exports = { suite: t };
new file mode 100644
new file mode 100644
@@ -0,0 +1,109 @@
@@ -0,0 +1,109 @@
+# OpenVibe GPUs (the GPU provisioner)
+
+The design (§2 "Our own GPUs", the Exchange remake) asks for a provisioner that rents a GPU from a
+marketplace, serves a model on it, registers the endpoint as an offer for the router and scales the
+whole thing to zero when nobody is using it - under a hard monthly cap the owner sets.
+
+This is that provisioner. It lives in AI (server/gpu/) and it is **off until the owner has set a
+marketplace API key and a monthly cap**.
+
+## The owner settings (the namespace `ai.gpus`)
+
+| Key | Class | What it is |
+| --- | --- | --- |
+| `vast.origin` / `vast.key` | internal / secret (bound to the origin) | the owner's Vast.ai API key, sealed to `https://console.vast.ai` |
+| `runpod.origin` / `runpod.key` | internal / secret (bound to the origin) | the owner's RunPod API key, sealed to `https://api.runpod.io` |
+| `monthly_cap` | money (`x-danger`, `nano_usd`) | the hard monthly cap. It needs a fresh sign-in to change |
+| `idle_minutes` | internal | how long an unused instance stays up (default 10) |
+| `max_runtime_minutes` | internal | how long one instance may run at most (default 720) |
+| `warm_window` | internal | an optional UTC window ("19:00-23:00") in which idle instances stay up |
+| `region` | internal | the region the offers must be in (default US) |
+
+The owner changes them the way every other AI setting changes: through Network's admin panel, which
+proxies to AI's `/internal/config/ai.gpus`. The keys are write-only (the panel never shows a value
+again), the origins are written by AI, and the cap is a money key.
+
+Without a key **or** without a cap the provisioner does nothing: no search, no instance, no money.
+
+## What it does
+
+1. **Cheapest fit.** It searches the marketplace for the offers that fit the model (VRAM, disk,
+ reliability, region) and takes the cheapest one. The order is deterministic: the same offers in
+ the same order always give the same instance.
+2. **Serve.** It rents the offer with the model's image and command (vLLM or llama.cpp with the
+ chosen weights) and waits for the endpoint to answer.
+3. **Register.** The endpoint becomes a normal provider (`kind: openai`, `provider_access:
+ OWN_DEPLOYMENT`) with the model on it, so the router, the Exchange catalogue, the health checks,
+ the breaker and the failover all work as they do for every other offer. Nothing new was written
+ for the router.
+4. **Health.** The endpoint is checked (`/v1/models`). After two failed checks the instance is dead:
+ the offer leaves the pool (the router fails over), the instance is given back and a new one is
+ rented.
+5. **Scale to zero.** An instance that served nobody for `idle_minutes` is given back (unless it is
+ inside the warm window). An instance that has run for `max_runtime_minutes` is given back too.
+6. **Cold start.** The time from the launch to the first answered request is measured and shown on the
+ console page (the design asks for the cold start to be said up front).
+
+## Orphans: an instance we paid for is never lost
+
+The row is written **before** the marketplace is called, and the instance is given the label of that
+row (`openvibe-<row id>`). Every pass:
+
+- the marketplace instances that carry our label are listed (Vast: `GET /api/v1/instances`, RunPod:
+ `GET /v2/pods`);
+- one whose label is a live row of ours is left alone - and **linked** when the row does not know its
+ marketplace id yet (a crash between the marketplace's answer and the row's update costs nothing: the
+ instance is found by its label and adopted);
+- one that no live row of ours owns is an **orphan**: it is given back and the fact is logged
+ (`gpu.orphan.destroyed` on the audit log).
+
+A destroy the marketplace refuses leaves the instance running and still being paid for. The try is
+counted on the row, the next pass tries again, and after **three** tries the operator is alerted: the
+audit row `gpu.orphan.destroy_failed`, the log, and the gauge `ai_gpu_destroy_failures` that the
+Host rule `OpenVibeAiGpuInstanceNotDestroyed` pages on
+([OpenVibe.Host #303](https://github.com/OpenVibers/OpenVibe.Host/pull/303)).
+
+## The money
+
+The money is integer nano-USD in BigInt, never a float (the same rule AI's Credits follow). Every
+instance holds a **reservation** - the budget it may cost - before it starts, and the reservation
+is part of the month's committed spend. A new instance is only started when its reservation still
+fits under the cap; if it does not, the launch is refused and the reason is returned. The instance is
+given back when its reservation has run out, so the money stops there. The worst case (the
+committed spend) therefore never passes the cap.
+
+Vast and RunPod bill **per second**, so the money is charged for the seconds an instance really ran
+(the ledger rounds up to the nano, so it never charges less than the GPU was paid for). The console page
+shows the price per hour and the price per second next to it.
+
+The ledger is `gpu_spend` (one row per settled instance); the instances, their reservations, their spent
+money, their destroy failures and their measured cold start are in `gpu_instances` (migration
+`0033_gpu.sql`).
+
+## The console page
+
+`GET /console/gpus` (the "GPUs" entry in the console) shows whether the provisioner is on and why not,
+the spend this month, the committed worst case and the cap, the models with their weights and their
+state, the measured cold start (from the launch to the first answered request), and every instance
+with its price (the hour and the second), endpoint, cold start and money. "Start" and "Stop" write a request row;
+the reconciler (one implementation, the same one the service runs) does the work.
+
+## The tests
+
+`test/gpu.test.js` runs the provisioner against the recorded marketplace answers
+(`test/fixtures/gpu/vast.json`) and a fake clock:
+
+- it is off without a key and without a cap,
+- the cheapest offer that fits is rented, registered as an offer and health-checked,
+- the monthly cap is never passed: the launch is refused and says why,
+- an idle instance scales to zero and the money is settled,
+- an instance that dies is replaced and the router gets the new endpoint,
+- an instance the marketplace accepted before the row knew its id is recovered, not thrown away,
+- an instance with our label and no row of ours is found and given back,
+- a destroy that keeps failing is tried again every pass and the operator is alerted.
+
+## What the owner has to check before the first launch
+
+The weights URL, the image and the container command are data in `server/gpu/catalog.js` - they are
+what the first launch really runs, and the console page shows them. The owner checks them (and
+corrects them there) before the first rented instance starts.
new file mode 100644
new file mode 100644
@@ -0,0 +1,70 @@
@@ -0,0 +1,70 @@
+-- phase: expand
+-- OpenVibe GPUs (the GPU provisioner): the marketplace instances AI rents, the money they cost and the
+-- owner's requests to start or stop them. Money is integer nano-USD in bigint (AI's D1 rule: never a
+-- float). Additive only: nothing existing is changed.
+CREATE TABLE IF NOT EXISTS gpu_instances (
+ id text COLLATE "C" PRIMARY KEY,
+ model_key text COLLATE "C" NOT NULL,
+ provider_key text COLLATE "C" NOT NULL,
+ marketplace text COLLATE "C" NOT NULL,
+ offer_id text COLLATE "C",
+ remote_id text COLLATE "C",
+ status text COLLATE "C" NOT NULL CHECK (status IN ('starting','running','stopping','destroyed','dead','refused','orphan')),
+ endpoint text COLLATE "C",
+ month text COLLATE "C" NOT NULL,
+ gpu_name text COLLATE "C",
+ gpu_ram_mb integer,
+ num_gpus integer,
+ disk_gb integer,
+ reliability double precision,
+ region text COLLATE "C",
+ price_per_hour text COLLATE "C", -- the marketplace's own decimal price, as it quoted it
+ hourly_nano bigint NOT NULL DEFAULT 0, -- that price as exact nano-USD per hour
+ reserved_nano bigint NOT NULL DEFAULT 0, -- the budget held for this instance (never above the cap)
+ spent_nano bigint NOT NULL DEFAULT 0, -- the money this instance really cost
+ max_runtime_minutes integer NOT NULL,
+ started_at timestamptz,
+ charged_at timestamptz, -- the moment the money was last put on the ledger
+ ready_at timestamptz,
+ last_used_at timestamptz,
+ last_health_at timestamptz,
+ health_failures integer NOT NULL DEFAULT 0,
+ destroy_failures integer NOT NULL DEFAULT 0, -- how many times in a row giving it back failed
+ label text COLLATE "C", -- the label we gave the marketplace instance
+ cold_start_ms bigint, -- from the launch to the first answered request
+ destroyed_at timestamptz,
+ error text COLLATE "C",
+ created_at timestamptz NOT NULL,
+ updated_at timestamptz NOT NULL
+);
+CREATE INDEX IF NOT EXISTS gpu_instances_model ON gpu_instances (model_key, status);
+CREATE INDEX IF NOT EXISTS gpu_instances_month ON gpu_instances (month);
+
+-- The money the GPU instances really cost. One row per settled instance (and per refund); the month's
+-- spend is their sum, so the ledger and the cap are the same numbers.
+CREATE TABLE IF NOT EXISTS gpu_spend (
+ id bigserial PRIMARY KEY,
+ at timestamptz NOT NULL,
+ month text COLLATE "C" NOT NULL,
+ kind text COLLATE "C" NOT NULL CHECK (kind IN ('charge','credit')),
+ nano_usd bigint NOT NULL,
+ instance_id text COLLATE "C",
+ detail text COLLATE "C",
+ actor text COLLATE "C",
+ created_at timestamptz NOT NULL
+);
+CREATE INDEX IF NOT EXISTS gpu_spend_month ON gpu_spend (month);
+
+-- What the owner asked for: start or stop a model's GPU. The reconciler fulfils it (the console and the
+-- admin API only write the request; one implementation acts on it).
+CREATE TABLE IF NOT EXISTS gpu_requests (
+ id bigserial PRIMARY KEY,
+ model_key text COLLATE "C" NOT NULL,
+ action text COLLATE "C" NOT NULL CHECK (action IN ('launch','stop')),
+ at timestamptz NOT NULL,
+ actor text COLLATE "C",
+ created_at timestamptz NOT NULL,
+ done_at timestamptz,
+ error text COLLATE "C"
+);
+CREATE INDEX IF NOT EXISTS gpu_requests_model ON gpu_requests (model_key, done_at);
@@ -24,7 +24,7 @@ const pkg = require('../package.json');
@@ -24,7 +24,7 @@ const pkg = require('../package.json');
const { adminRoutes } = require('openvibe-shared/config');
const { adminRoutes } = require('openvibe-shared/config');
-function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, memoryImport = null, community = null, presets = null, receipts = null, balance = null }) {
+function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env = process.env, clock = { now: () => Date.now() }, fetchImpl = globalThis.fetch, log = console, credentials = null, preferences = null, grants = null, vault = null, settings = null, benchEvents = null, runBudgets = null, memory = null, memoryImport = null, community = null, presets = null, receipts = null, balance = null, gpus = null }) {
const app = express();
const app = express();
// /.well-known/agent.json (openvibe-shared/agent-layer, SEO-2): what this site serves, for agents.
// /.well-known/agent.json (openvibe-shared/agent-layer, SEO-2): what this site serves, for agents.
require('openvibe-shared/agent-layer').mount(app, { descriptor: require('./agent-descriptor') });
require('openvibe-shared/agent-layer').mount(app, { descriptor: require('./agent-descriptor') });
@@ -36,7 +36,7 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
@@ -36,7 +36,7 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
// GET /metrics answers direct loopback callers only (Track O).
// GET /metrics answers direct loopback callers only (Track O).
const metrics = instrument(app, { service: 'ai', release: release.release });
const metrics = instrument(app, { service: 'ai', release: release.release });
app.use(require('openvibe-sdk/service').telemetryMiddleware); // request telemetry: a no-op until server/index.js calls telemetry.init()
app.use(require('openvibe-sdk/service').telemetryMiddleware); // request telemetry: a no-op until server/index.js calls telemetry.init()
- registerAiGauges(metrics.registry, { runs, registry, pool, balance });
+ registerAiGauges(metrics.registry, { runs, registry, pool, balance, gpus });
app.use(http.middleware());
app.use(http.middleware());
// Browser reads of the two public GETs (/stats, /release.json) only: an exact origin from
// Browser reads of the two public GETs (/stats, /release.json) only: an exact origin from
// AI_CORS_ORIGINS is echoed back, never credentials, and OPTIONS answers their preflight.
// AI_CORS_ORIGINS is echoed back, never credentials, and OPTIONS answers their preflight.
@@ -125,10 +125,10 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
@@ -125,10 +125,10 @@ function createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys
if (benchEvents) app.use(benchEvents.router);
if (benchEvents) app.use(benchEvents.router);
// D11 owner settings (openvibe-shared/config): Network's admin proxy, acting for the owner, reads and writes
// D11 owner settings (openvibe-shared/config): Network's admin proxy, acting for the owner, reads and writes
- // ai.exchange and ai.provider_keys. Loopback only (the /internal guard above), guarded by ai.config.manage; the
+ // ai.exchange, ai.provider_keys and ai.gpus (the GPU provisioner's marketplace keys and its monthly cap). Loopback only (the /internal guard above), guarded by ai.config.manage; the
// actor is Network's X-OV-Subject (usr_…), and a secret or x-danger change needs X-OV-Fresh: 1.
// actor is Network's X-OV-Subject (usr_…), and a secret or x-danger change needs X-OV-Fresh: 1.
if (settings) {
if (settings) {
- const configRoutes = adminRoutes([settings.exchange, settings.providerKeys.proxy], {
+ const configRoutes = adminRoutes([settings.exchange, settings.providerKeys.proxy, settings.gpus.proxy], {
basePath: '/internal/config',
basePath: '/internal/config',
requireAdmin: auth.requireCap(CAPS.configManage),
requireAdmin: auth.requireCap(CAPS.configManage),
actor: (req) => {
actor: (req) => {
@@ -115,6 +115,9 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
@@ -115,6 +115,9 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
const creditsOn = Boolean(config.credits && config.credits.enabled);
const creditsOn = Boolean(config.credits && config.credits.enabled);
const routingOn = Boolean(config.routingEvidence && config.routingEvidence.enabled);
const routingOn = Boolean(config.routingEvidence && config.routingEvidence.enabled);
const exchangeOn = creditsOn || routingOn;
const exchangeOn = creditsOn || routingOn;
+ // The GPU provisioner's state (the same module the service runs): the page shows the instances, the
+ // spend this month and the cap. The actions only write a request; the reconciler fulfils it.
+ const gpus = require('../gpu/provisioner').createGpuProvisioner({ db, registry, settings, clock, fetchImpl, log });
const r = express.Router();
const r = express.Router();
const scoped = ['/console', '/auth'];
const scoped = ['/console', '/auth'];
const baseOrigin = (() => { try { return new URL(config.baseUrl).origin; } catch { return null; } })();
const baseOrigin = (() => { try { return new URL(config.baseUrl).origin; } catch { return null; } })();
@@ -540,6 +543,19 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
@@ -540,6 +543,19 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
}));
}));
}));
}));
+ // ── OpenVibe GPUs: the rented instances, the spend this month and the cap ──
+ // The page is read-only; the two actions ask for a change (a request row) and the reconciler does it.
+ r.get('/console/gpus', read, wrap(async (req, res) => {
+ await send(res, 200, pages.gpus({
+ ...common(req), canManage: req.staff.caps.includes(CAPS.providerManage), notice: notice(req),
+ state: await gpus.state(),
+ }));
+ }));
+ r.post('/console/gpus/:model/:action', needs(CAPS.providerManage), wrap(async (req, res) => {
+ await gpus.request(String(req.params.model || ''), String(req.params.action || ''), req.staff.subject);
+ res.redirect(303, '/console/gpus?done=gpu_requested');
+ }));
+
// ── AI Exchange (P7): Credits settle queue and routing evidence ──
// ── AI Exchange (P7): Credits settle queue and routing evidence ──
// Read-only except "mark resolved", which acknowledges one review row (a 409/422 needs a person) and
// Read-only except "mark resolved", which acknowledges one review row (a 409/422 needs a person) and
// deliberately does not retry it. Both sections are absent unless their flag is on.
// deliberately does not retry it. Both sections are absent unless their flag is on.
@@ -122,6 +122,7 @@ const NAV = [
@@ -122,6 +122,7 @@ const NAV = [
['/console', 'Overview', 'overview'], ['/console/providers', 'Providers', 'providers'], ['/console/routes', 'Routes', 'routes'],
['/console', 'Overview', 'overview'], ['/console/providers', 'Providers', 'providers'], ['/console/routes', 'Routes', 'routes'],
['/console/templates', 'Templates', 'templates'], ['/console/workflows', 'Workflows', 'workflows'], ['/console/runs', 'Runs', 'runs'],
['/console/templates', 'Templates', 'templates'], ['/console/workflows', 'Workflows', 'workflows'], ['/console/runs', 'Runs', 'runs'],
['/console/quotas', 'Quotas', 'quotas'], ['/console/usage', 'Usage', 'usage'], ['/console/cache', 'Cache', 'cache'], ['/console/audit', 'Audit', 'audit'],
['/console/quotas', 'Quotas', 'quotas'], ['/console/usage', 'Usage', 'usage'], ['/console/cache', 'Cache', 'cache'], ['/console/audit', 'Audit', 'audit'],
+ ['/console/gpus', 'GPUs', 'gpus'],
];
];
// The console's pages are noindex; the one public page drawn with this layout (/stats) passes robots, a description and
// The console's pages are noindex; the one public page drawn with this layout (/stats) passes robots, a description and
@@ -558,6 +559,74 @@ ${routing ? routingSection({ routing }) : ''}`,
@@ -558,6 +559,74 @@ ${routing ? routingSection({ routing }) : ''}`,
});
});
}
}
+/** The GPU money as a dollar string: nano-USD in, exact dollars out (never a float). */
+function usdNano(nano) {
+ const n = BigInt(nano == null ? 0 : nano);
+ const abs = n < 0n ? -n : n;
+ const whole = abs / 1000000000n;
+ const frac = (abs % 1000000000n).toString().padStart(9, '0').replace(/0+$/, '');
+ return (n < 0n ? '-' : '') + '$' + whole.toString() + (frac ? '.' + frac : '');
+}
+
+/** A measured cold start: from the launch to the first answered request. */
+function coldStart(ms) {
+ const n = Math.max(0, Math.round(Number(ms) || 0));
+ if (n < 1000) return n + ' ms';
+ return (n / 1000).toFixed(1) + ' s';
+}
+
+/** The price of one second of a GPU, from its hourly nano-USD rate (Vast and RunPod bill per second). */
+function usdPerSecond(hourlyNano) {
+ const n = BigInt(hourlyNano == null ? 0 : hourlyNano);
+ return usdNano(n / 3600n);
+}
+
+function gpuModelRow(m, canManage, csrf) {
+ return html`<tr>
+ <td><code>${m.key}</code><br><span class="muted">${m.display_name}</span><br><span class="muted">${clip(m.weights_url, 90)}</span></td>
+ <td>${m.marketplace}</td><td>${m.runtime}</td>
+ <td>${m.instance ? m.instance.gpu_name + ' · $' + m.instance.price_per_hour + '/h' : '—'}</td>
+ <td>${m.instance ? badge(m.instance.status) : html`<span class="muted">stopped</span>`}</td>
+ <td>${m.cold_start_ms == null ? html`<span class="muted">not measured yet</span>` : coldStart(m.cold_start_ms)}</td>
+ <td>${canManage ? html`<form class="inline" method="post" action="/console/gpus/${enc(m.key)}/${m.instance ? 'stop' : 'launch'}">${csrfField(csrf)}<button type="submit">${m.instance ? 'Stop' : 'Start'}</button></form>` : ''}</td>
+</tr>`;
+}
+
+function gpuInstanceRow(i) {
+ return html`<tr>
+ <td><code>${i.model_key}</code></td><td>${i.marketplace}</td><td>${i.gpu_name}</td><td>$${i.price_per_hour}/h · ${usdPerSecond(i.hourly_nano)}/s</td>
+ <td>${badge(i.status)}</td><td>${i.endpoint ? html`<span class="muted">${clip(i.endpoint, 60)}</span>` : '—'}</td>
+ <td>${when(i.started_at)}</td><td>${i.cold_start_ms == null ? '—' : coldStart(i.cold_start_ms)}</td><td>${usdNano(i.reserved_nano)}</td><td>${usdNano(i.spent_nano)}</td>
+</tr>`;
+}
+
+/**
+ * The GPU page: the instances AI rents, what they cost this month and the cap they may never pass.
+ * The actions only ask (a request row); the reconciler does the work, so a click is never a second
+ * implementation of the provisioner.
+ */
+function gpus({ staff, csrf, state, canManage, notice, error }) {
+ const st = state || {};
+ const models = st.models || [];
+ const instances = st.instances || [];
+ const mk = st.marketplaces || {};
+ return layout({
+ title: 'OpenVibe GPUs', section: 'gpus', staff, csrf, notice, error, body: html`<h1>OpenVibe GPUs</h1>
+<p class="muted">The rented GPUs: what is running, what it cost this month, and the cap it may never pass. The provisioner is ${st.enabled ? 'on' : 'off'}${st.enabled ? '' : ' (' + st.reason + ')'}: it starts nothing until a marketplace API key and a monthly cap are set in the owner settings.</p>
+<div class="grid">
+ <div class="card"><div class="k">Status</div><div class="v">${st.enabled ? 'on' : 'off'}</div><div class="muted">${st.reason || 'a key and a cap are set'}</div></div>
+ <div class="card"><div class="k">Spent this month</div><div class="v">${usdNano(st.spent_nano)}</div><div class="muted">${st.month}</div></div>
+ <div class="card"><div class="k">Committed</div><div class="v">${usdNano(st.committed_nano)}</div><div class="muted">the worst case: what the running instances may still cost</div></div>
+ <div class="card"><div class="k">Monthly cap</div><div class="v">${st.cap_nano ? usdNano(st.cap_nano) : '—'}</div><div class="muted">never passed</div></div>
+</div>
+<p class="muted">Idle ${n(st.idle_minutes)} minute(s) before an instance scales to zero · max runtime ${n(st.max_runtime_minutes)} minute(s) · warm window ${st.warm_window || 'none'} · region ${st.region} · Vast.ai key ${mk.vast ? 'set' : 'not set'} · RunPod key ${mk.runpod ? 'set' : 'not set'}</p>
+<p class="muted">Vast and RunPod bill per second, so the ledger charges the seconds an instance really ran: the price per second is next to the hourly price. The cold start is measured from the launch to the first answered request.</p>
+${table('Models', ['Model', 'Marketplace', 'Runtime', 'GPU', 'Status', 'Cold start', 'Actions'], models.map((m) => gpuModelRow(m, canManage, csrf)), 'no model is known.')}
+${table('Instances', ['Model', 'Marketplace', 'GPU', 'Price / hour', 'Status', 'Endpoint', 'Started', 'Cold start', 'Reserved', 'Spent'], instances.map((i) => gpuInstanceRow(i)), 'no instance has been rented.')}
+`,
+ });
+}
+
function cache({ staff, csrf, stats, canManage, notice, error }) {
function cache({ staff, csrf, stats, canManage, notice, error }) {
const entries = stats.reduce((a, s) => a + s.entries, 0);
const entries = stats.reduce((a, s) => a + s.entries, 0);
const hits = stats.reduce((a, s) => a + (s.hits || 0), 0);
const hits = stats.reduce((a, s) => a + (s.hits || 0), 0);
@@ -665,4 +734,4 @@ function allowance(cards) {
@@ -665,4 +734,4 @@ function allowance(cards) {
/** The price and latency cells, as plain HTML strings for the public /stats page (server/home.js renderStats). */
/** The price and latency cells, as plain HTML strings for the public /stats page (server/home.js renderStats). */
const statsFormat = { ms, rate, price: (cards) => render(price(cards)), allowance: (cards) => render(allowance(cards)) };
const statsFormat = { ms, rate, price: (cards) => render(price(cards)), allowance: (cards) => render(allowance(cards)) };
-module.exports = { CSP, html, esc, safeUrl, statsFormat, pages: { signIn, message, overview, providers, providerNew, definitions, definition, runs, run, runReceipt, quotas, usage, cache, audit, exchange } };
+module.exports = { CSP, html, esc, safeUrl, statsFormat, pages: { signIn, message, overview, providers, providerNew, definitions, definition, runs, run, runReceipt, quotas, usage, cache, audit, exchange, gpus } };
new file mode 100644
new file mode 100644
@@ -0,0 +1,97 @@
@@ -0,0 +1,97 @@
+'use strict';
+/**
+ * The models the GPU provisioner knows how to serve, and what each one needs from a GPU.
+ *
+ * The first two are the design's starting point: a DeepSeek R1 32B distill at Q4 on one RTX 3090 or
+ * 4090 - once with the refusal training removed (abliterated) and once plain. A 32B Q4_K_M quant is
+ * about 20 GB of weights, so it fits one 24 GB card.
+ *
+ * The weights URL, the image and the command are data, not code: they are what the owner's first
+ * launch really runs, so they are shown on the console's GPU page and in the audit row. The owner
+ * checks them before the first launch and corrects them here.
+ *
+ * vram_mb the VRAM the weights and the KV cache need (the marketplace's filter)
+ * disk_gb the disk the image and the weights need
+ * gpu_names the cards that fit the model
+ * throughput the tokens an hour the card is expected to serve (a planning estimate; it prices the
+ * offer for the router)
+ */
+const MODELS = Object.freeze([
+ {
+ key: 'deepseek-r1-32b-abliterated',
+ display_name: 'DeepSeek R1 32B distill (abliterated), Q4',
+ provider_key: 'gpu.deepseek-r1-32b-abliterated',
+ model_key: 'deepseek-r1-32b-abliterated',
+ marketplace: 'vast',
+ context_window: 8192,
+ throughput_tokens_per_hour: 180000,
+ requirements: { vram_mb: 22000, disk_gb: 60, gpu_names: ['RTX 3090', 'RTX 4090'], num_gpus: 1, min_reliability: 0.99 },
+ runtime: {
+ kind: 'llama.cpp',
+ image: 'ghcr.io/ggml-org/llama.cpp:server-cuda',
+ port: 8000,
+ weights: {
+ repo: 'mradermacher/DeepSeek-R1-Distill-Llama-32B-abliterated-GGUF',
+ file: 'DeepSeek-R1-Distill-Llama-32B-abliterated-Q4_K_M.gguf',
+ },
+ // The container's command: fetch the weights, then serve them with the OpenAI API.
+ command: 'mkdir -p /models && curl -fL --retry 3 -o /models/{file} {weights_url} '
+ + '&& exec /app/llama-server --model /models/{file} --host 0.0.0.0 --port {port} '
+ + '--alias {model} --ctx-size {ctx} --n-gpu-layers 999',
+ },
+ },
+ {
+ key: 'deepseek-r1-32b',
+ display_name: 'DeepSeek R1 32B distill, Q4',
+ provider_key: 'gpu.deepseek-r1-32b',
+ model_key: 'deepseek-r1-32b',
+ marketplace: 'vast',
+ context_window: 8192,
+ throughput_tokens_per_hour: 180000,
+ requirements: { vram_mb: 22000, disk_gb: 60, gpu_names: ['RTX 3090', 'RTX 4090'], num_gpus: 1, min_reliability: 0.99 },
+ runtime: {
+ kind: 'llama.cpp',
+ image: 'ghcr.io/ggml-org/llama.cpp:server-cuda',
+ port: 8000,
+ weights: {
+ repo: 'bartowski/DeepSeek-R1-Distill-Llama-32B-GGUF',
+ file: 'DeepSeek-R1-Distill-Llama-32B-Q4_K_M.gguf',
+ },
+ command: 'mkdir -p /models && curl -fL --retry 3 -o /models/{file} {weights_url} '
+ + '&& exec /app/llama-server --model /models/{file} --host 0.0.0.0 --port {port} '
+ + '--alias {model} --ctx-size {ctx} --n-gpu-layers 999',
+ },
+ },
+]);
+
+const byKey = new Map(MODELS.map((m) => [m.key, m]));
+
+/** The model a key names, or null. */
+function find(key) {
+ return byKey.get(String(key || '')) || null;
+}
+
+/** Every model the provisioner knows. */
+function list() {
+ return [...MODELS];
+}
+
+/** The URL the weights come from (a public Hugging Face file). */
+function weightsUrl(spec) {
+ const w = (spec && spec.runtime && spec.runtime.weights) || {};
+ if (!w.repo || !w.file) return '';
+ return 'https://huggingface.co/' + w.repo + '/resolve/main/' + w.file;
+}
+
+/** The container command of a model, with its placeholders filled in. */
+function commandOf(spec) {
+ const r = (spec && spec.runtime) || {};
+ return String(r.command || '')
+ .replace(/\{file\}/g, (r.weights && r.weights.file) || '')
+ .replace(/\{weights_url\}/g, weightsUrl(spec))
+ .replace(/\{port\}/g, String(r.port || 8000))
+ .replace(/\{model\}/g, (spec && spec.model_key) || '')
+ .replace(/\{ctx\}/g, String((spec && spec.context_window) || 8192));
+}
+
+module.exports = { MODELS, find, list, weightsUrl, commandOf };
new file mode 100644
new file mode 100644
@@ -0,0 +1,28 @@
@@ -0,0 +1,28 @@
+'use strict';
+/**
+ * One HTTP call to a GPU marketplace. The API key never leaves this process: it goes in the
+ * Authorization header, never in a URL, and a failure never carries the key back.
+ */
+const { AiError } = require('../util');
+
+async function apiJson(fetchImpl, url, { method = 'GET', body = null, key = null, label = 'the marketplace' } = {}) {
+ const headers = { Accept: 'application/json' };
+ if (body != null) headers['Content-Type'] = 'application/json';
+ if (key) headers.Authorization = 'Bearer ' + key;
+ let res;
+ try {
+ res = await fetchImpl(url, { method, headers, body: body != null ? JSON.stringify(body) : undefined });
+ } catch (err) {
+ throw new AiError(502, 'gpu.marketplace_unreachable', label + ' did not answer: ' + String(err && err.message).slice(0, 200));
+ }
+ const text = await res.text().catch(() => '');
+ let json = null;
+ try { json = text ? JSON.parse(text) : null; } catch { json = null; }
+ if (!res.ok) {
+ const detail = (json && (json.detail || json.message || json.error)) || String(res.status);
+ throw new AiError(502, 'gpu.marketplace_error', label + ' refused: ' + String(detail).slice(0, 200) + ' (' + res.status + ')');
+ }
+ return json;
+}
+
+module.exports = { apiJson };
new file mode 100644
new file mode 100644
@@ -0,0 +1,62 @@
@@ -0,0 +1,62 @@
+'use strict';
+/**
+ * The money of the GPU provisioner: integer nano-USD in BigInt, never a float (AI's D1 rule; the same
+ * rule server/credits.js follows for Credits). Nothing here multiplies or divides a float:
+ *
+ * nanoCount() an amount the owner set (the monthly cap): an integer count of nano-USD.
+ * nanoUsd() a marketplace's own decimal price ("0.1348" dollars an hour) -> exact nano-USD.
+ * nanoForSeconds() what an instance at a known hourly rate costs for a known number of seconds,
+ * rounded up to the nano: the ledger never charges less than the GPU was paid for.
+ * usdText() nano-USD as a dollar string for a page (exact; a page never does arithmetic on it).
+ */
+const { AiError } = require('../util');
+const { parseCostUsd } = require('../credits');
+
+const NANO_PER_USD = 1000000000n;
+const NANO_PER_MILLION = 1000000000000n;
+const SECONDS_PER_HOUR = 3600n;
+
+/** An integer count of nano-USD (the settings' money unit). null when absent; never a float. */
+function nanoCount(value, field = 'amount') {
+ if (value == null || value === '') return null;
+ if (typeof value === 'bigint') return value;
+ const raw = String(value).trim();
+ if (!/^[0-9]+$/.test(raw)) throw new AiError(422, 'ai.invalid', field + ' must be an integer number of nano-USD');
+ return BigInt(raw);
+}
+
+/** A plain decimal dollar amount (a marketplace's price) -> exact nano-USD. */
+function nanoUsd(value, field = 'price') {
+ if (value == null || value === '') return null;
+ return parseCostUsd(value, field);
+}
+
+/** What an instance at hourlyNano costs for seconds: rounded up to the nano, never less. */
+function nanoForSeconds(hourlyNano, seconds) {
+ const s = BigInt(Math.max(0, Math.ceil(Number(seconds) || 0)));
+ if (s === 0n) return 0n;
+ return (BigInt(hourlyNano) * s + SECONDS_PER_HOUR - 1n) / SECONDS_PER_HOUR;
+}
+
+/** nano-USD as a dollar string (exact; for a page only). */
+function usdText(nano) {
+ const n = BigInt(nano == null ? 0 : nano);
+ const abs = n < 0n ? -n : n;
+ const whole = abs / NANO_PER_USD;
+ const frac = (abs % NANO_PER_USD).toString().padStart(9, '0').replace(/0+$/, '');
+ return (n < 0n ? '-' : '') + '$' + whole.toString() + (frac ? '.' + frac : '');
+}
+
+/**
+ * The price of one million tokens on a GPU that costs hourlyNano an hour and serves tokensPerHour
+ * tokens: the hourly rate spread over the tokens it can produce. It is an estimate for the router's
+ * price comparison (the authoritative money is the GPU ledger), computed from the exact nano rate so
+ * the two never drift apart.
+ */
+function usdPerMillionTokens(hourlyNano, tokensPerHour) {
+ const t = BigInt(Math.max(1, Math.trunc(Number(tokensPerHour) || 0)));
+ const nano = (BigInt(hourlyNano) * NANO_PER_MILLION + t - 1n) / t;
+ return Number(nano) / 1e9;
+}
+
+module.exports = { NANO_PER_USD, NANO_PER_MILLION, nanoCount, nanoUsd, nanoForSeconds, usdText, usdPerMillionTokens };
new file mode 100644
new file mode 100644
@@ -0,0 +1,797 @@
@@ -0,0 +1,797 @@
+'use strict';
+/**
+ * The GPU provisioner (the design's "OpenVibe GPUs").
+ *
+ * It rents the cheapest marketplace GPU that fits a model, serves the model's weights with vLLM or
+ * llama.cpp, registers the endpoint as an offer for the router, health-checks it, scales it to zero
+ * when it is idle, and never spends past the owner's monthly cap.
+ *
+ * The rules it keeps, in the order they matter:
+ *
+ * 1. It is off until the owner has set a marketplace API key AND a monthly cap. No key or no cap
+ * means no instance, no search and no money.
+ * 2. It never passes the monthly cap. Every instance holds a reservation (the budget it may cost)
+ * before it starts, the reservation is part of the month's committed spend, and a new instance
+ * is only started when the reservation still fits under the cap. When an instance's reservation
+ * has run out the instance is given back, so the money stops there.
+ * 3. The money is integer nano-USD in BigInt (server/gpu/money.js). Nothing here is a float.
+ * 4. It picks the cheapest offer that fits (VRAM, disk, reliability, region) and the choice is
+ * deterministic: the same offers in the same order always give the same instance.
+ */
+const { randomUUID } = require('node:crypto');
+const { AiError } = require('../util');
+const money = require('./money');
+const catalog = require('./catalog');
+const { vastAdapter } = require('./vast');
+const { runpodAdapter } = require('./runpod');
+
+const HEALTH_INTERVAL_MS = 30_000;
+const HEALTH_FAILURES = 2;
+const DEFAULT_IDLE_MINUTES = 10;
+const DEFAULT_MAX_RUNTIME_MINUTES = 720;
+const DEFAULT_REGION = 'US';
+const LIVE = ['starting', 'running', 'stopping'];
+const DEAD_STATES = ['exited', 'unknown', 'offline', 'dead', 'terminated', 'stopped', 'error'];
+// Every instance we rent carries our row id as its label (an openvibe- prefix), so the sweep can find
+// it again and give it back when no live row of ours owns it.
+const LABEL_PREFIX = 'openvibe-';
+const LABEL_RE = /^openvibe-[0-9a-f]{12}$/;
+// Three failed destroys in a row: the instance is still running and still being paid for, so the
+// operator is alerted (the gauge ai_gpu_destroy_failures and the rule in OpenVibe.Host's prometheus rules).
+const ALERT_AFTER = 3;
+// A row whose rental never came back is given up on after this long: its reservation is free again.
+const STALE_ROW_MS = 10 * 60_000;
+const PRICE_SOURCE = { vast: 'https://vast.ai/pricing', runpod: 'https://www.runpod.io/pricing' };
+
+function clampInt(value, min, max, fallback) {
+ if (value == null || value === '') return fallback;
+ const n = Number(value);
+ if (!Number.isFinite(n)) return fallback;
+ return Math.max(min, Math.min(max, Math.trunc(n)));
+}
+
+/** A database value as milliseconds, or null. */
+function toMs(value) {
+ if (value == null) return null;
+ if (value instanceof Date) return value.getTime();
+ const t = Date.parse(value);
+ return Number.isFinite(t) ? t : null;
+}
+
+/** A database value as exact nano-USD. A value that is not an integer is loud, never rounded. */
+function toNano(value) {
+ if (value == null) return 0n;
+ if (typeof value === 'bigint') return value;
+ const s = String(value).trim();
+ if (/^-?[0-9]+$/.test(s)) return BigInt(s);
+ throw new AiError(500, 'gpu.ledger_corrupt', 'the GPU ledger holds a value that is not an integer number of nano-USD');
+}
+
+/**
+ * The provisioner.
+ *
+ * db the database handle (open vibe sdk db)
+ * registry the provider registry: the instance is registered there as an offer for the router
+ * settings AI's settings; settings.gpus.values() is the owner's GPU settings
+ * clock { now(): number } - a fake clock in the tests
+ * fetchImpl the HTTP client the marketplace calls go through (a fake in the tests)
+ */
+function createGpuProvisioner({
+ db,
+ registry,
+ settings,
+ clock = { now: () => Date.now() },
+ fetchImpl = globalThis.fetch,
+ env = process.env,
+ log = console,
+ models = catalog.list(),
+} = {}) {
+ const now = () => clock.now();
+ const iso = (ms) => new Date(ms).toISOString();
+ const monthOf = (ms) => new Date(ms).toISOString().slice(0, 7);
+
+ // ── the owner's settings ──────────────────────────────────────────────
+
+ /** The owner's GPU settings, with the money exact and the defaults applied. */
+ function config() {
+ const v = (settings && settings.gpus && typeof settings.gpus.values === 'function') ? settings.gpus.values() : {};
+ let cap = null;
+ try {
+ cap = money.nanoCount(v.monthly_cap, 'the monthly GPU cap');
+ } catch {
+ cap = null;
+ }
+ const vastKey = String(v.vast_key || '');
+ const runpodKey = String(v.runpod_key || '');
+ let reason = '';
+ if (!vastKey && !runpodKey) reason = 'no Vast.ai or RunPod API key';
+ else if (cap == null || cap <= 0n) reason = 'no monthly cap';
+ return {
+ vast_key: vastKey,
+ runpod_key: runpodKey,
+ cap_nano: cap,
+ idle_minutes: clampInt(v.idle_minutes, 1, 1440, DEFAULT_IDLE_MINUTES),
+ max_runtime_minutes: clampInt(v.max_runtime_minutes, 1, 10080, DEFAULT_MAX_RUNTIME_MINUTES),
+ warm_window: String(v.warm_window || '').trim(),
+ region: String(v.region || DEFAULT_REGION).trim().toUpperCase() || DEFAULT_REGION,
+ enabled: Boolean(vastKey || runpodKey) && cap != null && cap > 0n,
+ reason,
+ };
+ }
+
+ /** The adapter of a marketplace, or null when its key is not set. */
+ function adapterFor(marketplace, cfg) {
+ if (marketplace === 'runpod') {
+ if (!cfg.runpod_key) return null;
+ return runpodAdapter({ fetchImpl, key: cfg.runpod_key });
+ }
+ if (!cfg.vast_key) return null;
+ return vastAdapter({ fetchImpl, key: cfg.vast_key });
+ }
+
+ /** Is the moment inside the owner's warm window (UTC)? Then an idle instance stays up. */
+ function inWarmWindow(window, at) {
+ const m = /^([0-2]\d):([0-5]\d)-([0-2]\d):([0-5]\d)$/.exec(String(window || '').trim());
+ if (!m) return false;
+ const minutes = (h, mi) => Number(h) * 60 + Number(mi);
+ const start = minutes(m[1], m[2]);
+ const end = minutes(m[3], m[4]);
+ if (start === end) return false;
+ const d = new Date(at);
+ const cur = d.getUTCHours() * 60 + d.getUTCMinutes();
+ if (start < end) return cur >= start && cur < end;
+ return cur >= start || cur < end;
+ }
+
+ // ── the money ─────────────────────────────────────────────────────────
+
+ /** The month's money: what was really spent, and what is committed (the worst case). */
+ async function moneySummary() {
+ const month = monthOf(now());
+ const cfg = config();
+ const ledger = await db.prepare('SELECT COALESCE(SUM(nano_usd), 0) AS total FROM gpu_spend WHERE month = ?').get(month);
+ const settled = toNano(ledger && ledger.total);
+ const rows = await db.prepare('SELECT * FROM gpu_instances WHERE month = ? AND status IN (?, ?, ?)').all(month, 'starting', 'running', 'stopping');
+ let reserved = 0n;
+ let running = 0n;
+ for (const row of rows) {
+ reserved += toNano(row.reserved_nano);
+ const started = toMs(row.started_at);
+ if (started != null) running += money.nanoForSeconds(toNano(row.hourly_nano), (now() - started) / 1000);
+ }
+ return {
+ month,
+ cap_nano: cfg.cap_nano,
+ spent_nano: settled + running,
+ committed_nano: settled + reserved,
+ reserved_nano: reserved,
+ settled_nano: settled,
+ };
+ }
+
+ /** Write the money an instance has cost since the last charge. */
+ async function charge(row, until, reason) {
+ const hourly = toNano(row.hourly_nano);
+ const from = toMs(row.charged_at) || toMs(row.started_at) || until;
+ const seconds = Math.max(0, (until - from) / 1000);
+ if (seconds <= 0) return 0n;
+ const amount = money.nanoForSeconds(hourly, seconds);
+ if (amount <= 0n) return 0n;
+ await db.prepare('INSERT INTO gpu_spend (at, month, kind, nano_usd, instance_id, detail, actor, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)')
+ .run(iso(until), monthOf(until), 'charge', amount.toString(), row.id, reason || 'the GPU ran', 'service:ai', iso(now()));
+ await db.prepare('UPDATE gpu_instances SET charged_at = ?, spent_nano = spent_nano + ?, updated_at = ? WHERE id = ?')
+ .run(iso(until), amount.toString(), iso(now()), row.id);
+ return amount;
+ }
+
+ // ── the instances ─────────────────────────────────────────────────────
+
+ async function liveInstance(modelKey) {
+ const row = await db.prepare('SELECT * FROM gpu_instances WHERE model_key = ? AND status IN (?, ?, ?) ORDER BY created_at DESC LIMIT 1')
+ .get(String(modelKey), 'starting', 'running', 'stopping');
+ return row || null;
+ }
+
+ async function instanceRows() {
+ const rows = await db.prepare('SELECT * FROM gpu_instances ORDER BY created_at DESC').all();
+ return rows || [];
+ }
+
+ /** The cheapest offer that fits. The same offers in the same order always give the same one. */
+ function cheapest(offers) {
+ const usable = (offers || []).filter((o) => o && o.price_per_hour != null && Number(o.price_per_hour) > 0);
+ usable.sort((a, b) => {
+ const pa = Number(a.price_per_hour);
+ const pb = Number(b.price_per_hour);
+ if (pa !== pb) return pa - pb;
+ return String(a.id) < String(b.id) ? -1 : String(a.id) > String(b.id) ? 1 : 0;
+ });
+ return usable[0] || null;
+ }
+
+ /** Ask for a model to start or stop. The reconciler does it; nothing here acts directly. */
+ async function request(modelKey, action, actor = 'service:ai') {
+ if (!catalog.find(modelKey) && !models.some((m) => m.key === modelKey)) throw new AiError(404, 'ai.not_found', 'no such GPU model');
+ if (action !== 'launch' && action !== 'stop') throw new AiError(422, 'ai.invalid', 'the action must be launch or stop');
+ await db.prepare('INSERT INTO gpu_requests (model_key, action, at, actor, created_at) VALUES (?, ?, ?, ?, ?)')
+ .run(String(modelKey), action, iso(now()), String(actor || '').slice(0, 200), iso(now()));
+ return { ok: true, model_key: modelKey, action };
+ }
+
+ /** Is the model wanted? It is wanted when the owner asked for it or when it was used recently. */
+ async function wanted(spec, cfg) {
+ const pending = await db.prepare("SELECT COUNT(*) AS n FROM gpu_requests WHERE model_key = ? AND action = 'launch' AND done_at IS NULL")
+ .get(spec.key);
+ if (pending && Number(pending.n) > 0) return true;
+ const used = await db.prepare('SELECT MAX(created_at) AS last FROM runs WHERE provider_key = ?').get(spec.provider_key);
+ const last = toMs(used && used.last);
+ if (last != null && now() - last <= cfg.idle_minutes * 60_000) return true;
+ return false;
+ }
+
+ /**
+ * Start an instance for a model: the cheapest offer that fits, inside the cap.
+ * Returns { ok, reason } - it never throws for the owner's reasons (off, no offer, cap).
+ */
+ async function launch(modelKey, { actor = 'service:ai' } = {}) {
+ const cfg = config();
+ if (!cfg.enabled) return { ok: false, code: 'gpu.off', reason: cfg.reason };
+ const spec = models.find((m) => m.key === modelKey) || catalog.find(modelKey);
+ if (!spec) throw new AiError(404, 'ai.not_found', 'no such GPU model');
+ const existing = await liveInstance(spec.key);
+ if (existing) return { ok: true, instance: describe(existing), note: 'already running' };
+ const adapter = adapterFor(spec.marketplace, cfg);
+ if (!adapter) return { ok: false, code: 'gpu.no_key', reason: 'no ' + spec.marketplace + ' API key' };
+
+ // 1. the cheapest offer that fits the model
+ let offers;
+ try {
+ offers = await adapter.searchOffers({
+ gpuNames: spec.requirements.gpu_names,
+ vramMb: spec.requirements.vram_mb,
+ diskGb: spec.requirements.disk_gb,
+ numGpus: spec.requirements.num_gpus,
+ minReliability: spec.requirements.min_reliability,
+ region: cfg.region,
+ maxPricePerHour: 10,
+ limit: 50,
+ });
+ } catch (err) {
+ return { ok: false, code: 'gpu.search_failed', reason: String(err && err.message).slice(0, 200) };
+ }
+ const offer = cheapest(offers);
+ if (!offer) {
+ return { ok: false, code: 'gpu.no_offer', reason: 'no ' + spec.marketplace + ' offer fits ' + spec.display_name };
+ }
+
+ // 2. the cap: the reservation has to fit under what is left of it
+ const hourly = money.nanoUsd(offer.price_per_hour, "the offer's price");
+ if (hourly == null || hourly <= 0n) {
+ return { ok: false, code: 'gpu.bad_price', reason: "the offer has no usable price" };
+ }
+ const sum = await moneySummary();
+ const cap = cfg.cap_nano;
+ if (sum.committed_nano >= cap) {
+ return { ok: false, code: 'gpu.cap', reason: 'the monthly cap is reached (' + money.usdText(sum.committed_nano) + ' of ' + money.usdText(cap) + ')' };
+ }
+ const affordableHours = (cap - sum.committed_nano) / hourly;
+ if (affordableHours < 1n) {
+ return { ok: false, code: 'gpu.cap', reason: 'the monthly cap leaves less than an hour of GPU time (' + money.usdText(cap - sum.committed_nano) + ' left)' };
+ }
+ const maxMinutes = Math.min(cfg.max_runtime_minutes, Number(affordableHours) * 60);
+ const reserved = hourly * BigInt(Math.ceil(maxMinutes / 60));
+ if (sum.committed_nano + reserved > cap) {
+ return { ok: false, code: 'gpu.cap', reason: 'the monthly cap leaves no room for ' + spec.display_name };
+ }
+
+ // 3. the row comes first. The marketplace is told the label of a row that already exists and
+ // holds the reservation, so an instance we paid for is never an orphan: the sweep finds it by
+ // the label and gives it back when no live row of ours owns it.
+ const id = LABEL_PREFIX + randomUUID().replace(/-/g, '').slice(0, 12);
+ const t = now();
+ await db.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,
+ gpu_name, gpu_ram_mb, num_gpus, disk_gb, reliability, region, price_per_hour, hourly_nano, reserved_nano, spent_nano,
+ max_runtime_minutes, started_at, charged_at, destroy_failures, label, cold_start_ms, health_failures, created_at, updated_at)
+ VALUES (@id, @model_key, @provider_key, @marketplace, @offer_id, NULL, 'starting', NULL, @month,
+ @gpu_name, @gpu_ram_mb, @num_gpus, @disk_gb, @reliability, @region, @price_per_hour, @hourly_nano, @reserved_nano, 0,
+ @max_runtime_minutes, @started_at, @charged_at, 0, @label, NULL, 0, @created_at, @updated_at)`)
+ .run({
+ id, model_key: spec.key, provider_key: spec.provider_key, marketplace: spec.marketplace,
+ offer_id: String(offer.id), month: monthOf(t),
+ gpu_name: String(offer.gpu_name || ''), gpu_ram_mb: Number(offer.gpu_ram_mb) || 0,
+ num_gpus: Number(offer.num_gpus) || 1, disk_gb: Number(offer.disk_gb) || 0,
+ reliability: Number(offer.reliability) || 0, region: String(offer.region || ''),
+ price_per_hour: String(offer.price_per_hour), hourly_nano: hourly.toString(), reserved_nano: reserved.toString(),
+ max_runtime_minutes: maxMinutes, started_at: iso(t), charged_at: iso(t), label: id,
+ created_at: iso(t), updated_at: iso(t),
+ });
+ await audit(actor, 'gpu.instance.requested', 'gpu_instance', id, {
+ model: spec.key, marketplace: spec.marketplace, offer_id: String(offer.id), hourly_nano: hourly.toString(),
+ reserved_nano: reserved.toString(), max_runtime_minutes: maxMinutes, label: id,
+ });
+
+ // 4. rent it. The marketplace is given the label of the row above, so the instance can always be
+ // found again. If the rental fails the row says so and the reservation is free again.
+ let created;
+ try {
+ created = await adapter.createInstance(offer, spec, {
+ command: catalog.commandOf(spec),
+ weights_url: catalog.weightsUrl(spec),
+ label: id,
+ });
+ } catch (err) {
+ const reason = String(err && err.message).slice(0, 200);
+ await db.prepare("UPDATE gpu_instances SET status = 'refused', error = ?, updated_at = ? WHERE id = ?")
+ .run(reason, iso(now()), id);
+ await audit(actor, 'gpu.instance.refused', 'gpu_instance', id, { reason });
+ return { ok: false, code: 'gpu.create_failed', reason };
+ }
+ await db.prepare('UPDATE gpu_instances SET remote_id = ?, updated_at = ? WHERE id = ?')
+ .run(String(created.remote_id || ''), iso(now()), id);
+ await audit(actor, 'gpu.instance.create', 'gpu_instance', id, { remote_id: String(created.remote_id || '') });
+ const row = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);
+ return { ok: true, instance: describe(row), offer, reserved_nano: reserved, hourly_nano: hourly };
+ }
+
+ /** The offer leaves the pool: the router fails over to the other providers. */
+ async function disableProvider(providerKey, reason) {
+ try {
+ await registry.upsertProvider({ key: providerKey, status: 'disabled' }, { actor: 'service:ai', origin: 'gpu' });
+ } catch (err) {
+ log.warn && log.warn('[gpu] could not disable the offer ' + providerKey + ': ' + String(err && err.message));
+ }
+ }
+
+ /** Register the instance's endpoint as an offer for the router. */
+ async function register(spec, row) {
+ const base = String(row.endpoint || '').replace(/\/+$/, '') + '/v1';
+ if (!/^https?:\/\/\S+$/i.test(base)) throw new AiError(502, 'gpu.no_endpoint', 'the instance has no usable endpoint');
+ const price = money.usdPerMillionTokens(toNano(row.hourly_nano), spec.throughput_tokens_per_hour);
+ const gpu = {
+ marketplace: row.marketplace,
+ offer_id: row.offer_id,
+ instance_id: row.id,
+ gpu_name: row.gpu_name,
+ price_per_hour: row.price_per_hour,
+ region: row.region,
+ runtime: spec.runtime.kind,
+ weights: catalog.weightsUrl(spec),
+ };
+ await registry.upsertProvider({
+ key: spec.provider_key,
+ display_name: spec.display_name,
+ kind: 'openai',
+ status: 'active',
+ base_url: base,
+ auth_mode: 'none',
+ secret_ref: null,
+ default_model: spec.model_key,
+ capabilities: ['chat'],
+ timeout_ms: 300000,
+ priority: 40,
+ billing_profile: 'metered',
+ provider_access: 'OWN_DEPLOYMENT',
+ metadata: { gpu },
+ }, { actor: 'service:ai', origin: 'gpu' });
+ await registry.upsertModel({
+ provider_key: spec.provider_key,
+ model_key: spec.model_key,
+ display_name: spec.display_name,
+ type: 'chat',
+ status: 'active',
+ context_window: spec.context_window,
+ cost: {
+ in_per_mtok: price,
+ out_per_mtok: price,
+ cached_per_mtok: price,
+ source: PRICE_SOURCE[row.marketplace] || 'urn:openvibe:ai:unverified:gpu',
+ verified_at: iso(now()).slice(0, 10),
+ },
+ supports: { json: false, tools: false, streaming: true, vision: false },
+ metadata: { gpu },
+ }, { actor: 'service:ai' });
+ return base;
+ }
+
+ /** Is the endpoint answering? The OpenAI-compatible /models is the cheapest honest check. */
+ async function healthy(endpoint) {
+ if (!endpoint) return false;
+ try {
+ const res = await fetchImpl(String(endpoint).replace(/\/+$/, '') + '/v1/models', { method: 'GET', headers: { Accept: 'application/json' } });
+ return Boolean(res && res.ok);
+ } catch {
+ return false;
+ }
+ }
+
+ /**
+ * Give the instance back and settle what it cost.
+ *
+ * A destroy that fails leaves the instance running (and being paid for): the row keeps its place,
+ * the try is counted, the next pass tries again and after three tries the operator is alerted
+ * (the gauge ai_gpu_destroy_failures and the rule in OpenVibe.Host's prometheus rules).
+ */
+ async function destroy(row, reason, { actor = 'service:ai' } = {}) {
+ const cfg = config();
+ const adapter = adapterFor(row.marketplace, cfg);
+ if (adapter && row.remote_id) {
+ try {
+ await adapter.destroyInstance(row.remote_id);
+ } catch (err) {
+ const failures = Number(row.destroy_failures || 0) + 1;
+ await db.prepare('UPDATE gpu_instances SET destroy_failures = ?, updated_at = ? WHERE id = ?')
+ .run(failures, iso(now()), row.id);
+ log.warn && log.warn('[gpu] could not give instance ' + row.id + ' back (' + failures + ' tries): ' + String(err && err.message));
+ if (failures >= ALERT_AFTER) {
+ await audit(actor, 'gpu.instance.destroy_failed', 'gpu_instance', row.id, { reason, failures, marketplace: row.marketplace });
+ log.error && log.error('[gpu] an instance could not be given back after ' + failures + ' tries: it is still running and still being paid for');
+ }
+ return { ok: false, failed: true, reason, failures };
+ }
+ }
+ const t = now();
+ // the freshest row: the money already on the ledger is not charged twice
+ const fresh = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(row.id);
+ await charge(fresh || row, t, reason);
+ await db.prepare("UPDATE gpu_instances SET status = 'destroyed', destroyed_at = ?, endpoint = NULL, destroy_failures = 0, updated_at = ? WHERE id = ?")
+ .run(iso(t), iso(t), row.id);
+ await disableProvider(row.provider_key, reason);
+ await audit(actor, 'gpu.instance.destroy', 'gpu_instance', row.id, { reason, model: row.model_key });
+ return { ok: true, reason };
+ }
+
+ /**
+ * Every pass: the marketplace instances that carry our label, and what to do with them.
+ *
+ * - one whose label is a live row of ours: left alone, and linked when the row does not know its
+ * marketplace id yet (a crash between the marketplace's answer and the row's update).
+ * - one no live row of ours owns: an orphan. It is given back and the fact is logged.
+ *
+ * A destroy that fails is tried again on the next pass; after three tries the operator is alerted.
+ */
+ async function sweepOrphans(cfg) {
+ const t = now();
+ let seen = 0;
+ for (const marketplace of ['vast', 'runpod']) {
+ const adapter = adapterFor(marketplace, cfg);
+ if (!adapter) continue;
+ let instances = [];
+ try {
+ instances = await adapter.listInstances();
+ } catch (err) {
+ log.warn && log.warn('[gpu] could not list the ' + marketplace + ' instances: ' + String(err && err.message).slice(0, 200));
+ continue;
+ }
+ for (const inst of instances) {
+ const label = String(inst.label || '');
+ if (!LABEL_RE.test(label)) continue; // not ours
+ seen += 1;
+ const row = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(label);
+ if (row && LIVE.includes(row.status)) {
+ // ours and alive: a crash after the marketplace accepted but before the row knew the
+ // id is repaired here, not thrown away.
+ if (!row.remote_id && inst.id) {
+ await db.prepare('UPDATE gpu_instances SET remote_id = ?, updated_at = ? WHERE id = ?')
+ .run(String(inst.id), iso(now()), row.id);
+ await audit('service:ai', 'gpu.instance.recovered', 'gpu_instance', row.id, { marketplace, remote_id: String(inst.id) });
+ log.log && log.log('[gpu] recovered the instance ' + inst.id + ' of the row ' + row.id + ' from the ' + marketplace + ' label');
+ }
+ continue;
+ }
+ await destroyOrphan({ marketplace, remote_id: String(inst.id), label }, row, cfg);
+ }
+ }
+ // The rows whose rental never came back (a crash before the marketplace answered, or an
+ // answer we never got): their reservation is free again, or the month would carry a budget
+ // for an instance that does not exist.
+ const stale = await db.prepare("SELECT * FROM gpu_instances WHERE status = 'starting' AND (remote_id IS NULL OR remote_id = '') AND started_at < ?").all(iso(t - STALE_ROW_MS));
+ for (const row of stale || []) {
+ await db.prepare("UPDATE gpu_instances SET status = 'refused', error = ?, updated_at = ? WHERE id = ?")
+ .run('the rental never came back', iso(t), row.id);
+ await audit('service:ai', 'gpu.instance.abandoned', 'gpu_instance', row.id, { marketplace: row.marketplace });
+ log.warn && log.warn('[gpu] gave up on the row ' + row.id + ': the rental never came back, its reservation is free again');
+ }
+ return seen;
+ }
+
+ /** An instance with our label that no live row owns: give it back, and keep the tries. */
+ async function destroyOrphan(inst, row, cfg) {
+ const adapter = adapterFor(inst.marketplace, cfg);
+ const id = row ? row.id : inst.label;
+ const t = now();
+ if (!row) {
+ // the row the orphan needs: its retries and its money need a home of their own
+ await db.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,
+ gpu_name, hourly_nano, reserved_nano, spent_nano, max_runtime_minutes, started_at, charged_at, destroy_failures, label, cold_start_ms, health_failures, created_at, updated_at)
+ VALUES (@id, '', '', @marketplace, '', @remote_id, 'orphan', NULL, @month, '', 0, 0, 0, 0, @started_at, @charged_at, 0, @label, NULL, 0, @created_at, @updated_at)`)
+ .run({
+ id, marketplace: inst.marketplace, remote_id: inst.remote_id, month: monthOf(t),
+ started_at: iso(t), charged_at: iso(t), label: inst.label, created_at: iso(t), updated_at: iso(t),
+ });
+ row = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);
+ }
+ // the orphan ran and was paid for: the seconds since the last charge are on the ledger
+ await charge(row, t, 'the orphan ran');
+ try {
+ await adapter.destroyInstance(inst.remote_id);
+ } catch (err) {
+ const failures = Number(row.destroy_failures || 0) + 1;
+ await db.prepare('UPDATE gpu_instances SET destroy_failures = ?, updated_at = ? WHERE id = ?')
+ .run(failures, iso(t), id);
+ log.warn && log.warn('[gpu] could not give the orphan ' + inst.remote_id + ' back (' + failures + ' tries): ' + String(err && err.message).slice(0, 200));
+ if (failures >= ALERT_AFTER) {
+ await audit('service:ai', 'gpu.orphan.destroy_failed', 'gpu_instance', id, { marketplace: inst.marketplace, remote_id: inst.remote_id, failures });
+ log.error && log.error('[gpu] an orphan instance could not be given back after ' + failures + ' tries: it is still running and still being paid for');
+ }
+ return { ok: false, failed: true };
+ }
+ await db.prepare("UPDATE gpu_instances SET status = 'destroyed', destroyed_at = ?, destroy_failures = 0, updated_at = ? WHERE id = ?")
+ .run(iso(t), iso(t), id);
+ await audit('service:ai', 'gpu.orphan.destroyed', 'gpu_instance', id, { marketplace: inst.marketplace, remote_id: inst.remote_id });
+ log.warn && log.warn('[gpu] gave back an orphan instance ' + inst.remote_id + ' (' + inst.marketplace + '): it carried our label but no live row owned it');
+ return { ok: true };
+ }
+
+ /** The destroy failures that are still open, per marketplace (the operator alert reads them). */
+ async function destroyFailures() {
+ const rows = await db.prepare('SELECT marketplace, MAX(destroy_failures) AS failures FROM gpu_instances WHERE destroy_failures > 0 GROUP BY marketplace').all();
+ return (rows || []).map((r) => ({ labels: { marketplace: r.marketplace }, value: Number(r.failures) || 0 }));
+ }
+
+ /** The instances with our label that no live row owns and that are still waiting to be given back. */
+ async function orphanCount() {
+ const rows = await db.prepare("SELECT marketplace, COUNT(*) AS n FROM gpu_instances WHERE status = 'orphan' GROUP BY marketplace").all();
+ return (rows || []).map((r) => ({ labels: { marketplace: r.marketplace }, value: Number(r.n) || 0 }));
+ }
+
+ /** One pass over the live instances: the reservation, the idle timer, the health. */
+ async function reconcileInstance(row, cfg) {
+ const spec = models.find((m) => m.key === row.model_key) || catalog.find(row.model_key);
+ if (!spec) return { changed: false };
+ const t = now();
+ const started = toMs(row.started_at) || t;
+
+ // the money that ran while we were away
+ await charge(row, t, 'the GPU ran');
+
+ // 1. the reservation ran out: the instance goes back, so the money stops there
+ if (t - started >= Number(row.max_runtime_minutes) * 60_000) {
+ await destroy({ ...row, charged_at: row.charged_at }, 'the reservation ran out');
+ return { changed: true, action: 'destroyed', reason: 'the reservation ran out' };
+ }
+
+ // 2. the idle timer: the offer scales to zero (unless it is inside the warm window). The clock runs
+ // from the moment the instance was ready, not from the moment it was rented: an instance that is still
+ // starting is never idle.
+ if (row.status === 'running') {
+ const lastUsed = toMs(row.last_used_at) || toMs(row.ready_at) || started;
+ if (t - lastUsed >= cfg.idle_minutes * 60_000 && !inWarmWindow(cfg.warm_window, t)) {
+ await destroy({ ...row, charged_at: row.charged_at }, 'idle');
+ return { changed: true, action: 'scaled_to_zero', reason: 'idle for ' + cfg.idle_minutes + ' minutes' };
+ }
+ }
+
+ // 3. the health of the instance
+ if (row.status === 'starting') {
+ const adapter = adapterFor(row.marketplace, cfg);
+ if (!adapter) return { changed: false };
+ // a row without a marketplace id: the crash between the rental and the row's update. The
+ // sweep finds the instance by the label and links it; until then there is nothing to poll.
+ if (!row.remote_id) return { changed: false };
+ let info = null;
+ try {
+ info = await adapter.showInstance(row.remote_id);
+ } catch (err) {
+ return { changed: false, error: String(err && err.message).slice(0, 200) };
+ }
+ const state = String((info && info.state) || 'unknown').toLowerCase();
+ if (DEAD_STATES.includes(state)) {
+ await destroy({ ...row, charged_at: row.charged_at }, 'the instance ' + state);
+ await request(spec.key, 'launch', 'service:ai');
+ return { changed: true, action: 'died', reason: 'the instance ' + state };
+ }
+ const endpoint = info && info.endpoint ? String(info.endpoint) : '';
+ if (endpoint && !(await healthy(endpoint))) {
+ // the endpoint is there but does not answer: the instance is dead, not slow
+ const failures = Number(row.health_failures || 0) + 1;
+ await db.prepare('UPDATE gpu_instances SET health_failures = ?, last_health_at = ?, updated_at = ? WHERE id = ?')
+ .run(failures, iso(t), iso(t), row.id);
+ if (failures >= HEALTH_FAILURES) {
+ await destroy({ ...row, charged_at: row.charged_at }, 'the endpoint never answered');
+ await request(spec.key, 'launch', 'service:ai');
+ return { changed: true, action: 'died', reason: 'the endpoint never answered' };
+ }
+ return { changed: false, health_failures: failures };
+ }
+ if (endpoint && await healthy(endpoint)) {
+ // the cold start: from the launch to the first answered request
+ await db.prepare("UPDATE gpu_instances SET status = 'running', endpoint = ?, ready_at = ?, last_health_at = ?, health_failures = 0, cold_start_ms = ?, updated_at = ? WHERE id = ?")
+ .run(endpoint, iso(t), iso(t), String(Math.max(0, t - started)), iso(t), row.id);
+ await register(spec, { ...row, endpoint });
+ return { changed: true, action: 'ready' };
+ }
+ return { changed: false };
+ }
+
+ if (row.status === 'running') {
+ const lastHealth = toMs(row.last_health_at) || 0;
+ if (t - lastHealth < HEALTH_INTERVAL_MS) return { changed: false };
+ const ok = await healthy(row.endpoint);
+ if (ok) {
+ await db.prepare('UPDATE gpu_instances SET health_failures = 0, last_health_at = ?, updated_at = ? WHERE id = ?')
+ .run(iso(t), iso(t), row.id);
+ return { changed: false };
+ }
+ const failures = Number(row.health_failures || 0) + 1;
+ await db.prepare('UPDATE gpu_instances SET health_failures = ?, last_health_at = ?, updated_at = ? WHERE id = ?')
+ .run(failures, iso(t), iso(t), row.id);
+ if (failures < HEALTH_FAILURES) return { changed: false, health_failures: failures };
+ // the instance died: the offer leaves the pool and a new one is rented
+ await destroy({ ...row, charged_at: row.charged_at }, 'the endpoint stopped answering');
+ await request(spec.key, 'launch', 'service:ai');
+ return { changed: true, action: 'died', reason: 'the endpoint stopped answering' };
+ }
+ return { changed: false };
+ }
+
+ /** The audit row. The registry's audit is used when it is there; the money rows are the record. */
+ async function audit(actor, action, kind, key, metadata) {
+ try {
+ if (registry && typeof registry.audit === 'function') {
+ await registry.audit(actor, action, kind, key, { metadata });
+ return;
+ }
+ } catch (err) {
+ log.warn && log.warn('[gpu] the audit row failed: ' + String(err && err.message));
+ }
+ }
+
+ /** A row as the console and the API show it. */
+ function describe(row) {
+ if (!row) return null;
+ return {
+ id: row.id,
+ model_key: row.model_key,
+ provider_key: row.provider_key,
+ marketplace: row.marketplace,
+ offer_id: row.offer_id,
+ remote_id: row.remote_id,
+ status: row.status,
+ endpoint: row.endpoint,
+ month: row.month,
+ gpu_name: row.gpu_name,
+ gpu_ram_mb: row.gpu_ram_mb,
+ num_gpus: row.num_gpus,
+ disk_gb: row.disk_gb,
+ reliability: row.reliability,
+ region: row.region,
+ price_per_hour: row.price_per_hour,
+ hourly_nano: String(toNano(row.hourly_nano)),
+ reserved_nano: String(toNano(row.reserved_nano)),
+ spent_nano: String(toNano(row.spent_nano)),
+ max_runtime_minutes: row.max_runtime_minutes,
+ label: row.label,
+ cold_start_ms: row.cold_start_ms == null ? null : String(row.cold_start_ms),
+ destroy_failures: Number(row.destroy_failures || 0),
+ started_at: row.started_at,
+ ready_at: row.ready_at,
+ last_used_at: row.last_used_at,
+ destroyed_at: row.destroyed_at,
+ error: row.error,
+ };
+ }
+
+ // ── the reconciler ────────────────────────────────────────────────────
+
+ /** One pass: the owner's requests, the live instances, and the models that are wanted. */
+ async function tick({ actor = 'service:ai' } = {}) {
+ const cfg = config();
+ if (!cfg.enabled) return { ok: false, reason: cfg.reason, launched: 0, stopped: 0, destroyed: 0 };
+
+ let launched = 0;
+ let stopped = 0;
+ let destroyed = 0;
+
+ // 0. the marketplace instances that carry our label: the ones no live row owns are given back
+ await sweepOrphans(cfg);
+
+ // 1. the owner's requests
+ const requests = await db.prepare('SELECT * FROM gpu_requests WHERE done_at IS NULL ORDER BY id').all();
+ for (const req of requests || []) {
+ try {
+ if (req.action === 'launch') {
+ const res = await launch(req.model_key, { actor: req.actor || actor });
+ if (!res.ok) {
+ await db.prepare('UPDATE gpu_requests SET done_at = ?, error = ? WHERE id = ?')
+ .run(iso(now()), String(res.reason).slice(0, 300), req.id);
+ continue;
+ }
+ launched += 1;
+ } else {
+ const row = await liveInstance(req.model_key);
+ if (row) {
+ await destroy(row, 'the owner stopped it', { actor: req.actor || actor });
+ stopped += 1;
+ }
+ }
+ await db.prepare('UPDATE gpu_requests SET done_at = ?, error = NULL WHERE id = ?').run(iso(now()), req.id);
+ } catch (err) {
+ await db.prepare('UPDATE gpu_requests SET done_at = ?, error = ? WHERE id = ?')
+ .run(iso(now()), String(err && err.message).slice(0, 300), req.id);
+ }
+ }
+
+ // 2. the live instances
+ const rows = await db.prepare('SELECT * FROM gpu_instances WHERE status IN (?, ?) ORDER BY created_at').all('starting', 'running');
+ for (const row of rows || []) {
+ const res = await reconcileInstance(row, cfg);
+ if (res && (res.action === 'destroyed' || res.action === 'scaled_to_zero' || res.action === 'died')) destroyed += 1;
+ }
+
+ // 3. the models that are wanted but have no instance
+ for (const spec of models) {
+ if (await liveInstance(spec.key)) continue;
+ if (!await wanted(spec, cfg)) continue;
+ const res = await launch(spec.key, { actor });
+ if (res.ok) launched += 1;
+ }
+
+ return { ok: true, launched, stopped, destroyed };
+ }
+
+ /** The state the console page and the admin API show. */
+ async function state() {
+ const cfg = config();
+ const sum = await moneySummary();
+ const rows = await instanceRows();
+ const list = models.map((spec) => {
+ const live = rows.find((r) => r.model_key === spec.key && LIVE.includes(r.status)) || null;
+ // the last measured cold start (the rows are newest first)
+ const measured = rows.find((r) => r.model_key === spec.key && r.cold_start_ms != null) || null;
+ return {
+ key: spec.key,
+ display_name: spec.display_name,
+ marketplace: spec.marketplace,
+ runtime: spec.runtime.kind,
+ requirements: spec.requirements,
+ weights_url: catalog.weightsUrl(spec),
+ command: catalog.commandOf(spec),
+ cold_start_ms: measured ? String(measured.cold_start_ms) : null,
+ instance: describe(live),
+ };
+ });
+ return {
+ enabled: cfg.enabled,
+ reason: cfg.reason,
+ month: sum.month,
+ cap_nano: cfg.cap_nano == null ? null : String(cfg.cap_nano),
+ spent_nano: String(sum.spent_nano),
+ committed_nano: String(sum.committed_nano),
+ reserved_nano: String(sum.reserved_nano),
+ idle_minutes: cfg.idle_minutes,
+ max_runtime_minutes: cfg.max_runtime_minutes,
+ warm_window: cfg.warm_window,
+ region: cfg.region,
+ marketplaces: { vast: Boolean(cfg.vast_key), runpod: Boolean(cfg.runpod_key) },
+ models: list,
+ instances: rows.map(describe),
+ };
+ }
+
+ return {
+ state,
+ tick,
+ launch,
+ request,
+ config,
+ moneySummary,
+ sweepOrphans,
+ destroyFailures,
+ orphanCount,
+ /** For the tests: the same thing the reconciler does with one instance. */
+ reconcileInstance,
+ inWarmWindow,
+ };
+}
+
+module.exports = { createGpuProvisioner, HEALTH_INTERVAL_MS, HEALTH_FAILURES };
new file mode 100644
new file mode 100644
@@ -0,0 +1,142 @@
@@ -0,0 +1,142 @@
+'use strict';
+/**
+ * RunPod: the marketplace for bigger cards (the H100 that fits a large model).
+ *
+ * The calls are the ones the RunPod v2 API documents:
+ * GET /v2/gpu-types the catalogue of cards and their prices
+ * POST /v2/pods rent a pod
+ * GET /v2/pods/{id} the pod
+ * DELETE /v2/pods/{id} give the pod back
+ *
+ * The API key goes in the Authorization header and nowhere else.
+ */
+const { apiJson } = require('./http');
+
+const BASE_URL = 'https://api.runpod.io';
+
+/** A RunPod adapter: searchOffers, createInstance, showInstance, destroyInstance. */
+function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BASE_URL } = {}) {
+ const auth = { key };
+
+ /** The endpoint of a running pod: its proxy URL, or the documented proxy address. */
+ function endpointOf(pod, port) {
+ const ports = (pod && pod.ports) || {};
+ const p = ports[port] || ports[String(port)];
+ if (p) {
+ if (p.proxyUrl) return String(p.proxyUrl);
+ if (p.publicPort && pod.ip) return 'https://' + pod.ip + ':' + p.publicPort;
+ }
+ if (pod && pod.id && port) return 'https://' + pod.id + '-' + port + '.proxy.runpod.io';
+ return '';
+ }
+
+ /** The card's price: the secure price when there is one, else the community price. */
+ function priceOf(gpu) {
+ const price = (gpu && gpu.price) || {};
+ const secure = Number(price.secure);
+ const community = Number(price.community);
+ if (Number.isFinite(secure) && secure > 0) return String(secure);
+ if (Number.isFinite(community) && community > 0) return String(community);
+ return '0';
+ }
+
+ return {
+ id: 'runpod',
+ label: 'RunPod',
+ baseUrl,
+
+ async searchOffers(req) {
+ const out = await apiJson(fetchImpl, baseUrl + '/v2/gpu-types', { ...auth, label: 'RunPod' });
+ const gpus = (out && out.gpus) || [];
+ const wanted = (req.gpuNames || []).map((n) => String(n).toLowerCase());
+ const offers = [];
+ for (const gpu of gpus) {
+ const name = String(gpu.name || '');
+ const lower = name.toLowerCase();
+ if (wanted.length && !wanted.some((w) => lower.includes(w.toLowerCase()))) continue;
+ if (Number(gpu.memory) < Number(req.vramMb)) continue;
+ if (Number(gpu.maxCount || 0) < Number(req.numGpus || 1)) continue;
+ const price = priceOf(gpu);
+ if (Number(price) > Number(req.maxPricePerHour)) continue;
+ for (const dc of (gpu.dataCenters || [])) {
+ if (dc.availability !== undefined && !dc.availability) continue;
+ offers.push({
+ id: String(gpu.id) + ':' + String(dc.id),
+ marketplace: 'runpod',
+ gpu_type_id: String(gpu.id),
+ data_center_id: String(dc.id),
+ gpu_name: name,
+ gpu_ram_mb: Number(gpu.memory) || 0,
+ num_gpus: Number(req.numGpus || 1),
+ disk_gb: Number(req.diskGb || 0),
+ price_per_hour: price,
+ reliability: 1,
+ region: String(dc.name || dc.id || ''),
+ });
+ }
+ }
+ return offers;
+ },
+
+ /**
+ * The pods of this account that carry our name. Runpod lists them at GET /v2/pods; the sweep
+ * reads them to find the ones no live row owns.
+ */
+ async listInstances() {
+ const body = await apiJson(fetchImpl, baseUrl + '/v2/pods', { ...auth, label: 'RunPod' });
+ const list = (body && (body.pods || body.items || body.data)) || [];
+ return list.map((pod) => ({
+ id: String(pod.id == null ? '' : pod.id),
+ label: String(pod.name || ''),
+ state: String(pod.status || 'unknown').toLowerCase(),
+ price_per_hour: pod.costPerHour == null ? '' : String(pod.costPerHour),
+ }));
+ },
+
+ async createInstance(offer, spec, opts = {}) {
+ const r = spec.runtime;
+ const body = {
+ // the name carries our row id: the sweep finds the pod again and gives it back
+ // when no live row of ours owns it.
+ name: String(opts.label || ('openvibe-' + spec.key)),
+ image: r.image,
+ gpu: { id: offer.gpu_type_id, count: spec.requirements.num_gpus },
+ disk: { size: spec.requirements.disk_gb, type: 'volume' },
+ ports: [String(r.port || 8000) + '/http'],
+ env: [
+ 'MODEL_FILE=' + ((r.weights && r.weights.file) || ''),
+ 'MODEL_URL=' + (opts.weights_url || ''),
+ 'MODEL_ALIAS=' + spec.model_key,
+ ],
+ start: true,
+ };
+ if (offer.data_center_id) body.dataCenterId = offer.data_center_id;
+ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods', { method: 'POST', body, ...auth, label: 'RunPod' });
+ const remoteId = String((out && out.id) || '');
+ if (!remoteId) {
+ const err = new Error('RunPod answered without a pod id');
+ err.status = 502;
+ throw err;
+ }
+ return { remote_id: remoteId, endpoint: '', state: String((out && out.status) || 'PROVISIONING').toLowerCase() };
+ },
+
+ async showInstance(remoteId) {
+ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { ...auth, label: 'RunPod' });
+ const status = String((out && out.status) || 'unknown').toLowerCase();
+ return {
+ remote_id: remoteId,
+ state: status,
+ endpoint: endpointOf(out, 8000),
+ raw: out,
+ };
+ },
+
+ async destroyInstance(remoteId) {
+ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'RunPod' });
+ return Boolean(out);
+ },
+ };
+}
+
+module.exports = { runpodAdapter, BASE_URL };
new file mode 100644
new file mode 100644
@@ -0,0 +1,151 @@
@@ -0,0 +1,151 @@
+'use strict';
+/**
+ * Vast.ai: the marketplace for consumer GPUs (the RTX 3090/4090 that fit a 32B Q4 distill).
+ *
+ * The calls are the ones the Vast API documents:
+ * POST /api/v0/bundles search the offers
+ * POST /api/v0/asks/{id}/ rent an offer (the answer's new_contract is the instance id)
+ * GET /api/v0/instances/{id} the instance (its actual_status and its ports)
+ * DELETE /api/v0/instances/{id} give the instance back
+ *
+ * The API key goes in the Authorization header and nowhere else.
+ */
+const { apiJson } = require('./http');
+
+const BASE_URL = 'https://console.vast.ai';
+
+/** A Vast adapter: searchOffers, createInstance, showInstance, destroyInstance. */
+function vastAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BASE_URL } = {}) {
+ const auth = { key };
+
+ /** The endpoint of a running instance: its proxy URL, or the direct address. */
+ function endpointOf(instance, port) {
+ const ports = (instance && instance.ports) || {};
+ const p = ports[port] || ports[String(port)];
+ if (p && p.proxy) return String(p.proxy);
+ const host = instance.ssh_node || instance.ssh_host || instance.public_ip || instance.ip;
+ const pr = p && p.port ? p.port : port;
+ if (host && pr) return 'http://' + host + ':' + pr;
+ return '';
+ }
+
+ return {
+ id: 'vast',
+ label: 'Vast.ai',
+ baseUrl,
+
+ async searchOffers(req) {
+ const body = {
+ type: 'ondemand',
+ verified: { eq: true },
+ rentable: { eq: true },
+ gpu_name: { in: req.gpuNames },
+ reliability: { gte: req.minReliability },
+ num_gpus: { gte: req.numGpus },
+ gpu_ram: { gte: req.vramMb },
+ disk_space: { gte: req.diskGb },
+ dph_total: { lte: req.maxPricePerHour },
+ geolocation: { in: [req.region] },
+ limit: req.limit || 50,
+ };
+ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/bundles', { method: 'POST', body, ...auth, label: 'Vast.ai' });
+ const offers = (out && out.offers) || [];
+ return offers.map((o) => ({
+ id: String(o.id),
+ marketplace: 'vast',
+ gpu_name: String(o.gpu_name || ''),
+ gpu_ram_mb: Number(o.gpu_ram) || 0,
+ num_gpus: Number(o.num_gpus) || 1,
+ disk_gb: Number(o.disk_space) || 0,
+ price_per_hour: String(o.dph_total == null ? '0' : o.dph_total),
+ reliability: Number(o.reliability) || 0,
+ region: String(o.geolocation || ''),
+ }));
+ },
+
+ /**
+ * The instances of this account that carry our label. Vast lists them at GET /api/v1/instances
+ * (25 per page, keyset pagination); the sweep reads them to find the ones no live row owns.
+ */
+ async listInstances() {
+ const out = [];
+ let after = null;
+ for (let page = 0; page < 20; page++) {
+ const url = baseUrl + '/api/v1/instances?limit=25&select_cols=id,label,actual_status,dph_total'
+ + (after ? '&after_token=' + encodeURIComponent(after) : '');
+ const body = await apiJson(fetchImpl, url, { ...auth, label: 'Vast.ai' });
+ const list = (body && (body.instances || body.results || body.data)) || [];
+ for (const inst of list) {
+ out.push({
+ id: String(inst.id == null ? '' : inst.id),
+ label: String(inst.label || ''),
+ state: String(inst.actual_status || inst.status || 'unknown').toLowerCase(),
+ price_per_hour: inst.dph_total == null ? '' : String(inst.dph_total),
+ });
+ }
+ after = body && body.next_token;
+ if (!after) break;
+ }
+ return out;
+ },
+
+ async createInstance(offer, spec, opts = {}) {
+ const r = spec.runtime;
+ const body = {
+ image: r.image,
+ args: opts.command || '',
+ env: {
+ MODEL_FILE: (r.weights && r.weights.file) || '',
+ MODEL_URL: opts.weights_url || '',
+ MODEL_ALIAS: spec.model_key,
+ },
+ disk: spec.requirements.disk_gb,
+ runtype: 'args',
+ // the label carries our row id: the sweep finds the instance again and gives it back
+ // when no live row of ours owns it.
+ label: String(opts.label || ('openvibe-' + spec.key)),
+ };
+ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/asks/' + offer.id + '/', { method: 'POST', body, ...auth, label: 'Vast.ai' });
+ if (!out || out.success === false) {
+ const err = new Error('Vast.ai did not rent the offer');
+ err.status = 502;
+ throw err;
+ }
+ const remoteId = String(out.new_contract || out.id || '');
+ if (!remoteId) {
+ const err = new Error('Vast.ai answered without an instance id');
+ err.status = 502;
+ throw err;
+ }
+ return { remote_id: remoteId, endpoint: '', state: 'starting' };
+ },
+
+ async showInstance(remoteId) {
+ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { ...auth, label: 'Vast.ai' });
+ const status = String((out && (out.actual_status || out.status)) || 'unknown');
+ return {
+ remote_id: remoteId,
+ state: status,
+ endpoint: endpointOf(out, (opts_port(out))),
+ raw: out,
+ };
+ },
+
+ async destroyInstance(remoteId) {
+ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'Vast.ai' });
+ return Boolean(out && out.success !== false);
+ },
+ };
+}
+
+// The port a Vast instance serves on is in its ports map; the first one that is not SSH is the app.
+function opts_port(instance) {
+ const ports = (instance && instance.ports) || {};
+ for (const key of Object.keys(ports)) {
+ const n = Number(key);
+ if (Number.isFinite(n) && n >= 8000) return n;
+ }
+ return 8000;
+}
+
+module.exports = { vastAdapter, BASE_URL };
@@ -53,6 +53,10 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -53,6 +53,10 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
// D11 owner settings: ai.exchange and ai.provider_keys on openvibe-shared/config. The provider-key store also
// D11 owner settings: ai.exchange and ai.provider_keys on openvibe-shared/config. The provider-key store also
// refreshes the synchronous vault cache, and its one-time migration runs before any provider is built.
// refreshes the synchronous vault cache, and its one-time migration runs before any provider is built.
const settings = await require('./settings').createSettings({ db, config, registry, clock, env, log, onExchange: () => ensureCredits() });
const settings = await require('./settings').createSettings({ db, config, registry, clock, env, log, onExchange: () => ensureCredits() });
+ // The GPU provisioner (the design's "OpenVibe GPUs"): it is off until the owner has set a marketplace
+ // API key and a monthly cap in the owner settings (ai.gpus). The reconciler rents, health-checks and
+ // scales the instances to zero; it never spends past the cap.
+ const gpus = require('./gpu/provisioner').createGpuProvisioner({ db, registry, settings, clock, fetchImpl, env, log });
const vault = require('./vault').createVault({ settings, clock, log });
const vault = require('./vault').createVault({ settings, clock, log });
await require('./vault').migrate({ db, config, settings, log }).catch((err) => log.warn(`[Vault] migration failed: ${err.message}`));
await require('./vault').migrate({ db, config, settings, log }).catch((err) => log.warn(`[Vault] migration failed: ${err.message}`));
await vault.load().catch((err) => log.warn(`[Vault] ${err.message}`));
await vault.load().catch((err) => log.warn(`[Vault] ${err.message}`));
@@ -146,7 +150,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -146,7 +150,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
clientSecret: env.OV_OAUTH_CLIENT_SECRET, fetch: fetchImpl,
clientSecret: env.OV_OAUTH_CLIENT_SECRET, fetch: fetchImpl,
}) : null;
}) : null;
const benchEvents = createBenchEvents({ bench, accountData, accountSend, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });
const benchEvents = createBenchEvents({ bench, accountData, accountSend, config, log, fetchImpl, clientSecret: env.OV_OAUTH_CLIENT_SECRET });
- const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, memoryImport, community, presets, receipts, balance });
+ const app = createApp({ config, db, registry, pool, quotas, cache, runs, auth, keys, env, clock, fetchImpl, log, credentials, preferences, grants, vault, settings, benchEvents, runBudgets, memory, memoryImport, community, presets, receipts, balance, gpus });
let jwks = null;
let jwks = null;
let keyLoaded = Promise.resolve(null);
let keyLoaded = Promise.resolve(null);
if (!keys.pinned && config.nodeEnv !== 'test') {
if (!keys.pinned && config.nodeEnv !== 'test') {
@@ -167,6 +171,16 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -167,6 +171,16 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
}, 60 * 60 * 1000);
}, 60 * 60 * 1000);
housekeeping.unref?.();
housekeeping.unref?.();
+ // The GPU reconciler: once a minute it fulfils the owner's requests, health-checks the running
+ // instances, scales the idle ones to zero and starts the models that are wanted. Never in test mode.
+ let gpuReconciler = null;
+ if (listen && config.nodeEnv !== 'test') {
+ gpuReconciler = setInterval(() => {
+ gpus.tick().catch((err) => log.warn('[gpu] the reconciler failed: ' + err.message));
+ }, 60 * 1000);
+ gpuReconciler.unref?.();
+ }
+
let server = null;
let server = null;
let benchSubscription = null;
let benchSubscription = null;
let accountSubscription = null;
let accountSubscription = null;
@@ -202,6 +216,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -202,6 +216,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
async function close() {
async function close() {
clearInterval(housekeeping);
clearInterval(housekeeping);
+ if (gpuReconciler) clearInterval(gpuReconciler);
balance.stop();
balance.stop();
if (benchSubscription) benchSubscription.stop();
if (benchSubscription) benchSubscription.stop();
if (accountSubscription) accountSubscription.stop();
if (accountSubscription) accountSubscription.stop();
@@ -222,7 +237,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
@@ -222,7 +237,7 @@ async function start({ config, db: givenDb = null, clock = { now: () => Date.now
if (!givenDb) await db.close();
if (!givenDb) await db.close();
}
}
- return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, bench, benchEvents, memory, memoryJobs, credits: creditsRef, settings, vault, community, close };
+ return { config, db, registry, pool, quotas, tiers, cache, fetcher, engine, runs, keys, keyLoaded, auth, app, server, bench, benchEvents, memory, memoryJobs, credits: creditsRef, settings, vault, community, gpus, close };
}
}
if (require.main === module) {
if (require.main === module) {
@@ -88,7 +88,7 @@ function createAiReadiness({ db, registry, pool, keys, runs, release = null }) {
@@ -88,7 +88,7 @@ function createAiReadiness({ db, registry, pool, keys, runs, release = null }) {
}
}
/** AI gauges on the openvibe-shared/metrics registry. */
/** AI gauges on the openvibe-shared/metrics registry. */
-function registerAiGauges(registry, { runs, registry: ai, pool, balance = null }) {
+function registerAiGauges(registry, { runs, registry: ai, pool, balance = null, gpus = null }) {
registry.gauge({
registry.gauge({
name: 'ai_runs', help: 'Runs waiting for a slot (queued) and holding one (running)', labelNames: ['state'],
name: 'ai_runs', help: 'Runs waiting for a slot (queued) and holding one (running)', labelNames: ['state'],
collect: () => {
collect: () => {
@@ -111,6 +111,17 @@ function registerAiGauges(registry, { runs, registry: ai, pool, balance = null }
@@ -111,6 +111,17 @@ function registerAiGauges(registry, { runs, registry: ai, pool, balance = null }
name: 'ai_provider_balance_usd', help: 'Provider account balance in USD, as last read by the hourly probe', labelNames: ['provider'],
name: 'ai_provider_balance_usd', help: 'Provider account balance in USD, as last read by the hourly probe', labelNames: ['provider'],
collect: () => (balance ? balance.values() : []).map(r => ({ labels: { provider: r.provider }, value: r.usd })),
collect: () => (balance ? balance.values() : []).map(r => ({ labels: { provider: r.provider }, value: r.usd })),
});
});
+ // The GPU provisioner (server/gpu/): a marketplace instance that could not be given back is still running
+ // and still being paid for. The operator alert fires at three tries in a row (OpenVibe.Host's prometheus
+ // rules), so the gauge counts the tries that are still open.
+ registry.gauge({
+ name: 'ai_gpu_destroy_failures', help: 'Marketplace GPU instances OpenVibe.AI could not give back, in a row (the operator alert fires at 3)', labelNames: ['marketplace'],
+ collect: async () => (gpus ? await gpus.destroyFailures() : []),
+ });
+ registry.gauge({
+ name: 'ai_gpu_orphan_instances', help: 'Marketplace GPU instances with our label that no live row owns and that are still waiting to be given back', labelNames: ['marketplace'],
+ collect: async () => (gpus ? await gpus.orphanCount() : []),
+ });
}
}
module.exports = { createAiReadiness, registerAiGauges };
module.exports = { createAiReadiness, registerAiGauges };
@@ -20,10 +20,15 @@ const { AiError } = require('./util');
@@ -20,10 +20,15 @@ const { AiError } = require('./util');
const EXCHANGE_NAMESPACE = 'ai.exchange';
const EXCHANGE_NAMESPACE = 'ai.exchange';
const PROVIDER_KEYS_NAMESPACE = 'ai.provider_keys';
const PROVIDER_KEYS_NAMESPACE = 'ai.provider_keys';
+const GPU_NAMESPACE = 'ai.gpus';
const SERVICE = 'ai';
const SERVICE = 'ai';
const SELF = Object.freeze({ type: 'service', id: SERVICE });
const SELF = Object.freeze({ type: 'service', id: SERVICE });
const GROUP = 'AI Exchange';
const GROUP = 'AI Exchange';
const PROVIDER_GROUP = 'Provider keys';
const PROVIDER_GROUP = 'Provider keys';
+const GPU_GROUP = 'OpenVibe GPUs';
+// The marketplaces the GPU provisioner rents from. AI owns these origins and writes them into the
+// namespace, so the owner's key is sealed to the marketplace it was entered for.
+const GPU_ORIGINS = Object.freeze({ vast: 'https://console.vast.ai', runpod: 'https://api.runpod.io' });
/** Bounds from server/config.js (load()), so the schema and the code agree. */
/** Bounds from server/config.js (load()), so the schema and the code agree. */
const EXCHANGE_SCHEMA = Object.freeze({
const EXCHANGE_SCHEMA = Object.freeze({
@@ -108,6 +113,69 @@ function originOf(baseUrl) {
@@ -108,6 +113,69 @@ function originOf(baseUrl) {
} catch { return ''; }
} catch { return ''; }
}
}
+/**
+ * The ai.gpus namespace: the owner's GPU settings.
+ *
+ * vast.origin / vast.key Vast.ai: the origin AI owns and the owner's write-only API key, sealed
+ * to that origin (a key can never be sent to another host).
+ * runpod.origin / runpod.key the same for RunPod.
+ * monthly_cap the hard monthly cap in nano-USD. The provisioner never passes it: it
+ * refuses to start an instance and says why. It is a money key, so it
+ * needs a fresh sign-in to change.
+ * idle_minutes how long an unused instance stays up before it scales to zero.
+ * max_runtime_minutes how long one instance may run at most (its reservation).
+ * warm_window an optional window (UTC, "19:00-23:00") in which idle instances stay up.
+ * region the region the marketplace offers must be in.
+ */
+const GPU_SCHEMA = Object.freeze({
+ $schema: 'https://json-schema.org/draft/2020-12/schema',
+ title: 'OpenVibe GPUs',
+ type: 'object',
+ additionalProperties: false,
+ properties: {
+ 'vast.origin': {
+ type: 'string', title: 'Vast.ai origin', 'x-group': GPU_GROUP,
+ description: 'The origin of the Vast.ai API. AI writes it; the key below is sealed to it.',
+ },
+ 'vast.key': {
+ type: 'string', title: 'Vast.ai API key', 'x-group': GPU_GROUP, 'x-bind': 'vast.origin',
+ description: "The owner's write-only Vast.ai API key, sealed to the origin above.",
+ },
+ 'runpod.origin': {
+ type: 'string', title: 'RunPod origin', 'x-group': GPU_GROUP,
+ description: 'The origin of the RunPod API. AI writes it; the key below is sealed to it.',
+ },
+ 'runpod.key': {
+ type: 'string', title: 'RunPod API key', 'x-group': GPU_GROUP, 'x-bind': 'runpod.origin',
+ description: "The owner's write-only RunPod API key, sealed to the origin above.",
+ },
+ monthly_cap: {
+ type: 'integer', title: 'Monthly GPU cap', 'x-group': GPU_GROUP, 'x-unit': 'nano_usd', 'x-danger': true,
+ minimum: 0,
+ description: 'The hard monthly cap for the rented GPUs, in nano-USD. The provisioner never spends past it: it refuses to start an instance and says why. Without a cap the provisioner stays off.',
+ },
+ idle_minutes: {
+ type: 'integer', title: 'Idle minutes', 'x-group': GPU_GROUP,
+ minimum: 1, maximum: 1440,
+ description: 'How long an instance serves nobody before it scales to zero (the default is 10).',
+ },
+ max_runtime_minutes: {
+ type: 'integer', title: 'Max runtime (minutes)', 'x-group': GPU_GROUP,
+ minimum: 1, maximum: 10080,
+ description: 'How long one instance may run at most. When the time is up the instance is given back, so the money never runs past the cap (the default is 720).',
+ },
+ warm_window: {
+ type: 'string', title: 'Warm window (UTC)', 'x-group': GPU_GROUP,
+ pattern: '^([0-2][0-9]):[0-5][0-9]-([0-2][0-9]):[0-5][0-9]$',
+ description: 'An optional window in which idle instances stay up, as "19:00-23:00" (UTC). Empty means no warm window.',
+ },
+ region: {
+ type: 'string', title: 'Region', 'x-group': GPU_GROUP,
+ description: 'The region the marketplace offers must be in (the default is US).',
+ },
+ },
+});
+
/** The namespace for the providers there are today: <p>.origin (internal) and <p>.key (secret, bound to it). */
/** The namespace for the providers there are today: <p>.origin (internal) and <p>.key (secret, bound to it). */
function providerKeysSchema(providers) {
function providerKeysSchema(providers) {
const properties = {};
const properties = {};
@@ -251,6 +319,45 @@ function createProviderKeys({ db, registry, env = process.env, log = console, on
@@ -251,6 +319,45 @@ function createProviderKeys({ db, registry, env = process.env, log = console, on
};
};
}
}
+/**
+ * The ai.gpus store: the owner's GPU settings. The marketplace origins are written once (AI owns them),
+ * the keys and the cap are the owner's.
+ */
+async function createGpuSettings({ db, env = process.env, log = console, onChange }) {
+ const store = await createConfigStore({
+ db, service: SERVICE, namespace: GPU_NAMESPACE, schema: GPU_SCHEMA, classify, env, log, onChange,
+ });
+ const origins = {};
+ for (const [name, origin] of Object.entries(GPU_ORIGINS)) origins[name + '.origin'] = origin;
+ const changes = await store.changes(origins, { merge: true });
+ if (changes.length) await store.apply(origins, { merge: true, actor: SELF, reason: 'the GPU marketplace origins' });
+ if (store.sealed) await store.reseal().catch((err) => log.warn('[config] ' + GPU_NAMESPACE + ': reseal failed: ' + err.message));
+
+ /** The owner's GPU settings. The keys are the secrets; they never leave this process. */
+ function values() {
+ const v = store.get() || {};
+ return {
+ vast_origin: String(v['vast.origin'] || ''),
+ vast_key: String(v['vast.key'] || ''),
+ runpod_origin: String(v['runpod.origin'] || ''),
+ runpod_key: String(v['runpod.key'] || ''),
+ monthly_cap: v.monthly_cap == null ? null : String(v.monthly_cap),
+ idle_minutes: v.idle_minutes == null ? null : Number(v.idle_minutes),
+ max_runtime_minutes: v.max_runtime_minutes == null ? null : Number(v.max_runtime_minutes),
+ warm_window: String(v.warm_window || ''),
+ region: String(v.region || ''),
+ };
+ }
+
+ return {
+ store,
+ proxy: store,
+ namespace: GPU_NAMESPACE,
+ values,
+ get sealed() { return Boolean(store.sealed); },
+ };
+}
+
/**
/**
* Create both namespaces (the DDL is migrations/0028_config.sql), apply ai.exchange to the live config object,
* Create both namespaces (the DDL is migrations/0028_config.sql), apply ai.exchange to the live config object,
* load the provider-key cache and reseal plaintext/old-KEK rows once when OV_CONFIG_KEKS is set.
* load the provider-key cache and reseal plaintext/old-KEK rows once when OV_CONFIG_KEKS is set.
@@ -270,12 +377,13 @@ async function createSettings({ db, config, registry, clock = { now: () => Date.
@@ -270,12 +377,13 @@ async function createSettings({ db, config, registry, clock = { now: () => Date.
applyExchange(config, exchange.get());
applyExchange(config, exchange.get());
const providerKeys = createProviderKeys({ db, registry, env, log, onChange });
const providerKeys = createProviderKeys({ db, registry, env, log, onChange });
await providerKeys.refresh();
await providerKeys.refresh();
+ const gpus = await createGpuSettings({ db, env, log, onChange });
if (exchange.sealed) await exchange.reseal().catch((err) => log.warn(`[config] ${EXCHANGE_NAMESPACE}: reseal failed: ${err.message}`));
if (exchange.sealed) await exchange.reseal().catch((err) => log.warn(`[config] ${EXCHANGE_NAMESPACE}: reseal failed: ${err.message}`));
if (providerKeys.sealed) await providerKeys.store.reseal().catch((err) => log.warn(`[config] ${PROVIDER_KEYS_NAMESPACE}: reseal failed: ${err.message}`));
if (providerKeys.sealed) await providerKeys.store.reseal().catch((err) => log.warn(`[config] ${PROVIDER_KEYS_NAMESPACE}: reseal failed: ${err.message}`));
- return { exchange, providerKeys, onChange };
+ return { exchange, providerKeys, gpus, onChange };
}
}
module.exports = {
module.exports = {
- EXCHANGE_NAMESPACE, PROVIDER_KEYS_NAMESPACE, SERVICE, SELF, EXCHANGE_SCHEMA,
- applyExchange, originOf, providerKeysSchema, createSettings,
+ EXCHANGE_NAMESPACE, PROVIDER_KEYS_NAMESPACE, GPU_NAMESPACE, SERVICE, SELF, EXCHANGE_SCHEMA, GPU_SCHEMA, GPU_ORIGINS,
+ applyExchange, originOf, providerKeysSchema, createGpuSettings, createSettings,
};
};
new file mode 100644
new file mode 100644
@@ -0,0 +1,70 @@
@@ -0,0 +1,70 @@
+'use strict';
+/**
+ * The recorded marketplace: the answers Vast.ai gave when the offer search, the rental, the instance,
+ * the list and the health check ran (test/fixtures/gpu/vast.json). The fake serves them, so no test
+ * reaches the network and every run sees the same offers. The instance answers are the recorded ones
+ * with the port the rental was given.
+ *
+ * api.dead the first endpoint stops answering (the instance died)
+ * api.destroyFails every destroy is refused (the marketplace will not give the instance back)
+ * api.extra the instances the account holds that carry our label (the orphans)
+ */
+const fs = require('fs');
+const path = require('path');
+
+const RECORDED = JSON.parse(fs.readFileSync(path.join(__dirname, 'gpu', 'vast.json'), 'utf8'));
+
+/** A fake Vast.ai. */
+function createGpuFetch() {
+ const calls = { search: 0, create: [], show: 0, destroy: [], health: 0, list: 0 };
+ const rented = new Map(); // the instances this account rented, id -> { id, label, state }
+ let created = 0;
+ const api = {
+ calls,
+ dead: false,
+ destroyFails: false,
+ extra: [],
+ fetchImpl: async (url, opts = {}) => {
+ const u = String(url);
+ const method = String(opts.method || 'GET');
+ const reply = (status, body) => ({ ok: status >= 200 && status < 300, status, text: async () => JSON.stringify(body) });
+ if (u.includes('/api/v0/bundles')) {
+ calls.search += 1;
+ return reply(200, RECORDED.search);
+ }
+ if (u.includes('/api/v0/asks/')) {
+ const body = JSON.parse(opts.body || '{}');
+ calls.create.push({ url: u, body });
+ created += 1;
+ const id = 'inst-' + created;
+ rented.set(id, { id, label: String(body.label || ''), state: 'running' });
+ return reply(200, { success: true, new_contract: id });
+ }
+ if (u.includes('/api/v1/instances')) {
+ calls.list += 1;
+ return reply(200, { instances: [...rented.values(), ...api.extra] });
+ }
+ if (u.includes('/api/v0/instances/')) {
+ if (method === 'DELETE') {
+ calls.destroy.push(u);
+ if (api.destroyFails) return reply(500, { detail: 'the marketplace refused to give the instance back' });
+ rented.delete(u.split('/').pop());
+ return reply(200, { success: true, msg: 'destroyed' });
+ }
+ calls.show += 1;
+ const id = u.split('/').pop();
+ const port = 12345 + Math.max(0, created - 1);
+ return reply(200, { ...RECORDED.instance, id, ports: { 8000: { port, proxy: 'https://gw.vast.ai:' + port } } });
+ }
+ if (u.includes('/v1/models')) {
+ calls.health += 1;
+ if (api.dead && u.includes(':12345')) return reply(503, { error: 'the endpoint stopped answering' });
+ return reply(200, { object: 'list', data: [{ id: 'deepseek-r1-32b' }] });
+ }
+ return reply(404, { detail: 'no recorded answer for ' + method + ' ' + u });
+ },
+ };
+ return api;
+}
+
+module.exports = { createGpuFetch, RECORDED };
new file mode 100644
new file mode 100644
@@ -0,0 +1,52 @@
@@ -0,0 +1,52 @@
+{
+ "search": {
+ "offers": [
+ {
+ "id": 222,
+ "gpu_name": "RTX 3090",
+ "gpu_ram": 24576,
+ "num_gpus": 1,
+ "disk_space": 120,
+ "dph_total": "0.30",
+ "reliability": 0.995,
+ "geolocation": "US",
+ "rentable": true,
+ "verified": "verified"
+ },
+ {
+ "id": 111,
+ "gpu_name": "RTX 3090",
+ "gpu_ram": 24576,
+ "num_gpus": 1,
+ "disk_space": 100,
+ "dph_total": "0.1348",
+ "reliability": 0.998,
+ "geolocation": "US",
+ "rentable": true,
+ "verified": "verified"
+ },
+ {
+ "id": 333,
+ "gpu_name": "RTX 4090",
+ "gpu_ram": 24576,
+ "num_gpus": 1,
+ "disk_space": 200,
+ "dph_total": "0.20",
+ "reliability": 0.999,
+ "geolocation": "US",
+ "rentable": true,
+ "verified": "verified"
+ }
+ ]
+ },
+ "instance": {
+ "actual_status": "running",
+ "ssh_node": "ssh1.vast.ai",
+ "ports": {
+ "8000": {
+ "port": 12345,
+ "proxy": "https://gw.vast.ai:12345"
+ }
+ }
+ }
+}
\ No newline at end of file
\ No newline at end of file
new file mode 100644
new file mode 100644
@@ -0,0 +1,256 @@
@@ -0,0 +1,256 @@
+'use strict';
+/**
+ * The GPU provisioner: the cheapest offer that fits the model, the monthly cap it never passes, the
+ * idle instance that scales to zero and the instance that dies. The marketplace answers are recorded
+ * (test/fixtures/gpu/vast.json) and the clock is a fake one, so no test reaches the network and no
+ * test waits for real time.
+ */
+const assert = require('assert');
+const { boot, suite } = require('./helpers');
+const { createGpuFetch } = require('./fixtures/gpu-fetch');
+
+const T0 = Date.parse('2026-11-06T12:00:00Z');
+const OWNER = { type: 'user', id: 'usr_' + 'A'.repeat(26) };
+
+/** The owner's GPU settings: the marketplace key (a secret) and the monthly cap (money). */
+async function settings(h, { key = 'vast-test-key', cap = '50000000000', idle = null, maxRuntime = null } = {}) {
+ const store = h.settings.gpus.store;
+ const values = { monthly_cap: Number(cap) };
+ if (idle != null) values.idle_minutes = idle;
+ if (maxRuntime != null) values.max_runtime_minutes = maxRuntime;
+ await store.apply(values, { merge: true, actor: OWNER, reason: 'the test' });
+ if (key) await store.setSecret('vast.key', key, { actor: OWNER, reason: 'the test' });
+}
+
+
+/**
+ * The row a crash between the marketplace's answer and the row's update leaves behind: the row is
+ * there (it was written before the rental) but it does not know the marketplace id yet.
+ */
+async function crashedRow(h, id, at) {
+ const iso = (ms) => new Date(ms).toISOString();
+ await h.db.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,
+ gpu_name, gpu_ram_mb, num_gpus, disk_gb, reliability, region, price_per_hour, hourly_nano, reserved_nano, spent_nano,
+ max_runtime_minutes, started_at, charged_at, destroy_failures, label, cold_start_ms, health_failures, created_at, updated_at)
+ VALUES (?, ?, ?, 'vast', '111', NULL, 'starting', NULL, ?, 'RTX 3090', 24576, 1, 100, 0.998, 'US', '0.1348', '134800000', '134800000', 0,
+ 720, ?, ?, 0, ?, NULL, 0, ?, ?)`)
+ .run(id, 'deepseek-r1-32b', 'gpu.deepseek-r1-32b', '2026-11', iso(at), iso(at), id, iso(at), iso(at));
+}
+
+const t = suite('gpu');
+
+t.test('the provisioner is off without a key and without a cap', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ let res = await h.gpus.launch('deepseek-r1-32b');
+ assert.strictEqual(res.ok, false);
+ assert.strictEqual(res.code, 'gpu.off');
+ assert.match(res.reason, /API key/);
+ assert.strictEqual(api.calls.search, 0);
+ // a key, but no cap: still off
+ await settings(h, { cap: '0' });
+ res = await h.gpus.launch('deepseek-r1-32b');
+ assert.strictEqual(res.ok, false);
+ assert.strictEqual(res.code, 'gpu.off');
+ assert.strictEqual(api.calls.create.length, 0);
+ const st = await h.gpus.state();
+ assert.strictEqual(st.enabled, false);
+ } finally { await h.stop(); }
+});
+
+t.test('the cheapest offer that fits is rented, registered as an offer and health-checked', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '50000000000' }); // $50
+ const res = await h.gpus.launch('deepseek-r1-32b');
+ assert.ok(res.ok, res.reason);
+ // offer 111 (0.1348/h) is the cheapest of the three that fit
+ assert.strictEqual(api.calls.create.length, 1);
+ assert.ok(api.calls.create[0].url.includes('/asks/111/'), api.calls.create[0].url);
+ const st = await h.gpus.state();
+ assert.strictEqual(st.instances.length, 1);
+ assert.strictEqual(st.instances[0].offer_id, '111');
+ assert.strictEqual(st.instances[0].hourly_nano, '134800000');
+ assert.strictEqual(st.instances[0].status, 'starting');
+ assert.ok(BigInt(st.committed_nano) <= BigInt(st.cap_nano), 'the reservation is inside the cap');
+
+ // the next pass finds the endpoint, checks it and registers the offer for the router
+ now += 60_000;
+ await h.gpus.tick();
+ assert.ok(api.calls.health >= 1, 'the endpoint was health-checked');
+ const provider = await h.registry.getProvider('gpu.deepseek-r1-32b');
+ assert.ok(provider, 'the offer is registered');
+ assert.strictEqual(provider.status, 'active');
+ assert.strictEqual(provider.base_url, 'https://gw.vast.ai:12345/v1');
+ assert.strictEqual(provider.provider_access, 'OWN_DEPLOYMENT');
+ const model = await h.registry.getModel('gpu.deepseek-r1-32b', 'deepseek-r1-32b');
+ assert.ok(model, 'the model is registered');
+ assert.ok(model.cost.out_per_mtok > 0, 'the offer is priced');
+ } finally { await h.stop(); }
+});
+
+t.test('the monthly cap is never passed: the launch is refused and says why', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '1000000' }); // $0.001: less than an hour of a 0.1348/h GPU
+ const res = await h.gpus.launch('deepseek-r1-32b');
+ assert.strictEqual(res.ok, false);
+ assert.strictEqual(res.code, 'gpu.cap');
+ assert.match(res.reason, /cap/);
+ assert.strictEqual(api.calls.create.length, 0, 'nothing was rented');
+ const st = await h.gpus.state();
+ assert.strictEqual(st.instances.length, 0);
+ assert.strictEqual(st.spent_nano, '0');
+ } finally { await h.stop(); }
+});
+
+t.test('an idle instance scales to zero and the money is settled', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '50000000000', idle: 1 });
+ await h.gpus.launch('deepseek-r1-32b');
+ now += 60_000;
+ await h.gpus.tick(); // the instance comes up
+ assert.strictEqual((await h.gpus.state()).instances[0].status, 'running');
+ now += 90_000; // it has served nobody for more than an idle minute
+ await h.gpus.tick();
+ assert.strictEqual(api.calls.destroy.length, 1, 'the instance was given back');
+ const st = await h.gpus.state();
+ assert.strictEqual(st.instances[0].status, 'destroyed');
+ assert.ok(BigInt(st.spent_nano) > 0n, 'the money it cost is on the ledger');
+ assert.ok(BigInt(st.committed_nano) <= BigInt(st.cap_nano), 'the money is inside the cap');
+ const provider = await h.registry.getProvider('gpu.deepseek-r1-32b');
+ assert.strictEqual(provider.status, 'disabled', 'the offer leaves the pool');
+ } finally { await h.stop(); }
+});
+
+t.test('an instance that dies is replaced and the router gets the new endpoint', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '50000000000' });
+ await h.gpus.launch('deepseek-r1-32b');
+ now += 60_000;
+ await h.gpus.tick();
+ assert.strictEqual((await h.gpus.state()).instances[0].status, 'running');
+
+ api.dead = true; // the endpoint stops answering
+ now += 60_000;
+ await h.gpus.tick(); // the first failed check
+ now += 60_000;
+ await h.gpus.tick(); // the second: the instance is dead
+ assert.strictEqual(api.calls.destroy.length, 1, 'the dead instance was given back');
+ assert.strictEqual((await h.registry.getProvider('gpu.deepseek-r1-32b')).status, 'disabled');
+
+ now += 60_000;
+ await h.gpus.tick(); // a new instance is rented
+ assert.strictEqual(api.calls.create.length, 2, 'a second instance was rented');
+ now += 60_000;
+ await h.gpus.tick(); // and comes up
+ const provider = await h.registry.getProvider('gpu.deepseek-r1-32b');
+ assert.strictEqual(provider.status, 'active');
+ assert.strictEqual(provider.base_url, 'https://gw.vast.ai:12346/v1', 'the router got the new endpoint');
+ const st = await h.gpus.state();
+ assert.ok(BigInt(st.committed_nano) <= BigInt(st.cap_nano), 'the money is inside the cap');
+ } finally { await h.stop(); }
+});
+
+
+t.test('an instance the marketplace accepted before the row knew its id is recovered, not thrown away', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '50000000000' });
+ // the crash: the marketplace accepted the instance, the row never learned its id
+ const id = 'openvibe-abc123def456';
+ api.extra.push({ id: 'inst-crash', label: id, state: 'running' });
+ await crashedRow(h, id, now);
+ now += 60_000;
+ await h.gpus.tick();
+ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);
+ assert.strictEqual(row.remote_id, 'inst-crash', 'the row knows its instance again');
+ assert.strictEqual(row.status, 'running', 'and the instance serves');
+ assert.strictEqual(api.calls.destroy.length, 0, 'a paid instance of ours is never thrown away');
+ const audit = await h.registry.listAudit({ action: 'gpu.instance.recovered', targetId: id });
+ assert.strictEqual(audit.length, 1, 'the recovery is on the audit log');
+ } finally { await h.stop(); }
+});
+
+t.test('an instance with our label and no row of ours is found and given back', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '50000000000' });
+ const label = 'openvibe-111111111111';
+ api.extra.push({ id: 'inst-orphan', label, state: 'running' });
+ now += 60_000;
+ await h.gpus.tick();
+ assert.ok(api.calls.destroy.some((u) => u.includes('inst-orphan')), 'the orphan was given back');
+ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(label);
+ assert.strictEqual(row.status, 'destroyed');
+ const audit = await h.registry.listAudit({ action: 'gpu.orphan.destroyed' });
+ assert.strictEqual(audit.length, 1, 'the orphan is on the audit log');
+ assert.deepStrictEqual(await h.gpus.orphanCount(), [], 'no orphan is left waiting');
+ } finally { await h.stop(); }
+});
+
+t.test('a destroy that keeps failing is tried again every pass and the operator is alerted', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '50000000000' });
+ api.destroyFails = true;
+ const label = 'openvibe-222222222222';
+ api.extra.push({ id: 'inst-stuck', label, state: 'running' });
+ for (let i = 0; i < 3; i++) {
+ now += 60_000;
+ await h.gpus.tick();
+ }
+ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(label);
+ assert.strictEqual(Number(row.destroy_failures), 3, 'the tries are counted');
+ assert.strictEqual(row.status, 'orphan', 'and the instance is still running');
+ assert.strictEqual(api.calls.destroy.filter((u) => u.includes('inst-stuck')).length, 3, 'every pass tries again');
+ const gauges = await h.gpus.destroyFailures();
+ assert.strictEqual(gauges.length, 1);
+ assert.strictEqual(gauges[0].labels.marketplace, 'vast');
+ assert.strictEqual(gauges[0].value, 3, 'the gauge the operator alert reads');
+ const audit = await h.registry.listAudit({ action: 'gpu.orphan.destroy_failed' });
+ assert.strictEqual(audit.length, 1, 'the operator alert is on the audit log');
+ } finally { await h.stop(); }
+});
+
+t.test('a row whose rental never came back gives its reservation back', async () => {
+ const api = createGpuFetch();
+ let now = T0;
+ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
+ try {
+ await settings(h, { cap: '50000000000' });
+ // the crash before the marketplace answered: the row holds a reservation for an instance
+ // that does not exist
+ const id = 'openvibe-def456abc789';
+ await crashedRow(h, id, now);
+ const before = await h.gpus.moneySummary();
+ assert.ok(BigInt(before.committed_nano) > 0n, 'the row holds its reservation');
+ now += 11 * 60_000; // past the give-up time
+ await h.gpus.tick();
+ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);
+ assert.strictEqual(row.status, 'refused');
+ const after = await h.gpus.moneySummary();
+ assert.strictEqual(BigInt(after.committed_nano), 0n, 'the reservation is free again');
+ } finally { await h.stop(); }
+});
+
+if (require.main === module) t.run();
+module.exports = { suite: t };