openvibers/openvibe.ai Commits

Add GPU rental strategies and preserve financial reservations on uncertain calls (#369)

OP OpenVibers committed 5 days ago · d495ca8863 · 20 files, +429 −133

* Add GPU strategy previews and protect reservations across uncertain rentals

* Require bounded GPU variable costs and document quote API

* Confirm GPU deletion before releasing financial commitments

* Merge Shared dependency update into GPU broker branch

* Pin openvibe-sdk v0.100.0 (was commit fe19b9d9)

Replace the openvibe-sdk commit pin with the released v0.100.0 tag and
refresh the lockfile. Update STATUS.json's SDK line.

* Merge origin/main; keep openvibe-sdk v0.100.0 with shared v4.153.0

main moved to openvibe-shared v4.153.0 while this branch pins openvibe-sdk
v0.100.0. Resolve the pin conflict in package.json/STATUS.json keeping the
new SDK tag plus main's shared tag, and regenerate the lockfile.

Changes

ddiff --git a/STATUS.json b/STATUS.json
iindex 459b34e..da69262 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -8,7 +8,7 @@
"port": 4700,
"domain": "ai.openvibe.services (the old ai.openvibe.network address was retired on 2026-10-10); the service runs on openvibe-ovh (/opt/openvibe.ai, unit openvibe-ai) on 127.0.0.1:4700 behind nginx; OpenVibe.Live and OpenVibe.Network call it on loopback",
"contracts": "openvibe-contracts v0.338.0 (ai.config.manage and the common.config-* contracts carry D11; billing.credit-hold@1 takes a nullable project_id for personal holds)",
-- "sdk": "openvibe-sdk v0.94.0",
++ "sdk": "openvibe-sdk v0.100.0",
"shared": "openvibe-shared v4.153.0",
"capabilities": [
"ai.run.create",
ddiff --git a/docs/ai-gpus.md b/docs/ai-gpus.md
iindex 41a941d..73cb609 100644
---- a/docs/ai-gpus.md
++++ b/docs/ai-gpus.md
@@@ -2,10 +2,10 @@
The design (§2 "Our own GPUs", the Exchange remake) asks for a provisioner that rents a GPU from a
marketplace, serves a model on it, registers the endpoint as an offer for the router and scales the
--whole thing to zero when nobody is using it - under a hard monthly cap the owner sets.
++whole thing to zero when nobody is using it - under an owner-set monthly reservation cap.
This is that provisioner. It lives in AI (server/gpu/) and it is **off until the owner has set a
--marketplace API key and a monthly cap**.
++marketplace API key, a monthly cap and a confirmed bound on extra storage/network costs**.
## The owner settings (the namespace `ai.gpus`)
@@@ -23,7 +23,7 @@ The owner changes them the way every other AI setting changes: through Network's
proxies to AI's `/internal/config/ai.gpus`. The keys are write-only (the panel never shows a value
again), the origins are written by AI, and the cap is a money key.
--Without a key **or** without a cap the provisioner does nothing: no search, no instance, no money.
++Without a key **or** without a cap no new instance is rented. Read-only quotes work separately; cleanup of existing instances continues even if the cap is disabled.
## What it does
@@@ -69,8 +69,8 @@ The money is integer nano-USD in BigInt, never a float (the same rule AI's Credi
instance holds a **reservation** - the budget it may cost - before it starts, and the reservation
is part of the month's committed spend. A new instance is only started when its reservation still
fits under the cap; if it does not, the launch is refused and the reason is returned. The instance is
--given back when its reservation has run out, so the money stops there. The worst case (the
--committed spend) therefore never passes the cap.
++scheduled for release when its reservation runs out. New reservations cannot pass the cap;
++provider deletion failures can continue billing beyond it and require operator intervention.
Vast and RunPod bill **per second**, so the money is charged for the seconds an instance really ran
(the ledger rounds up to the nano, so it never charges less than the GPU was paid for). The console page
@@@ -114,7 +114,7 @@ The feature is off while **any** of the three is missing, and the public
`GET /api/v1/exchange/endpoints` answers the plain reason the Exchange page shows:
1. `ai.gpus` `vast.key` or `runpod.key` (the owner's GPU provider key),
--2. `ai.gpus` `monthly_cap` (the hard monthly cap),
++2. `ai.gpus` `monthly_cap` (the monthly reservation cap), plus `costs_bounded=true` and an explicit `extra_hourly_nano`,
3. Balance billing (`AI_CREDITS` and Billing), because an endpoint is paid for per second.
The first two are `x-owner-input` fields (group `GPUs`, `x-required-for: "own endpoints"`) so the admin
@@@ -127,7 +127,7 @@ panel's Setup tab lists them with what they turn on and where to get them.
- it is off without a key and without a cap,
- the cheapest offer that fits is rented, registered as an offer and health-checked,
--- the monthly cap is never passed: the launch is refused and says why,
++- a launch that would exceed the monthly reservation cap is refused with a reason,
- an idle instance scales to zero and the money is settled,
- an instance that dies is replaced and the router gets the new endpoint,
- an instance the marketplace accepted before the row knew its id is recovered, not thrown away,
@@@ -150,3 +150,42 @@ local fake Billing and a canned GPU answer:
The weights URL, the image and the container command are data in `server/gpu/catalog.js` - they are
what the first launch really runs, and the console page shows them. The owner checks them (and
corrects them there) before the first rented instance starts.
++
++
++## Compute broker strategies and quote previews
++
++`ai.gpus.strategy` selects `cheapest` (default), `fastest`, or `automatic` across configured
++Vast and RunPod marketplaces. `private` and `own` refuse third-party GPU rentals; those need
++an existing Link/OpenVibe endpoint. Region remains a hard constraint. Without measured
++latency, Fastest explicitly reports that it falls back to price within the required region.
++`ai.gpus.dry_run=true` makes shared-model launch requests quote-only and disables new private
++endpoints. Keys and the positive `monthly_cap` (nano-USD) are still required for live rentals.
++Also explicitly set `ai.gpus.extra_hourly_nano` to a conservative per-instance hourly bound
++for disk/network charges (including zero if appropriate), and `ai.gpus.costs_bounded=true`
++only after enforcing those limits in the runtime/network. Both default to blocking live
++rentals. The allowance is added to reservations, hourly prices and metered cost estimates;
++these estimates are not a reconciliation against the provider invoice. Existing paid
++instances continue cleanup when these switches are unset. Quote previews show the base GPU
++price; the live placement audit records base, allowance and total separately.
++No new accounts, payment methods or provider keys are created.
++
++`GET /console/gpus/:model/quote?strategy=cheapest` previews without a Billing hold or provider
++purchase. Exchange uses the public, minute-cached `/api/v1/exchange/gpu-quotes/:model` and can
++preview even when rentals are disabled. A private-endpoint create may include `strategy`;
++the choice persists on the endpoint. Every live GPU choice is recorded as `gpu.placement`
++in the existing audit log, including marketplace, offer, strategy, reasons and hourly cost.
++
++Reservations are serialized through `gpu_budget_lock`. An uncertain creation result keeps
++its row and reservation for label recovery, including across month boundaries; an empty or
++failed instance listing cannot prove a purchase never happened. Disabling the cap continues
++remote cleanup. Metering splits elapsed time at UTC month boundaries and avoids counting
++already-charged runtime twice in the live summary. Release begins a minute before the runtime
++limit; an accepted DELETE retains the reservation until a follow-up lookup confirms absence.
++Failed or pending deletion remains a billable risk;
++the existing destroy-failure alert must be monitored.
++
++RunPod uses the official v2 catalogue (`/v2/catalog/gpus?include=AVAILABILITY&product=POD`),
++GiB-to-MiB conversion, secure-cloud GPU counts, region filtering, numeric disk size and an env
++map. Vast rents with PUT, not POST. Tests stub both APIs; no live purchase was made.
++
++Source: https://github.com/runpod/runpod-plugins-official/blob/main/plugins/runpod/skills/runpod-migrate/reference/graphql-to-v2.md
ddiff --git a/migrations/0048_gpu_budget_lock.sql b/migrations/0048_gpu_budget_lock.sql
nnew file mode 100644
iindex 0000000..2592fd2
---- /dev/null
++++ b/migrations/0048_gpu_budget_lock.sql
@@@ -0,0 +1,4 @@
++-- phase: expand
++CREATE TABLE IF NOT EXISTS gpu_budget_lock (id integer PRIMARY KEY);
++INSERT INTO gpu_budget_lock (id) VALUES (1) ON CONFLICT DO NOTHING;
++ALTER TABLE own_endpoints ADD COLUMN IF NOT EXISTS strategy text NOT NULL DEFAULT 'cheapest';
ddiff --git a/package-lock.json b/package-lock.json
iindex 6650839..ede3c25 100644
---- a/package-lock.json
++++ b/package-lock.json
@@@ -14,7 +14,7 @@
"express": "^4.21.2",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.338.0",
-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.94.0",
++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.100.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.153.0",
"pg": "^8.23.0"
},
@@@ -1559,9 +1559,9 @@
}
},
"node_modules/openvibe-sdk": {
-- "version": "0.94.0",
-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.94.0",
-- "integrity": "sha512-p3buZSOSFV81OdZgs5GoSIfMCQNJjW23eAIXIt0RVFIOvh3TByNNEMmuCanxRybFIbzh+9IDMbBwHJrBLl3JeA==",
++ "version": "0.100.0",
++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.100.0",
++ "integrity": "sha512-P7Mi0jqC4PACvNZKgZQqGumUYSAin5X1B1ZTpyvcZ/6P9CIg6xDZew6Zmn/Hqp5VafqVKg09Y6VbEBjj2krtQg==",
"license": "MIT",
"bin": {
"openvibe-download": "bin/openvibe-download.js"
ddiff --git a/package.json b/package.json
iindex 8a1217d..02f9ff3 100644
---- a/package.json
++++ b/package.json
@@@ -30,7 +30,7 @@
"express": "^4.21.2",
"iovalkey": "^0.4.0",
"openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.338.0",
-- "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.94.0",
++ "openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.100.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.153.0",
"pg": "^8.23.0"
},
ddiff --git a/server/api/exchange.js b/server/api/exchange.js
iindex 74b80a4..6c0b696 100644
---- a/server/api/exchange.js
++++ b/server/api/exchange.js
@@@ -160,6 +160,25 @@ function exchangeRouter({ db, registry, pool, quotas = null, env = process.env,
// Your own endpoint: whether it is on and, when it is off, the plain reason. Public: the Exchange page
// reads it before anyone signs in, and nothing here is per person.
++ // Public, non-personal, read-only previews. Cache each model/strategy for one minute to avoid
++ // turning this public page into unlimited provider searches. Never expose account keys or ids.
++ const gpuQuotes = new Map();
++ r.get('/api/v1/exchange/gpu-quotes/:model', publicCache(30), async (req, res, next) => {
++ try {
++ if (!ownEndpoints) return res.status(503).json({ error: 'GPU quotes unavailable' });
++ const model = String(req.params.model);
++ const strategy = String(req.query.strategy || 'cheapest');
++ if (!ownEndpoints.status().models.some(m => m.key === model) || !['automatic','cheapest','fastest','private','own'].includes(strategy)) return res.status(422).json({ error: 'unknown model or strategy' });
++ const key = model + ':' + strategy;
++ let entry = gpuQuotes.get(key);
++ if (!entry || entry.until < clock.now()) {
++ entry = { until: clock.now() + 60000, promise: ownEndpoints.preview(model, { strategy }) };
++ gpuQuotes.set(key, entry);
++ }
++ res.json(await entry.promise);
++ } catch (err) { next(err); }
++ });
++
r.get('/api/v1/exchange/endpoints', publicCache(30), (req, res) => {
res.json(ownEndpoints ? ownEndpoints.status() : {
enabled: false, reason: 'The GPU provider is not set up yet, so no endpoint can start.', reasons: ['The GPU provider is not set up yet, so no endpoint can start.'],
ddiff --git a/server/console/index.js b/server/console/index.js
iindex 7daf9aa..9ea84ed 100644
---- a/server/console/index.js
++++ b/server/console/index.js
@@@ -562,6 +562,9 @@ function consoleRouter({ config, db, registry, pool, quotas, cache, runs, ops, e
state: await gpus.state(),
}));
}));
++ r.get('/console/gpus/:model/quote', read, wrap(async (req, res) => {
++ res.json(await gpus.preview(String(req.params.model), { strategy: req.query.strategy || undefined }));
++ }));
r.post('/console/gpus/:model/:action', needs(CAPS.providerManage), wrap(async (req, res) => {
await gpus.request(String(req.params.model || ''), String(req.params.action || ''), req.staff.subject);
res.redirect(303, '/console/gpus?done=gpu_requested');
ddiff --git a/server/console/views.js b/server/console/views.js
iindex 5569d38..605b1ca 100644
---- a/server/console/views.js
++++ b/server/console/views.js
@@@ -694,9 +694,11 @@ function gpus({ staff, csrf, state, canManage, notice, error }) {
<div class="card"><div class="k">Could still cost</div><div class="v">${usdNano(st.committed_nano)}</div><div class="muted">the most the GPUs running now may cost before they are given back</div></div>
<div class="card"><div class="k">Monthly cap</div><div class="v">${st.cap_nano ? usdNano(st.cap_nano) : 'Not set'}</div><div class="muted">${st.cap_nano ? 'AI never starts a GPU that could pass it' : 'without a cap nothing is rented'}</div></div>
</div>
++<p>Placement: ${st.strategy || 'cheapest'} · ${st.dry_run ? 'Quote only' : 'Renting allowed within the cap'}. <a href="https://openvibe.run/api/v1/compute">Desktop and browser pools</a></p>
<p class="muted">An unused GPU is given back after ${n(st.idle_minutes)} minute${Number(st.idle_minutes) === 1 ? '' : 's'}; one GPU runs at most ${n(st.max_runtime_minutes)} minutes; ${st.warm_window ? `GPUs stay ready ${st.warm_window} UTC` : 'no hours keep GPUs ready'}; GPUs are rented in ${st.region}. Vast.ai key: ${mk.vast ? 'set' : 'not set'} · RunPod key: ${mk.runpod ? 'set' : 'not set'}.</p>
<p class="muted">Vast.ai and RunPod bill by the second, so only the seconds a GPU really ran are charged; the price per second is next to the price per hour. "First answer after" is the time from renting a GPU to its first answer.</p>
${table('Models that can run on a rented GPU', ['Model', 'Marketplace', 'Runs with', 'GPU', 'Status', 'First answer after', 'Actions'], models.map((m) => gpuModelRow(m, canManage, csrf)), 'no model is set up to run on a rented GPU yet.')}
++<p>Preview without renting: ${models.map(m => html`<a href="/console/gpus/${enc(m.key)}/quote">${m.display_name}</a> `)}</p>
${table('GPUs rented', ['Model', 'Marketplace', 'GPU', 'Price per hour', 'Status', 'Address', 'Started', 'First answer after', 'Most it may cost', 'Spent'], instances.map((i) => gpuInstanceRow(i)), 'no GPU has been rented.')}
`,
});
ddiff --git a/server/gpu/endpoints.js b/server/gpu/endpoints.js
iindex 01d73a7..7f76df2 100644
---- a/server/gpu/endpoints.js
++++ b/server/gpu/endpoints.js
@@@ -87,7 +87,9 @@ function createOwnEndpoints({ db, gpus, credits, config = null, clock = { now: (
if (!billing) reasons.push(BILLING_OFF);
const marketplace = cfg.vast_key ? 'vast' : (cfg.runpod_key ? 'runpod' : null);
return {
-- enabled: Boolean(cfg.enabled && billing),
++ enabled: Boolean(cfg.enabled && billing && !cfg.dry_run),
++ strategy: cfg.strategy || 'cheapest',
++ dry_run: Boolean(cfg.dry_run),
reason: reasons.join(' '),
reasons,
marketplace,
@@@ -131,6 +133,7 @@ function createOwnEndpoints({ db, gpus, credits, config = null, clock = { now: (
id: row.id,
status: row.status,
source: row.source,
++ strategy: row.strategy || 'cheapest',
display_name: row.display_name,
catalog_key: row.catalog_key,
weights: row.source === 'weights' ? { repo: row.weights_repo, file: row.weights_file, lora_repo: row.lora_repo, lora_file: row.lora_file } : null,
@@@ -219,6 +222,8 @@ function createOwnEndpoints({ db, gpus, credits, config = null, clock = { now: (
const st = status();
if (!st.enabled) throw new AiError(409, 'own_endpoint.off', st.reason || 'Your own endpoint is off.');
const cfg = gpuConfig();
++ const strategy = body.strategy || cfg.strategy || 'cheapest';
++ if (!['automatic', 'cheapest', 'fastest'].includes(strategy)) throw new AiError(422, 'input.invalid', 'marketplace rentals support automatic, cheapest or fastest; private and own require a Link machine');
const active = await db.prepare(`SELECT COUNT(*) AS n FROM own_endpoints WHERE subject = ? AND status IN (${ACTIVE.map(() => '?').join(', ')})`)
.get(String(subject), ...ACTIVE);
if (Number(active && active.n) >= MAX_ACTIVE) throw new AiError(409, 'own_endpoint.limit', 'You already have ' + MAX_ACTIVE + ' endpoints running or starting. Stop one first.');
@@@ -251,11 +256,11 @@ function createOwnEndpoints({ db, gpus, credits, config = null, clock = { now: (
const t = now();
const id = 'oep_' + ids.ulid();
await db.prepare(`INSERT INTO own_endpoints (id, subject, status, source, catalog_key, display_name, weights_repo, weights_file,
-- lora_repo, lora_file, context_window, idle_minutes, max_runtime_minutes, month, created_at, updated_at)
-- VALUES (?, ?, 'requested', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(
++ lora_repo, lora_file, context_window, idle_minutes, max_runtime_minutes, month, created_at, updated_at, strategy)
++ VALUES (?, ?, 'requested', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(
id, String(subject), source, spec ? spec.key : null, display,
weights ? weights.repo : null, weights ? weights.file : null, weights ? weights.loraRepo || null : null, weights ? weights.loraFile || null : null,
-- spec ? spec.context_window : 8192, idleMinutes, maxRuntime, at(t).slice(0, 7), at(t), at(t));
++ spec ? spec.context_window : 8192, idleMinutes, maxRuntime, at(t).slice(0, 7), at(t), at(t), strategy);
return describe(await rowById(id));
}
@@@ -300,6 +305,7 @@ function createOwnEndpoints({ db, gpus, credits, config = null, clock = { now: (
const res = await gpus.launchSpec(spec, {
actor: 'user:' + row.subject,
ownEndpointId: row.id,
++ strategy: row.strategy || undefined,
idleMinutes: row.idle_minutes,
maxRuntimeMinutes: row.max_runtime_minutes,
beforeCreate: async ({ offer, hourly, reserved, maxMinutes, marketplace }) => {
@@@ -491,7 +497,7 @@ function createOwnEndpoints({ db, gpus, credits, config = null, clock = { now: (
return { status: res.status, contentType: res.headers.get('content-type') || 'application/json', body: res.body };
}
-- return { status, create, list, get, stop, tick, specFor, ready, afterDestroy, forward, describe, settle };
++ return { status, create, list, get, stop, tick, specFor, ready, afterDestroy, forward, describe, settle, preview: (key, options) => gpus.preview(key, options) };
}
module.exports = { createOwnEndpoints, MAX_ACTIVE, MAX_IDLE_MINUTES, MAX_RUNTIME_MINUTES, BILLING_OFF, plainGpuReason, reasonText };
ddiff --git a/server/gpu/http.js b/server/gpu/http.js
iindex 8e82867..1416b71 100644
---- a/server/gpu/http.js
++++ b/server/gpu/http.js
@@@ -5,22 +5,23 @@
*/
const { AiError } = require('../util');
--async function apiJson(fetchImpl, url, { method = 'GET', body = null, key = null, label = 'the marketplace' } = {}) {
++async function apiJson(fetchImpl, url, { method = 'GET', body = null, key = null, label = 'the marketplace', allowMissing = false } = {}) {
const headers = { Accept: 'application/json' };
if (body != null) headers['Content-Type'] = 'application/json';
if (key) headers.Authorization = 'Bearer ' + key;
let res;
try {
-- res = await fetchImpl(url, { method, headers, body: body != null ? JSON.stringify(body) : undefined });
++ res = await fetchImpl(url, { method, headers, redirect: 'error', signal: AbortSignal.timeout(15000), body: body != null ? JSON.stringify(body) : undefined });
} catch (err) {
-- throw new AiError(502, 'gpu.marketplace_unreachable', label + ' did not answer: ' + String(err && err.message).slice(0, 200));
++ throw new AiError(502, 'gpu.marketplace_unreachable', label + ' did not answer');
}
const text = await res.text().catch(() => '');
let json = null;
try { json = text ? JSON.parse(text) : null; } catch { json = null; }
++ if (res.status === 404 && allowMissing) return { missing: true };
if (!res.ok) {
const detail = (json && (json.detail || json.message || json.error)) || String(res.status);
-- throw new AiError(502, 'gpu.marketplace_error', label + ' refused: ' + String(detail).slice(0, 200) + ' (' + res.status + ')');
++ throw new AiError(502, 'gpu.marketplace_error', label + ' refused (' + res.status + ')' );
}
return json;
}
ddiff --git a/server/gpu/provisioner.js b/server/gpu/provisioner.js
iindex 35b8f6f..2cef339 100644
---- a/server/gpu/provisioner.js
++++ b/server/gpu/provisioner.js
@@@ -19,6 +19,7 @@
* deterministic: the same offers in the same order always give the same instance.
*/
const { randomUUID } = require('node:crypto');
++const { choose, STRATEGIES } = require('openvibe-sdk/compute');
const { AiError } = require('../util');
const money = require('./money');
const catalog = require('./catalog');
@@@ -39,8 +40,6 @@ const LABEL_RE = /^openvibe-[0-9a-f]{12}$/;
// Three failed destroys in a row: the instance is still running and still being paid for, so the
// operator is alerted (the gauge ai_gpu_destroy_failures and the rule in OpenVibe.Host's prometheus rules).
const ALERT_AFTER = 3;
--// A row whose rental never came back is given up on after this long: its reservation is free again.
--const STALE_ROW_MS = 10 * 60_000;
const PRICE_SOURCE = { vast: 'https://vast.ai/pricing', runpod: 'https://www.runpod.io/pricing' };
function clampInt(value, min, max, fallback) {
@@@ -102,20 +101,27 @@ function createGpuProvisioner({
} catch {
cap = null;
}
++ let extraHourly = null;
++ try { extraHourly = money.nanoCount(v.extra_hourly_nano, 'the hourly storage and network allowance'); } catch {}
++ const bounded = v.costs_bounded === true && extraHourly != null && extraHourly >= 0n;
const vastKey = String(v.vast_key || '');
const runpodKey = String(v.runpod_key || '');
let reason = '';
if (!vastKey && !runpodKey) reason = 'no Vast.ai or RunPod API key';
else if (cap == null || cap <= 0n) reason = 'no monthly cap';
++ else if (!bounded) reason = 'storage and network costs need an explicit hourly bound';
return {
++ strategy: STRATEGIES.includes(v.strategy) ? v.strategy : 'cheapest',
++ dry_run: v.dry_run === true,
vast_key: vastKey,
runpod_key: runpodKey,
cap_nano: cap,
++ extra_hourly_nano: extraHourly,
idle_minutes: clampInt(v.idle_minutes, 1, 1440, DEFAULT_IDLE_MINUTES),
max_runtime_minutes: clampInt(v.max_runtime_minutes, 1, 10080, DEFAULT_MAX_RUNTIME_MINUTES),
warm_window: String(v.warm_window || '').trim(),
region: String(v.region || DEFAULT_REGION).trim().toUpperCase() || DEFAULT_REGION,
-- enabled: Boolean(vastKey || runpodKey) && cap != null && cap > 0n,
++ enabled: Boolean(vastKey || runpodKey) && cap != null && cap > 0n && bounded,
reason,
};
}
@@@ -147,18 +153,21 @@ function createGpuProvisioner({
// ── the money ─────────────────────────────────────────────────────────
/** The month's money: what was really spent, and what is committed (the worst case). */
-- async function moneySummary() {
++ async function moneySummary(handle = db) {
const month = monthOf(now());
const cfg = config();
-- const ledger = await db.prepare('SELECT COALESCE(SUM(nano_usd), 0) AS total FROM gpu_spend WHERE month = ?').get(month);
++ const ledger = await handle.prepare('SELECT COALESCE(SUM(nano_usd), 0) AS total FROM gpu_spend WHERE month = ?').get(month);
const settled = toNano(ledger && ledger.total);
-- const rows = await db.prepare('SELECT * FROM gpu_instances WHERE month = ? AND status IN (?, ?, ?)').all(month, 'starting', 'running', 'stopping');
++ const rows = await handle.prepare('SELECT * FROM gpu_instances WHERE status IN (?, ?, ?)').all('starting', 'running', 'stopping');
let reserved = 0n;
let running = 0n;
for (const row of rows) {
-- reserved += toNano(row.reserved_nano);
++ const remaining = toNano(row.reserved_nano) > toNano(row.spent_nano) ? toNano(row.reserved_nano) - toNano(row.spent_nano) : 0n;
const started = toMs(row.started_at);
-- if (started != null) running += money.nanoForSeconds(toNano(row.hourly_nano), (now() - started) / 1000);
++ const accrued = started == null ? 0n : money.nanoForSeconds(toNano(row.hourly_nano), Math.max(0, now() - Math.max(toMs(row.charged_at) || started, Date.parse(month + '-01T00:00:00Z'))) / 1000);
++ running += accrued;
++ // Pending deletion may run beyond its reservation. Never make that overspend look like free budget.
++ reserved += remaining > accrued ? remaining : accrued;
}
return {
month,
@@@ -172,17 +181,24 @@ function createGpuProvisioner({
/** Write the money an instance has cost since the last charge. */
async function charge(row, until, reason) {
++ return db.tx(async (tdb) => {
++ row = await tdb.prepare('SELECT * FROM gpu_instances WHERE id = ? FOR UPDATE').get(row.id) || row;
const hourly = toNano(row.hourly_nano);
const from = toMs(row.charged_at) || toMs(row.started_at) || until;
-- const seconds = Math.max(0, (until - from) / 1000);
-- if (seconds <= 0) return 0n;
-- const amount = money.nanoForSeconds(hourly, seconds);
-- if (amount <= 0n) return 0n;
-- await db.prepare('INSERT INTO gpu_spend (at, month, kind, nano_usd, instance_id, detail, actor, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)')
-- .run(iso(until), monthOf(until), 'charge', amount.toString(), row.id, reason || 'the GPU ran', 'service:ai', iso(now()));
-- await db.prepare('UPDATE gpu_instances SET charged_at = ?, spent_nano = spent_nano + ?, updated_at = ? WHERE id = ?')
++ if (until <= from) return 0n;
++ let amount = 0n;
++ for (let start = from; start < until;) {
++ const d = new Date(start);
++ const end = Math.min(until, Date.UTC(d.getUTCFullYear(), d.getUTCMonth() + 1, 1));
++ const part = money.nanoForSeconds(hourly, (end - start) / 1000);
++ await tdb.prepare('INSERT INTO gpu_spend (at, month, kind, nano_usd, instance_id, detail, actor, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?)')
++ .run(iso(end), monthOf(start), 'charge', part.toString(), row.id, reason || 'the GPU ran', 'service:ai', iso(now()));
++ amount += part; start = end;
++ }
++ await tdb.prepare('UPDATE gpu_instances SET charged_at = ?, spent_nano = spent_nano + ?, updated_at = ? WHERE id = ?')
.run(iso(until), amount.toString(), iso(now()), row.id);
return amount;
++ });
}
// ── the instances ─────────────────────────────────────────────────────
@@@ -198,16 +214,35 @@ function createGpuProvisioner({
return rows || [];
}
-- /** The cheapest offer that fits. The same offers in the same order always give the same one. */
-- function cheapest(offers) {
-- const usable = (offers || []).filter((o) => o && o.price_per_hour != null && Number(o.price_per_hour) > 0);
-- usable.sort((a, b) => {
-- const pa = Number(a.price_per_hour);
-- const pb = Number(b.price_per_hour);
-- if (pa !== pb) return pa - pb;
-- return String(a.id) < String(b.id) ? -1 : String(a.id) > String(b.id) ? 1 : 0;
-- });
-- return usable[0] || null;
++ /** A quote never creates a request, takes a Billing hold or rents anything. */
++ async function preview(modelKey, { strategy = config().strategy, region = config().region, runtimeSpec = null, rentalOnly = false } = {}) {
++ const spec = runtimeSpec || models.find(m => m.key === modelKey) || catalog.find(modelKey);
++ if (!spec) throw new AiError(404, 'ai.not_found', 'no such GPU model');
++ if (!STRATEGIES.includes(strategy)) throw new AiError(422, 'ai.invalid', 'unknown compute strategy');
++ if (strategy === 'own' || strategy === 'private') return { dry_run: true, strategy, selected: null, offers: [], reasons: ['rented marketplace GPUs are not own or OpenVibe machines; use a Link endpoint'] };
++ const cfg = config();
++ const names = ['vast', 'runpod'].filter(name => !rentalOnly || cfg[name + '_key']);
++ const offers = [];
++ const failures = [];
++ for (const name of names) {
++ const adapter = name === 'vast' ? vastAdapter({ fetchImpl, key: cfg.vast_key || null }) : runpodAdapter({ fetchImpl, key: cfg.runpod_key || null });
++ try {
++ const found = await adapter.searchOffers({ gpuNames: spec.requirements.gpu_names, vramMb: spec.requirements.vram_mb,
++ diskGb: spec.requirements.disk_gb, numGpus: spec.requirements.num_gpus,
++ minReliability: spec.requirements.min_reliability, region, maxPricePerHour: 10, limit: 50 });
++ // Never trust marketplace-side filters alone.
++ offers.push(...found.filter(o => o.gpu_ram_mb >= spec.requirements.vram_mb && o.num_gpus >= spec.requirements.num_gpus
++ && o.disk_gb >= spec.requirements.disk_gb && o.reliability >= spec.requirements.min_reliability
++ && Number(o.price_per_hour) > 0 && Number.isFinite(Number(o.price_per_hour))
++ && (!region || String(o.region).toUpperCase() === String(region).toUpperCase())));
++ } catch { failures.push(name + ': quote API unavailable'); }
++ }
++ const candidates = offers.map(o => ({ id: o.marketplace + ':' + o.id, tier: 'partner', health: 'up', region: o.region,
++ hourly: Number(o.price_per_hour), currency: 'USD', free_ram_mb: 2048, free_slots: 1, gpu_mb: o.gpu_ram_mb,
++ latency_ms: o.latency_ms }));
++ const decision = choose({ strategy, region }, candidates, { currency: 'USD' }, { now: now() });
++ const selected = offers.find(o => o.marketplace + ':' + o.id === decision.selected) || null;
++ return { dry_run: true, strategy, selected, offers, reasons: [...decision.reasons, ...(strategy === 'fastest' && !offers.some(o => o.latency_ms != null) ? ['latency unmeasured; choosing by price within the required region'] : []), ...failures], expires_at: iso(now() + 60000) };
}
/** Ask for a model to start or stop. The reconciler does it; nothing here acts directly. */
@@@ -255,37 +290,25 @@ function createGpuProvisioner({
* beforeCreate({ offer, hourly, reserved, maxMinutes, marketplace }) the endpoint's Billing hold, if
* any, is taken here - before the marketplace is called, so nothing runs that the person cannot pay.
*/
-- async function launchSpec(spec, { actor = 'service:ai', ownEndpointId = null, idleMinutes = null, maxRuntimeMinutes = null, beforeCreate = null } = {}) {
++ async function launchSpec(spec, { actor = 'service:ai', ownEndpointId = null, idleMinutes = null, maxRuntimeMinutes = null, beforeCreate = null, strategy = config().strategy, dryRun = false } = {}) {
const cfg = config();
++ if (cfg.dry_run && ownEndpointId) return { ok: false, code: 'gpu.dry_run', reason: 'GPU rentals are in quote-only mode' };
++ if (dryRun || cfg.dry_run) return { ok: true, ...await preview(spec.key, { strategy, runtimeSpec: spec }) };
if (!cfg.enabled) return { ok: false, code: 'gpu.off', reason: cfg.reason };
++ if (!STRATEGIES.includes(strategy)) throw new AiError(422, 'ai.invalid', 'unknown compute strategy');
++ if (strategy === 'private' || strategy === 'own') return { ok: false, code: 'gpu.no_offer', reason: 'this strategy requires an existing Link or OpenVibe machine' };
const existing = ownEndpointId ? await liveInstanceForEndpoint(ownEndpointId) : await liveInstance(spec.key);
if (existing) return { ok: true, instance: describe(existing), note: 'already running' };
-- const adapter = adapterFor(spec.marketplace, cfg);
-- if (!adapter) return { ok: false, code: 'gpu.no_key', reason: 'no ' + spec.marketplace + ' API key' };
--
-- // 1. the cheapest offer that fits the model
-- let offers;
-- try {
-- offers = await adapter.searchOffers({
-- gpuNames: spec.requirements.gpu_names,
-- vramMb: spec.requirements.vram_mb,
-- diskGb: spec.requirements.disk_gb,
-- numGpus: spec.requirements.num_gpus,
-- minReliability: spec.requirements.min_reliability,
-- region: cfg.region,
-- maxPricePerHour: 10,
-- limit: 50,
-- });
-- } catch (err) {
-- return { ok: false, code: 'gpu.search_failed', reason: String(err && err.message).slice(0, 200) };
-- }
-- const offer = cheapest(offers);
-- if (!offer) {
-- return { ok: false, code: 'gpu.no_offer', reason: 'no ' + spec.marketplace + ' offer fits ' + spec.display_name };
-- }
++ const quote = await preview(spec.key, { strategy, runtimeSpec: spec, rentalOnly: true });
++ const offer = quote.selected;
++ if (!offer) return { ok: false, code: 'gpu.no_offer', reason: quote.reasons.join('; ') };
++ const marketplace = offer.marketplace;
++ const adapter = adapterFor(marketplace, cfg);
++ if (!adapter) return { ok: false, code: 'gpu.no_key', reason: 'no ' + marketplace + ' API key' };
// 2. the cap: the reservation has to fit under what is left of it
-- const hourly = money.nanoUsd(offer.price_per_hour, "the offer's price");
++ const baseHourly = money.nanoUsd(offer.price_per_hour, "the offer's price");
++ const hourly = baseHourly == null ? null : baseHourly + cfg.extra_hourly_nano;
if (hourly == null || hourly <= 0n) {
return { ok: false, code: 'gpu.bad_price', reason: "the offer has no usable price" };
}
@@@ -310,7 +333,7 @@ function createGpuProvisioner({
// instance never runs that the person cannot pay for.
if (beforeCreate) {
try {
-- await beforeCreate({ offer, hourly, reserved, maxMinutes, marketplace: spec.marketplace });
++ await beforeCreate({ offer, hourly, reserved, maxMinutes, marketplace });
} catch (err) {
return { ok: false, code: (err && err.code) || 'gpu.billing', reason: String((err && err.message) || err).slice(0, 300) };
}
@@@ -321,7 +344,14 @@ function createGpuProvisioner({
// the label and gives it back when no live row of ours owns it.
const id = LABEL_PREFIX + randomUUID().replace(/-/g, '').slice(0, 12);
const t = now();
-- await db.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,
++ try {
++ await db.tx(async (tdb) => {
++ await tdb.maybe('SELECT id FROM gpu_budget_lock WHERE id=1 FOR UPDATE');
++ const latest = await moneySummary(tdb);
++ if (latest.committed_nano + reserved > cap) throw new AiError(409, 'gpu.cap', 'the monthly cap has no room for another reservation');
++ const unknown = await tdb.prepare("SELECT id FROM gpu_instances WHERE (status='starting' AND remote_id IS NULL AND error IS NOT NULL) OR status='orphan' LIMIT 1").get();
++ if (unknown) throw new AiError(409, 'gpu.uncertain', 'an unknown provisioning outcome or orphan must be reconciled first');
++ await tdb.prepare(`INSERT INTO gpu_instances (id, model_key, provider_key, marketplace, offer_id, remote_id, status, endpoint, month,
gpu_name, gpu_ram_mb, num_gpus, disk_gb, reliability, region, price_per_hour, hourly_nano, reserved_nano, spent_nano,
max_runtime_minutes, started_at, charged_at, destroy_failures, label, cold_start_ms, health_failures, created_at, updated_at,
own_endpoint_id, idle_minutes)
@@@ -330,18 +360,24 @@ function createGpuProvisioner({
@max_runtime_minutes, @started_at, @charged_at, 0, @label, NULL, 0, @created_at, @updated_at,
@own_endpoint_id, @idle_minutes)`)
.run({
-- id, model_key: spec.key, provider_key: spec.provider_key, marketplace: spec.marketplace,
++ id, model_key: spec.key, provider_key: spec.provider_key, marketplace,
offer_id: String(offer.id), month: monthOf(t),
gpu_name: String(offer.gpu_name || ''), gpu_ram_mb: Number(offer.gpu_ram_mb) || 0,
num_gpus: Number(offer.num_gpus) || 1, disk_gb: Number(offer.disk_gb) || 0,
reliability: Number(offer.reliability) || 0, region: String(offer.region || ''),
-- price_per_hour: String(offer.price_per_hour), hourly_nano: hourly.toString(), reserved_nano: reserved.toString(),
++ price_per_hour: money.usdText(hourly).slice(1), hourly_nano: hourly.toString(), reserved_nano: reserved.toString(),
max_runtime_minutes: maxMinutes, started_at: iso(t), charged_at: iso(t), label: id,
created_at: iso(t), updated_at: iso(t),
own_endpoint_id: ownEndpointId, idle_minutes: idleMinutes == null ? null : clampInt(idleMinutes, 1, 1440, cfg.idle_minutes),
});
++ });
++ } catch (err) {
++ if (err.code === 'gpu.cap' || err.code === 'gpu.uncertain') return { ok: false, code: err.code, reason: err.message };
++ throw err;
++ }
++ await audit(actor, 'gpu.placement', 'gpu_instance', id, { strategy, marketplace, offer_id: offer.id, reasons: quote.reasons, base_hourly_nano: baseHourly.toString(), extra_hourly_nano: cfg.extra_hourly_nano.toString(), hourly_nano: hourly.toString() });
await audit(actor, 'gpu.instance.requested', 'gpu_instance', id, {
-- model: spec.key, marketplace: spec.marketplace, offer_id: String(offer.id), hourly_nano: hourly.toString(),
++ model: spec.key, marketplace, offer_id: String(offer.id), hourly_nano: hourly.toString(),
reserved_nano: reserved.toString(), max_runtime_minutes: maxMinutes, label: id,
...(ownEndpointId ? { own_endpoint_id: ownEndpointId } : {}),
});
@@@ -356,11 +392,12 @@ function createGpuProvisioner({
label: id,
});
} catch (err) {
-- const reason = String(err && err.message).slice(0, 200);
-- await db.prepare("UPDATE gpu_instances SET status = 'refused', error = ?, updated_at = ? WHERE id = ?")
++ const reason = 'provision outcome unknown; reservation retained for reconciliation';
++ await db.prepare("UPDATE gpu_instances SET status = 'starting', error = ?, updated_at = ? WHERE id = ?")
.run(reason, iso(now()), id);
await audit(actor, 'gpu.instance.refused', 'gpu_instance', id, { reason });
-- return { ok: false, code: 'gpu.create_failed', reason };
++ const pending = await db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);
++ return { ok: true, pending: true, instance: describe(pending), reason };
}
await db.prepare('UPDATE gpu_instances SET remote_id = ?, updated_at = ? WHERE id = ?')
.run(String(created.remote_id || ''), iso(now()), id);
@@@ -450,9 +487,12 @@ function createGpuProvisioner({
async function destroy(row, reason, { actor = 'service:ai' } = {}) {
const cfg = config();
const adapter = adapterFor(row.marketplace, cfg);
++ if (!adapter || !row.remote_id) return { ok: false, reason: 'provider key or remote id missing; reservation retained' };
if (adapter && row.remote_id) {
try {
-- await adapter.destroyInstance(row.remote_id);
++ if (await adapter.destroyInstance(row.remote_id) === false) throw new Error('provider did not accept deletion');
++ const afterDelete = await adapter.showInstance(row.remote_id);
++ if (!afterDelete.deleted) throw new Error('provider deletion is not yet confirmed');
} catch (err) {
const failures = Number(row.destroy_failures || 0) + 1;
await db.prepare('UPDATE gpu_instances SET destroy_failures = ?, updated_at = ? WHERE id = ?')
@@@ -522,16 +562,8 @@ function createGpuProvisioner({
await destroyOrphan({ marketplace, remote_id: String(inst.id), label }, row, cfg);
}
}
-- // The rows whose rental never came back (a crash before the marketplace answered, or an
-- // answer we never got): their reservation is free again, or the month would carry a budget
-- // for an instance that does not exist.
-- const stale = await db.prepare("SELECT * FROM gpu_instances WHERE status = 'starting' AND (remote_id IS NULL OR remote_id = '') AND started_at < ?").all(iso(t - STALE_ROW_MS));
-- for (const row of stale || []) {
-- await db.prepare("UPDATE gpu_instances SET status = 'refused', error = ?, updated_at = ? WHERE id = ?")
-- .run('the rental never came back', iso(t), row.id);
-- await audit('service:ai', 'gpu.instance.abandoned', 'gpu_instance', row.id, { marketplace: row.marketplace });
-- log.warn && log.warn('[gpu] gave up on the row ' + row.id + ': the rental never came back, its reservation is free again');
-- }
++ // Unknown creation outcomes retain their reservation until the remote instance is recovered.
++ // A failed/truncated listing cannot prove that a purchase never happened.
return seen;
}
@@@ -554,7 +586,7 @@ function createGpuProvisioner({
// the orphan ran and was paid for: the seconds since the last charge are on the ledger
await charge(row, t, 'the orphan ran');
try {
-- await adapter.destroyInstance(inst.remote_id);
++ if (await adapter.destroyInstance(inst.remote_id) === false || !(await adapter.showInstance(inst.remote_id)).deleted) throw new Error('orphan deletion is not confirmed');
} catch (err) {
const failures = Number(row.destroy_failures || 0) + 1;
await db.prepare('UPDATE gpu_instances SET destroy_failures = ?, updated_at = ? WHERE id = ?')
@@@ -587,6 +619,7 @@ function createGpuProvisioner({
/** One pass over the live instances: the reservation, the idle timer, the health. */
async function reconcileInstance(row, cfg) {
++ if (!row.remote_id) return { changed: false, reason: 'creation outcome unknown; reservation retained' };
const priv = Boolean(row.own_endpoint_id);
const spec = priv
? (ownEndpoints && typeof ownEndpoints.specFor === 'function' ? ownEndpoints.specFor(row) : null)
@@@ -599,8 +632,9 @@ function createGpuProvisioner({
// the money that ran while we were away
await charge(row, t, 'the GPU ran');
-- // 1. the reservation ran out: the instance goes back, so the money stops there
-- if (t - started >= Number(row.max_runtime_minutes) * 60_000) {
++ // Begin deletion a minute early to allow the reconciliation interval and provider latency.
++ // A provider outage can still exceed this allowance; retain the commitment until confirmed gone.
++ if (t - started >= Math.max(0, Number(row.max_runtime_minutes) * 60_000 - 60_000)) {
await destroy({ ...row, charged_at: row.charged_at }, 'the reservation ran out');
return { changed: true, action: 'destroyed', reason: 'the reservation ran out' };
}
@@@ -738,14 +772,26 @@ function createGpuProvisioner({
/** One pass: the owner's requests, the live instances, and the models that are wanted. */
async function tick({ actor = 'service:ai' } = {}) {
const cfg = config();
-- if (!cfg.enabled) return { ok: false, reason: cfg.reason, launched: 0, stopped: 0, destroyed: 0 };
++ // Recovery and orphan cleanup also run while new rental spending is disabled.
++ await sweepOrphans(cfg);
++ if (!cfg.enabled) {
++ // Lowering the cap to zero disables new launches, but must not strand paid instances.
++ const rows = await db.prepare("SELECT * FROM gpu_instances WHERE status IN ('starting','running','stopping')").all();
++ for (const row of rows) if (row.remote_id && adapterFor(row.marketplace, cfg)) await destroy(row, 'GPU budget disabled');
++ return { ok: false, reason: cfg.reason, launched: 0, stopped: 0, destroyed: 0 };
++ }
let launched = 0;
let stopped = 0;
let destroyed = 0;
-- // 0. the marketplace instances that carry our label: the ones no live row owns are given back
-- await sweepOrphans(cfg);
++ if ((await moneySummary()).committed_nano > cfg.cap_nano) {
++ for (const row of await db.prepare("SELECT * FROM gpu_instances WHERE status IN ('starting','running','stopping')").all()) {
++ if (row.remote_id && adapterFor(row.marketplace, cfg)) await destroy(row, 'monthly cap lowered');
++ }
++ return { ok: false, reason: 'monthly cap lowered; paid instances are being released', launched: 0, stopped: 0, destroyed: 0 };
++ }
++
// 1. the owner's requests
const requests = await db.prepare('SELECT * FROM gpu_requests WHERE done_at IS NULL ORDER BY id').all();
@@@ -758,7 +804,7 @@ function createGpuProvisioner({
.run(iso(now()), String(res.reason).slice(0, 300), req.id);
continue;
}
-- launched += 1;
++ if (!res.dry_run) launched += 1;
} else {
const row = await liveInstance(req.model_key);
if (row) {
@@@ -785,7 +831,7 @@ function createGpuProvisioner({
if (await liveInstance(spec.key)) continue;
if (!await wanted(spec, cfg)) continue;
const res = await launch(spec.key, { actor });
-- if (res.ok) launched += 1;
++ if (res.ok && !res.dry_run) launched += 1;
}
return { ok: true, launched, stopped, destroyed };
@@@ -813,6 +859,8 @@ function createGpuProvisioner({
};
});
return {
++ strategy: cfg.strategy,
++ dry_run: cfg.dry_run,
enabled: cfg.enabled,
reason: cfg.reason,
month: sum.month,
@@@ -858,6 +906,7 @@ function createGpuProvisioner({
return {
state,
++ preview,
tick,
launch,
attachEndpoints,
ddiff --git a/server/gpu/runpod.js b/server/gpu/runpod.js
iindex a396c74..df698af 100644
---- a/server/gpu/runpod.js
++++ b/server/gpu/runpod.js
@@@ -34,9 +34,8 @@ function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BAS
function priceOf(gpu) {
const price = (gpu && gpu.price) || {};
const secure = Number(price.secure);
-- const community = Number(price.community);
++
if (Number.isFinite(secure) && secure > 0) return String(secure);
-- if (Number.isFinite(community) && community > 0) return String(community);
return '0';
}
@@@ -46,7 +45,7 @@ function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BAS
baseUrl,
async searchOffers(req) {
-- const out = await apiJson(fetchImpl, baseUrl + '/v2/gpu-types', { ...auth, label: 'RunPod' });
++ const out = await apiJson(fetchImpl, baseUrl + '/v2/catalog/gpus?include=AVAILABILITY&product=POD', { ...auth, label: 'RunPod' });
const gpus = (out && out.gpus) || [];
const wanted = (req.gpuNames || []).map((n) => String(n).toLowerCase());
const offers = [];
@@@ -54,24 +53,26 @@ function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BAS
const name = String(gpu.name || '');
const lower = name.toLowerCase();
if (wanted.length && !wanted.some((w) => lower.includes(w.toLowerCase()))) continue;
-- if (Number(gpu.memory) < Number(req.vramMb)) continue;
-- if (Number(gpu.maxCount || 0) < Number(req.numGpus || 1)) continue;
-- const price = priceOf(gpu);
++ if (Number(gpu.memory) * 1024 < Number(req.vramMb)) continue;
++ if (Number(gpu.maxCount && gpu.maxCount.secure || 0) < Number(req.numGpus || 1)) continue;
++ const price = String(Number(priceOf(gpu)) * Number(req.numGpus || 1));
if (Number(price) > Number(req.maxPricePerHour)) continue;
for (const dc of (gpu.dataCenters || [])) {
-- if (dc.availability !== undefined && !dc.availability) continue;
++ if (!dc.availability || dc.availability === 'NONE') continue;
++ const region = String(dc.countryCode || dc.region || String(dc.id).split('-')[0]);
++ if (req.region && region.toUpperCase() !== String(req.region).toUpperCase()) continue;
offers.push({
id: String(gpu.id) + ':' + String(dc.id),
marketplace: 'runpod',
gpu_type_id: String(gpu.id),
data_center_id: String(dc.id),
gpu_name: name,
-- gpu_ram_mb: Number(gpu.memory) || 0,
++ gpu_ram_mb: Number(gpu.memory) * 1024 || 0,
num_gpus: Number(req.numGpus || 1),
disk_gb: Number(req.diskGb || 0),
price_per_hour: price,
reliability: 1,
-- region: String(dc.name || dc.id || ''),
++ region,
});
}
}
@@@ -84,7 +85,7 @@ function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BAS
*/
async listInstances() {
const body = await apiJson(fetchImpl, baseUrl + '/v2/pods', { ...auth, label: 'RunPod' });
-- const list = (body && (body.pods || body.items || body.data)) || [];
++ const list = Array.isArray(body) ? body : (body && (body.pods || body.items || body.data)) || [];
return list.map((pod) => ({
id: String(pod.id == null ? '' : pod.id),
label: String(pod.name || ''),
@@@ -101,13 +102,11 @@ function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BAS
name: String(opts.label || ('openvibe-' + spec.key)),
image: r.image,
gpu: { id: offer.gpu_type_id, count: spec.requirements.num_gpus },
-- disk: { size: spec.requirements.disk_gb, type: 'volume' },
++ disk: spec.requirements.disk_gb,
++ cloud: 'SECURE',
++ args: opts.command || '',
ports: [String(r.port || 8000) + '/http'],
-- env: [
-- 'MODEL_FILE=' + ((r.weights && r.weights.file) || ''),
-- 'MODEL_URL=' + (opts.weights_url || ''),
-- 'MODEL_ALIAS=' + spec.model_key,
-- ],
++ env: { MODEL_FILE: (r.weights && r.weights.file) || '', MODEL_URL: opts.weights_url || '', MODEL_ALIAS: spec.model_key },
start: true,
};
if (offer.data_center_id) body.dataCenterId = offer.data_center_id;
@@@ -122,7 +121,8 @@ function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BAS
},
async showInstance(remoteId) {
-- const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { ...auth, label: 'RunPod' });
++ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { ...auth, label: 'RunPod', allowMissing: true });
++ if (out && out.missing) return { remote_id: remoteId, state: 'deleted', deleted: true, endpoint: '' };
const status = String((out && out.status) || 'unknown').toLowerCase();
return {
remote_id: remoteId,
@@@ -133,8 +133,8 @@ function runpodAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BAS
},
async destroyInstance(remoteId) {
-- const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'RunPod' });
-- return Boolean(out);
++ const out = await apiJson(fetchImpl, baseUrl + '/v2/pods/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'RunPod', allowMissing: true });
++ return out == null || out.success !== false;
},
};
}
ddiff --git a/server/gpu/vast.js b/server/gpu/vast.js
iindex 8320ad2..8b8f225 100644
---- a/server/gpu/vast.js
++++ b/server/gpu/vast.js
@@@ -105,7 +105,7 @@ function vastAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BASE_
// when no live row of ours owns it.
label: String(opts.label || ('openvibe-' + spec.key)),
};
-- const out = await apiJson(fetchImpl, baseUrl + '/api/v0/asks/' + offer.id + '/', { method: 'POST', body, ...auth, label: 'Vast.ai' });
++ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/asks/' + offer.id + '/', { method: 'PUT', body, ...auth, label: 'Vast.ai' });
if (!out || out.success === false) {
const err = new Error('Vast.ai did not rent the offer');
err.status = 502;
@@@ -121,18 +121,20 @@ function vastAdapter({ fetchImpl = globalThis.fetch, key = null, baseUrl = BASE_
},
async showInstance(remoteId) {
-- const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { ...auth, label: 'Vast.ai' });
-- const status = String((out && (out.actual_status || out.status)) || 'unknown');
++ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { ...auth, label: 'Vast.ai', allowMissing: true });
++ if (out && out.missing) return { remote_id: remoteId, state: 'deleted', deleted: true, endpoint: '' };
++ const instance = out && out.instances ? out.instances : out;
++ const status = String((instance && (instance.actual_status || instance.status)) || 'unknown');
return {
remote_id: remoteId,
state: status,
-- endpoint: endpointOf(out, (opts_port(out))),
++ endpoint: endpointOf(instance, (opts_port(instance))),
raw: out,
};
},
async destroyInstance(remoteId) {
-- const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'Vast.ai' });
++ const out = await apiJson(fetchImpl, baseUrl + '/api/v0/instances/' + encodeURIComponent(remoteId), { method: 'DELETE', ...auth, label: 'Vast.ai', allowMissing: true });
return Boolean(out && out.success !== false);
},
};
ddiff --git a/server/openapi.js b/server/openapi.js
iindex e13e18b..4425cd8 100644
---- a/server/openapi.js
++++ b/server/openapi.js
@@@ -475,9 +475,9 @@ route('get', '/own-endpoints', {
});
route('post', '/own-endpoints', {
operationId: 'createOwnEndpoint', summary: 'Start a private endpoint on a chosen model or your own weights', tags: ['exchange'], security: [{ bearer: [] }],
-- description: 'Only a person. Name `model` (a key from GET /exchange/endpoints) or `weights` ({ repo, file, lora_repo?, lora_file? }: a Hugging Face .gguf and an optional LoRA, fetched by the rented container). The endpoint is started by the reconciler on the cheapest fit, after a Billing hold of its worst case; a start that would pass the owner\'s monthly cap is refused with a plain reason, and nothing is rented. 409 own_endpoint.off carries the same reason the public status does.',
++ description: 'Only a person. Name `model` (a key from GET /exchange/endpoints) or `weights` ({ repo, file, lora_repo?, lora_file? }: a Hugging Face .gguf and an optional LoRA, fetched by the rented container). The endpoint is started by the reconciler using the requested strategy (Cheapest by default), after a Billing hold of its worst case; a start that would pass the owner\'s monthly cap is refused with a plain reason, and nothing is rented. 409 own_endpoint.off carries the same reason the public status does.',
requestBody: { required: true, content: { 'application/json': { schema: { type: 'object', properties: {
-- model: str(), idle_minutes: integer({ minimum: 1, maximum: 120 }),
++ model: str(), strategy: str({ enum: ['automatic', 'cheapest', 'fastest'] }), idle_minutes: integer({ minimum: 1, maximum: 120 }),
weights: { type: 'object', required: ['repo', 'file'], properties: { repo: str(), file: str(), lora_repo: str(), lora_file: str() } },
display_name: str(),
} } } } },
@@@ -495,6 +495,18 @@ route('post', '/own-endpoints/{id}/stop', {
parameters: [path('id', str({ pattern: '^oep_[0-9A-HJKMNP-TV-Z]{26}$' }))],
responses: { 200: json('The endpoint', { type: 'object', required: ['endpoint'], properties: { endpoint: { type: 'object' } } }), 404: problem, ...errors([401, 403]) },
});
++route('get', '/exchange/gpu-quotes/{model}', {
++ operationId: 'exchangeGpuQuote', summary: 'Preview GPU marketplace prices without renting', tags: ['exchange'], security: [],
++ description: 'Public read-only preview for a known endpoint model. Base GPU rates exclude the operator storage/network allowance. No Billing hold or purchase; cached for one minute by model and strategy.',
++ parameters: [path('model', str(), 'Model key from /exchange/endpoints'), query('strategy', str({ enum: ['automatic', 'cheapest', 'fastest', 'private', 'own'], default: 'cheapest' }))],
++ responses: {
++ 200: json('Quote preview', { type: 'object', properties: {
++ dry_run: { type: 'boolean' }, strategy: str(), selected: nullable('object'), offers: arrayOf({ type: 'object' }), reasons: arrayOf(str()), expires_at: str(),
++ } }),
++ 422: json('Unknown model or strategy', { type: 'object', properties: { error: str() } }),
++ 503: json('GPU quotes unavailable', { type: 'object', properties: { error: str() } }),
++ },
++});
route('get', '/exchange/models/{slug}', {
operationId: 'exchangeModel', summary: 'One model and its offers', tags: ['exchange'], security: [],
description: 'Public. The slug is "<org>/<model>", lowercased, e.g. deepseek/deepseek-flash.',
ddiff --git a/server/settings.js b/server/settings.js
iindex 36503db..28f20f5 100644
---- a/server/settings.js
++++ b/server/settings.js
@@@ -179,6 +179,26 @@ const GPU_SCHEMA = Object.freeze({
type: 'object',
additionalProperties: false,
properties: {
++ strategy: {
++ type: 'string', enum: ['automatic', 'cheapest', 'fastest', 'private', 'own'], default: 'cheapest', title: 'GPU placement strategy', 'x-class': 'internal',
++ description: 'How GPU offers are selected within the required region, model resources and spending cap. Own and private strategies refuse marketplace rentals.',
++ 'x-enum-labels': {
++ automatic: { label: 'Automatic', description: 'Use the first eligible compute tier, then choose by price.' },
++ cheapest: { label: 'Cheapest', description: 'Choose the lowest hourly price that fits the model.' },
++ fastest: { label: 'Fastest', description: 'Prefer measured latency; use regional price when unmeasured.' },
++ private: { label: 'Most private', description: 'Require existing OpenVibe or own capacity; refuse marketplaces.' },
++ own: { label: 'On my machine', description: 'Require existing own capacity; refuse marketplace rentals.' },
++ },
++ },
++ dry_run: { type: 'boolean', default: false, title: 'Quote without renting', 'x-class': 'internal', description: 'Preview GPU choices and costs without creating a new paid instance. Existing paid instances still receive cleanup.' },
++ costs_bounded: {
++ type: 'boolean', default: false, title: 'Storage and network costs are bounded', 'x-group': GPU_GROUP, 'x-danger': true,
++ description: 'Enable only after enforcing storage and network limits that fit the hourly allowance below. Without this confirmation GPU rentals remain off.',
++ },
++ extra_hourly_nano: {
++ type: 'integer', minimum: 0, title: 'Storage and network allowance per hour', 'x-group': GPU_GROUP, 'x-unit': 'nano_usd', 'x-danger': true,
++ description: 'A conservative upper bound for extra storage and network charges per instance per hour, in billionths of a dollar. Added to every reservation and metered estimate. Must be explicitly set, even when zero.',
++ },
'vast.origin': {
type: 'string', title: 'Address the Vast.ai key is locked to', 'x-group': GPU_GROUP, 'x-advanced': true,
description: 'The Vast.ai address AI sends the key to. AI writes it itself; the key below only opens for this address, so it can never be sent anywhere else.',
@@@ -216,7 +236,7 @@ const GPU_SCHEMA = Object.freeze({
monthly_cap: {
type: 'integer', title: 'Most the GPUs may cost in a month', 'x-group': GPU_GROUP, 'x-unit': 'nano_usd', 'x-danger': true,
minimum: 0, 'x-example': 50000000000,
-- description: 'The most the rented GPUs may cost in one calendar month (in billionths of a dollar: 50000000000 is $50). AI never passes it: it refuses to start a GPU and says why. Unset, nothing is rented.',
++ description: 'The most the rented GPUs may cost in one calendar month (in billionths of a dollar: 50000000000 is $50). AI refuses new reservations above it; failed provider deletion can continue billing and needs operator intervention. Unset, nothing is rented.',
'x-owner-input': {
group: 'gpus',
unlocks: 'Lets AI rent GPUs up to this amount each month. Without a cap nothing is rented, even with a key.',
@@@ -447,6 +467,10 @@ async function createGpuSettings({ db, env = process.env, log = console, onChang
function values() {
const v = store.get() || {};
return {
++ strategy: v.strategy || 'cheapest',
++ dry_run: v.dry_run === true,
++ costs_bounded: v.costs_bounded === true,
++ extra_hourly_nano: v.extra_hourly_nano == null ? null : String(v.extra_hourly_nano),
vast_origin: String(v['vast.origin'] || ''),
vast_key: String(v['vast.key'] || ''),
runpod_origin: String(v['runpod.origin'] || ''),
ddiff --git a/test/fixtures/gpu-fetch.js b/test/fixtures/gpu-fetch.js
iindex c5e2288..2c575e2 100644
---- a/test/fixtures/gpu-fetch.js
++++ b/test/fixtures/gpu-fetch.js
@@@ -48,11 +48,15 @@ function createGpuFetch() {
if (method === 'DELETE') {
calls.destroy.push(u);
if (api.destroyFails) return reply(500, { detail: 'the marketplace refused to give the instance back' });
-- rented.delete(u.split('/').pop());
++ if (!api.deletePending) {
++ rented.delete(u.split('/').pop());
++ api.extra = api.extra.filter(x => String(x.id) !== u.split('/').pop());
++ }
return reply(200, { success: true, msg: 'destroyed' });
}
calls.show += 1;
const id = u.split('/').pop();
++ if (!rented.has(id) && !api.extra.some(x => String(x.id) === id)) return reply(404, { detail: 'instance not found' });
const port = 12345 + Math.max(0, created - 1);
return reply(200, { ...RECORDED.instance, id, ports: { 8000: { port, proxy: 'https://gw.vast.ai:' + port } } });
}
ddiff --git a/test/gpu-adapters.test.js b/test/gpu-adapters.test.js
nnew file mode 100644
iindex 0000000..f7fb73d
---- /dev/null
++++ b/test/gpu-adapters.test.js
@@@ -0,0 +1,26 @@
++'use strict';
++const assert=require('node:assert/strict');
++const {runpodAdapter}=require('../server/gpu/runpod');
++const {vastAdapter}=require('../server/gpu/vast');
++(async()=>{
++ const calls=[];
++ const fetchImpl=async(url,options={})=>{
++ calls.push({url,...options});let data={};
++ if(url.includes('/catalog/gpus'))data={gpus:[{id:'gpu',name:'H100',memory:80,maxCount:{secure:8},price:{secure:2},dataCenters:[{id:'US-1',availability:'HIGH'},{id:'DE-1',availability:'HIGH'},{id:'US-2',availability:'NONE'}]}]};
++ else if(url.endsWith('/pods') && options.method==='POST')data={id:'pod'};
++ else if(url.includes('/asks/'))data={new_contract:'instance'};
++ if(options.method==='DELETE')return {ok:true,status:204,text:async()=>''};
++ return {ok:true,status:200,text:async()=>JSON.stringify(data)};
++ };
++ const p=runpodAdapter({fetchImpl,key:'stub'});
++ const offers=await p.searchOffers({gpuNames:['H100'],vramMb:80000,numGpus:2,diskGb:60,maxPricePerHour:10,region:'US'});
++ assert.equal(offers.length,1);assert.equal(offers[0].price_per_hour,'4');assert.equal(offers[0].gpu_ram_mb,81920);
++ await p.createInstance(offers[0],{key:'model',model_key:'model',requirements:{num_gpus:2,disk_gb:60},runtime:{image:'image'}},{command:'run'});
++ const body=JSON.parse(calls.find(x=>x.url.endsWith('/pods')).body);
++ assert.equal(body.disk,60);assert.equal(body.cloud,'SECURE');assert.equal(body.args,'run');assert.equal(Array.isArray(body.env),false);
++ assert.equal(await p.destroyInstance('pod'),true);
++ await vastAdapter({fetchImpl,key:'stub'}).createInstance({id:'1'},{key:'m',runtime:{image:'image'},requirements:{disk_gb:60}});
++ assert.equal(calls.find(x=>x.url.includes('/asks/')).method,'PUT');
++ assert.ok(calls.every(x=>x.redirect==='error' && x.signal));
++ console.log('GPU provider wire methods, region, count/VRAM units, request shapes and 204 deletion: passed');
++})().catch(e=>{console.error(e);process.exit(1);});
ddiff --git a/test/gpu.test.js b/test/gpu.test.js
iindex 0231a4f..715db35 100644
---- a/test/gpu.test.js
++++ b/test/gpu.test.js
@@@ -15,7 +15,7 @@ const OWNER = { type: 'user', id: 'usr_' + 'A'.repeat(26) };
/** The owner's GPU settings: the marketplace key (a secret) and the monthly cap (money). */
async function settings(h, { key = 'vast-test-key', cap = '50000000000', idle = null, maxRuntime = null } = {}) {
const store = h.settings.gpus.store;
-- const values = { monthly_cap: Number(cap) };
++ const values = { monthly_cap: Number(cap), costs_bounded: true, extra_hourly_nano: 0 };
if (idle != null) values.idle_minutes = idle;
if (maxRuntime != null) values.max_runtime_minutes = maxRuntime;
await store.apply(values, { merge: true, actor: OWNER, reason: 'the test' });
@@@ -228,10 +228,11 @@ t.test('a destroy that keeps failing is tried again every pass and the operator
assert.strictEqual(gauges[0].value, 3, 'the gauge the operator alert reads');
const audit = await h.registry.listAudit({ action: 'gpu.orphan.destroy_failed' });
assert.strictEqual(audit.length, 1, 'the operator alert is on the audit log');
++ assert.strictEqual((await h.gpus.launch('deepseek-r1-32b')).code, 'gpu.uncertain', 'an unaccounted orphan blocks new spending');
} finally { await h.stop(); }
});
--t.test('a row whose rental never came back gives its reservation back', async () => {
++t.test('an unknown rental retains its reservation until reconciled', async () => {
const api = createGpuFetch();
let now = T0;
const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
@@@ -246,9 +247,113 @@ t.test('a row whose rental never came back gives its reservation back', async ()
now += 11 * 60_000; // past the give-up time
await h.gpus.tick();
const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id = ?').get(id);
-- assert.strictEqual(row.status, 'refused');
++ assert.strictEqual(row.status, 'starting');
const after = await h.gpus.moneySummary();
-- assert.strictEqual(BigInt(after.committed_nano), 0n, 'the reservation is free again');
++ assert.ok(BigInt(after.committed_nano) >= BigInt(before.committed_nano), 'an empty listing cannot release an uncertain financial commitment');
++ } finally { await h.stop(); }
++});
++
++t.test('unbounded extras block rentals and the allowance counts against the cap', async () => {
++ const api = createGpuFetch();
++ const h = await boot({ clock: { now: () => T0 }, fetchImpl: api.fetchImpl });
++ try {
++ await settings(h, { cap: '1000000000', maxRuntime: 60 });
++ await h.settings.gpus.store.apply({ costs_bounded: false }, { merge: true, actor: OWNER });
++ const blocked = await h.gpus.launch('deepseek-r1-32b');
++ assert.strictEqual(blocked.ok, false);assert.match(blocked.reason, /hourly bound/);
++ assert.strictEqual(api.calls.create.length, 0);
++ await h.settings.gpus.store.apply({ costs_bounded: true, extra_hourly_nano: 1000000000 }, { merge: true, actor: OWNER });
++ assert.strictEqual((await h.gpus.launch('deepseek-r1-32b')).code, 'gpu.cap');
++ assert.strictEqual(api.calls.create.length, 0);
++ await h.settings.gpus.store.apply({ extra_hourly_nano: 100000000 }, { merge: true, actor: OWNER });
++ const rented = await h.gpus.launch('deepseek-r1-32b');
++ assert.ok(rented.ok, rented.reason);
++ assert.strictEqual(String(rented.hourly_nano), '234800000');
++ assert.strictEqual(rented.instance.price_per_hour, '0.2348');
++ } finally { await h.stop(); }
++});
++
++t.test('accepted deletion retains the reservation until the instance is confirmed gone', async () => {
++ const api = createGpuFetch();
++ let now = T0;
++ const h = await boot({ clock: { now: () => now }, fetchImpl: api.fetchImpl });
++ try {
++ await settings(h);
++ const rented = await h.gpus.launch('deepseek-r1-32b');
++ api.deletePending = true;
++ await settings(h, { cap: '0' });
++ await h.gpus.tick();
++ let row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id=?').get(rented.instance.id);
++ assert.notStrictEqual(row.status, 'destroyed');
++ assert.ok(BigInt((await h.gpus.moneySummary()).committed_nano) > 0n);
++ now += 24 * 3600000;
++ const overdue = await h.gpus.moneySummary();
++ assert.ok(BigInt(overdue.committed_nano) >= BigInt(overdue.spent_nano), 'unconfirmed deletion cannot understate the commitment');
++ api.deletePending = false;
++ await h.gpus.tick();
++ row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id=?').get(rented.instance.id);
++ assert.strictEqual(row.status, 'destroyed');
++ } finally { await h.stop(); }
++});
++
++t.test('quotes and dry runs work without a cap and never create an instance', async () => {
++ const api = createGpuFetch();
++ const h = await boot({ clock: { now: () => T0 }, fetchImpl: api.fetchImpl });
++ try {
++ const q = await h.gpus.preview('deepseek-r1-32b');
++ assert.strictEqual(q.selected.id, '111');
++ const dry = await h.gpus.launch('deepseek-r1-32b', { dryRun: true });
++ assert.strictEqual(dry.dry_run, true);
++ assert.strictEqual(api.calls.create.length, 0);
++ for (const strategy of ['own', 'private']) {
++ const p = await h.gpus.preview('deepseek-r1-32b', { strategy });
++ assert.strictEqual(p.selected, null);
++ }
++ assert.strictEqual((await h.gpus.state()).instances.length, 0);
++ } finally { await h.stop(); }
++});
++
++t.test('disabling the cap continues cleanup of paid instances', async () => {
++ const api = createGpuFetch();
++ const h = await boot({ clock: { now: () => T0 }, fetchImpl: api.fetchImpl });
++ try {
++ await settings(h);
++ await h.gpus.launch('deepseek-r1-32b');
++ await settings(h, { cap: '0' });
++ await h.gpus.tick();
++ assert.ok(api.calls.destroy.length > 0);
++ } finally { await h.stop(); }
++});
++
++t.test('concurrent models cannot over-reserve the same monthly cap', async () => {
++ const api = createGpuFetch();
++ const h = await boot({ clock: { now: () => T0 }, fetchImpl: api.fetchImpl });
++ try {
++ await settings(h, { cap: '150000000', maxRuntime: 60 });
++ const results = await Promise.all([h.gpus.launch('deepseek-r1-32b'), h.gpus.launch('deepseek-r1-32b-abliterated')]);
++ assert.strictEqual(results.filter(r => r.ok).length, 1);
++ assert.strictEqual(api.calls.create.length, 1);
++ assert.ok(BigInt((await h.gpus.moneySummary()).committed_nano) <= 150000000n);
++ } finally { await h.stop(); }
++});
++
++t.test('a lost create response retains its reservation and is recovered by label', async () => {
++ const api = createGpuFetch();
++ const fetchImpl = async (url, options) => {
++ const response = await api.fetchImpl(url, options);
++ if (String(url).includes('/asks/')) throw new Error('response lost');
++ return response;
++ };
++ const h = await boot({ clock: { now: () => T0 }, fetchImpl });
++ try {
++ await settings(h);
++ const pending = await h.gpus.launch('deepseek-r1-32b');
++ assert.strictEqual(pending.pending, true);
++ assert.ok(BigInt((await h.gpus.moneySummary()).committed_nano) > 0n);
++ await h.gpus.tick();
++ const row = await h.db.prepare('SELECT * FROM gpu_instances WHERE id=?').get(pending.instance.id);
++ assert.ok(row.remote_id, 'the label recovers the purchase rather than issuing a second one');
++ assert.strictEqual(api.calls.create.length, 1);
} finally { await h.stop(); }
});
ddiff --git a/test/own-endpoints.test.js b/test/own-endpoints.test.js
iindex 4229bec..c75afc2 100644
---- a/test/own-endpoints.test.js
++++ b/test/own-endpoints.test.js
@@@ -95,7 +95,7 @@ function marketplace(billing) {
async function setGpus(h, { key = 'vast-test-key', cap = '50000000000', idle = null } = {}) {
const store = h.settings.gpus.store;
-- const values = { monthly_cap: Number(cap) };
++ const values = { monthly_cap: Number(cap), costs_bounded: true, extra_hourly_nano: 0 };
if (idle != null) values.idle_minutes = idle;
await store.apply(values, { merge: true, actor: OWNER, reason: 'the test' });
if (key) await store.setSecret('vast.key', key, { actor: OWNER, reason: 'the test' });
ddiff --git a/test/run.js b/test/run.js
iindex 30a54bf..b19f571 100644
---- a/test/run.js
++++ b/test/run.js
@@@ -13,4 +13,4 @@
* ○ and not counted as passed (openvibe-shared/test-runner).
*/
'use strict';
--require('openvibe-shared/test-runner').main({ dir: __dirname, timeoutMs: 60000, pad: 32, parallel: 1 });
++require('openvibe-shared/test-runner').main({ dir: __dirname, timeoutMs: 180000, pad: 32, parallel: 1 });