openvibers/openvibe.ai Commits
A second local chat model: provider local2 (#306)
Changes
@@@ -159,6 +159,11 @@ AI_MODEL=#AI_MEDIA_STORAGE_HOSTS=*.backblazeb2.com,*.r2.cloudflarestorage.com,edge.openvibe.media#AI_LOCAL_LLM_MODEL=qwen2.5-1.5b-instruct#AI_LOCAL_LLM_TIMEOUT_MS=180000++# A second local chat model server (its own llama.cpp process and another model family): provider `local2`,++# so a caller can pin two local offers, e.g. Games' AI-vs-AI matches. Same rules: no key, free, no JSON.++#AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1++#AI_LOCAL_LLM2_MODEL=llama-3.2-1b-instruct++#AI_LOCAL_LLM2_TIMEOUT_MS=180000# A second local OpenAI-compatible server for embeddings, when the chat model's llama.cpp process cannot also# serve the embedding GGUF (llama.cpp serves one model per process). Default: AI_LOCAL_LLM_URL.#AI_LOCAL_EMBED_URL=http://127.0.0.1:8091/v1@@@ -445,6 +445,7 @@ definition, but never over an admin's version.| `http` | The local HTTP seam: `POST {operation, …}` → `{text, json, usage}` | `AI_HTTP_SEAM_URL` or the admin API || `whisper` | whisper.cpp on this host, with Live's VAD, hallucination filter, multilingual model and live/batch lanes; NVIDIA Parakeet TDT 0.6B v3 (`parakeet-cli`) for English and 24 European languages when `PARAKEET_MODEL` is set, falling back to whisper | `WHISPER_*`, `PARAKEET_*`, `ASR_ENGINE` — Live's names || `openai`, key `local` | A model server on this host speaking the OpenAI API (llama.cpp's `llama-server`, Ollama's `/v1`): no key, never paid; route `media.local` | `AI_LOCAL_LLM_URL`, `AI_LOCAL_LLM_MODEL`, `AI_LOCAL_LLM_TIMEOUT_MS` |++| `openai`, key `local2` | A second local chat server, another llama.cpp process and model family: no key, never paid, no JSON or embeddings. Two local offers let a caller pin a match between them (Games' AI-vs-AI job) | `AI_LOCAL_LLM2_URL`, `AI_LOCAL_LLM2_MODEL`, `AI_LOCAL_LLM2_TIMEOUT_MS` |Routing: a route either names a `capability` and the router builds its pool from every provider thatcan serve it, ordered by `openvibe-sdk/placement` (objective, price, measured latency, health, with@@@ -693,6 +694,15 @@ streams always win: 2 threads, at most 1.5 CPUs, nice 19, 3 GB. It uses about 0.28 tokens/s. Without it, `media.analyze` writes an extractive overview and says so. To swap the model, put thefile beside it, change `--model`/`--alias` and `AI_LOCAL_LLM_MODEL`, and restart both units.++`deploy/systemd/openvibe-llm2.service` is the optional second chat server: the same llama.cpp build and models++directory on `127.0.0.1:8092`, a different model family, and the provider `local2`++(`AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1`, `AI_LOCAL_LLM2_MODEL`). It is held to 2 threads, at most 1 CPU,++nice 19 and 2 GB, and it never advertises `json` or `embed`. Games' AI-vs-AI job pins one local offer per side,++so the two models must be different families and both aliases must be the `model` in its `GAMES_AI_MATCHES_OFFERS`++pair. On the production host since 2026-11-06: Llama-3.2-1B-Instruct Q4_K_M (Llama 3.2 Community License,++Hugging Face sha256 `6f85a640a97cf2bf5b8e764087b1e83da0fdb51d7c9fab7d0fece9385611df83`, 807,694,464 bytes),++checked against that hash before install; it uses about 0.8 GB.++Local embeddings are off unless `AI_LOCAL_EMBED_MODEL` names an embedding model the server at `AI_LOCAL_LLM_URL`answers at `/embeddings`. Then `local` advertises `embed`, the model is registered as an `embedding` model, and`default.embedding` (a capability pool) offers it beside the shared provider's `AI_EMBEDDING_MODEL`; `local` isnnew file mode 100644@@@ -0,0 +1,34 @@++[Unit]++Description=OpenVibe local model 2 — llama.cpp llama-server, Llama 3.2 1B on 127.0.0.1:8092, for AI-vs-AI matches++# OpenVibe.AI reaches it as the provider `local2` (AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1). Loopback only,++# no key. It is the second side of an AI-vs-AI match (Games' GAMES_AI_MATCHES_OFFERS): a different model family++# from the Qwen unit on 8090, so the match is not a model playing itself. It shares the host with live streams,++# so it is held back: 2 threads, at most 1 CPU, nice 19 and 2 GB of memory.++# Install (README "Local model"): llama.cpp's CPU build under /opt/llama.cpp/<build> with /opt/llama.cpp/current,++# the model in /var/lib/openvibe-llm/models, both checked against their published sha256.++After=network.target++++[Service]++Type=simple++User=ubuntu++Group=ubuntu++ExecStart=/opt/llama.cpp/current/llama-server \++ --model /var/lib/openvibe-llm/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \++ --alias llama-3.2-1b-instruct \++ --host 127.0.0.1 --port 8092 \++ --threads 2 --ctx-size 8192 --parallel 1 \++ --no-webui++Nice=19++CPUQuota=100%++MemoryMax=2G++Restart=on-failure++RestartSec=10++TimeoutStopSec=15++NoNewPrivileges=true++PrivateTmp=true++ProtectSystem=strict++ProtectHome=true++ReadOnlyPaths=/var/lib/openvibe-llm /opt/llama.cpp++++[Install]++WantedBy=multi-user.target@@@ -176,6 +176,14 @@ function load(env = process.env) {embedModel: String(env.AI_LOCAL_EMBED_MODEL || '').trim(),timeoutMs: int(env.AI_LOCAL_LLM_TIMEOUT_MS, 180000),},++ // A second local chat server (its own llama.cpp process, another model family): the provider `local2`.++ // Same rules as `local`: no key, never paid, no JSON. Two local offers let a caller pin a match++ // between models without sending either side to a paid provider.++ localLlm2: {++ url: trimUrl(env.AI_LOCAL_LLM2_URL),++ model: String(env.AI_LOCAL_LLM2_MODEL || '').trim(),++ timeoutMs: int(env.AI_LOCAL_LLM2_TIMEOUT_MS, 180000),++ },// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },@@@ -55,13 +55,18 @@ function providerRecords(config, env) {sharedProviderRecord(config, env),{ key: 'whisper', display_name: 'whisper.cpp (local speech-to-text)', kind: 'whisper', status: 'active', auth_mode: 'none', default_model: 'whisper.cpp', capabilities: ['transcribe'], timeout_ms: 600000, priority: 20, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'WHISPER_*', billing_profile: 'free' } },];++ // A local model server (llama.cpp, Ollama) speaking the OpenAI API: no key, never paid (WS-O task 5).++ // No json: the small local models must not be offered the product JSON routes. Embed only with an embedding model.++ const localTextCaps = ['chat', 'generate', 'summarize', 'classify', 'extract'];if (config.localLlm && config.localLlm.url) {-- // A local model server (llama.cpp, Ollama) speaking the OpenAI API: no key, never paid (WS-O task 5).-- // No json: the small local model must not be offered the product JSON routes. Embed only with an embedding model.-- const localCaps = ['chat', 'generate', 'summarize', 'classify', 'extract'];++ const localCaps = [...localTextCaps];if (config.localLlm.embedModel) localCaps.push('embed');out.push({ key: 'local', display_name: 'Local model (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm.model || 'local', capabilities: localCaps, timeout_ms: config.localLlm.timeoutMs, priority: 5, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'AI_LOCAL_LLM_URL', embed_url: config.localLlm.embedUrl || config.localLlm.url, local: true, paid: false, billing_profile: 'free' } });}++ if (config.localLlm2 && config.localLlm2.url) {++ // The second local chat model: another llama.cpp process and another family, so a match has two sides.++ out.push({ key: 'local2', display_name: 'Local model 2 (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm2.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm2.model || 'local2', capabilities: [...localTextCaps], timeout_ms: config.localLlm2.timeoutMs, priority: 5, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'AI_LOCAL_LLM2_URL', local: true, paid: false, billing_profile: 'free' } });++ }if (config.httpSeamUrl) {out.push({ key: 'http-seam', display_name: 'Local HTTP seam', kind: 'http', status: 'active', base_url: config.httpSeamUrl, auth_mode: 'none', default_model: null, capabilities: [], timeout_ms: 30000, priority: 60, metadata: { configured_by: 'AI_HTTP_SEAM_URL', billing_profile: 'metered' } });}nnew file mode 100644@@@ -0,0 +1,53 @@++'use strict';++// Two local chat servers are two offered models: `local` and `local2`. A first-party generate run can pin++// one local offer for each side of an AI-vs-AI match, and neither pin reaches the other side's server.++const assert = require('assert');++const { boot, request, token, suite, seamServer, ALL } = require('./helpers');++++const t = suite('local-models');++const firstParty = token('games', ['ai.run.create', 'ai.run.read'], { ns: ['ai.*'] });++const answer = (model) => ({ model, choices: [{ message: { content: model } }], usage: { prompt_tokens: 3, completion_tokens: 2 } });++++let h; let qwen; let llama;++const calls = { qwen: 0, llama: 0 };++++t.test('boot with two local chat servers', async () => {++ qwen = await seamServer(() => { calls.qwen++; return answer('qwen-test'); });++ llama = await seamServer(() => { calls.llama++; return answer('llama-test'); });++ h = await boot({ env: {++ AI_STUB_FALLBACK: 'false', AI_PROVIDER_RETRIES: '0',++ AI_LOCAL_LLM_URL: `${qwen.url}/v1`, AI_LOCAL_LLM_MODEL: 'qwen-test',++ AI_LOCAL_LLM2_URL: `${llama.url}/v1`, AI_LOCAL_LLM2_MODEL: 'llama-test',++ } });++});++++t.test('GET /api/v1/routes offers both local models', async () => {++ const r = await request(h.base, 'GET', '/api/v1/routes?operation=generate', { tok: firstParty });++ assert.strictEqual(r.status, 200, r.text);++ const ids = (r.body.offers || []).map((o) => `${o.provider}:${o.model}`);++ assert.ok(ids.includes('local:qwen-test'), `local offer missing from ${ids.join(', ')}`);++ assert.ok(ids.includes('local2:llama-test'), `local2 offer missing from ${ids.join(', ')}`);++});++++t.test('a pin to each local offer calls only that model', async () => {++ for (const [provider, model, other] of [['local2', 'llama-test', 'qwen'], ['local', 'qwen-test', 'llama']]) {++ const before = { qwen: calls.qwen, llama: calls.llama };++ const r = await request(h.base, 'POST', '/api/v1/generate?wait=5000', { tok: firstParty, body: {++ prompt: 'pick a hero', route: { provider, model }, max_tokens: 30, options: { cache: false },++ } });++ assert.strictEqual(r.status, 201, r.text);++ assert.strictEqual(r.body.run.provenance.provider, provider);++ assert.strictEqual(r.body.run.provenance.model, model);++ assert.strictEqual(r.body.run.provenance.fallback_used, false);++ assert.match(r.body.run.explain.reasons.join(' '), new RegExp(`pinned to ${provider}/${model}`));++ assert.strictEqual(calls[other], before[other], `the ${other} server was not called`);++ }++});++++t.test('shutdown', async () => {++ await h.stop();++ await qwen.close();++ await llama.close();++});++++t.run();@@@ -172,6 +172,19 @@ t.test('local provider: never json, embed only with AI_LOCAL_EMBED_MODEL', () =>assert.strictEqual(load({ AI_LOCAL_EMBED_MODEL: ' nomic-embed-text ' }).localLlm.embedModel, 'nomic-embed-text');});++t.test('second local chat model: provider local2, another family, no json or embed', () => {++ const records = (env) => providerRecords(load(env), env);++ assert.strictEqual(records({}).find(x => x.key === 'local2'), undefined, 'no local2 without AI_LOCAL_LLM2_URL');++ const p = records({ AI_LOCAL_LLM2_URL: 'http://127.0.0.1:8092/v1', AI_LOCAL_LLM2_MODEL: ' llama-3.2-1b-instruct ' }).find(x => x.key === 'local2');++ assert.ok(p, 'local2 is seeded beside local');++ assert.strictEqual(p.base_url, 'http://127.0.0.1:8092/v1');++ assert.strictEqual(p.default_model, 'llama-3.2-1b-instruct');++ assert.deepStrictEqual(p.capabilities, ['chat', 'generate', 'summarize', 'classify', 'extract'], 'never json or embed');++ assert.strictEqual(p.timeout_ms, 180000);++ assert.strictEqual(p.priority, 5);++ assert.deepStrictEqual([p.metadata.local, p.metadata.paid, p.metadata.billing_profile], [true, false, 'free']);++});++t.test('image: off without AI_IMAGE_MODEL; the Images API body, per-image usage and the stub PNG', async () => {const shared = (env) => providerRecords(load(env), env).find(x => x.key === 'shared');assert.ok(!shared({}).capabilities.includes('image'), 'no image capability without AI_IMAGE_MODEL');
@@ -159,6 +159,11 @@ AI_MODEL=
@@ -159,6 +159,11 @@ AI_MODEL=
#AI_MEDIA_STORAGE_HOSTS=*.backblazeb2.com,*.r2.cloudflarestorage.com,edge.openvibe.media
#AI_MEDIA_STORAGE_HOSTS=*.backblazeb2.com,*.r2.cloudflarestorage.com,edge.openvibe.media
#AI_LOCAL_LLM_MODEL=qwen2.5-1.5b-instruct
#AI_LOCAL_LLM_MODEL=qwen2.5-1.5b-instruct
#AI_LOCAL_LLM_TIMEOUT_MS=180000
#AI_LOCAL_LLM_TIMEOUT_MS=180000
+# A second local chat model server (its own llama.cpp process and another model family): provider `local2`,
+# so a caller can pin two local offers, e.g. Games' AI-vs-AI matches. Same rules: no key, free, no JSON.
+#AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1
+#AI_LOCAL_LLM2_MODEL=llama-3.2-1b-instruct
+#AI_LOCAL_LLM2_TIMEOUT_MS=180000
# A second local OpenAI-compatible server for embeddings, when the chat model's llama.cpp process cannot also
# A second local OpenAI-compatible server for embeddings, when the chat model's llama.cpp process cannot also
# serve the embedding GGUF (llama.cpp serves one model per process). Default: AI_LOCAL_LLM_URL.
# serve the embedding GGUF (llama.cpp serves one model per process). Default: AI_LOCAL_LLM_URL.
#AI_LOCAL_EMBED_URL=http://127.0.0.1:8091/v1
#AI_LOCAL_EMBED_URL=http://127.0.0.1:8091/v1
@@ -445,6 +445,7 @@ definition, but never over an admin's version.
@@ -445,6 +445,7 @@ definition, but never over an admin's version.
| `http` | The local HTTP seam: `POST {operation, …}` → `{text, json, usage}` | `AI_HTTP_SEAM_URL` or the admin API |
| `http` | The local HTTP seam: `POST {operation, …}` → `{text, json, usage}` | `AI_HTTP_SEAM_URL` or the admin API |
| `whisper` | whisper.cpp on this host, with Live's VAD, hallucination filter, multilingual model and live/batch lanes; NVIDIA Parakeet TDT 0.6B v3 (`parakeet-cli`) for English and 24 European languages when `PARAKEET_MODEL` is set, falling back to whisper | `WHISPER_*`, `PARAKEET_*`, `ASR_ENGINE` — Live's names |
| `whisper` | whisper.cpp on this host, with Live's VAD, hallucination filter, multilingual model and live/batch lanes; NVIDIA Parakeet TDT 0.6B v3 (`parakeet-cli`) for English and 24 European languages when `PARAKEET_MODEL` is set, falling back to whisper | `WHISPER_*`, `PARAKEET_*`, `ASR_ENGINE` — Live's names |
| `openai`, key `local` | A model server on this host speaking the OpenAI API (llama.cpp's `llama-server`, Ollama's `/v1`): no key, never paid; route `media.local` | `AI_LOCAL_LLM_URL`, `AI_LOCAL_LLM_MODEL`, `AI_LOCAL_LLM_TIMEOUT_MS` |
| `openai`, key `local` | A model server on this host speaking the OpenAI API (llama.cpp's `llama-server`, Ollama's `/v1`): no key, never paid; route `media.local` | `AI_LOCAL_LLM_URL`, `AI_LOCAL_LLM_MODEL`, `AI_LOCAL_LLM_TIMEOUT_MS` |
+| `openai`, key `local2` | A second local chat server, another llama.cpp process and model family: no key, never paid, no JSON or embeddings. Two local offers let a caller pin a match between them (Games' AI-vs-AI job) | `AI_LOCAL_LLM2_URL`, `AI_LOCAL_LLM2_MODEL`, `AI_LOCAL_LLM2_TIMEOUT_MS` |
Routing: a route either names a `capability` and the router builds its pool from every provider that
Routing: a route either names a `capability` and the router builds its pool from every provider that
can serve it, ordered by `openvibe-sdk/placement` (objective, price, measured latency, health, with
can serve it, ordered by `openvibe-sdk/placement` (objective, price, measured latency, health, with
@@ -693,6 +694,15 @@ streams always win: 2 threads, at most 1.5 CPUs, nice 19, 3 GB. It uses about 0.
@@ -693,6 +694,15 @@ streams always win: 2 threads, at most 1.5 CPUs, nice 19, 3 GB. It uses about 0.
28 tokens/s. Without it, `media.analyze` writes an extractive overview and says so. To swap the model, put the
28 tokens/s. Without it, `media.analyze` writes an extractive overview and says so. To swap the model, put the
file beside it, change `--model`/`--alias` and `AI_LOCAL_LLM_MODEL`, and restart both units.
file beside it, change `--model`/`--alias` and `AI_LOCAL_LLM_MODEL`, and restart both units.
+`deploy/systemd/openvibe-llm2.service` is the optional second chat server: the same llama.cpp build and models
+directory on `127.0.0.1:8092`, a different model family, and the provider `local2`
+(`AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1`, `AI_LOCAL_LLM2_MODEL`). It is held to 2 threads, at most 1 CPU,
+nice 19 and 2 GB, and it never advertises `json` or `embed`. Games' AI-vs-AI job pins one local offer per side,
+so the two models must be different families and both aliases must be the `model` in its `GAMES_AI_MATCHES_OFFERS`
+pair. On the production host since 2026-11-06: Llama-3.2-1B-Instruct Q4_K_M (Llama 3.2 Community License,
+Hugging Face sha256 `6f85a640a97cf2bf5b8e764087b1e83da0fdb51d7c9fab7d0fece9385611df83`, 807,694,464 bytes),
+checked against that hash before install; it uses about 0.8 GB.
+
Local embeddings are off unless `AI_LOCAL_EMBED_MODEL` names an embedding model the server at `AI_LOCAL_LLM_URL`
Local embeddings are off unless `AI_LOCAL_EMBED_MODEL` names an embedding model the server at `AI_LOCAL_LLM_URL`
answers at `/embeddings`. Then `local` advertises `embed`, the model is registered as an `embedding` model, and
answers at `/embeddings`. Then `local` advertises `embed`, the model is registered as an `embedding` model, and
`default.embedding` (a capability pool) offers it beside the shared provider's `AI_EMBEDDING_MODEL`; `local` is
`default.embedding` (a capability pool) offers it beside the shared provider's `AI_EMBEDDING_MODEL`; `local` is
new file mode 100644
new file mode 100644
@@ -0,0 +1,34 @@
@@ -0,0 +1,34 @@
+[Unit]
+Description=OpenVibe local model 2 — llama.cpp llama-server, Llama 3.2 1B on 127.0.0.1:8092, for AI-vs-AI matches
+# OpenVibe.AI reaches it as the provider `local2` (AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1). Loopback only,
+# no key. It is the second side of an AI-vs-AI match (Games' GAMES_AI_MATCHES_OFFERS): a different model family
+# from the Qwen unit on 8090, so the match is not a model playing itself. It shares the host with live streams,
+# so it is held back: 2 threads, at most 1 CPU, nice 19 and 2 GB of memory.
+# Install (README "Local model"): llama.cpp's CPU build under /opt/llama.cpp/<build> with /opt/llama.cpp/current,
+# the model in /var/lib/openvibe-llm/models, both checked against their published sha256.
+After=network.target
+
+[Service]
+Type=simple
+User=ubuntu
+Group=ubuntu
+ExecStart=/opt/llama.cpp/current/llama-server \
+ --model /var/lib/openvibe-llm/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
+ --alias llama-3.2-1b-instruct \
+ --host 127.0.0.1 --port 8092 \
+ --threads 2 --ctx-size 8192 --parallel 1 \
+ --no-webui
+Nice=19
+CPUQuota=100%
+MemoryMax=2G
+Restart=on-failure
+RestartSec=10
+TimeoutStopSec=15
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+ReadOnlyPaths=/var/lib/openvibe-llm /opt/llama.cpp
+
+[Install]
+WantedBy=multi-user.target
@@ -176,6 +176,14 @@ function load(env = process.env) {
@@ -176,6 +176,14 @@ function load(env = process.env) {
embedModel: String(env.AI_LOCAL_EMBED_MODEL || '').trim(),
embedModel: String(env.AI_LOCAL_EMBED_MODEL || '').trim(),
timeoutMs: int(env.AI_LOCAL_LLM_TIMEOUT_MS, 180000),
timeoutMs: int(env.AI_LOCAL_LLM_TIMEOUT_MS, 180000),
},
},
+ // A second local chat server (its own llama.cpp process, another model family): the provider `local2`.
+ // Same rules as `local`: no key, never paid, no JSON. Two local offers let a caller pin a match
+ // between models without sending either side to a paid provider.
+ localLlm2: {
+ url: trimUrl(env.AI_LOCAL_LLM2_URL),
+ model: String(env.AI_LOCAL_LLM2_MODEL || '').trim(),
+ timeoutMs: int(env.AI_LOCAL_LLM2_TIMEOUT_MS, 180000),
+ },
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
@@ -55,13 +55,18 @@ function providerRecords(config, env) {
@@ -55,13 +55,18 @@ function providerRecords(config, env) {
sharedProviderRecord(config, env),
sharedProviderRecord(config, env),
{ key: 'whisper', display_name: 'whisper.cpp (local speech-to-text)', kind: 'whisper', status: 'active', auth_mode: 'none', default_model: 'whisper.cpp', capabilities: ['transcribe'], timeout_ms: 600000, priority: 20, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'WHISPER_*', billing_profile: 'free' } },
{ key: 'whisper', display_name: 'whisper.cpp (local speech-to-text)', kind: 'whisper', status: 'active', auth_mode: 'none', default_model: 'whisper.cpp', capabilities: ['transcribe'], timeout_ms: 600000, priority: 20, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'WHISPER_*', billing_profile: 'free' } },
];
];
+ // A local model server (llama.cpp, Ollama) speaking the OpenAI API: no key, never paid (WS-O task 5).
+ // No json: the small local models must not be offered the product JSON routes. Embed only with an embedding model.
+ const localTextCaps = ['chat', 'generate', 'summarize', 'classify', 'extract'];
if (config.localLlm && config.localLlm.url) {
if (config.localLlm && config.localLlm.url) {
- // A local model server (llama.cpp, Ollama) speaking the OpenAI API: no key, never paid (WS-O task 5).
- // No json: the small local model must not be offered the product JSON routes. Embed only with an embedding model.
- const localCaps = ['chat', 'generate', 'summarize', 'classify', 'extract'];
+ const localCaps = [...localTextCaps];
if (config.localLlm.embedModel) localCaps.push('embed');
if (config.localLlm.embedModel) localCaps.push('embed');
out.push({ key: 'local', display_name: 'Local model (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm.model || 'local', capabilities: localCaps, timeout_ms: config.localLlm.timeoutMs, priority: 5, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'AI_LOCAL_LLM_URL', embed_url: config.localLlm.embedUrl || config.localLlm.url, local: true, paid: false, billing_profile: 'free' } });
out.push({ key: 'local', display_name: 'Local model (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm.model || 'local', capabilities: localCaps, timeout_ms: config.localLlm.timeoutMs, priority: 5, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'AI_LOCAL_LLM_URL', embed_url: config.localLlm.embedUrl || config.localLlm.url, local: true, paid: false, billing_profile: 'free' } });
}
}
+ if (config.localLlm2 && config.localLlm2.url) {
+ // The second local chat model: another llama.cpp process and another family, so a match has two sides.
+ out.push({ key: 'local2', display_name: 'Local model 2 (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm2.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm2.model || 'local2', capabilities: [...localTextCaps], timeout_ms: config.localLlm2.timeoutMs, priority: 5, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'AI_LOCAL_LLM2_URL', local: true, paid: false, billing_profile: 'free' } });
+ }
if (config.httpSeamUrl) {
if (config.httpSeamUrl) {
out.push({ key: 'http-seam', display_name: 'Local HTTP seam', kind: 'http', status: 'active', base_url: config.httpSeamUrl, auth_mode: 'none', default_model: null, capabilities: [], timeout_ms: 30000, priority: 60, metadata: { configured_by: 'AI_HTTP_SEAM_URL', billing_profile: 'metered' } });
out.push({ key: 'http-seam', display_name: 'Local HTTP seam', kind: 'http', status: 'active', base_url: config.httpSeamUrl, auth_mode: 'none', default_model: null, capabilities: [], timeout_ms: 30000, priority: 60, metadata: { configured_by: 'AI_HTTP_SEAM_URL', billing_profile: 'metered' } });
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,53 @@
@@ -0,0 +1,53 @@
+'use strict';
+// Two local chat servers are two offered models: `local` and `local2`. A first-party generate run can pin
+// one local offer for each side of an AI-vs-AI match, and neither pin reaches the other side's server.
+const assert = require('assert');
+const { boot, request, token, suite, seamServer, ALL } = require('./helpers');
+
+const t = suite('local-models');
+const firstParty = token('games', ['ai.run.create', 'ai.run.read'], { ns: ['ai.*'] });
+const answer = (model) => ({ model, choices: [{ message: { content: model } }], usage: { prompt_tokens: 3, completion_tokens: 2 } });
+
+let h; let qwen; let llama;
+const calls = { qwen: 0, llama: 0 };
+
+t.test('boot with two local chat servers', async () => {
+ qwen = await seamServer(() => { calls.qwen++; return answer('qwen-test'); });
+ llama = await seamServer(() => { calls.llama++; return answer('llama-test'); });
+ h = await boot({ env: {
+ AI_STUB_FALLBACK: 'false', AI_PROVIDER_RETRIES: '0',
+ AI_LOCAL_LLM_URL: `${qwen.url}/v1`, AI_LOCAL_LLM_MODEL: 'qwen-test',
+ AI_LOCAL_LLM2_URL: `${llama.url}/v1`, AI_LOCAL_LLM2_MODEL: 'llama-test',
+ } });
+});
+
+t.test('GET /api/v1/routes offers both local models', async () => {
+ const r = await request(h.base, 'GET', '/api/v1/routes?operation=generate', { tok: firstParty });
+ assert.strictEqual(r.status, 200, r.text);
+ const ids = (r.body.offers || []).map((o) => `${o.provider}:${o.model}`);
+ assert.ok(ids.includes('local:qwen-test'), `local offer missing from ${ids.join(', ')}`);
+ assert.ok(ids.includes('local2:llama-test'), `local2 offer missing from ${ids.join(', ')}`);
+});
+
+t.test('a pin to each local offer calls only that model', async () => {
+ for (const [provider, model, other] of [['local2', 'llama-test', 'qwen'], ['local', 'qwen-test', 'llama']]) {
+ const before = { qwen: calls.qwen, llama: calls.llama };
+ const r = await request(h.base, 'POST', '/api/v1/generate?wait=5000', { tok: firstParty, body: {
+ prompt: 'pick a hero', route: { provider, model }, max_tokens: 30, options: { cache: false },
+ } });
+ assert.strictEqual(r.status, 201, r.text);
+ assert.strictEqual(r.body.run.provenance.provider, provider);
+ assert.strictEqual(r.body.run.provenance.model, model);
+ assert.strictEqual(r.body.run.provenance.fallback_used, false);
+ assert.match(r.body.run.explain.reasons.join(' '), new RegExp(`pinned to ${provider}/${model}`));
+ assert.strictEqual(calls[other], before[other], `the ${other} server was not called`);
+ }
+});
+
+t.test('shutdown', async () => {
+ await h.stop();
+ await qwen.close();
+ await llama.close();
+});
+
+t.run();
@@ -172,6 +172,19 @@ t.test('local provider: never json, embed only with AI_LOCAL_EMBED_MODEL', () =>
@@ -172,6 +172,19 @@ t.test('local provider: never json, embed only with AI_LOCAL_EMBED_MODEL', () =>
assert.strictEqual(load({ AI_LOCAL_EMBED_MODEL: ' nomic-embed-text ' }).localLlm.embedModel, 'nomic-embed-text');
assert.strictEqual(load({ AI_LOCAL_EMBED_MODEL: ' nomic-embed-text ' }).localLlm.embedModel, 'nomic-embed-text');
});
});
+t.test('second local chat model: provider local2, another family, no json or embed', () => {
+ const records = (env) => providerRecords(load(env), env);
+ assert.strictEqual(records({}).find(x => x.key === 'local2'), undefined, 'no local2 without AI_LOCAL_LLM2_URL');
+ const p = records({ AI_LOCAL_LLM2_URL: 'http://127.0.0.1:8092/v1', AI_LOCAL_LLM2_MODEL: ' llama-3.2-1b-instruct ' }).find(x => x.key === 'local2');
+ assert.ok(p, 'local2 is seeded beside local');
+ assert.strictEqual(p.base_url, 'http://127.0.0.1:8092/v1');
+ assert.strictEqual(p.default_model, 'llama-3.2-1b-instruct');
+ assert.deepStrictEqual(p.capabilities, ['chat', 'generate', 'summarize', 'classify', 'extract'], 'never json or embed');
+ assert.strictEqual(p.timeout_ms, 180000);
+ assert.strictEqual(p.priority, 5);
+ assert.deepStrictEqual([p.metadata.local, p.metadata.paid, p.metadata.billing_profile], [true, false, 'free']);
+});
+
t.test('image: off without AI_IMAGE_MODEL; the Images API body, per-image usage and the stub PNG', async () => {
t.test('image: off without AI_IMAGE_MODEL; the Images API body, per-image usage and the stub PNG', async () => {
const shared = (env) => providerRecords(load(env), env).find(x => x.key === 'shared');
const shared = (env) => providerRecords(load(env), env).find(x => x.key === 'shared');
assert.ok(!shared({}).capabilities.includes('image'), 'no image capability without AI_IMAGE_MODEL');
assert.ok(!shared({}).capabilities.includes('image'), 'no image capability without AI_IMAGE_MODEL');