openvibers/openvibe.ai Commits

A second local chat model: provider local2 (#306)

OP OpenVibers committed 7 days ago · 00ce68f826 · 7 files, +131 −3

Two local llama.cpp servers can now answer as two offered models, so a
first-party generate run can pin one local model per side without a paid
provider. local2 has the same free/no-key rules as local and never
advertises json or embed. Adds the unit for the Llama 3.2 1B server on
127.0.0.1:8092 and the env names, README and tests.

Changes

ddiff --git a/.env.example b/.env.example
iindex fa24146..32e496f 100644
---- a/.env.example
++++ b/.env.example
@@@ -159,6 +159,11 @@ AI_MODEL=
#AI_MEDIA_STORAGE_HOSTS=*.backblazeb2.com,*.r2.cloudflarestorage.com,edge.openvibe.media
#AI_LOCAL_LLM_MODEL=qwen2.5-1.5b-instruct
#AI_LOCAL_LLM_TIMEOUT_MS=180000
++# A second local chat model server (its own llama.cpp process and another model family): provider `local2`,
++# so a caller can pin two local offers, e.g. Games' AI-vs-AI matches. Same rules: no key, free, no JSON.
++#AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1
++#AI_LOCAL_LLM2_MODEL=llama-3.2-1b-instruct
++#AI_LOCAL_LLM2_TIMEOUT_MS=180000
# A second local OpenAI-compatible server for embeddings, when the chat model's llama.cpp process cannot also
# serve the embedding GGUF (llama.cpp serves one model per process). Default: AI_LOCAL_LLM_URL.
#AI_LOCAL_EMBED_URL=http://127.0.0.1:8091/v1
ddiff --git a/README.md b/README.md
iindex 24131e6..452690d 100644
---- a/README.md
++++ b/README.md
@@@ -445,6 +445,7 @@ definition, but never over an admin's version.
| `http` | The local HTTP seam: `POST {operation, …}` → `{text, json, usage}` | `AI_HTTP_SEAM_URL` or the admin API |
| `whisper` | whisper.cpp on this host, with Live's VAD, hallucination filter, multilingual model and live/batch lanes; NVIDIA Parakeet TDT 0.6B v3 (`parakeet-cli`) for English and 24 European languages when `PARAKEET_MODEL` is set, falling back to whisper | `WHISPER_*`, `PARAKEET_*`, `ASR_ENGINE` — Live's names |
| `openai`, key `local` | A model server on this host speaking the OpenAI API (llama.cpp's `llama-server`, Ollama's `/v1`): no key, never paid; route `media.local` | `AI_LOCAL_LLM_URL`, `AI_LOCAL_LLM_MODEL`, `AI_LOCAL_LLM_TIMEOUT_MS` |
++| `openai`, key `local2` | A second local chat server, another llama.cpp process and model family: no key, never paid, no JSON or embeddings. Two local offers let a caller pin a match between them (Games' AI-vs-AI job) | `AI_LOCAL_LLM2_URL`, `AI_LOCAL_LLM2_MODEL`, `AI_LOCAL_LLM2_TIMEOUT_MS` |
Routing: a route either names a `capability` and the router builds its pool from every provider that
can serve it, ordered by `openvibe-sdk/placement` (objective, price, measured latency, health, with
@@@ -693,6 +694,15 @@ streams always win: 2 threads, at most 1.5 CPUs, nice 19, 3 GB. It uses about 0.
28 tokens/s. Without it, `media.analyze` writes an extractive overview and says so. To swap the model, put the
file beside it, change `--model`/`--alias` and `AI_LOCAL_LLM_MODEL`, and restart both units.
++`deploy/systemd/openvibe-llm2.service` is the optional second chat server: the same llama.cpp build and models
++directory on `127.0.0.1:8092`, a different model family, and the provider `local2`
++(`AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1`, `AI_LOCAL_LLM2_MODEL`). It is held to 2 threads, at most 1 CPU,
++nice 19 and 2 GB, and it never advertises `json` or `embed`. Games' AI-vs-AI job pins one local offer per side,
++so the two models must be different families and both aliases must be the `model` in its `GAMES_AI_MATCHES_OFFERS`
++pair. On the production host since 2026-11-06: Llama-3.2-1B-Instruct Q4_K_M (Llama 3.2 Community License,
++Hugging Face sha256 `6f85a640a97cf2bf5b8e764087b1e83da0fdb51d7c9fab7d0fece9385611df83`, 807,694,464 bytes),
++checked against that hash before install; it uses about 0.8 GB.
++
Local embeddings are off unless `AI_LOCAL_EMBED_MODEL` names an embedding model the server at `AI_LOCAL_LLM_URL`
answers at `/embeddings`. Then `local` advertises `embed`, the model is registered as an `embedding` model, and
`default.embedding` (a capability pool) offers it beside the shared provider's `AI_EMBEDDING_MODEL`; `local` is
ddiff --git a/deploy/systemd/openvibe-llm2.service b/deploy/systemd/openvibe-llm2.service
nnew file mode 100644
iindex 0000000..5de96da
---- /dev/null
++++ b/deploy/systemd/openvibe-llm2.service
@@@ -0,0 +1,34 @@
++[Unit]
++Description=OpenVibe local model 2 — llama.cpp llama-server, Llama 3.2 1B on 127.0.0.1:8092, for AI-vs-AI matches
++# OpenVibe.AI reaches it as the provider `local2` (AI_LOCAL_LLM2_URL=http://127.0.0.1:8092/v1). Loopback only,
++# no key. It is the second side of an AI-vs-AI match (Games' GAMES_AI_MATCHES_OFFERS): a different model family
++# from the Qwen unit on 8090, so the match is not a model playing itself. It shares the host with live streams,
++# so it is held back: 2 threads, at most 1 CPU, nice 19 and 2 GB of memory.
++# Install (README "Local model"): llama.cpp's CPU build under /opt/llama.cpp/<build> with /opt/llama.cpp/current,
++# the model in /var/lib/openvibe-llm/models, both checked against their published sha256.
++After=network.target
++
++[Service]
++Type=simple
++User=ubuntu
++Group=ubuntu
++ExecStart=/opt/llama.cpp/current/llama-server \
++ --model /var/lib/openvibe-llm/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
++ --alias llama-3.2-1b-instruct \
++ --host 127.0.0.1 --port 8092 \
++ --threads 2 --ctx-size 8192 --parallel 1 \
++ --no-webui
++Nice=19
++CPUQuota=100%
++MemoryMax=2G
++Restart=on-failure
++RestartSec=10
++TimeoutStopSec=15
++NoNewPrivileges=true
++PrivateTmp=true
++ProtectSystem=strict
++ProtectHome=true
++ReadOnlyPaths=/var/lib/openvibe-llm /opt/llama.cpp
++
++[Install]
++WantedBy=multi-user.target
ddiff --git a/server/config.js b/server/config.js
iindex 092f681..0eada9c 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -176,6 +176,14 @@ function load(env = process.env) {
embedModel: String(env.AI_LOCAL_EMBED_MODEL || '').trim(),
timeoutMs: int(env.AI_LOCAL_LLM_TIMEOUT_MS, 180000),
},
++ // A second local chat server (its own llama.cpp process, another model family): the provider `local2`.
++ // Same rules as `local`: no key, never paid, no JSON. Two local offers let a caller pin a match
++ // between models without sending either side to a paid provider.
++ localLlm2: {
++ url: trimUrl(env.AI_LOCAL_LLM2_URL),
++ model: String(env.AI_LOCAL_LLM2_MODEL || '').trim(),
++ timeoutMs: int(env.AI_LOCAL_LLM2_TIMEOUT_MS, 180000),
++ },
// Image generation (POST /api/v1/image, workflow ai.image): off until AI_IMAGE_MODEL names the shared
// provider's image model (dall-e-3, gpt-image-1); then boot seeds that model and route default.image.
image: { model: String(env.AI_IMAGE_MODEL || '').trim(), size: String(env.AI_IMAGE_SIZE || '').trim() || '1024x1024' },
ddiff --git a/server/workflows/seed.js b/server/workflows/seed.js
iindex 87ed013..1351c30 100644
---- a/server/workflows/seed.js
++++ b/server/workflows/seed.js
@@@ -55,13 +55,18 @@ function providerRecords(config, env) {
sharedProviderRecord(config, env),
{ key: 'whisper', display_name: 'whisper.cpp (local speech-to-text)', kind: 'whisper', status: 'active', auth_mode: 'none', default_model: 'whisper.cpp', capabilities: ['transcribe'], timeout_ms: 600000, priority: 20, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'WHISPER_*', billing_profile: 'free' } },
];
++ // A local model server (llama.cpp, Ollama) speaking the OpenAI API: no key, never paid (WS-O task 5).
++ // No json: the small local models must not be offered the product JSON routes. Embed only with an embedding model.
++ const localTextCaps = ['chat', 'generate', 'summarize', 'classify', 'extract'];
if (config.localLlm && config.localLlm.url) {
-- // A local model server (llama.cpp, Ollama) speaking the OpenAI API: no key, never paid (WS-O task 5).
-- // No json: the small local model must not be offered the product JSON routes. Embed only with an embedding model.
-- const localCaps = ['chat', 'generate', 'summarize', 'classify', 'extract'];
++ const localCaps = [...localTextCaps];
if (config.localLlm.embedModel) localCaps.push('embed');
out.push({ key: 'local', display_name: 'Local model (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm.model || 'local', capabilities: localCaps, timeout_ms: config.localLlm.timeoutMs, priority: 5, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'AI_LOCAL_LLM_URL', embed_url: config.localLlm.embedUrl || config.localLlm.url, local: true, paid: false, billing_profile: 'free' } });
}
++ if (config.localLlm2 && config.localLlm2.url) {
++ // The second local chat model: another llama.cpp process and another family, so a match has two sides.
++ out.push({ key: 'local2', display_name: 'Local model 2 (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm2.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm2.model || 'local2', capabilities: [...localTextCaps], timeout_ms: config.localLlm2.timeoutMs, priority: 5, provider_access: 'OWN_DEPLOYMENT', metadata: { configured_by: 'AI_LOCAL_LLM2_URL', local: true, paid: false, billing_profile: 'free' } });
++ }
if (config.httpSeamUrl) {
out.push({ key: 'http-seam', display_name: 'Local HTTP seam', kind: 'http', status: 'active', base_url: config.httpSeamUrl, auth_mode: 'none', default_model: null, capabilities: [], timeout_ms: 30000, priority: 60, metadata: { configured_by: 'AI_HTTP_SEAM_URL', billing_profile: 'metered' } });
}
ddiff --git a/test/local-models.test.js b/test/local-models.test.js
nnew file mode 100644
iindex 0000000..72d1123
---- /dev/null
++++ b/test/local-models.test.js
@@@ -0,0 +1,53 @@
++'use strict';
++// Two local chat servers are two offered models: `local` and `local2`. A first-party generate run can pin
++// one local offer for each side of an AI-vs-AI match, and neither pin reaches the other side's server.
++const assert = require('assert');
++const { boot, request, token, suite, seamServer, ALL } = require('./helpers');
++
++const t = suite('local-models');
++const firstParty = token('games', ['ai.run.create', 'ai.run.read'], { ns: ['ai.*'] });
++const answer = (model) => ({ model, choices: [{ message: { content: model } }], usage: { prompt_tokens: 3, completion_tokens: 2 } });
++
++let h; let qwen; let llama;
++const calls = { qwen: 0, llama: 0 };
++
++t.test('boot with two local chat servers', async () => {
++ qwen = await seamServer(() => { calls.qwen++; return answer('qwen-test'); });
++ llama = await seamServer(() => { calls.llama++; return answer('llama-test'); });
++ h = await boot({ env: {
++ AI_STUB_FALLBACK: 'false', AI_PROVIDER_RETRIES: '0',
++ AI_LOCAL_LLM_URL: `${qwen.url}/v1`, AI_LOCAL_LLM_MODEL: 'qwen-test',
++ AI_LOCAL_LLM2_URL: `${llama.url}/v1`, AI_LOCAL_LLM2_MODEL: 'llama-test',
++ } });
++});
++
++t.test('GET /api/v1/routes offers both local models', async () => {
++ const r = await request(h.base, 'GET', '/api/v1/routes?operation=generate', { tok: firstParty });
++ assert.strictEqual(r.status, 200, r.text);
++ const ids = (r.body.offers || []).map((o) => `${o.provider}:${o.model}`);
++ assert.ok(ids.includes('local:qwen-test'), `local offer missing from ${ids.join(', ')}`);
++ assert.ok(ids.includes('local2:llama-test'), `local2 offer missing from ${ids.join(', ')}`);
++});
++
++t.test('a pin to each local offer calls only that model', async () => {
++ for (const [provider, model, other] of [['local2', 'llama-test', 'qwen'], ['local', 'qwen-test', 'llama']]) {
++ const before = { qwen: calls.qwen, llama: calls.llama };
++ const r = await request(h.base, 'POST', '/api/v1/generate?wait=5000', { tok: firstParty, body: {
++ prompt: 'pick a hero', route: { provider, model }, max_tokens: 30, options: { cache: false },
++ } });
++ assert.strictEqual(r.status, 201, r.text);
++ assert.strictEqual(r.body.run.provenance.provider, provider);
++ assert.strictEqual(r.body.run.provenance.model, model);
++ assert.strictEqual(r.body.run.provenance.fallback_used, false);
++ assert.match(r.body.run.explain.reasons.join(' '), new RegExp(`pinned to ${provider}/${model}`));
++ assert.strictEqual(calls[other], before[other], `the ${other} server was not called`);
++ }
++});
++
++t.test('shutdown', async () => {
++ await h.stop();
++ await qwen.close();
++ await llama.close();
++});
++
++t.run();
ddiff --git a/test/providers.test.js b/test/providers.test.js
iindex 4b1caa9..24d5bd7 100644
---- a/test/providers.test.js
++++ b/test/providers.test.js
@@@ -172,6 +172,19 @@ t.test('local provider: never json, embed only with AI_LOCAL_EMBED_MODEL', () =>
assert.strictEqual(load({ AI_LOCAL_EMBED_MODEL: ' nomic-embed-text ' }).localLlm.embedModel, 'nomic-embed-text');
});
++t.test('second local chat model: provider local2, another family, no json or embed', () => {
++ const records = (env) => providerRecords(load(env), env);
++ assert.strictEqual(records({}).find(x => x.key === 'local2'), undefined, 'no local2 without AI_LOCAL_LLM2_URL');
++ const p = records({ AI_LOCAL_LLM2_URL: 'http://127.0.0.1:8092/v1', AI_LOCAL_LLM2_MODEL: ' llama-3.2-1b-instruct ' }).find(x => x.key === 'local2');
++ assert.ok(p, 'local2 is seeded beside local');
++ assert.strictEqual(p.base_url, 'http://127.0.0.1:8092/v1');
++ assert.strictEqual(p.default_model, 'llama-3.2-1b-instruct');
++ assert.deepStrictEqual(p.capabilities, ['chat', 'generate', 'summarize', 'classify', 'extract'], 'never json or embed');
++ assert.strictEqual(p.timeout_ms, 180000);
++ assert.strictEqual(p.priority, 5);
++ assert.deepStrictEqual([p.metadata.local, p.metadata.paid, p.metadata.billing_profile], [true, false, 'free']);
++});
++
t.test('image: off without AI_IMAGE_MODEL; the Images API body, per-image usage and the stub PNG', async () => {
const shared = (env) => providerRecords(load(env), env).find(x => x.key === 'shared');
assert.ok(!shared({}).capabilities.includes('image'), 'no image capability without AI_IMAGE_MODEL');