openvibers/openvibe.ai Commits

Local models run two slots (plan BA4) (#358)

OP OpenVibers committed 6 days ago · fe31bb76a5 · 3 files, +12 −2

llama.cpp batches the slots into each step; the 8192-token context is split (4096 a slot, ample for media.analyze,
Games and the background tier's small jobs). Threads, CPU and memory limits are unchanged, so a stream still wins.

Changes

ddiff --git a/README.md b/README.md
iindex fc29aea..5a98e86 100644
---- a/README.md
++++ b/README.md
@@@ -892,6 +892,10 @@ the release's sha256) in `/opt/llama.cpp/b11222` with `/opt/llama.cpp/current`,
  streams always win: 2 threads, at most 1.5 CPUs, nice 19, 3 GB. It uses about 0.9 GB and generates about
  28 tokens/s. Without it, `media.analyze` writes an extractive overview and says so. To swap the model, put the
  file beside it, change `--model`/`--alias` and `AI_LOCAL_LLM_MODEL`, and restart both units.
++Both servers run two slots (`--parallel 2`, 2026-11-07): llama.cpp batches the slots into each step, so a second
++request no longer waits for the first; the 8192-token context is split, 4096 a slot. Install a changed unit by hand:
++`sudo install -m 0644 deploy/systemd/openvibe-llm*.service /etc/systemd/system/ && sudo systemctl daemon-reload &&
++sudo systemctl restart openvibe-llm openvibe-llm2` (AI falls over to `shared` for the seconds they restart).
  
  `deploy/systemd/openvibe-llm2.service` is the optional second chat server: the same llama.cpp build and models
  directory on `127.0.0.1:8092`, a different model family, and the provider `local2`
ddiff --git a/deploy/systemd/openvibe-llm.service b/deploy/systemd/openvibe-llm.service
iindex 8bbd556..7b3f302 100644
---- a/deploy/systemd/openvibe-llm.service
++++ b/deploy/systemd/openvibe-llm.service
@@@ -10,6 +10,9 @@ Description=OpenVibe local model server — llama.cpp llama-server, OpenAI API o
  # restricts the server to embeddings, so this chat unit runs without it.
  After=network.target
  
++# Two slots (2026-11-07, plan BA4): llama.cpp batches both into each step, so a second request no longer waits for
++# the first to finish. The 8192-token context is split (4096 a slot, ample for these routes); threads, CPU and memory
++# limits are unchanged, so a stream still always wins.
  [Service]
  Type=simple
  User=ubuntu
@@@ -18,7 +21,7 @@ ExecStart=/opt/llama.cpp/current/llama-server \
      --model /var/lib/openvibe-llm/models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
      --alias qwen2.5-1.5b-instruct \
      --host 127.0.0.1 --port 8090 \
--    --threads 2 --ctx-size 8192 --parallel 1 \
++    --threads 2 --ctx-size 8192 --parallel 2 \
      --no-webui
  Nice=19
  CPUQuota=150%
ddiff --git a/deploy/systemd/openvibe-llm2.service b/deploy/systemd/openvibe-llm2.service
iindex 5de96da..f6e9ab4 100644
---- a/deploy/systemd/openvibe-llm2.service
++++ b/deploy/systemd/openvibe-llm2.service
@@@ -8,6 +8,9 @@ Description=OpenVibe local model 2 — llama.cpp llama-server, Llama 3.2 1B on 1
  # the model in /var/lib/openvibe-llm/models, both checked against their published sha256.
  After=network.target
  
++# Two slots (2026-11-07, plan BA4): llama.cpp batches both into each step, so a second request no longer waits for
++# the first to finish. The 8192-token context is split (4096 a slot, ample for these routes); threads, CPU and memory
++# limits are unchanged, so a stream still always wins.
  [Service]
  Type=simple
  User=ubuntu
@@@ -16,7 +19,7 @@ ExecStart=/opt/llama.cpp/current/llama-server \
      --model /var/lib/openvibe-llm/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
      --alias llama-3.2-1b-instruct \
      --host 127.0.0.1 --port 8092 \
--    --threads 2 --ctx-size 8192 --parallel 1 \
++    --threads 2 --ctx-size 8192 --parallel 2 \
      --no-webui
  Nice=19
  CPUQuota=100%