openvibers/openvibe.link Commits
Vibe by voice: a voice plugin hears the call word on the machine and sends only what follows (voice_ask); the core routes Bot's voice_answer back and passes the owner's voice setting from config (contracts 0.214.0) (#58)
Changes
@@@ -98,8 +98,9 @@ jobs:- run: python -m pytest -q plugins/relay/tests- run: python -m pytest -q plugins/ros2/tests- run: python -m pytest -q plugins/computer/tests++ - run: python -m pytest -q plugins/voice/tests- name: packages install-- run: pip install ./plugins/sdk ./plugins/dryrun ./plugins/adeept_adr036 ./plugins/cozmo ./plugins/relay ./plugins/ros2 ./plugins/computer++ run: pip install ./plugins/sdk ./plugins/dryrun ./plugins/adeept_adr036 ./plugins/cozmo ./plugins/relay ./plugins/ros2 ./plugins/computer ./plugins/voiceshell:name: shellcheck@@@ -23,7 +23,7 @@ jobs:- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true- run: go vet ./... && go test -count=1 ./...- run: |-- for p in sdk dryrun adeept_adr036 cozmo relay ros2 computer; do python -m pytest -q plugins/$p/tests; done++ for p in sdk dryrun adeept_adr036 cozmo relay ros2 computer voice; do python -m pytest -q plugins/$p/tests; done- run: scripts/dist.sh "${GITHUB_REF_NAME}"# Every release must carry the installer OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are# documented to serve (releases/latest/download/install.sh), the plugin bundle the installer downloads, and the@@@ -55,6 +55,8 @@ describes itself and waits for EOF without opening any hardware.| `resume` | | accept motion again || `approval` | `id`, `approved` bool | the owner's answer to an `approval` event it raised || `tray` | `attached` bool, `ttl_s` | (computer plugin only) the tray shows its approvals for `ttl_s` s unless told again: its own desktop prompts may stay quiet |++| `voice` | `on` bool, `call_word` | (plugins that describe a `voice` capability) the owner's Vibe by voice setting from Bot's config: listen for `call_word` only while `on` ([voice.md](voice.md)) |++| `voice_answer` | `id`, `state` (working, answered, refused), `text`, `url` | (voice plugins) Vibe's answer to a `voice_ask` event the plugin raised: speak and show it |`value` has already been validated and clamped to the owner's limits by the core ([protocol.md](protocol.md#commands)).`deadline_ms` is relative to receipt and already capped.@@@ -87,6 +89,11 @@ Python: `cmd.turn_epoch` (an int, or `None`).| `video` | `format` (`jpeg`), `path`, optional `width`, `height`, `seq` || `local_stop` | `reason` — the driver's own stop (a button, a sensor it trusts): latches the Node's local stop, see below |++Two `event` names are the core's own, for voice plugins: `voice_ready` asks for the owner's setting (the core answers++with `voice`), and `voice_ask` (`id`, `text`, optional `lang`) is what the person said after the call word. The core sends++it to Bot as a `voice_ask` frame and never as a telemetry event, and routes the `voice_answer` frames back to the plugin++that asked.++For `video`, the plugin writes each frame to a file (write to a temporary name, then rename, and rotate a few names soa slow reader never sees a partial file) and announces it with `{"op": "video", "format": "jpeg", "path": …}`(`emit_video` in [`plugins/sdk`](../plugins/sdk)). The core reads the file at once and encodes it: with `ffmpeg` ifnnew file mode 100644@@@ -0,0 +1,41 @@++# Vibe by voice++++Say your machine's call word ("hey vibe" unless you chose another), then ask: Vibe, OpenVibe's assistant, answers out++loud on the machine. The question and the answer are also kept in your "Voice on <machine>" chat on openvibe.chat.++++## What leaves the machine++++Only what you say **after** the call word. The voice plugin listens on the machine itself and recognises speech++locally. Words before the call word are dropped as they are heard, nothing is recorded, and no audio is sent anywhere.++The text of your question goes to OpenVibe.Bot as a `voice_ask` frame (contracts 0.214.0). Bot asks Vibe as you, the++machine's owner, through OpenVibe.Chat, and sends the answer back as `voice_answer` frames. Answers count against your++daily Vibe allowance like any other.++++## Turning it on++++1. Add the plugin to the Node's config (`config.json`, `plugins`):++++ ```json++ { "name": "voice", "config": { "model": "/home/you/vosk-model-small-en-us-0.15", "lang": "en-US" } }++ ```++++2. Install what it uses: `pip install vosk`, a model from https://alphacephei.com/vosk/models (the small one for your++ language is enough), `arecord` (alsa-utils) or `parec` (PulseAudio or PipeWire) for the microphone, and something to++ speak with (`espeak-ng`, or Piper with `piper_model` set).++3. On https://openvibe.bot/machines, press **Let Vibe answer my voice here** on the machine and choose its call word.++ The Node hears about it in Bot's config at once. **Stop Vibe answering my voice here** turns listening off again.++++## Plugin settings++++| Setting | Default | What |++|---|---|---|++| `engine` | `vosk` | `vosk` (a local model, from the microphone) or `file` (each new line of `heard_file` is one thing heard: tests, or another recogniser feeding it) |++| `model` | | the Vosk model's directory |++| `lang` | | the language you speak, BCP 47 (`en-US`) |++| `speak` | `auto` | `auto` (Piper when `piper_model` is set, else espeak-ng, else spd-say), `none`, or an argv that reads the text on stdin |++| `piper_model` | | a Piper voice (`.onnx`), spoken through `aplay` |++| `notify` | `true` | also show answers as desktop notifications (notify-send) |++| `follow_up_s` | `8` | after the call word alone, how long the next thing you say counts as the question |++++The call word counts only when said whole ("hey vibration" is not "hey vibe"). Speech-to-text's other spellings of a++greeting count ("okay vibe", "hi vibe"), as does the plural ("hey vibes").@@@ -343,6 +343,7 @@ var Bundled = map[string]string{"relay": "openvibe_relay","ros2": "openvibe_ros2","computer": "openvibe_computer",++ "voice": "openvibe_voice",}// Default is the config written by `openvibe-node pair` when none exists: the dry-run plugin and its test pattern.@@@ -78,12 +78,17 @@ type Node struct {allowed map[string]booltelemetry map[string]map[string]any // per plugin, latestevents []protocol.Event-- telemMS int-- jpeg *video.JPEGSource-- jpegFrom string-- videoSrc string-- lastTelem time.Time-- started time.Time++ // Vibe by voice (contracts 0.214.0): the owner's setting from Bot's config, and which plugin asked each voice_ask++ // still waiting for its answer.++ voiceOn bool++ callWord string++ voiceAskedBy map[string]string++ telemMS int++ jpeg *video.JPEGSource++ jpegFrom string++ videoSrc string++ lastTelem time.Time++ started time.Time// configured is set by the first config on a connection; until then only halt runs, so a latch the owner set// while the link was down is applied before any motion.@@@ -310,6 +315,8 @@ func (n *Node) Frame(f protocol.Frame) {n.job(m)case protocol.JobCancel:n.jobCancel(m)++ case protocol.VoiceAnswer:++ n.voiceAnswer(m)case protocol.JobExitAck:if n.worker != nil {n.worker.ExitAck(m.ID)@@@ -370,6 +377,65 @@ func (n *Node) applyConfig(c protocol.Config) {}n.log.Info("config", "max_speed", l.MaxSpeed, "max_turn", l.MaxTurn, "max_command_ms", l.MaxCommandMS,"allowed", c.AllowedCommands, "estop_latched", c.EstopLatched)++ n.mu.Lock()++ n.voiceOn, n.callWord = c.Voice, c.CallWord++ n.mu.Unlock()++ n.tellVoice(nil)++}++++// tellVoice gives the voice plugins (or just p) the owner's voice setting: they listen for the call word only while on.++func (n *Node) tellVoice(only *plugins.Plugin) {++ n.mu.Lock()++ on, word := n.voiceOn, n.callWord++ n.mu.Unlock()++ for _, p := range n.mgr.Plugins() {++ if (only == nil || p == only) && p.HasCapability("voice") {++ if err := p.Voice(on, word); err != nil {++ n.log.Debug("voice setting not delivered", "plugin", p.Name(), "err", err)++ }++ }++ }++}++++// voiceAsk sends a voice plugin's question to Bot (never into telemetry: what the person said is theirs) and remembers++// which plugin asked, so the answers reach it.++func (n *Node) voiceAsk(pluginName string, data map[string]any) {++ id, _ := data["id"].(string)++ text, _ := data["text"].(string)++ lang, _ := data["lang"].(string)++ if id == "" || text == "" || len(id) > 64 || len(text) > 2000 {++ n.log.Warn("voice_ask from a plugin without id or text, or too long: dropped", "plugin", pluginName)++ return++ }++ n.mu.Lock()++ if n.voiceAskedBy == nil {++ n.voiceAskedBy = map[string]string{}++ }++ if len(n.voiceAskedBy) > 100 {++ n.voiceAskedBy = map[string]string{}++ }++ n.voiceAskedBy[id] = pluginName++ n.mu.Unlock()++ n.log.Info("voice question to Vibe", "plugin", pluginName, "id", id, "chars", len(text))++ n.send(protocol.VoiceAsk{ID: id, Text: text, Lang: lang})++}++++// voiceAnswer hands Bot's answer to the plugin that asked; a final answer (not working) forgets the question.++func (n *Node) voiceAnswer(a protocol.VoiceAnswer) {++ n.mu.Lock()++ name := n.voiceAskedBy[a.ID]++ if a.State != "working" {++ delete(n.voiceAskedBy, a.ID)++ }++ n.mu.Unlock()++ p := n.mgr.Get(name)++ if name == "" || p == nil {++ n.log.Debug("voice_answer for no waiting question", "id", a.ID)++ return++ }++ if err := p.VoiceAnswer(a.ID, a.State, a.Text, a.URL); err != nil {++ n.log.Warn("voice answer not delivered", "plugin", name, "err", err)++ }}// command validates and routes one command. The synchronous part (checks, clamping, queueing to the plugin) keeps@@@ -696,6 +762,16 @@ func (n *Node) output(o plugins.Output) {n.mu.Unlock()case plugins.OutEvent:name, _ := o.Data["name"].(string)++ // Vibe by voice: a question goes to Bot on its own (never as a telemetry event); a voice plugin that has just++ // started asks for the owner's setting.++ if name == "voice_ask" {++ n.voiceAsk(o.Plugin, o.Data)++ return++ }++ if name == "voice_ready" {++ n.tellVoice(n.mgr.Get(o.Plugin))++ return++ }fields := map[string]any{}for k, v := range o.Data {if k != "name" {nnew file mode 100644@@@ -0,0 +1,138 @@++//go:build !windows++++package node++++import (++ "context"++ "encoding/json"++ "os"++ "os/exec"++ "path/filepath"++ "runtime"++ "strings"++ "testing"++ "time"++++ "github.com/OpenVibers/OpenVibe.Node/internal/config"++ "github.com/OpenVibers/OpenVibe.Node/internal/fakebot"++ "github.com/OpenVibers/OpenVibe.Node/internal/link"++ "github.com/OpenVibers/OpenVibe.Node/internal/plugins"++ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"++)++++// TestVoiceEndToEnd runs the real voice plugin (its file engine stands in for the microphone) under a Node paired with++// the fake Bot: nothing goes to Bot until the owner's config turns voice on; then only what follows the call word goes,++// as a voice_ask frame (never as a telemetry event), and Bot's answer reaches the plugin, which speaks it.++func TestVoiceEndToEnd(t *testing.T) {++ if runtime.GOOS == "windows" {++ t.Skip("end-to-end tests run on unix")++ }++ py, err := exec.LookPath("python3")++ if err != nil {++ t.Skip("python3 not installed")++ }++ root, _ := filepath.Abs("../..")++ home := t.TempDir()++ heard := filepath.Join(home, "heard.txt")++ said := filepath.Join(home, "said.txt")++ paths := config.HomePaths(home)++ srv := fakebot.New()++ t.Cleanup(srv.Close)++ srv.Config = protocol.Config{HeartbeatMS: 100, Limits: protocol.Limits{MaxCommandMS: 1000}, AllowedCommands: []string{protocol.KindHalt}}++ srv.AddCode("VOICE234")++ creds, err := link.Pair(context.Background(), nil, srv.URL(), protocol.PairRequest{Code: "VOICE234", AgentVersion: "test", DeviceKind: "onboard", Drivers: []string{"voice"}})++ if err != nil {++ t.Fatal(err)++ }++ cfg := config.Default()++ cfg.Python = py++ cfg.Plugins = []config.PluginConfig{{Name: "voice", Config: map[string]any{"engine": "file", "heard_file": heard, "notify": false,++ "speak": []string{"sh", "-c", `cat >> "$0"`, said}},++ Env: map[string]string{"PYTHONPATH": filepath.Join(root, "plugins/sdk") + string(os.PathListSeparator) + filepath.Join(root, "plugins/voice")}}}++ cfg.Video = config.VideoConfig{Source: "off"}++ n, err := New(Options{Config: cfg, Paths: paths, Creds: creds, Version: "test", HeartbeatInterval: 100 * time.Millisecond,++ LinkBackoffMin: 50 * time.Millisecond, LatchPoll: 50 * time.Millisecond, Log: testLog()})++ if err != nil {++ t.Fatal(err)++ }++ ctx, cancel := context.WithCancel(context.Background())++ done := make(chan struct{})++ go func() { n.Run(ctx); close(done) }()++ e := &env{t: t, srv: srv, node: n, paths: paths, cancel: cancel, done: done}++ t.Cleanup(e.stop)++ e.conn = e.nextConn()++ if _, err := e.conn.Expect(protocol.TypeStatus, 20*time.Second, func(m protocol.Message) bool {++ s := m.(protocol.Status)++ return len(s.Drivers) == 1 && s.Drivers[0].State == plugins.StateReady++ }); err != nil {++ t.Fatal(err)++ }++ appendLine := func(s string) {++ f, err := os.OpenFile(heard, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600)++ if err != nil {++ t.Fatal(err)++ }++ _, _ = f.WriteString(s + "\n")++ f.Close()++ }++++ // Voice off (the first config did not say voice): the plugin is not listening, nothing reaches Bot.++ appendLine("hey vibe are you there")++ if _, err := e.conn.Expect(protocol.TypeVoiceAsk, 700*time.Millisecond, nil); err == nil {++ t.Fatal("a voice_ask while voice is off")++ }++++ // The owner turns voice on: the core passes it to the plugin, which listens for the call word.++ on := srv.Config++ on.Voice, on.CallWord = true, "hey desk"++ if err := e.conn.Send(on); err != nil {++ t.Fatal(err)++ }++ time.Sleep(400 * time.Millisecond)++ appendLine("so anyway the meeting ran long")++ appendLine("Hey desk, what is on my calendar this afternoon?")++ f, err := e.conn.Expect(protocol.TypeVoiceAsk, 10*time.Second, nil)++ if err != nil {++ t.Fatal(err)++ }++ ask := f.Msg.(protocol.VoiceAsk)++ if ask.Text != "what is on my calendar this afternoon" || !strings.HasPrefix(ask.ID, "va-") {++ t.Fatalf("voice_ask %+v", ask)++ }++ for _, fr := range e.conn.Frames() {++ if tel, ok := fr.Msg.(protocol.Telemetry); ok {++ for _, ev := range tel.Events {++ if ev.Name == "voice_ask" {++ t.Fatal("the question went out as a telemetry event")++ }++ }++ }++ if fr.Type == protocol.TypeVoiceAsk && strings.Contains(string(mustJSON(fr.Msg)), "meeting") {++ t.Fatal("what was not for Vibe left the machine")++ }++ }++++ // Bot answers working, then with Vibe's answer: the plugin speaks the answer, not the working line.++ if err := e.conn.Send(protocol.VoiceAnswer{ID: ask.ID, State: "working", Text: "On it."}); err != nil {++ t.Fatal(err)++ }++ if err := e.conn.Send(protocol.VoiceAnswer{ID: ask.ID, State: "answered", Text: "A call at three, nothing after four.", URL: "https://openvibe.chat/vibe/v-0123456789ab"}); err != nil {++ t.Fatal(err)++ }++ deadline := time.Now().Add(10 * time.Second)++ for {++ b, _ := os.ReadFile(said)++ if string(b) == "A call at three, nothing after four." {++ break++ }++ if time.Now().After(deadline) {++ t.Fatalf("spoken %q", string(b))++ }++ time.Sleep(50 * time.Millisecond)++ }++}++++func mustJSON(v any) []byte {++ b, _ := json.Marshal(v)++ return b++}@@@ -206,6 +206,27 @@ func (p *Plugin) Answer(id string, approved bool) error {return p.send(map[string]any{"op": "approval", "id": id, "approved": approved})}++// Voice tells a voice plugin whether its owner lets Vibe answer their voice here, and the call word to listen for.++func (p *Plugin) Voice(on bool, callWord string) error {++ return p.send(map[string]any{"op": "voice", "on": on, "call_word": callWord})++}++++// VoiceAnswer gives a voice plugin Vibe's answer to a voice_ask it raised (to speak and show).++func (p *Plugin) VoiceAnswer(id, state, text, url string) error {++ return p.send(map[string]any{"op": "voice_answer", "id": id, "state": state, "text": text, "url": url})++}++++// HasCapability says whether the plugin's describe lists the capability (false before it described itself).++func (p *Plugin) HasCapability(name string) bool {++ p.mu.Lock()++ defer p.mu.Unlock()++ if p.desc == nil {++ return false++ }++ _, ok := p.desc.Capabilities[name]++ return ok++}++// Tray tells the plugin whether the Node's tray shows its approvals, for the next ttlS seconds unless told again.func (p *Plugin) Tray(on bool, ttlS int) error {return p.send(map[string]any{"op": "tray", "attached": on, "ttl_s": ttlS})@@@ -39,6 +39,8 @@ const (TypeEstop = "estop" // latched true sets the e-stop, false is the owner clearing itTypeHeartbeatAck = "heartbeat_ack"TypeError = "error"++ // TypeVoiceAnswer is Vibe's answer to a voice_ask (contracts 0.214.0): working, answered or refused, to speak.++ TypeVoiceAnswer = "voice_answer")// Close codes the server ends the device socket with.@@@ -63,6 +65,8 @@ const (TypeHeartbeat = "heartbeat"TypeReauth = "reauth" // a Network-paired machine's fresh node token, before the last one expiresTypeEstopState = "estop_state"++ // TypeVoiceAsk is what the owner said after the machine's call word, transcribed on the machine (contracts 0.214.0).++ TypeVoiceAsk = "voice_ask")// Job frames (OpenVibe.Contracts platform.job-frame@1). Server → device: job, job_cancel, job_exit_ack. Device →@@@ -299,6 +303,26 @@ type Config struct {// empty list allows only halt.AllowedCommands []string `json:"allowed_commands"`EstopLatched bool `json:"estop_latched"`++ // Voice says the owner lets Vibe answer their voice on this machine (contracts 0.214.0): a voice plugin listens for++ // CallWord only while it is true.++ Voice bool `json:"voice,omitempty"`++ CallWord string `json:"call_word,omitempty"`++}++++// VoiceAsk is the owner's question after the machine's call word (device → server).++type VoiceAsk struct {++ ID string `json:"id"`++ Text string `json:"text"`++ Lang string `json:"lang,omitempty"`++ RobotID string `json:"robot_id,omitempty"`++}++++// VoiceAnswer is Vibe's answer to a VoiceAsk with the same ID (server → device): State is working, answered or refused.++type VoiceAnswer struct {++ ID string `json:"id"`++ State string `json:"state"`++ Text string `json:"text,omitempty"`++ URL string `json:"url,omitempty"`}// Operator is who sent a command, as the server's gate saw them.@@@ -649,6 +673,8 @@ func (Nack) MessageType() string { return TypeNack }func (Heartbeat) MessageType() string { return TypeHeartbeat }func (Reauth) MessageType() string { return TypeReauth }func (EstopState) MessageType() string { return TypeEstopState }++func (VoiceAsk) MessageType() string { return TypeVoiceAsk }++func (VoiceAnswer) MessageType() string { return TypeVoiceAnswer }func (JobRequest) MessageType() string { return TypeJob }func (JobCancel) MessageType() string { return TypeJobCancel }func (JobExitAck) MessageType() string { return TypeJobExitAck }@@@ -678,6 +704,8 @@ var registry = map[string]func() Message{TypeJobStdout: func() Message { return &JobStdout{} },TypeJobUsage: func() Message { return &JobUsage{} },TypeJobExit: func() Message { return &JobExit{} },++ TypeVoiceAsk: func() Message { return &VoiceAsk{} },++ TypeVoiceAnswer: func() Message { return &VoiceAnswer{} },}// ErrUnknownType is returned (wrapped) by Decode for a frame whose type is not in this version's message set.@@@ -777,6 +805,10 @@ func derefMessage(m Message) Message {return *vcase *JobExit:return *v++ case *VoiceAsk:++ return *v++ case *VoiceAnswer:++ return *v}return m}@@@ -137,6 +137,15 @@ class Plugin:"""The owner answered, on the machine (`openvibe-node approve|deny <id>`), an approval this plugin asked forwith emit_event("approval", id=…). Runs on the runtime's main thread."""++ def on_voice(self, on, call_word):++ """Vibe by voice (contracts 0.214.0): the owner lets Vibe answer their voice on this machine (on) and chose++ call_word, or stopped it. Sent to plugins whose describe lists the `voice` capability, after every config from++ Bot and when the plugin emits the voice_ready event. A voice plugin listens for the call word only while on."""++++ def on_voice_answer(self, cid, state, text, url):++ """Vibe's answer to a voice_ask this plugin emitted (emit_event("voice_ask", id=…, text=…)): state is working,++ answered or refused; text is what to say, url the Vibe chat that keeps it."""++def on_tray(self, attached, ttl_s):"""The Node's tray (`openvibe-node tray`, in the owner's graphical session) is showing this plugin's approvals,for the next ttl_s seconds unless told again. A plugin that notifies the desktop itself can stay quiet then.@@@ -293,6 +302,16 @@ class Runtime:self.plugin.on_tray(msg.get("attached") is True, msg.get("ttl_s"))except Exception as e:self.plugin.log("tray handler failed: %r", e)++ elif op == "voice":++ try:++ self.plugin.on_voice(msg.get("on") is True, str(msg.get("call_word") or ""))++ except Exception as e:++ self.plugin.log("voice handler failed: %r", e)++ elif op == "voice_answer":++ try:++ self.plugin.on_voice_answer(str(msg.get("id") or ""), str(msg.get("state") or ""), str(msg.get("text") or ""), str(msg.get("url") or ""))++ except Exception as e:++ self.plugin.log("voice answer handler failed: %r", e)elif op == "resume":if self.local_stop_pending:self.plugin.log("resume ignored: the local stop has not reached the core yet")nnew file mode 100644@@@ -0,0 +1,256 @@++"""OpenVibe Node voice plugin: Vibe by voice on the person's own machine (contracts 0.214.0; consumer plan §8).++++The owner lets Vibe answer their voice on this machine and chooses a call word on openvibe.bot/machines; Bot's config++says so (the core passes it on as the ``voice`` op). While it is on, this plugin listens on the machine itself, and++only what the person says *after* the call word leaves it: as a ``voice_ask`` event the core sends to Bot, which asks++Vibe as the owner and sends ``voice_answer`` frames back. The answer is spoken here and shown as a desktop notification,++and kept in the owner's "Voice on <machine>" chat on openvibe.chat.++++Nothing is recorded and no audio leaves the machine: recognition runs locally, and the words before the call word are++dropped as they are heard.++++Config (the plugin's block in config.json):++++* ``engine``: ``vosk`` (the default: a local Vosk model, ``model`` = its directory, audio from ``arecord`` or ``parec``)++ or ``file`` (each new line appended to ``heard_file`` is one thing heard; for tests and for wiring other recognisers).++* ``lang``: the language of the model, BCP 47 (``en-US``).++* ``speak``: ``auto`` (Piper when ``piper_model`` is set, else espeak-ng, else spd-say), ``none``, or an argv list that++ reads the text on stdin (``["sh", "-c", "cat >> said.txt"]``).++* ``notify``: show answers as desktop notifications (notify-send), true by default.++* ``follow_up_s``: after the call word alone, how long the next thing heard counts as the question (8 s).++"""++++import json++import os++import shutil++import subprocess++import threading++import time++import unicodedata++import uuid++++from openvibe_plugin import Fault, Plugin++++VERSION = "0.1.0"++SAMPLE_RATE = 16000++# Speech-to-text writes a spoken greeting several ways (as Live's stream call word, server/ai/vibe-voice.js).++GREETINGS = {"hey", "hi", "hay", "heyy", "ok", "okay", "yo"}++++++def norm(s):++ """Lower case, accents off, punctuation to spaces, one space between words."""++ s = unicodedata.normalize("NFKD", str(s or "").lower())++ s = "".join(c for c in s if not unicodedata.combining(c))++ return " ".join("".join(c if (c.isalnum() or c in "' ") else " " for c in s).split())++++++def word_matches(heard, want, first):++ h = heard[:-2] if heard.endswith("'s") else heard++ h, w = h.replace("'", ""), want.replace("'", "")++ if h == w or h == w + "s" or h + "s" == w:++ return True++ return first and w in GREETINGS and h in GREETINGS++++++def after_call_word(phrase, call_word):++ """The words after the call word (possibly ''), or None when the phrase does not say it (whole words only)."""++ words, want = norm(phrase).split(), norm(call_word).split()++ if not want:++ return None++ for i in range(0, len(words) - len(want) + 1):++ if all(word_matches(words[i + j], want[j], j == 0) for j in range(len(want))):++ return " ".join(words[i + len(want):]).strip(" ,.")++ return None++++++class Voice(Plugin):++ driver = "voice"++ version = VERSION++ motion_kinds = frozenset() # it moves nothing++++ def __init__(self):++ super().__init__()++ self.on = False++ self.call_word = "hey vibe"++ self.armed_until = 0.0++ self.listener = None++ self.stopping = threading.Event()++ self.speaker_lock = threading.Lock()++ self.cfg = {}++ self.engine = "vosk"++++ # ---- describe / setup ----++ def describe(self, config):++ engine = str((config or {}).get("engine") or "vosk")++ return {"voice": {"engine": engine, "speak": self._speaker_name(config or {})}}++++ def setup(self, config):++ self.cfg = config or {}++ self.engine = str(self.cfg.get("engine") or "vosk")++ if self.engine == "vosk":++ try:++ import vosk # noqa: F401++ except ImportError:++ raise Fault("unsupported", "the voice plugin needs Vosk: pip install vosk, and a model in its config (model)")++ if not self.cfg.get("model") or not os.path.isdir(str(self.cfg.get("model"))):++ raise Fault("unsupported", "set model to a Vosk model directory (alphacephei.com/vosk/models, the small one is enough)")++ if not (shutil.which("arecord") or shutil.which("parec")):++ raise Fault("unsupported", "the voice plugin reads the microphone with arecord (alsa-utils) or parec (PulseAudio/PipeWire)")++ elif self.engine == "file":++ if not self.cfg.get("heard_file"):++ raise Fault("bad_value", "engine file needs heard_file")++ else:++ raise Fault("bad_value", "engine is vosk or file")++ # Ask for the owner's setting: the core answers with the voice op (on, call word).++ self.emit_event("voice_ready")++++ def handle(self, cmd):++ raise Fault("unsupported", "the voice plugin takes no commands")++++ def close(self):++ self._stop_listening()++++ # ---- the owner's setting ----++ def on_voice(self, on, call_word):++ self.call_word = norm(call_word) or "hey vibe"++ if on and not self.on:++ self.on = True++ self._start_listening()++ self.log("voice on: listening for %r", self.call_word)++ elif not on and self.on:++ self.on = False++ self._stop_listening()++ self.log("voice off")++++ # ---- hearing ----++ def heard(self, phrase, now=None):++ """One finished thing heard on the machine. Returns the question sent to Vibe, or None."""++ if not self.on:++ return None++ now = time.monotonic() if now is None else now++ rest = after_call_word(phrase, self.call_word)++ if rest is None:++ if self.armed_until and now <= self.armed_until and norm(phrase):++ self.armed_until = 0.0++ return self._ask(norm(phrase))++ return None # not for Vibe: dropped, never sent or kept++ if not rest:++ self.armed_until = now + float(self.cfg.get("follow_up_s") or 8)++ return None++ self.armed_until = 0.0++ return self._ask(rest)++++ def _ask(self, text):++ text = text[:2000]++ cid = "va-" + uuid.uuid4().hex[:16]++ fields = {"id": cid, "text": text}++ if self.cfg.get("lang"):++ fields["lang"] = str(self.cfg.get("lang"))++ self.emit_event("voice_ask", **fields)++ return text++++ def _start_listening(self):++ self.stopping.clear()++ target = self._listen_file if self.engine == "file" else self._listen_vosk++ self.listener = threading.Thread(target=target, name="voice-listen", daemon=True)++ self.listener.start()++++ def _stop_listening(self):++ self.stopping.set()++ self.listener = None++++ def _listen_file(self):++ path = str(self.cfg.get("heard_file"))++ pos = os.path.getsize(path) if os.path.exists(path) else 0++ while not self.stopping.is_set():++ try:++ size = os.path.getsize(path) if os.path.exists(path) else 0++ if size < pos:++ pos = 0++ if size > pos:++ with open(path, "r", encoding="utf-8", errors="replace") as f:++ f.seek(pos)++ data = f.read()++ pos = f.tell()++ for line in data.splitlines():++ if line.strip():++ self.heard(line)++ except OSError as e:++ self.log("heard_file: %s", e)++ self.stopping.wait(0.1)++++ def _listen_vosk(self):++ import vosk++ vosk.SetLogLevel(-1)++ model = vosk.Model(str(self.cfg.get("model")))++ rec = vosk.KaldiRecognizer(model, SAMPLE_RATE)++ argv = (["arecord", "-q", "-f", "S16_LE", "-r", str(SAMPLE_RATE), "-c", "1", "-t", "raw"] if shutil.which("arecord")++ else ["parec", "--format=s16le", "--rate=%d" % SAMPLE_RATE, "--channels=1"])++ proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)++ try:++ while not self.stopping.is_set():++ chunk = proc.stdout.read(4000)++ if not chunk:++ break++ if rec.AcceptWaveform(chunk):++ text = json.loads(rec.Result()).get("text", "")++ if text:++ self.heard(text)++ finally:++ proc.terminate()++++ # ---- answering ----++ def on_voice_answer(self, cid, state, text, url):++ if state == "working":++ return++ if text:++ self.log("Vibe %s (%s)", state, cid)++ self.say(text)++ self._notify(text, url)++++ def _speaker_name(self, cfg):++ speak = cfg.get("speak", "auto")++ if isinstance(speak, list):++ return "command"++ if speak == "none":++ return None++ if cfg.get("piper_model") and shutil.which("piper"):++ return "piper"++ for name in ("espeak-ng", "spd-say"):++ if shutil.which(name):++ return name++ return None++++ def say(self, text):++ speak = self.cfg.get("speak", "auto")++ if speak == "none":++ return++ if isinstance(speak, list) and speak:++ argv, data = [str(a) for a in speak], text++ else:++ name = self._speaker_name(self.cfg)++ if name == "piper":++ argv, data = ["sh", "-c", 'piper --model "$0" --output-raw | aplay -q -r 22050 -f S16_LE -t raw -', str(self.cfg.get("piper_model"))], text++ elif name == "espeak-ng":++ argv, data = ["espeak-ng", "--stdin"], text++ elif name == "spd-say":++ argv, data = ["spd-say", "-w", text], None++ else:++ self.log("nothing to speak with (install espeak-ng, or set piper_model)")++ return++ with self.speaker_lock:++ try:++ subprocess.run(argv, input=data.encode("utf-8") if data is not None else None, timeout=120, check=False,++ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)++ except (OSError, subprocess.SubprocessError) as e:++ self.log("speaking failed: %s", e)++++ def _notify(self, text, url):++ if self.cfg.get("notify", True) is False or not shutil.which("notify-send"):++ return++ body = text if not url else "%s\n%s" % (text, url)++ try:++ subprocess.run(["notify-send", "-a", "OpenVibe", "Vibe", body[:500]], timeout=5, check=False,++ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)++ except (OSError, subprocess.SubprocessError):++ passnnew file mode 100644@@@ -0,0 +1,6 @@++import sys++++from openvibe_plugin import run++from openvibe_voice import Voice++++sys.exit(run(Voice()))nnew file mode 100644@@@ -0,0 +1,17 @@++[build-system]++requires = ["setuptools>=61"]++build-backend = "setuptools.build_meta"++++[project]++name = "openvibe-voice"++version = "0.1.0"++description = "OpenVibe Node voice plugin: Vibe by voice on your own machine; a call word heard locally, only what follows goes to Vibe"++requires-python = ">=3.9"++license = { text = "AGPL-3.0-or-later" }++dependencies = ["openvibe-plugin"]++++[project.optional-dependencies]++vosk = ["vosk"]++++[tool.setuptools]++packages = ["openvibe_voice"]nnew file mode 100644@@@ -0,0 +1,111 @@++import io++import json++import os++import sys++import tempfile++import time++import unittest++++HERE = os.path.dirname(os.path.abspath(__file__))++sys.path.insert(0, os.path.dirname(HERE))++sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(HERE)), "sdk"))++++from openvibe_plugin import Runtime # noqa: E402++from openvibe_voice import Voice, after_call_word # noqa: E402++++++class CallWordTest(unittest.TestCase):++ def test_the_words_after_the_call_word_whole_words_only(self):++ self.assertEqual(after_call_word("Hey Vibe, what time is it?", "hey vibe"), "what time is it")++ self.assertEqual(after_call_word("okay vibe turn the lights off", "hey vibe"), "turn the lights off", "a greeting written another way")++ self.assertEqual(after_call_word("hey vibes play some jazz", "hey vibe"), "play some jazz", "the plural speech-to-text writes")++ self.assertEqual(after_call_word("hey vibe", "hey vibe"), "")++ self.assertIsNone(after_call_word("that's a vibe right there", "hey vibe"))++ self.assertIsNone(after_call_word("hey vibration", "hey vibe"), "whole words only")++ self.assertEqual(after_call_word("um hey desk open my mail", "hey desk"), "open my mail")++++++class VoicePluginTest(unittest.TestCase):++ def setUp(self):++ self.dir = tempfile.mkdtemp()++ self.heard = os.path.join(self.dir, "heard.txt")++ self.said = os.path.join(self.dir, "said.txt")++ self.out = io.StringIO()++ self.p = Voice()++ self.p.log = lambda *a: None++ self.rt = Runtime(self.p, infile=io.StringIO(""), outfile=self.out)++ self.send({"op": "hello", "config": {"engine": "file", "heard_file": self.heard, "lang": "en-US", "notify": False,++ "speak": ["sh", "-c", 'cat >> "$0"', self.said]}})++++ def tearDown(self):++ self.p.close()++++ def send(self, m):++ self.rt.feed_line(json.dumps(m))++++ def lines(self):++ return [json.loads(x) for x in self.out.getvalue().splitlines()]++++ def events(self, name):++ return [m for m in self.lines() if m.get("op") == "event" and m.get("name") == name]++++ def test_describes_voice_and_asks_for_the_owners_setting(self):++ d = [m for m in self.lines() if m.get("op") == "describe"][0]++ self.assertEqual(d["driver"], "voice")++ self.assertEqual(d["capabilities"]["voice"]["engine"], "file")++ self.assertEqual(d["motion_kinds"], [])++ self.assertEqual(len(self.events("voice_ready")), 1)++++ def test_off_nothing_is_sent_on_only_what_follows_the_call_word(self):++ self.assertIsNone(self.p.heard("hey vibe what time is it"), "voice off: nothing")++ self.send({"op": "voice", "on": True, "call_word": "Hey Vibe"})++ self.assertIsNone(self.p.heard("so I told him about the weekend"), "not for Vibe: dropped")++ self.assertEqual(self.p.heard("hey vibe what time is it"), "what time is it")++ asks = self.events("voice_ask")++ self.assertEqual(len(asks), 1)++ self.assertEqual(asks[0]["text"], "what time is it")++ self.assertEqual(asks[0]["lang"], "en-US")++ self.assertRegex(asks[0]["id"], r"^va-[0-9a-f]{16}$")++ self.assertNotIn("weekend", self.out.getvalue(), "what was not for Vibe never left the plugin")++++ def test_the_call_word_alone_then_the_question(self):++ self.send({"op": "voice", "on": True, "call_word": "hey vibe"})++ t = time.monotonic()++ self.assertIsNone(self.p.heard("hey vibe", now=t))++ self.assertEqual(self.p.heard("turn on the kitchen lights", now=t + 3), "turn on the kitchen lights")++ self.assertIsNone(self.p.heard("and something else", now=t + 4), "the follow-up is one question")++ self.assertIsNone(self.p.heard("hey vibe", now=t + 10))++ self.assertIsNone(self.p.heard("too late now", now=t + 30), "after follow_up_s it is not a question")++++ def test_heard_from_the_file_engine(self):++ self.send({"op": "voice", "on": True, "call_word": "hey vibe"})++ with open(self.heard, "a") as f:++ f.write("background noise\nhey vibe how far is the moon\n")++ deadline = time.monotonic() + 5++ while time.monotonic() < deadline and not self.events("voice_ask"):++ time.sleep(0.05)++ self.assertEqual([a["text"] for a in self.events("voice_ask")], ["how far is the moon"])++ self.send({"op": "voice", "on": False, "call_word": "hey vibe"})++ with open(self.heard, "a") as f:++ f.write("hey vibe are you still there\n")++ time.sleep(0.3)++ self.assertEqual(len(self.events("voice_ask")), 1, "off: the listener stopped")++++ def test_the_answer_is_spoken_working_is_not(self):++ self.send({"op": "voice_answer", "id": "va-1", "state": "working", "text": "On it."})++ self.assertFalse(os.path.exists(self.said))++ self.send({"op": "voice_answer", "id": "va-1", "state": "answered", "text": "It is about 384,000 kilometres away.", "url": "https://openvibe.chat/vibe/v-0123456789ab"})++ with open(self.said) as f:++ self.assertEqual(f.read(), "It is about 384,000 kilometres away.")++ self.send({"op": "voice_answer", "id": "va-2", "state": "refused", "text": "Voice is off for this machine."})++ with open(self.said) as f:++ self.assertIn("Voice is off for this machine.", f.read())++++ def test_takes_no_commands(self):++ self.send({"op": "command", "id": "c1", "kind": "say", "value": {"text": "x"}, "deadline_ms": 1000})++ last = self.lines()[-1]++ self.assertEqual((last["op"], last["fault_code"]), ("nack", "unsupported"))++++++if __name__ == "__main__":++ unittest.main()@@@ -24,7 +24,7 @@ done# The plugin bundle: the Python packages without their tests.tar -czf dist/openvibe-node-plugins.tar.gz -C plugins \--exclude=tests --exclude=__pycache__ --exclude='*.egg-info' --exclude=.pytest_cache --exclude=build \-- sdk dryrun adeept_adr036 cozmo relay ros2 computer++ sdk dryrun adeept_adr036 cozmo relay ros2 computer voice# OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are documented to serve this asset name:# https://github.com/OpenVibers/OpenVibe.Node/releases/latest/download/install.sh — do not rename it.cp install/install.sh dist/install.sh
@@ -98,8 +98,9 @@ jobs:
@@ -98,8 +98,9 @@ jobs:
- run: python -m pytest -q plugins/relay/tests
- run: python -m pytest -q plugins/relay/tests
- run: python -m pytest -q plugins/ros2/tests
- run: python -m pytest -q plugins/ros2/tests
- run: python -m pytest -q plugins/computer/tests
- run: python -m pytest -q plugins/computer/tests
+ - run: python -m pytest -q plugins/voice/tests
- name: packages install
- name: packages install
- run: pip install ./plugins/sdk ./plugins/dryrun ./plugins/adeept_adr036 ./plugins/cozmo ./plugins/relay ./plugins/ros2 ./plugins/computer
+ run: pip install ./plugins/sdk ./plugins/dryrun ./plugins/adeept_adr036 ./plugins/cozmo ./plugins/relay ./plugins/ros2 ./plugins/computer ./plugins/voice
shell:
shell:
name: shellcheck
name: shellcheck
@@ -23,7 +23,7 @@ jobs:
@@ -23,7 +23,7 @@ jobs:
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
- run: go vet ./... && go test -count=1 ./...
- run: go vet ./... && go test -count=1 ./...
- run: |
- run: |
- for p in sdk dryrun adeept_adr036 cozmo relay ros2 computer; do python -m pytest -q plugins/$p/tests; done
+ for p in sdk dryrun adeept_adr036 cozmo relay ros2 computer voice; do python -m pytest -q plugins/$p/tests; done
- run: scripts/dist.sh "${GITHUB_REF_NAME}"
- run: scripts/dist.sh "${GITHUB_REF_NAME}"
# Every release must carry the installer OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are
# Every release must carry the installer OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are
# documented to serve (releases/latest/download/install.sh), the plugin bundle the installer downloads, and the
# documented to serve (releases/latest/download/install.sh), the plugin bundle the installer downloads, and the
@@ -55,6 +55,8 @@ describes itself and waits for EOF without opening any hardware.
@@ -55,6 +55,8 @@ describes itself and waits for EOF without opening any hardware.
| `resume` | | accept motion again |
| `resume` | | accept motion again |
| `approval` | `id`, `approved` bool | the owner's answer to an `approval` event it raised |
| `approval` | `id`, `approved` bool | the owner's answer to an `approval` event it raised |
| `tray` | `attached` bool, `ttl_s` | (computer plugin only) the tray shows its approvals for `ttl_s` s unless told again: its own desktop prompts may stay quiet |
| `tray` | `attached` bool, `ttl_s` | (computer plugin only) the tray shows its approvals for `ttl_s` s unless told again: its own desktop prompts may stay quiet |
+| `voice` | `on` bool, `call_word` | (plugins that describe a `voice` capability) the owner's Vibe by voice setting from Bot's config: listen for `call_word` only while `on` ([voice.md](voice.md)) |
+| `voice_answer` | `id`, `state` (working, answered, refused), `text`, `url` | (voice plugins) Vibe's answer to a `voice_ask` event the plugin raised: speak and show it |
`value` has already been validated and clamped to the owner's limits by the core ([protocol.md](protocol.md#commands)).
`value` has already been validated and clamped to the owner's limits by the core ([protocol.md](protocol.md#commands)).
`deadline_ms` is relative to receipt and already capped.
`deadline_ms` is relative to receipt and already capped.
@@ -87,6 +89,11 @@ Python: `cmd.turn_epoch` (an int, or `None`).
@@ -87,6 +89,11 @@ Python: `cmd.turn_epoch` (an int, or `None`).
| `video` | `format` (`jpeg`), `path`, optional `width`, `height`, `seq` |
| `video` | `format` (`jpeg`), `path`, optional `width`, `height`, `seq` |
| `local_stop` | `reason` — the driver's own stop (a button, a sensor it trusts): latches the Node's local stop, see below |
| `local_stop` | `reason` — the driver's own stop (a button, a sensor it trusts): latches the Node's local stop, see below |
+Two `event` names are the core's own, for voice plugins: `voice_ready` asks for the owner's setting (the core answers
+with `voice`), and `voice_ask` (`id`, `text`, optional `lang`) is what the person said after the call word. The core sends
+it to Bot as a `voice_ask` frame and never as a telemetry event, and routes the `voice_answer` frames back to the plugin
+that asked.
+
For `video`, the plugin writes each frame to a file (write to a temporary name, then rename, and rotate a few names so
For `video`, the plugin writes each frame to a file (write to a temporary name, then rename, and rotate a few names so
a slow reader never sees a partial file) and announces it with `{"op": "video", "format": "jpeg", "path": …}`
a slow reader never sees a partial file) and announces it with `{"op": "video", "format": "jpeg", "path": …}`
(`emit_video` in [`plugins/sdk`](../plugins/sdk)). The core reads the file at once and encodes it: with `ffmpeg` if
(`emit_video` in [`plugins/sdk`](../plugins/sdk)). The core reads the file at once and encodes it: with `ffmpeg` if
new file mode 100644
new file mode 100644
@@ -0,0 +1,41 @@
@@ -0,0 +1,41 @@
+# Vibe by voice
+
+Say your machine's call word ("hey vibe" unless you chose another), then ask: Vibe, OpenVibe's assistant, answers out
+loud on the machine. The question and the answer are also kept in your "Voice on <machine>" chat on openvibe.chat.
+
+## What leaves the machine
+
+Only what you say **after** the call word. The voice plugin listens on the machine itself and recognises speech
+locally. Words before the call word are dropped as they are heard, nothing is recorded, and no audio is sent anywhere.
+The text of your question goes to OpenVibe.Bot as a `voice_ask` frame (contracts 0.214.0). Bot asks Vibe as you, the
+machine's owner, through OpenVibe.Chat, and sends the answer back as `voice_answer` frames. Answers count against your
+daily Vibe allowance like any other.
+
+## Turning it on
+
+1. Add the plugin to the Node's config (`config.json`, `plugins`):
+
+ ```json
+ { "name": "voice", "config": { "model": "/home/you/vosk-model-small-en-us-0.15", "lang": "en-US" } }
+ ```
+
+2. Install what it uses: `pip install vosk`, a model from https://alphacephei.com/vosk/models (the small one for your
+ language is enough), `arecord` (alsa-utils) or `parec` (PulseAudio or PipeWire) for the microphone, and something to
+ speak with (`espeak-ng`, or Piper with `piper_model` set).
+3. On https://openvibe.bot/machines, press **Let Vibe answer my voice here** on the machine and choose its call word.
+ The Node hears about it in Bot's config at once. **Stop Vibe answering my voice here** turns listening off again.
+
+## Plugin settings
+
+| Setting | Default | What |
+|---|---|---|
+| `engine` | `vosk` | `vosk` (a local model, from the microphone) or `file` (each new line of `heard_file` is one thing heard: tests, or another recogniser feeding it) |
+| `model` | | the Vosk model's directory |
+| `lang` | | the language you speak, BCP 47 (`en-US`) |
+| `speak` | `auto` | `auto` (Piper when `piper_model` is set, else espeak-ng, else spd-say), `none`, or an argv that reads the text on stdin |
+| `piper_model` | | a Piper voice (`.onnx`), spoken through `aplay` |
+| `notify` | `true` | also show answers as desktop notifications (notify-send) |
+| `follow_up_s` | `8` | after the call word alone, how long the next thing you say counts as the question |
+
+The call word counts only when said whole ("hey vibration" is not "hey vibe"). Speech-to-text's other spellings of a
+greeting count ("okay vibe", "hi vibe"), as does the plural ("hey vibes").
@@ -343,6 +343,7 @@ var Bundled = map[string]string{
@@ -343,6 +343,7 @@ var Bundled = map[string]string{
"relay": "openvibe_relay",
"relay": "openvibe_relay",
"ros2": "openvibe_ros2",
"ros2": "openvibe_ros2",
"computer": "openvibe_computer",
"computer": "openvibe_computer",
+ "voice": "openvibe_voice",
}
}
// Default is the config written by `openvibe-node pair` when none exists: the dry-run plugin and its test pattern.
// Default is the config written by `openvibe-node pair` when none exists: the dry-run plugin and its test pattern.
@@ -78,12 +78,17 @@ type Node struct {
@@ -78,12 +78,17 @@ type Node struct {
allowed map[string]bool
allowed map[string]bool
telemetry map[string]map[string]any // per plugin, latest
telemetry map[string]map[string]any // per plugin, latest
events []protocol.Event
events []protocol.Event
- telemMS int
- jpeg *video.JPEGSource
- jpegFrom string
- videoSrc string
- lastTelem time.Time
- started time.Time
+ // Vibe by voice (contracts 0.214.0): the owner's setting from Bot's config, and which plugin asked each voice_ask
+ // still waiting for its answer.
+ voiceOn bool
+ callWord string
+ voiceAskedBy map[string]string
+ telemMS int
+ jpeg *video.JPEGSource
+ jpegFrom string
+ videoSrc string
+ lastTelem time.Time
+ started time.Time
// configured is set by the first config on a connection; until then only halt runs, so a latch the owner set
// configured is set by the first config on a connection; until then only halt runs, so a latch the owner set
// while the link was down is applied before any motion.
// while the link was down is applied before any motion.
@@ -310,6 +315,8 @@ func (n *Node) Frame(f protocol.Frame) {
@@ -310,6 +315,8 @@ func (n *Node) Frame(f protocol.Frame) {
n.job(m)
n.job(m)
case protocol.JobCancel:
case protocol.JobCancel:
n.jobCancel(m)
n.jobCancel(m)
+ case protocol.VoiceAnswer:
+ n.voiceAnswer(m)
case protocol.JobExitAck:
case protocol.JobExitAck:
if n.worker != nil {
if n.worker != nil {
n.worker.ExitAck(m.ID)
n.worker.ExitAck(m.ID)
@@ -370,6 +377,65 @@ func (n *Node) applyConfig(c protocol.Config) {
@@ -370,6 +377,65 @@ func (n *Node) applyConfig(c protocol.Config) {
}
}
n.log.Info("config", "max_speed", l.MaxSpeed, "max_turn", l.MaxTurn, "max_command_ms", l.MaxCommandMS,
n.log.Info("config", "max_speed", l.MaxSpeed, "max_turn", l.MaxTurn, "max_command_ms", l.MaxCommandMS,
"allowed", c.AllowedCommands, "estop_latched", c.EstopLatched)
"allowed", c.AllowedCommands, "estop_latched", c.EstopLatched)
+ n.mu.Lock()
+ n.voiceOn, n.callWord = c.Voice, c.CallWord
+ n.mu.Unlock()
+ n.tellVoice(nil)
+}
+
+// tellVoice gives the voice plugins (or just p) the owner's voice setting: they listen for the call word only while on.
+func (n *Node) tellVoice(only *plugins.Plugin) {
+ n.mu.Lock()
+ on, word := n.voiceOn, n.callWord
+ n.mu.Unlock()
+ for _, p := range n.mgr.Plugins() {
+ if (only == nil || p == only) && p.HasCapability("voice") {
+ if err := p.Voice(on, word); err != nil {
+ n.log.Debug("voice setting not delivered", "plugin", p.Name(), "err", err)
+ }
+ }
+ }
+}
+
+// voiceAsk sends a voice plugin's question to Bot (never into telemetry: what the person said is theirs) and remembers
+// which plugin asked, so the answers reach it.
+func (n *Node) voiceAsk(pluginName string, data map[string]any) {
+ id, _ := data["id"].(string)
+ text, _ := data["text"].(string)
+ lang, _ := data["lang"].(string)
+ if id == "" || text == "" || len(id) > 64 || len(text) > 2000 {
+ n.log.Warn("voice_ask from a plugin without id or text, or too long: dropped", "plugin", pluginName)
+ return
+ }
+ n.mu.Lock()
+ if n.voiceAskedBy == nil {
+ n.voiceAskedBy = map[string]string{}
+ }
+ if len(n.voiceAskedBy) > 100 {
+ n.voiceAskedBy = map[string]string{}
+ }
+ n.voiceAskedBy[id] = pluginName
+ n.mu.Unlock()
+ n.log.Info("voice question to Vibe", "plugin", pluginName, "id", id, "chars", len(text))
+ n.send(protocol.VoiceAsk{ID: id, Text: text, Lang: lang})
+}
+
+// voiceAnswer hands Bot's answer to the plugin that asked; a final answer (not working) forgets the question.
+func (n *Node) voiceAnswer(a protocol.VoiceAnswer) {
+ n.mu.Lock()
+ name := n.voiceAskedBy[a.ID]
+ if a.State != "working" {
+ delete(n.voiceAskedBy, a.ID)
+ }
+ n.mu.Unlock()
+ p := n.mgr.Get(name)
+ if name == "" || p == nil {
+ n.log.Debug("voice_answer for no waiting question", "id", a.ID)
+ return
+ }
+ if err := p.VoiceAnswer(a.ID, a.State, a.Text, a.URL); err != nil {
+ n.log.Warn("voice answer not delivered", "plugin", name, "err", err)
+ }
}
}
// command validates and routes one command. The synchronous part (checks, clamping, queueing to the plugin) keeps
// command validates and routes one command. The synchronous part (checks, clamping, queueing to the plugin) keeps
@@ -696,6 +762,16 @@ func (n *Node) output(o plugins.Output) {
@@ -696,6 +762,16 @@ func (n *Node) output(o plugins.Output) {
n.mu.Unlock()
n.mu.Unlock()
case plugins.OutEvent:
case plugins.OutEvent:
name, _ := o.Data["name"].(string)
name, _ := o.Data["name"].(string)
+ // Vibe by voice: a question goes to Bot on its own (never as a telemetry event); a voice plugin that has just
+ // started asks for the owner's setting.
+ if name == "voice_ask" {
+ n.voiceAsk(o.Plugin, o.Data)
+ return
+ }
+ if name == "voice_ready" {
+ n.tellVoice(n.mgr.Get(o.Plugin))
+ return
+ }
fields := map[string]any{}
fields := map[string]any{}
for k, v := range o.Data {
for k, v := range o.Data {
if k != "name" {
if k != "name" {
new file mode 100644
new file mode 100644
@@ -0,0 +1,138 @@
@@ -0,0 +1,138 @@
+//go:build !windows
+
+package node
+
+import (
+ "context"
+ "encoding/json"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/OpenVibers/OpenVibe.Node/internal/config"
+ "github.com/OpenVibers/OpenVibe.Node/internal/fakebot"
+ "github.com/OpenVibers/OpenVibe.Node/internal/link"
+ "github.com/OpenVibers/OpenVibe.Node/internal/plugins"
+ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"
+)
+
+// TestVoiceEndToEnd runs the real voice plugin (its file engine stands in for the microphone) under a Node paired with
+// the fake Bot: nothing goes to Bot until the owner's config turns voice on; then only what follows the call word goes,
+// as a voice_ask frame (never as a telemetry event), and Bot's answer reaches the plugin, which speaks it.
+func TestVoiceEndToEnd(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("end-to-end tests run on unix")
+ }
+ py, err := exec.LookPath("python3")
+ if err != nil {
+ t.Skip("python3 not installed")
+ }
+ root, _ := filepath.Abs("../..")
+ home := t.TempDir()
+ heard := filepath.Join(home, "heard.txt")
+ said := filepath.Join(home, "said.txt")
+ paths := config.HomePaths(home)
+ srv := fakebot.New()
+ t.Cleanup(srv.Close)
+ srv.Config = protocol.Config{HeartbeatMS: 100, Limits: protocol.Limits{MaxCommandMS: 1000}, AllowedCommands: []string{protocol.KindHalt}}
+ srv.AddCode("VOICE234")
+ creds, err := link.Pair(context.Background(), nil, srv.URL(), protocol.PairRequest{Code: "VOICE234", AgentVersion: "test", DeviceKind: "onboard", Drivers: []string{"voice"}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ cfg := config.Default()
+ cfg.Python = py
+ cfg.Plugins = []config.PluginConfig{{Name: "voice", Config: map[string]any{"engine": "file", "heard_file": heard, "notify": false,
+ "speak": []string{"sh", "-c", `cat >> "$0"`, said}},
+ Env: map[string]string{"PYTHONPATH": filepath.Join(root, "plugins/sdk") + string(os.PathListSeparator) + filepath.Join(root, "plugins/voice")}}}
+ cfg.Video = config.VideoConfig{Source: "off"}
+ n, err := New(Options{Config: cfg, Paths: paths, Creds: creds, Version: "test", HeartbeatInterval: 100 * time.Millisecond,
+ LinkBackoffMin: 50 * time.Millisecond, LatchPoll: 50 * time.Millisecond, Log: testLog()})
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithCancel(context.Background())
+ done := make(chan struct{})
+ go func() { n.Run(ctx); close(done) }()
+ e := &env{t: t, srv: srv, node: n, paths: paths, cancel: cancel, done: done}
+ t.Cleanup(e.stop)
+ e.conn = e.nextConn()
+ if _, err := e.conn.Expect(protocol.TypeStatus, 20*time.Second, func(m protocol.Message) bool {
+ s := m.(protocol.Status)
+ return len(s.Drivers) == 1 && s.Drivers[0].State == plugins.StateReady
+ }); err != nil {
+ t.Fatal(err)
+ }
+ appendLine := func(s string) {
+ f, err := os.OpenFile(heard, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, _ = f.WriteString(s + "\n")
+ f.Close()
+ }
+
+ // Voice off (the first config did not say voice): the plugin is not listening, nothing reaches Bot.
+ appendLine("hey vibe are you there")
+ if _, err := e.conn.Expect(protocol.TypeVoiceAsk, 700*time.Millisecond, nil); err == nil {
+ t.Fatal("a voice_ask while voice is off")
+ }
+
+ // The owner turns voice on: the core passes it to the plugin, which listens for the call word.
+ on := srv.Config
+ on.Voice, on.CallWord = true, "hey desk"
+ if err := e.conn.Send(on); err != nil {
+ t.Fatal(err)
+ }
+ time.Sleep(400 * time.Millisecond)
+ appendLine("so anyway the meeting ran long")
+ appendLine("Hey desk, what is on my calendar this afternoon?")
+ f, err := e.conn.Expect(protocol.TypeVoiceAsk, 10*time.Second, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ask := f.Msg.(protocol.VoiceAsk)
+ if ask.Text != "what is on my calendar this afternoon" || !strings.HasPrefix(ask.ID, "va-") {
+ t.Fatalf("voice_ask %+v", ask)
+ }
+ for _, fr := range e.conn.Frames() {
+ if tel, ok := fr.Msg.(protocol.Telemetry); ok {
+ for _, ev := range tel.Events {
+ if ev.Name == "voice_ask" {
+ t.Fatal("the question went out as a telemetry event")
+ }
+ }
+ }
+ if fr.Type == protocol.TypeVoiceAsk && strings.Contains(string(mustJSON(fr.Msg)), "meeting") {
+ t.Fatal("what was not for Vibe left the machine")
+ }
+ }
+
+ // Bot answers working, then with Vibe's answer: the plugin speaks the answer, not the working line.
+ if err := e.conn.Send(protocol.VoiceAnswer{ID: ask.ID, State: "working", Text: "On it."}); err != nil {
+ t.Fatal(err)
+ }
+ if err := e.conn.Send(protocol.VoiceAnswer{ID: ask.ID, State: "answered", Text: "A call at three, nothing after four.", URL: "https://openvibe.chat/vibe/v-0123456789ab"}); err != nil {
+ t.Fatal(err)
+ }
+ deadline := time.Now().Add(10 * time.Second)
+ for {
+ b, _ := os.ReadFile(said)
+ if string(b) == "A call at three, nothing after four." {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatalf("spoken %q", string(b))
+ }
+ time.Sleep(50 * time.Millisecond)
+ }
+}
+
+func mustJSON(v any) []byte {
+ b, _ := json.Marshal(v)
+ return b
+}
@@ -206,6 +206,27 @@ func (p *Plugin) Answer(id string, approved bool) error {
@@ -206,6 +206,27 @@ func (p *Plugin) Answer(id string, approved bool) error {
return p.send(map[string]any{"op": "approval", "id": id, "approved": approved})
return p.send(map[string]any{"op": "approval", "id": id, "approved": approved})
}
}
+// Voice tells a voice plugin whether its owner lets Vibe answer their voice here, and the call word to listen for.
+func (p *Plugin) Voice(on bool, callWord string) error {
+ return p.send(map[string]any{"op": "voice", "on": on, "call_word": callWord})
+}
+
+// VoiceAnswer gives a voice plugin Vibe's answer to a voice_ask it raised (to speak and show).
+func (p *Plugin) VoiceAnswer(id, state, text, url string) error {
+ return p.send(map[string]any{"op": "voice_answer", "id": id, "state": state, "text": text, "url": url})
+}
+
+// HasCapability says whether the plugin's describe lists the capability (false before it described itself).
+func (p *Plugin) HasCapability(name string) bool {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if p.desc == nil {
+ return false
+ }
+ _, ok := p.desc.Capabilities[name]
+ return ok
+}
+
// Tray tells the plugin whether the Node's tray shows its approvals, for the next ttlS seconds unless told again.
// Tray tells the plugin whether the Node's tray shows its approvals, for the next ttlS seconds unless told again.
func (p *Plugin) Tray(on bool, ttlS int) error {
func (p *Plugin) Tray(on bool, ttlS int) error {
return p.send(map[string]any{"op": "tray", "attached": on, "ttl_s": ttlS})
return p.send(map[string]any{"op": "tray", "attached": on, "ttl_s": ttlS})
@@ -39,6 +39,8 @@ const (
@@ -39,6 +39,8 @@ const (
TypeEstop = "estop" // latched true sets the e-stop, false is the owner clearing it
TypeEstop = "estop" // latched true sets the e-stop, false is the owner clearing it
TypeHeartbeatAck = "heartbeat_ack"
TypeHeartbeatAck = "heartbeat_ack"
TypeError = "error"
TypeError = "error"
+ // TypeVoiceAnswer is Vibe's answer to a voice_ask (contracts 0.214.0): working, answered or refused, to speak.
+ TypeVoiceAnswer = "voice_answer"
)
)
// Close codes the server ends the device socket with.
// Close codes the server ends the device socket with.
@@ -63,6 +65,8 @@ const (
@@ -63,6 +65,8 @@ const (
TypeHeartbeat = "heartbeat"
TypeHeartbeat = "heartbeat"
TypeReauth = "reauth" // a Network-paired machine's fresh node token, before the last one expires
TypeReauth = "reauth" // a Network-paired machine's fresh node token, before the last one expires
TypeEstopState = "estop_state"
TypeEstopState = "estop_state"
+ // TypeVoiceAsk is what the owner said after the machine's call word, transcribed on the machine (contracts 0.214.0).
+ TypeVoiceAsk = "voice_ask"
)
)
// Job frames (OpenVibe.Contracts platform.job-frame@1). Server → device: job, job_cancel, job_exit_ack. Device →
// Job frames (OpenVibe.Contracts platform.job-frame@1). Server → device: job, job_cancel, job_exit_ack. Device →
@@ -299,6 +303,26 @@ type Config struct {
@@ -299,6 +303,26 @@ type Config struct {
// empty list allows only halt.
// empty list allows only halt.
AllowedCommands []string `json:"allowed_commands"`
AllowedCommands []string `json:"allowed_commands"`
EstopLatched bool `json:"estop_latched"`
EstopLatched bool `json:"estop_latched"`
+ // Voice says the owner lets Vibe answer their voice on this machine (contracts 0.214.0): a voice plugin listens for
+ // CallWord only while it is true.
+ Voice bool `json:"voice,omitempty"`
+ CallWord string `json:"call_word,omitempty"`
+}
+
+// VoiceAsk is the owner's question after the machine's call word (device → server).
+type VoiceAsk struct {
+ ID string `json:"id"`
+ Text string `json:"text"`
+ Lang string `json:"lang,omitempty"`
+ RobotID string `json:"robot_id,omitempty"`
+}
+
+// VoiceAnswer is Vibe's answer to a VoiceAsk with the same ID (server → device): State is working, answered or refused.
+type VoiceAnswer struct {
+ ID string `json:"id"`
+ State string `json:"state"`
+ Text string `json:"text,omitempty"`
+ URL string `json:"url,omitempty"`
}
}
// Operator is who sent a command, as the server's gate saw them.
// Operator is who sent a command, as the server's gate saw them.
@@ -649,6 +673,8 @@ func (Nack) MessageType() string { return TypeNack }
@@ -649,6 +673,8 @@ func (Nack) MessageType() string { return TypeNack }
func (Heartbeat) MessageType() string { return TypeHeartbeat }
func (Heartbeat) MessageType() string { return TypeHeartbeat }
func (Reauth) MessageType() string { return TypeReauth }
func (Reauth) MessageType() string { return TypeReauth }
func (EstopState) MessageType() string { return TypeEstopState }
func (EstopState) MessageType() string { return TypeEstopState }
+func (VoiceAsk) MessageType() string { return TypeVoiceAsk }
+func (VoiceAnswer) MessageType() string { return TypeVoiceAnswer }
func (JobRequest) MessageType() string { return TypeJob }
func (JobRequest) MessageType() string { return TypeJob }
func (JobCancel) MessageType() string { return TypeJobCancel }
func (JobCancel) MessageType() string { return TypeJobCancel }
func (JobExitAck) MessageType() string { return TypeJobExitAck }
func (JobExitAck) MessageType() string { return TypeJobExitAck }
@@ -678,6 +704,8 @@ var registry = map[string]func() Message{
@@ -678,6 +704,8 @@ var registry = map[string]func() Message{
TypeJobStdout: func() Message { return &JobStdout{} },
TypeJobStdout: func() Message { return &JobStdout{} },
TypeJobUsage: func() Message { return &JobUsage{} },
TypeJobUsage: func() Message { return &JobUsage{} },
TypeJobExit: func() Message { return &JobExit{} },
TypeJobExit: func() Message { return &JobExit{} },
+ TypeVoiceAsk: func() Message { return &VoiceAsk{} },
+ TypeVoiceAnswer: func() Message { return &VoiceAnswer{} },
}
}
// ErrUnknownType is returned (wrapped) by Decode for a frame whose type is not in this version's message set.
// ErrUnknownType is returned (wrapped) by Decode for a frame whose type is not in this version's message set.
@@ -777,6 +805,10 @@ func derefMessage(m Message) Message {
@@ -777,6 +805,10 @@ func derefMessage(m Message) Message {
return *v
return *v
case *JobExit:
case *JobExit:
return *v
return *v
+ case *VoiceAsk:
+ return *v
+ case *VoiceAnswer:
+ return *v
}
}
return m
return m
}
}
@@ -137,6 +137,15 @@ class Plugin:
@@ -137,6 +137,15 @@ class Plugin:
"""The owner answered, on the machine (`openvibe-node approve|deny <id>`), an approval this plugin asked for
"""The owner answered, on the machine (`openvibe-node approve|deny <id>`), an approval this plugin asked for
with emit_event("approval", id=…). Runs on the runtime's main thread."""
with emit_event("approval", id=…). Runs on the runtime's main thread."""
+ def on_voice(self, on, call_word):
+ """Vibe by voice (contracts 0.214.0): the owner lets Vibe answer their voice on this machine (on) and chose
+ call_word, or stopped it. Sent to plugins whose describe lists the `voice` capability, after every config from
+ Bot and when the plugin emits the voice_ready event. A voice plugin listens for the call word only while on."""
+
+ def on_voice_answer(self, cid, state, text, url):
+ """Vibe's answer to a voice_ask this plugin emitted (emit_event("voice_ask", id=…, text=…)): state is working,
+ answered or refused; text is what to say, url the Vibe chat that keeps it."""
+
def on_tray(self, attached, ttl_s):
def on_tray(self, attached, ttl_s):
"""The Node's tray (`openvibe-node tray`, in the owner's graphical session) is showing this plugin's approvals,
"""The Node's tray (`openvibe-node tray`, in the owner's graphical session) is showing this plugin's approvals,
for the next ttl_s seconds unless told again. A plugin that notifies the desktop itself can stay quiet then.
for the next ttl_s seconds unless told again. A plugin that notifies the desktop itself can stay quiet then.
@@ -293,6 +302,16 @@ class Runtime:
@@ -293,6 +302,16 @@ class Runtime:
self.plugin.on_tray(msg.get("attached") is True, msg.get("ttl_s"))
self.plugin.on_tray(msg.get("attached") is True, msg.get("ttl_s"))
except Exception as e:
except Exception as e:
self.plugin.log("tray handler failed: %r", e)
self.plugin.log("tray handler failed: %r", e)
+ elif op == "voice":
+ try:
+ self.plugin.on_voice(msg.get("on") is True, str(msg.get("call_word") or ""))
+ except Exception as e:
+ self.plugin.log("voice handler failed: %r", e)
+ elif op == "voice_answer":
+ try:
+ self.plugin.on_voice_answer(str(msg.get("id") or ""), str(msg.get("state") or ""), str(msg.get("text") or ""), str(msg.get("url") or ""))
+ except Exception as e:
+ self.plugin.log("voice answer handler failed: %r", e)
elif op == "resume":
elif op == "resume":
if self.local_stop_pending:
if self.local_stop_pending:
self.plugin.log("resume ignored: the local stop has not reached the core yet")
self.plugin.log("resume ignored: the local stop has not reached the core yet")
new file mode 100644
new file mode 100644
@@ -0,0 +1,256 @@
@@ -0,0 +1,256 @@
+"""OpenVibe Node voice plugin: Vibe by voice on the person's own machine (contracts 0.214.0; consumer plan §8).
+
+The owner lets Vibe answer their voice on this machine and chooses a call word on openvibe.bot/machines; Bot's config
+says so (the core passes it on as the ``voice`` op). While it is on, this plugin listens on the machine itself, and
+only what the person says *after* the call word leaves it: as a ``voice_ask`` event the core sends to Bot, which asks
+Vibe as the owner and sends ``voice_answer`` frames back. The answer is spoken here and shown as a desktop notification,
+and kept in the owner's "Voice on <machine>" chat on openvibe.chat.
+
+Nothing is recorded and no audio leaves the machine: recognition runs locally, and the words before the call word are
+dropped as they are heard.
+
+Config (the plugin's block in config.json):
+
+* ``engine``: ``vosk`` (the default: a local Vosk model, ``model`` = its directory, audio from ``arecord`` or ``parec``)
+ or ``file`` (each new line appended to ``heard_file`` is one thing heard; for tests and for wiring other recognisers).
+* ``lang``: the language of the model, BCP 47 (``en-US``).
+* ``speak``: ``auto`` (Piper when ``piper_model`` is set, else espeak-ng, else spd-say), ``none``, or an argv list that
+ reads the text on stdin (``["sh", "-c", "cat >> said.txt"]``).
+* ``notify``: show answers as desktop notifications (notify-send), true by default.
+* ``follow_up_s``: after the call word alone, how long the next thing heard counts as the question (8 s).
+"""
+
+import json
+import os
+import shutil
+import subprocess
+import threading
+import time
+import unicodedata
+import uuid
+
+from openvibe_plugin import Fault, Plugin
+
+VERSION = "0.1.0"
+SAMPLE_RATE = 16000
+# Speech-to-text writes a spoken greeting several ways (as Live's stream call word, server/ai/vibe-voice.js).
+GREETINGS = {"hey", "hi", "hay", "heyy", "ok", "okay", "yo"}
+
+
+def norm(s):
+ """Lower case, accents off, punctuation to spaces, one space between words."""
+ s = unicodedata.normalize("NFKD", str(s or "").lower())
+ s = "".join(c for c in s if not unicodedata.combining(c))
+ return " ".join("".join(c if (c.isalnum() or c in "' ") else " " for c in s).split())
+
+
+def word_matches(heard, want, first):
+ h = heard[:-2] if heard.endswith("'s") else heard
+ h, w = h.replace("'", ""), want.replace("'", "")
+ if h == w or h == w + "s" or h + "s" == w:
+ return True
+ return first and w in GREETINGS and h in GREETINGS
+
+
+def after_call_word(phrase, call_word):
+ """The words after the call word (possibly ''), or None when the phrase does not say it (whole words only)."""
+ words, want = norm(phrase).split(), norm(call_word).split()
+ if not want:
+ return None
+ for i in range(0, len(words) - len(want) + 1):
+ if all(word_matches(words[i + j], want[j], j == 0) for j in range(len(want))):
+ return " ".join(words[i + len(want):]).strip(" ,.")
+ return None
+
+
+class Voice(Plugin):
+ driver = "voice"
+ version = VERSION
+ motion_kinds = frozenset() # it moves nothing
+
+ def __init__(self):
+ super().__init__()
+ self.on = False
+ self.call_word = "hey vibe"
+ self.armed_until = 0.0
+ self.listener = None
+ self.stopping = threading.Event()
+ self.speaker_lock = threading.Lock()
+ self.cfg = {}
+ self.engine = "vosk"
+
+ # ---- describe / setup ----
+ def describe(self, config):
+ engine = str((config or {}).get("engine") or "vosk")
+ return {"voice": {"engine": engine, "speak": self._speaker_name(config or {})}}
+
+ def setup(self, config):
+ self.cfg = config or {}
+ self.engine = str(self.cfg.get("engine") or "vosk")
+ if self.engine == "vosk":
+ try:
+ import vosk # noqa: F401
+ except ImportError:
+ raise Fault("unsupported", "the voice plugin needs Vosk: pip install vosk, and a model in its config (model)")
+ if not self.cfg.get("model") or not os.path.isdir(str(self.cfg.get("model"))):
+ raise Fault("unsupported", "set model to a Vosk model directory (alphacephei.com/vosk/models, the small one is enough)")
+ if not (shutil.which("arecord") or shutil.which("parec")):
+ raise Fault("unsupported", "the voice plugin reads the microphone with arecord (alsa-utils) or parec (PulseAudio/PipeWire)")
+ elif self.engine == "file":
+ if not self.cfg.get("heard_file"):
+ raise Fault("bad_value", "engine file needs heard_file")
+ else:
+ raise Fault("bad_value", "engine is vosk or file")
+ # Ask for the owner's setting: the core answers with the voice op (on, call word).
+ self.emit_event("voice_ready")
+
+ def handle(self, cmd):
+ raise Fault("unsupported", "the voice plugin takes no commands")
+
+ def close(self):
+ self._stop_listening()
+
+ # ---- the owner's setting ----
+ def on_voice(self, on, call_word):
+ self.call_word = norm(call_word) or "hey vibe"
+ if on and not self.on:
+ self.on = True
+ self._start_listening()
+ self.log("voice on: listening for %r", self.call_word)
+ elif not on and self.on:
+ self.on = False
+ self._stop_listening()
+ self.log("voice off")
+
+ # ---- hearing ----
+ def heard(self, phrase, now=None):
+ """One finished thing heard on the machine. Returns the question sent to Vibe, or None."""
+ if not self.on:
+ return None
+ now = time.monotonic() if now is None else now
+ rest = after_call_word(phrase, self.call_word)
+ if rest is None:
+ if self.armed_until and now <= self.armed_until and norm(phrase):
+ self.armed_until = 0.0
+ return self._ask(norm(phrase))
+ return None # not for Vibe: dropped, never sent or kept
+ if not rest:
+ self.armed_until = now + float(self.cfg.get("follow_up_s") or 8)
+ return None
+ self.armed_until = 0.0
+ return self._ask(rest)
+
+ def _ask(self, text):
+ text = text[:2000]
+ cid = "va-" + uuid.uuid4().hex[:16]
+ fields = {"id": cid, "text": text}
+ if self.cfg.get("lang"):
+ fields["lang"] = str(self.cfg.get("lang"))
+ self.emit_event("voice_ask", **fields)
+ return text
+
+ def _start_listening(self):
+ self.stopping.clear()
+ target = self._listen_file if self.engine == "file" else self._listen_vosk
+ self.listener = threading.Thread(target=target, name="voice-listen", daemon=True)
+ self.listener.start()
+
+ def _stop_listening(self):
+ self.stopping.set()
+ self.listener = None
+
+ def _listen_file(self):
+ path = str(self.cfg.get("heard_file"))
+ pos = os.path.getsize(path) if os.path.exists(path) else 0
+ while not self.stopping.is_set():
+ try:
+ size = os.path.getsize(path) if os.path.exists(path) else 0
+ if size < pos:
+ pos = 0
+ if size > pos:
+ with open(path, "r", encoding="utf-8", errors="replace") as f:
+ f.seek(pos)
+ data = f.read()
+ pos = f.tell()
+ for line in data.splitlines():
+ if line.strip():
+ self.heard(line)
+ except OSError as e:
+ self.log("heard_file: %s", e)
+ self.stopping.wait(0.1)
+
+ def _listen_vosk(self):
+ import vosk
+ vosk.SetLogLevel(-1)
+ model = vosk.Model(str(self.cfg.get("model")))
+ rec = vosk.KaldiRecognizer(model, SAMPLE_RATE)
+ argv = (["arecord", "-q", "-f", "S16_LE", "-r", str(SAMPLE_RATE), "-c", "1", "-t", "raw"] if shutil.which("arecord")
+ else ["parec", "--format=s16le", "--rate=%d" % SAMPLE_RATE, "--channels=1"])
+ proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+ try:
+ while not self.stopping.is_set():
+ chunk = proc.stdout.read(4000)
+ if not chunk:
+ break
+ if rec.AcceptWaveform(chunk):
+ text = json.loads(rec.Result()).get("text", "")
+ if text:
+ self.heard(text)
+ finally:
+ proc.terminate()
+
+ # ---- answering ----
+ def on_voice_answer(self, cid, state, text, url):
+ if state == "working":
+ return
+ if text:
+ self.log("Vibe %s (%s)", state, cid)
+ self.say(text)
+ self._notify(text, url)
+
+ def _speaker_name(self, cfg):
+ speak = cfg.get("speak", "auto")
+ if isinstance(speak, list):
+ return "command"
+ if speak == "none":
+ return None
+ if cfg.get("piper_model") and shutil.which("piper"):
+ return "piper"
+ for name in ("espeak-ng", "spd-say"):
+ if shutil.which(name):
+ return name
+ return None
+
+ def say(self, text):
+ speak = self.cfg.get("speak", "auto")
+ if speak == "none":
+ return
+ if isinstance(speak, list) and speak:
+ argv, data = [str(a) for a in speak], text
+ else:
+ name = self._speaker_name(self.cfg)
+ if name == "piper":
+ argv, data = ["sh", "-c", 'piper --model "$0" --output-raw | aplay -q -r 22050 -f S16_LE -t raw -', str(self.cfg.get("piper_model"))], text
+ elif name == "espeak-ng":
+ argv, data = ["espeak-ng", "--stdin"], text
+ elif name == "spd-say":
+ argv, data = ["spd-say", "-w", text], None
+ else:
+ self.log("nothing to speak with (install espeak-ng, or set piper_model)")
+ return
+ with self.speaker_lock:
+ try:
+ subprocess.run(argv, input=data.encode("utf-8") if data is not None else None, timeout=120, check=False,
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ except (OSError, subprocess.SubprocessError) as e:
+ self.log("speaking failed: %s", e)
+
+ def _notify(self, text, url):
+ if self.cfg.get("notify", True) is False or not shutil.which("notify-send"):
+ return
+ body = text if not url else "%s\n%s" % (text, url)
+ try:
+ subprocess.run(["notify-send", "-a", "OpenVibe", "Vibe", body[:500]], timeout=5, check=False,
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ except (OSError, subprocess.SubprocessError):
+ pass
new file mode 100644
new file mode 100644
@@ -0,0 +1,6 @@
@@ -0,0 +1,6 @@
+import sys
+
+from openvibe_plugin import run
+from openvibe_voice import Voice
+
+sys.exit(run(Voice()))
new file mode 100644
new file mode 100644
@@ -0,0 +1,17 @@
@@ -0,0 +1,17 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "openvibe-voice"
+version = "0.1.0"
+description = "OpenVibe Node voice plugin: Vibe by voice on your own machine; a call word heard locally, only what follows goes to Vibe"
+requires-python = ">=3.9"
+license = { text = "AGPL-3.0-or-later" }
+dependencies = ["openvibe-plugin"]
+
+[project.optional-dependencies]
+vosk = ["vosk"]
+
+[tool.setuptools]
+packages = ["openvibe_voice"]
new file mode 100644
new file mode 100644
@@ -0,0 +1,111 @@
@@ -0,0 +1,111 @@
+import io
+import json
+import os
+import sys
+import tempfile
+import time
+import unittest
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(HERE))
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(HERE)), "sdk"))
+
+from openvibe_plugin import Runtime # noqa: E402
+from openvibe_voice import Voice, after_call_word # noqa: E402
+
+
+class CallWordTest(unittest.TestCase):
+ def test_the_words_after_the_call_word_whole_words_only(self):
+ self.assertEqual(after_call_word("Hey Vibe, what time is it?", "hey vibe"), "what time is it")
+ self.assertEqual(after_call_word("okay vibe turn the lights off", "hey vibe"), "turn the lights off", "a greeting written another way")
+ self.assertEqual(after_call_word("hey vibes play some jazz", "hey vibe"), "play some jazz", "the plural speech-to-text writes")
+ self.assertEqual(after_call_word("hey vibe", "hey vibe"), "")
+ self.assertIsNone(after_call_word("that's a vibe right there", "hey vibe"))
+ self.assertIsNone(after_call_word("hey vibration", "hey vibe"), "whole words only")
+ self.assertEqual(after_call_word("um hey desk open my mail", "hey desk"), "open my mail")
+
+
+class VoicePluginTest(unittest.TestCase):
+ def setUp(self):
+ self.dir = tempfile.mkdtemp()
+ self.heard = os.path.join(self.dir, "heard.txt")
+ self.said = os.path.join(self.dir, "said.txt")
+ self.out = io.StringIO()
+ self.p = Voice()
+ self.p.log = lambda *a: None
+ self.rt = Runtime(self.p, infile=io.StringIO(""), outfile=self.out)
+ self.send({"op": "hello", "config": {"engine": "file", "heard_file": self.heard, "lang": "en-US", "notify": False,
+ "speak": ["sh", "-c", 'cat >> "$0"', self.said]}})
+
+ def tearDown(self):
+ self.p.close()
+
+ def send(self, m):
+ self.rt.feed_line(json.dumps(m))
+
+ def lines(self):
+ return [json.loads(x) for x in self.out.getvalue().splitlines()]
+
+ def events(self, name):
+ return [m for m in self.lines() if m.get("op") == "event" and m.get("name") == name]
+
+ def test_describes_voice_and_asks_for_the_owners_setting(self):
+ d = [m for m in self.lines() if m.get("op") == "describe"][0]
+ self.assertEqual(d["driver"], "voice")
+ self.assertEqual(d["capabilities"]["voice"]["engine"], "file")
+ self.assertEqual(d["motion_kinds"], [])
+ self.assertEqual(len(self.events("voice_ready")), 1)
+
+ def test_off_nothing_is_sent_on_only_what_follows_the_call_word(self):
+ self.assertIsNone(self.p.heard("hey vibe what time is it"), "voice off: nothing")
+ self.send({"op": "voice", "on": True, "call_word": "Hey Vibe"})
+ self.assertIsNone(self.p.heard("so I told him about the weekend"), "not for Vibe: dropped")
+ self.assertEqual(self.p.heard("hey vibe what time is it"), "what time is it")
+ asks = self.events("voice_ask")
+ self.assertEqual(len(asks), 1)
+ self.assertEqual(asks[0]["text"], "what time is it")
+ self.assertEqual(asks[0]["lang"], "en-US")
+ self.assertRegex(asks[0]["id"], r"^va-[0-9a-f]{16}$")
+ self.assertNotIn("weekend", self.out.getvalue(), "what was not for Vibe never left the plugin")
+
+ def test_the_call_word_alone_then_the_question(self):
+ self.send({"op": "voice", "on": True, "call_word": "hey vibe"})
+ t = time.monotonic()
+ self.assertIsNone(self.p.heard("hey vibe", now=t))
+ self.assertEqual(self.p.heard("turn on the kitchen lights", now=t + 3), "turn on the kitchen lights")
+ self.assertIsNone(self.p.heard("and something else", now=t + 4), "the follow-up is one question")
+ self.assertIsNone(self.p.heard("hey vibe", now=t + 10))
+ self.assertIsNone(self.p.heard("too late now", now=t + 30), "after follow_up_s it is not a question")
+
+ def test_heard_from_the_file_engine(self):
+ self.send({"op": "voice", "on": True, "call_word": "hey vibe"})
+ with open(self.heard, "a") as f:
+ f.write("background noise\nhey vibe how far is the moon\n")
+ deadline = time.monotonic() + 5
+ while time.monotonic() < deadline and not self.events("voice_ask"):
+ time.sleep(0.05)
+ self.assertEqual([a["text"] for a in self.events("voice_ask")], ["how far is the moon"])
+ self.send({"op": "voice", "on": False, "call_word": "hey vibe"})
+ with open(self.heard, "a") as f:
+ f.write("hey vibe are you still there\n")
+ time.sleep(0.3)
+ self.assertEqual(len(self.events("voice_ask")), 1, "off: the listener stopped")
+
+ def test_the_answer_is_spoken_working_is_not(self):
+ self.send({"op": "voice_answer", "id": "va-1", "state": "working", "text": "On it."})
+ self.assertFalse(os.path.exists(self.said))
+ self.send({"op": "voice_answer", "id": "va-1", "state": "answered", "text": "It is about 384,000 kilometres away.", "url": "https://openvibe.chat/vibe/v-0123456789ab"})
+ with open(self.said) as f:
+ self.assertEqual(f.read(), "It is about 384,000 kilometres away.")
+ self.send({"op": "voice_answer", "id": "va-2", "state": "refused", "text": "Voice is off for this machine."})
+ with open(self.said) as f:
+ self.assertIn("Voice is off for this machine.", f.read())
+
+ def test_takes_no_commands(self):
+ self.send({"op": "command", "id": "c1", "kind": "say", "value": {"text": "x"}, "deadline_ms": 1000})
+ last = self.lines()[-1]
+ self.assertEqual((last["op"], last["fault_code"]), ("nack", "unsupported"))
+
+
+if __name__ == "__main__":
+ unittest.main()
@@ -24,7 +24,7 @@ done
@@ -24,7 +24,7 @@ done
# The plugin bundle: the Python packages without their tests.
# The plugin bundle: the Python packages without their tests.
tar -czf dist/openvibe-node-plugins.tar.gz -C plugins \
tar -czf dist/openvibe-node-plugins.tar.gz -C plugins \
--exclude=tests --exclude=__pycache__ --exclude='*.egg-info' --exclude=.pytest_cache --exclude=build \
--exclude=tests --exclude=__pycache__ --exclude='*.egg-info' --exclude=.pytest_cache --exclude=build \
- sdk dryrun adeept_adr036 cozmo relay ros2 computer
+ sdk dryrun adeept_adr036 cozmo relay ros2 computer voice
# OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are documented to serve this asset name:
# OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are documented to serve this asset name:
# https://github.com/OpenVibers/OpenVibe.Node/releases/latest/download/install.sh — do not rename it.
# https://github.com/OpenVibers/OpenVibe.Node/releases/latest/download/install.sh — do not rename it.
cp install/install.sh dist/install.sh
cp install/install.sh dist/install.sh