openvibers/openvibe.link Commits

Vibe by voice: a voice plugin hears the call word on the machine and sends only what follows (voice_ask); the core routes Bot's voice_answer back and passes the owner's voice setting from config (contracts 0.214.0) (#58)

OP OpenVibers committed 19 days ago · 801c816b92 · 15 files, +735 −9

Changes

ddiff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
iindex b603bcb..a778ba0 100644
---- a/.github/workflows/ci.yml
++++ b/.github/workflows/ci.yml
@@@ -98,8 +98,9 @@ jobs:
- run: python -m pytest -q plugins/relay/tests
- run: python -m pytest -q plugins/ros2/tests
- run: python -m pytest -q plugins/computer/tests
++ - run: python -m pytest -q plugins/voice/tests
- name: packages install
-- run: pip install ./plugins/sdk ./plugins/dryrun ./plugins/adeept_adr036 ./plugins/cozmo ./plugins/relay ./plugins/ros2 ./plugins/computer
++ run: pip install ./plugins/sdk ./plugins/dryrun ./plugins/adeept_adr036 ./plugins/cozmo ./plugins/relay ./plugins/ros2 ./plugins/computer ./plugins/voice
shell:
name: shellcheck
ddiff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
iindex c31d98e..df2f5d6 100644
---- a/.github/workflows/release.yml
++++ b/.github/workflows/release.yml
@@@ -23,7 +23,7 @@ jobs:
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
- run: go vet ./... && go test -count=1 ./...
- run: |
-- for p in sdk dryrun adeept_adr036 cozmo relay ros2 computer; do python -m pytest -q plugins/$p/tests; done
++ for p in sdk dryrun adeept_adr036 cozmo relay ros2 computer voice; do python -m pytest -q plugins/$p/tests; done
- run: scripts/dist.sh "${GITHUB_REF_NAME}"
# Every release must carry the installer OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are
# documented to serve (releases/latest/download/install.sh), the plugin bundle the installer downloads, and the
ddiff --git a/docs/plugins.md b/docs/plugins.md
iindex 493b5dc..eb2301a 100644
---- a/docs/plugins.md
++++ b/docs/plugins.md
@@@ -55,6 +55,8 @@ describes itself and waits for EOF without opening any hardware.
| `resume` | | accept motion again |
| `approval` | `id`, `approved` bool | the owner's answer to an `approval` event it raised |
| `tray` | `attached` bool, `ttl_s` | (computer plugin only) the tray shows its approvals for `ttl_s` s unless told again: its own desktop prompts may stay quiet |
++| `voice` | `on` bool, `call_word` | (plugins that describe a `voice` capability) the owner's Vibe by voice setting from Bot's config: listen for `call_word` only while `on` ([voice.md](voice.md)) |
++| `voice_answer` | `id`, `state` (working, answered, refused), `text`, `url` | (voice plugins) Vibe's answer to a `voice_ask` event the plugin raised: speak and show it |
`value` has already been validated and clamped to the owner's limits by the core ([protocol.md](protocol.md#commands)).
`deadline_ms` is relative to receipt and already capped.
@@@ -87,6 +89,11 @@ Python: `cmd.turn_epoch` (an int, or `None`).
| `video` | `format` (`jpeg`), `path`, optional `width`, `height`, `seq` |
| `local_stop` | `reason` — the driver's own stop (a button, a sensor it trusts): latches the Node's local stop, see below |
++Two `event` names are the core's own, for voice plugins: `voice_ready` asks for the owner's setting (the core answers
++with `voice`), and `voice_ask` (`id`, `text`, optional `lang`) is what the person said after the call word. The core sends
++it to Bot as a `voice_ask` frame and never as a telemetry event, and routes the `voice_answer` frames back to the plugin
++that asked.
++
For `video`, the plugin writes each frame to a file (write to a temporary name, then rename, and rotate a few names so
a slow reader never sees a partial file) and announces it with `{"op": "video", "format": "jpeg", "path": …}`
(`emit_video` in [`plugins/sdk`](../plugins/sdk)). The core reads the file at once and encodes it: with `ffmpeg` if
ddiff --git a/docs/voice.md b/docs/voice.md
nnew file mode 100644
iindex 0000000..80d9dc2
---- /dev/null
++++ b/docs/voice.md
@@@ -0,0 +1,41 @@
++# Vibe by voice
++
++Say your machine's call word ("hey vibe" unless you chose another), then ask: Vibe, OpenVibe's assistant, answers out
++loud on the machine. The question and the answer are also kept in your "Voice on <machine>" chat on openvibe.chat.
++
++## What leaves the machine
++
++Only what you say **after** the call word. The voice plugin listens on the machine itself and recognises speech
++locally. Words before the call word are dropped as they are heard, nothing is recorded, and no audio is sent anywhere.
++The text of your question goes to OpenVibe.Bot as a `voice_ask` frame (contracts 0.214.0). Bot asks Vibe as you, the
++machine's owner, through OpenVibe.Chat, and sends the answer back as `voice_answer` frames. Answers count against your
++daily Vibe allowance like any other.
++
++## Turning it on
++
++1. Add the plugin to the Node's config (`config.json`, `plugins`):
++
++ ```json
++ { "name": "voice", "config": { "model": "/home/you/vosk-model-small-en-us-0.15", "lang": "en-US" } }
++ ```
++
++2. Install what it uses: `pip install vosk`, a model from https://alphacephei.com/vosk/models (the small one for your
++ language is enough), `arecord` (alsa-utils) or `parec` (PulseAudio or PipeWire) for the microphone, and something to
++ speak with (`espeak-ng`, or Piper with `piper_model` set).
++3. On https://openvibe.bot/machines, press **Let Vibe answer my voice here** on the machine and choose its call word.
++ The Node hears about it in Bot's config at once. **Stop Vibe answering my voice here** turns listening off again.
++
++## Plugin settings
++
++| Setting | Default | What |
++|---|---|---|
++| `engine` | `vosk` | `vosk` (a local model, from the microphone) or `file` (each new line of `heard_file` is one thing heard: tests, or another recogniser feeding it) |
++| `model` | | the Vosk model's directory |
++| `lang` | | the language you speak, BCP 47 (`en-US`) |
++| `speak` | `auto` | `auto` (Piper when `piper_model` is set, else espeak-ng, else spd-say), `none`, or an argv that reads the text on stdin |
++| `piper_model` | | a Piper voice (`.onnx`), spoken through `aplay` |
++| `notify` | `true` | also show answers as desktop notifications (notify-send) |
++| `follow_up_s` | `8` | after the call word alone, how long the next thing you say counts as the question |
++
++The call word counts only when said whole ("hey vibration" is not "hey vibe"). Speech-to-text's other spellings of a
++greeting count ("okay vibe", "hi vibe"), as does the plural ("hey vibes").
ddiff --git a/internal/config/config.go b/internal/config/config.go
iindex 02ee3f2..3e67bb1 100644
---- a/internal/config/config.go
++++ b/internal/config/config.go
@@@ -343,6 +343,7 @@ var Bundled = map[string]string{
"relay": "openvibe_relay",
"ros2": "openvibe_ros2",
"computer": "openvibe_computer",
++ "voice": "openvibe_voice",
}
// Default is the config written by `openvibe-node pair` when none exists: the dry-run plugin and its test pattern.
ddiff --git a/internal/node/node.go b/internal/node/node.go
iindex a58a649..93ce179 100644
---- a/internal/node/node.go
++++ b/internal/node/node.go
@@@ -78,12 +78,17 @@ type Node struct {
allowed map[string]bool
telemetry map[string]map[string]any // per plugin, latest
events []protocol.Event
-- telemMS int
-- jpeg *video.JPEGSource
-- jpegFrom string
-- videoSrc string
-- lastTelem time.Time
-- started time.Time
++ // Vibe by voice (contracts 0.214.0): the owner's setting from Bot's config, and which plugin asked each voice_ask
++ // still waiting for its answer.
++ voiceOn bool
++ callWord string
++ voiceAskedBy map[string]string
++ telemMS int
++ jpeg *video.JPEGSource
++ jpegFrom string
++ videoSrc string
++ lastTelem time.Time
++ started time.Time
// configured is set by the first config on a connection; until then only halt runs, so a latch the owner set
// while the link was down is applied before any motion.
@@@ -310,6 +315,8 @@ func (n *Node) Frame(f protocol.Frame) {
n.job(m)
case protocol.JobCancel:
n.jobCancel(m)
++ case protocol.VoiceAnswer:
++ n.voiceAnswer(m)
case protocol.JobExitAck:
if n.worker != nil {
n.worker.ExitAck(m.ID)
@@@ -370,6 +377,65 @@ func (n *Node) applyConfig(c protocol.Config) {
}
n.log.Info("config", "max_speed", l.MaxSpeed, "max_turn", l.MaxTurn, "max_command_ms", l.MaxCommandMS,
"allowed", c.AllowedCommands, "estop_latched", c.EstopLatched)
++ n.mu.Lock()
++ n.voiceOn, n.callWord = c.Voice, c.CallWord
++ n.mu.Unlock()
++ n.tellVoice(nil)
++}
++
++// tellVoice gives the voice plugins (or just p) the owner's voice setting: they listen for the call word only while on.
++func (n *Node) tellVoice(only *plugins.Plugin) {
++ n.mu.Lock()
++ on, word := n.voiceOn, n.callWord
++ n.mu.Unlock()
++ for _, p := range n.mgr.Plugins() {
++ if (only == nil || p == only) && p.HasCapability("voice") {
++ if err := p.Voice(on, word); err != nil {
++ n.log.Debug("voice setting not delivered", "plugin", p.Name(), "err", err)
++ }
++ }
++ }
++}
++
++// voiceAsk sends a voice plugin's question to Bot (never into telemetry: what the person said is theirs) and remembers
++// which plugin asked, so the answers reach it.
++func (n *Node) voiceAsk(pluginName string, data map[string]any) {
++ id, _ := data["id"].(string)
++ text, _ := data["text"].(string)
++ lang, _ := data["lang"].(string)
++ if id == "" || text == "" || len(id) > 64 || len(text) > 2000 {
++ n.log.Warn("voice_ask from a plugin without id or text, or too long: dropped", "plugin", pluginName)
++ return
++ }
++ n.mu.Lock()
++ if n.voiceAskedBy == nil {
++ n.voiceAskedBy = map[string]string{}
++ }
++ if len(n.voiceAskedBy) > 100 {
++ n.voiceAskedBy = map[string]string{}
++ }
++ n.voiceAskedBy[id] = pluginName
++ n.mu.Unlock()
++ n.log.Info("voice question to Vibe", "plugin", pluginName, "id", id, "chars", len(text))
++ n.send(protocol.VoiceAsk{ID: id, Text: text, Lang: lang})
++}
++
++// voiceAnswer hands Bot's answer to the plugin that asked; a final answer (not working) forgets the question.
++func (n *Node) voiceAnswer(a protocol.VoiceAnswer) {
++ n.mu.Lock()
++ name := n.voiceAskedBy[a.ID]
++ if a.State != "working" {
++ delete(n.voiceAskedBy, a.ID)
++ }
++ n.mu.Unlock()
++ p := n.mgr.Get(name)
++ if name == "" || p == nil {
++ n.log.Debug("voice_answer for no waiting question", "id", a.ID)
++ return
++ }
++ if err := p.VoiceAnswer(a.ID, a.State, a.Text, a.URL); err != nil {
++ n.log.Warn("voice answer not delivered", "plugin", name, "err", err)
++ }
}
// command validates and routes one command. The synchronous part (checks, clamping, queueing to the plugin) keeps
@@@ -696,6 +762,16 @@ func (n *Node) output(o plugins.Output) {
n.mu.Unlock()
case plugins.OutEvent:
name, _ := o.Data["name"].(string)
++ // Vibe by voice: a question goes to Bot on its own (never as a telemetry event); a voice plugin that has just
++ // started asks for the owner's setting.
++ if name == "voice_ask" {
++ n.voiceAsk(o.Plugin, o.Data)
++ return
++ }
++ if name == "voice_ready" {
++ n.tellVoice(n.mgr.Get(o.Plugin))
++ return
++ }
fields := map[string]any{}
for k, v := range o.Data {
if k != "name" {
ddiff --git a/internal/node/voice_test.go b/internal/node/voice_test.go
nnew file mode 100644
iindex 0000000..52387b0
---- /dev/null
++++ b/internal/node/voice_test.go
@@@ -0,0 +1,138 @@
++//go:build !windows
++
++package node
++
++import (
++ "context"
++ "encoding/json"
++ "os"
++ "os/exec"
++ "path/filepath"
++ "runtime"
++ "strings"
++ "testing"
++ "time"
++
++ "github.com/OpenVibers/OpenVibe.Node/internal/config"
++ "github.com/OpenVibers/OpenVibe.Node/internal/fakebot"
++ "github.com/OpenVibers/OpenVibe.Node/internal/link"
++ "github.com/OpenVibers/OpenVibe.Node/internal/plugins"
++ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"
++)
++
++// TestVoiceEndToEnd runs the real voice plugin (its file engine stands in for the microphone) under a Node paired with
++// the fake Bot: nothing goes to Bot until the owner's config turns voice on; then only what follows the call word goes,
++// as a voice_ask frame (never as a telemetry event), and Bot's answer reaches the plugin, which speaks it.
++func TestVoiceEndToEnd(t *testing.T) {
++ if runtime.GOOS == "windows" {
++ t.Skip("end-to-end tests run on unix")
++ }
++ py, err := exec.LookPath("python3")
++ if err != nil {
++ t.Skip("python3 not installed")
++ }
++ root, _ := filepath.Abs("../..")
++ home := t.TempDir()
++ heard := filepath.Join(home, "heard.txt")
++ said := filepath.Join(home, "said.txt")
++ paths := config.HomePaths(home)
++ srv := fakebot.New()
++ t.Cleanup(srv.Close)
++ srv.Config = protocol.Config{HeartbeatMS: 100, Limits: protocol.Limits{MaxCommandMS: 1000}, AllowedCommands: []string{protocol.KindHalt}}
++ srv.AddCode("VOICE234")
++ creds, err := link.Pair(context.Background(), nil, srv.URL(), protocol.PairRequest{Code: "VOICE234", AgentVersion: "test", DeviceKind: "onboard", Drivers: []string{"voice"}})
++ if err != nil {
++ t.Fatal(err)
++ }
++ cfg := config.Default()
++ cfg.Python = py
++ cfg.Plugins = []config.PluginConfig{{Name: "voice", Config: map[string]any{"engine": "file", "heard_file": heard, "notify": false,
++ "speak": []string{"sh", "-c", `cat >> "$0"`, said}},
++ Env: map[string]string{"PYTHONPATH": filepath.Join(root, "plugins/sdk") + string(os.PathListSeparator) + filepath.Join(root, "plugins/voice")}}}
++ cfg.Video = config.VideoConfig{Source: "off"}
++ n, err := New(Options{Config: cfg, Paths: paths, Creds: creds, Version: "test", HeartbeatInterval: 100 * time.Millisecond,
++ LinkBackoffMin: 50 * time.Millisecond, LatchPoll: 50 * time.Millisecond, Log: testLog()})
++ if err != nil {
++ t.Fatal(err)
++ }
++ ctx, cancel := context.WithCancel(context.Background())
++ done := make(chan struct{})
++ go func() { n.Run(ctx); close(done) }()
++ e := &env{t: t, srv: srv, node: n, paths: paths, cancel: cancel, done: done}
++ t.Cleanup(e.stop)
++ e.conn = e.nextConn()
++ if _, err := e.conn.Expect(protocol.TypeStatus, 20*time.Second, func(m protocol.Message) bool {
++ s := m.(protocol.Status)
++ return len(s.Drivers) == 1 && s.Drivers[0].State == plugins.StateReady
++ }); err != nil {
++ t.Fatal(err)
++ }
++ appendLine := func(s string) {
++ f, err := os.OpenFile(heard, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600)
++ if err != nil {
++ t.Fatal(err)
++ }
++ _, _ = f.WriteString(s + "\n")
++ f.Close()
++ }
++
++ // Voice off (the first config did not say voice): the plugin is not listening, nothing reaches Bot.
++ appendLine("hey vibe are you there")
++ if _, err := e.conn.Expect(protocol.TypeVoiceAsk, 700*time.Millisecond, nil); err == nil {
++ t.Fatal("a voice_ask while voice is off")
++ }
++
++ // The owner turns voice on: the core passes it to the plugin, which listens for the call word.
++ on := srv.Config
++ on.Voice, on.CallWord = true, "hey desk"
++ if err := e.conn.Send(on); err != nil {
++ t.Fatal(err)
++ }
++ time.Sleep(400 * time.Millisecond)
++ appendLine("so anyway the meeting ran long")
++ appendLine("Hey desk, what is on my calendar this afternoon?")
++ f, err := e.conn.Expect(protocol.TypeVoiceAsk, 10*time.Second, nil)
++ if err != nil {
++ t.Fatal(err)
++ }
++ ask := f.Msg.(protocol.VoiceAsk)
++ if ask.Text != "what is on my calendar this afternoon" || !strings.HasPrefix(ask.ID, "va-") {
++ t.Fatalf("voice_ask %+v", ask)
++ }
++ for _, fr := range e.conn.Frames() {
++ if tel, ok := fr.Msg.(protocol.Telemetry); ok {
++ for _, ev := range tel.Events {
++ if ev.Name == "voice_ask" {
++ t.Fatal("the question went out as a telemetry event")
++ }
++ }
++ }
++ if fr.Type == protocol.TypeVoiceAsk && strings.Contains(string(mustJSON(fr.Msg)), "meeting") {
++ t.Fatal("what was not for Vibe left the machine")
++ }
++ }
++
++ // Bot answers working, then with Vibe's answer: the plugin speaks the answer, not the working line.
++ if err := e.conn.Send(protocol.VoiceAnswer{ID: ask.ID, State: "working", Text: "On it."}); err != nil {
++ t.Fatal(err)
++ }
++ if err := e.conn.Send(protocol.VoiceAnswer{ID: ask.ID, State: "answered", Text: "A call at three, nothing after four.", URL: "https://openvibe.chat/vibe/v-0123456789ab"}); err != nil {
++ t.Fatal(err)
++ }
++ deadline := time.Now().Add(10 * time.Second)
++ for {
++ b, _ := os.ReadFile(said)
++ if string(b) == "A call at three, nothing after four." {
++ break
++ }
++ if time.Now().After(deadline) {
++ t.Fatalf("spoken %q", string(b))
++ }
++ time.Sleep(50 * time.Millisecond)
++ }
++}
++
++func mustJSON(v any) []byte {
++ b, _ := json.Marshal(v)
++ return b
++}
ddiff --git a/internal/plugins/plugin.go b/internal/plugins/plugin.go
iindex 18bc064..7360705 100644
---- a/internal/plugins/plugin.go
++++ b/internal/plugins/plugin.go
@@@ -206,6 +206,27 @@ func (p *Plugin) Answer(id string, approved bool) error {
return p.send(map[string]any{"op": "approval", "id": id, "approved": approved})
}
++// Voice tells a voice plugin whether its owner lets Vibe answer their voice here, and the call word to listen for.
++func (p *Plugin) Voice(on bool, callWord string) error {
++ return p.send(map[string]any{"op": "voice", "on": on, "call_word": callWord})
++}
++
++// VoiceAnswer gives a voice plugin Vibe's answer to a voice_ask it raised (to speak and show).
++func (p *Plugin) VoiceAnswer(id, state, text, url string) error {
++ return p.send(map[string]any{"op": "voice_answer", "id": id, "state": state, "text": text, "url": url})
++}
++
++// HasCapability says whether the plugin's describe lists the capability (false before it described itself).
++func (p *Plugin) HasCapability(name string) bool {
++ p.mu.Lock()
++ defer p.mu.Unlock()
++ if p.desc == nil {
++ return false
++ }
++ _, ok := p.desc.Capabilities[name]
++ return ok
++}
++
// Tray tells the plugin whether the Node's tray shows its approvals, for the next ttlS seconds unless told again.
func (p *Plugin) Tray(on bool, ttlS int) error {
return p.send(map[string]any{"op": "tray", "attached": on, "ttl_s": ttlS})
ddiff --git a/internal/protocol/protocol.go b/internal/protocol/protocol.go
iindex 9c4121b..e7dcf8b 100644
---- a/internal/protocol/protocol.go
++++ b/internal/protocol/protocol.go
@@@ -39,6 +39,8 @@ const (
TypeEstop = "estop" // latched true sets the e-stop, false is the owner clearing it
TypeHeartbeatAck = "heartbeat_ack"
TypeError = "error"
++ // TypeVoiceAnswer is Vibe's answer to a voice_ask (contracts 0.214.0): working, answered or refused, to speak.
++ TypeVoiceAnswer = "voice_answer"
)
// Close codes the server ends the device socket with.
@@@ -63,6 +65,8 @@ const (
TypeHeartbeat = "heartbeat"
TypeReauth = "reauth" // a Network-paired machine's fresh node token, before the last one expires
TypeEstopState = "estop_state"
++ // TypeVoiceAsk is what the owner said after the machine's call word, transcribed on the machine (contracts 0.214.0).
++ TypeVoiceAsk = "voice_ask"
)
// Job frames (OpenVibe.Contracts platform.job-frame@1). Server → device: job, job_cancel, job_exit_ack. Device →
@@@ -299,6 +303,26 @@ type Config struct {
// empty list allows only halt.
AllowedCommands []string `json:"allowed_commands"`
EstopLatched bool `json:"estop_latched"`
++ // Voice says the owner lets Vibe answer their voice on this machine (contracts 0.214.0): a voice plugin listens for
++ // CallWord only while it is true.
++ Voice bool `json:"voice,omitempty"`
++ CallWord string `json:"call_word,omitempty"`
++}
++
++// VoiceAsk is the owner's question after the machine's call word (device → server).
++type VoiceAsk struct {
++ ID string `json:"id"`
++ Text string `json:"text"`
++ Lang string `json:"lang,omitempty"`
++ RobotID string `json:"robot_id,omitempty"`
++}
++
++// VoiceAnswer is Vibe's answer to a VoiceAsk with the same ID (server → device): State is working, answered or refused.
++type VoiceAnswer struct {
++ ID string `json:"id"`
++ State string `json:"state"`
++ Text string `json:"text,omitempty"`
++ URL string `json:"url,omitempty"`
}
// Operator is who sent a command, as the server's gate saw them.
@@@ -649,6 +673,8 @@ func (Nack) MessageType() string { return TypeNack }
func (Heartbeat) MessageType() string { return TypeHeartbeat }
func (Reauth) MessageType() string { return TypeReauth }
func (EstopState) MessageType() string { return TypeEstopState }
++func (VoiceAsk) MessageType() string { return TypeVoiceAsk }
++func (VoiceAnswer) MessageType() string { return TypeVoiceAnswer }
func (JobRequest) MessageType() string { return TypeJob }
func (JobCancel) MessageType() string { return TypeJobCancel }
func (JobExitAck) MessageType() string { return TypeJobExitAck }
@@@ -678,6 +704,8 @@ var registry = map[string]func() Message{
TypeJobStdout: func() Message { return &JobStdout{} },
TypeJobUsage: func() Message { return &JobUsage{} },
TypeJobExit: func() Message { return &JobExit{} },
++ TypeVoiceAsk: func() Message { return &VoiceAsk{} },
++ TypeVoiceAnswer: func() Message { return &VoiceAnswer{} },
}
// ErrUnknownType is returned (wrapped) by Decode for a frame whose type is not in this version's message set.
@@@ -777,6 +805,10 @@ func derefMessage(m Message) Message {
return *v
case *JobExit:
return *v
++ case *VoiceAsk:
++ return *v
++ case *VoiceAnswer:
++ return *v
}
return m
}
ddiff --git a/plugins/sdk/openvibe_plugin/__init__.py b/plugins/sdk/openvibe_plugin/__init__.py
iindex fdd0507..aaacb87 100644
---- a/plugins/sdk/openvibe_plugin/__init__.py
++++ b/plugins/sdk/openvibe_plugin/__init__.py
@@@ -137,6 +137,15 @@ class Plugin:
"""The owner answered, on the machine (`openvibe-node approve|deny <id>`), an approval this plugin asked for
with emit_event("approval", id=…). Runs on the runtime's main thread."""
++ def on_voice(self, on, call_word):
++ """Vibe by voice (contracts 0.214.0): the owner lets Vibe answer their voice on this machine (on) and chose
++ call_word, or stopped it. Sent to plugins whose describe lists the `voice` capability, after every config from
++ Bot and when the plugin emits the voice_ready event. A voice plugin listens for the call word only while on."""
++
++ def on_voice_answer(self, cid, state, text, url):
++ """Vibe's answer to a voice_ask this plugin emitted (emit_event("voice_ask", id=…, text=…)): state is working,
++ answered or refused; text is what to say, url the Vibe chat that keeps it."""
++
def on_tray(self, attached, ttl_s):
"""The Node's tray (`openvibe-node tray`, in the owner's graphical session) is showing this plugin's approvals,
for the next ttl_s seconds unless told again. A plugin that notifies the desktop itself can stay quiet then.
@@@ -293,6 +302,16 @@ class Runtime:
self.plugin.on_tray(msg.get("attached") is True, msg.get("ttl_s"))
except Exception as e:
self.plugin.log("tray handler failed: %r", e)
++ elif op == "voice":
++ try:
++ self.plugin.on_voice(msg.get("on") is True, str(msg.get("call_word") or ""))
++ except Exception as e:
++ self.plugin.log("voice handler failed: %r", e)
++ elif op == "voice_answer":
++ try:
++ self.plugin.on_voice_answer(str(msg.get("id") or ""), str(msg.get("state") or ""), str(msg.get("text") or ""), str(msg.get("url") or ""))
++ except Exception as e:
++ self.plugin.log("voice answer handler failed: %r", e)
elif op == "resume":
if self.local_stop_pending:
self.plugin.log("resume ignored: the local stop has not reached the core yet")
ddiff --git a/plugins/voice/openvibe_voice/__init__.py b/plugins/voice/openvibe_voice/__init__.py
nnew file mode 100644
iindex 0000000..652e6f3
---- /dev/null
++++ b/plugins/voice/openvibe_voice/__init__.py
@@@ -0,0 +1,256 @@
++"""OpenVibe Node voice plugin: Vibe by voice on the person's own machine (contracts 0.214.0; consumer plan §8).
++
++The owner lets Vibe answer their voice on this machine and chooses a call word on openvibe.bot/machines; Bot's config
++says so (the core passes it on as the ``voice`` op). While it is on, this plugin listens on the machine itself, and
++only what the person says *after* the call word leaves it: as a ``voice_ask`` event the core sends to Bot, which asks
++Vibe as the owner and sends ``voice_answer`` frames back. The answer is spoken here and shown as a desktop notification,
++and kept in the owner's "Voice on <machine>" chat on openvibe.chat.
++
++Nothing is recorded and no audio leaves the machine: recognition runs locally, and the words before the call word are
++dropped as they are heard.
++
++Config (the plugin's block in config.json):
++
++* ``engine``: ``vosk`` (the default: a local Vosk model, ``model`` = its directory, audio from ``arecord`` or ``parec``)
++ or ``file`` (each new line appended to ``heard_file`` is one thing heard; for tests and for wiring other recognisers).
++* ``lang``: the language of the model, BCP 47 (``en-US``).
++* ``speak``: ``auto`` (Piper when ``piper_model`` is set, else espeak-ng, else spd-say), ``none``, or an argv list that
++ reads the text on stdin (``["sh", "-c", "cat >> said.txt"]``).
++* ``notify``: show answers as desktop notifications (notify-send), true by default.
++* ``follow_up_s``: after the call word alone, how long the next thing heard counts as the question (8 s).
++"""
++
++import json
++import os
++import shutil
++import subprocess
++import threading
++import time
++import unicodedata
++import uuid
++
++from openvibe_plugin import Fault, Plugin
++
++VERSION = "0.1.0"
++SAMPLE_RATE = 16000
++# Speech-to-text writes a spoken greeting several ways (as Live's stream call word, server/ai/vibe-voice.js).
++GREETINGS = {"hey", "hi", "hay", "heyy", "ok", "okay", "yo"}
++
++
++def norm(s):
++ """Lower case, accents off, punctuation to spaces, one space between words."""
++ s = unicodedata.normalize("NFKD", str(s or "").lower())
++ s = "".join(c for c in s if not unicodedata.combining(c))
++ return " ".join("".join(c if (c.isalnum() or c in "' ") else " " for c in s).split())
++
++
++def word_matches(heard, want, first):
++ h = heard[:-2] if heard.endswith("'s") else heard
++ h, w = h.replace("'", ""), want.replace("'", "")
++ if h == w or h == w + "s" or h + "s" == w:
++ return True
++ return first and w in GREETINGS and h in GREETINGS
++
++
++def after_call_word(phrase, call_word):
++ """The words after the call word (possibly ''), or None when the phrase does not say it (whole words only)."""
++ words, want = norm(phrase).split(), norm(call_word).split()
++ if not want:
++ return None
++ for i in range(0, len(words) - len(want) + 1):
++ if all(word_matches(words[i + j], want[j], j == 0) for j in range(len(want))):
++ return " ".join(words[i + len(want):]).strip(" ,.")
++ return None
++
++
++class Voice(Plugin):
++ driver = "voice"
++ version = VERSION
++ motion_kinds = frozenset() # it moves nothing
++
++ def __init__(self):
++ super().__init__()
++ self.on = False
++ self.call_word = "hey vibe"
++ self.armed_until = 0.0
++ self.listener = None
++ self.stopping = threading.Event()
++ self.speaker_lock = threading.Lock()
++ self.cfg = {}
++ self.engine = "vosk"
++
++ # ---- describe / setup ----
++ def describe(self, config):
++ engine = str((config or {}).get("engine") or "vosk")
++ return {"voice": {"engine": engine, "speak": self._speaker_name(config or {})}}
++
++ def setup(self, config):
++ self.cfg = config or {}
++ self.engine = str(self.cfg.get("engine") or "vosk")
++ if self.engine == "vosk":
++ try:
++ import vosk # noqa: F401
++ except ImportError:
++ raise Fault("unsupported", "the voice plugin needs Vosk: pip install vosk, and a model in its config (model)")
++ if not self.cfg.get("model") or not os.path.isdir(str(self.cfg.get("model"))):
++ raise Fault("unsupported", "set model to a Vosk model directory (alphacephei.com/vosk/models, the small one is enough)")
++ if not (shutil.which("arecord") or shutil.which("parec")):
++ raise Fault("unsupported", "the voice plugin reads the microphone with arecord (alsa-utils) or parec (PulseAudio/PipeWire)")
++ elif self.engine == "file":
++ if not self.cfg.get("heard_file"):
++ raise Fault("bad_value", "engine file needs heard_file")
++ else:
++ raise Fault("bad_value", "engine is vosk or file")
++ # Ask for the owner's setting: the core answers with the voice op (on, call word).
++ self.emit_event("voice_ready")
++
++ def handle(self, cmd):
++ raise Fault("unsupported", "the voice plugin takes no commands")
++
++ def close(self):
++ self._stop_listening()
++
++ # ---- the owner's setting ----
++ def on_voice(self, on, call_word):
++ self.call_word = norm(call_word) or "hey vibe"
++ if on and not self.on:
++ self.on = True
++ self._start_listening()
++ self.log("voice on: listening for %r", self.call_word)
++ elif not on and self.on:
++ self.on = False
++ self._stop_listening()
++ self.log("voice off")
++
++ # ---- hearing ----
++ def heard(self, phrase, now=None):
++ """One finished thing heard on the machine. Returns the question sent to Vibe, or None."""
++ if not self.on:
++ return None
++ now = time.monotonic() if now is None else now
++ rest = after_call_word(phrase, self.call_word)
++ if rest is None:
++ if self.armed_until and now <= self.armed_until and norm(phrase):
++ self.armed_until = 0.0
++ return self._ask(norm(phrase))
++ return None # not for Vibe: dropped, never sent or kept
++ if not rest:
++ self.armed_until = now + float(self.cfg.get("follow_up_s") or 8)
++ return None
++ self.armed_until = 0.0
++ return self._ask(rest)
++
++ def _ask(self, text):
++ text = text[:2000]
++ cid = "va-" + uuid.uuid4().hex[:16]
++ fields = {"id": cid, "text": text}
++ if self.cfg.get("lang"):
++ fields["lang"] = str(self.cfg.get("lang"))
++ self.emit_event("voice_ask", **fields)
++ return text
++
++ def _start_listening(self):
++ self.stopping.clear()
++ target = self._listen_file if self.engine == "file" else self._listen_vosk
++ self.listener = threading.Thread(target=target, name="voice-listen", daemon=True)
++ self.listener.start()
++
++ def _stop_listening(self):
++ self.stopping.set()
++ self.listener = None
++
++ def _listen_file(self):
++ path = str(self.cfg.get("heard_file"))
++ pos = os.path.getsize(path) if os.path.exists(path) else 0
++ while not self.stopping.is_set():
++ try:
++ size = os.path.getsize(path) if os.path.exists(path) else 0
++ if size < pos:
++ pos = 0
++ if size > pos:
++ with open(path, "r", encoding="utf-8", errors="replace") as f:
++ f.seek(pos)
++ data = f.read()
++ pos = f.tell()
++ for line in data.splitlines():
++ if line.strip():
++ self.heard(line)
++ except OSError as e:
++ self.log("heard_file: %s", e)
++ self.stopping.wait(0.1)
++
++ def _listen_vosk(self):
++ import vosk
++ vosk.SetLogLevel(-1)
++ model = vosk.Model(str(self.cfg.get("model")))
++ rec = vosk.KaldiRecognizer(model, SAMPLE_RATE)
++ argv = (["arecord", "-q", "-f", "S16_LE", "-r", str(SAMPLE_RATE), "-c", "1", "-t", "raw"] if shutil.which("arecord")
++ else ["parec", "--format=s16le", "--rate=%d" % SAMPLE_RATE, "--channels=1"])
++ proc = subprocess.Popen(argv, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
++ try:
++ while not self.stopping.is_set():
++ chunk = proc.stdout.read(4000)
++ if not chunk:
++ break
++ if rec.AcceptWaveform(chunk):
++ text = json.loads(rec.Result()).get("text", "")
++ if text:
++ self.heard(text)
++ finally:
++ proc.terminate()
++
++ # ---- answering ----
++ def on_voice_answer(self, cid, state, text, url):
++ if state == "working":
++ return
++ if text:
++ self.log("Vibe %s (%s)", state, cid)
++ self.say(text)
++ self._notify(text, url)
++
++ def _speaker_name(self, cfg):
++ speak = cfg.get("speak", "auto")
++ if isinstance(speak, list):
++ return "command"
++ if speak == "none":
++ return None
++ if cfg.get("piper_model") and shutil.which("piper"):
++ return "piper"
++ for name in ("espeak-ng", "spd-say"):
++ if shutil.which(name):
++ return name
++ return None
++
++ def say(self, text):
++ speak = self.cfg.get("speak", "auto")
++ if speak == "none":
++ return
++ if isinstance(speak, list) and speak:
++ argv, data = [str(a) for a in speak], text
++ else:
++ name = self._speaker_name(self.cfg)
++ if name == "piper":
++ argv, data = ["sh", "-c", 'piper --model "$0" --output-raw | aplay -q -r 22050 -f S16_LE -t raw -', str(self.cfg.get("piper_model"))], text
++ elif name == "espeak-ng":
++ argv, data = ["espeak-ng", "--stdin"], text
++ elif name == "spd-say":
++ argv, data = ["spd-say", "-w", text], None
++ else:
++ self.log("nothing to speak with (install espeak-ng, or set piper_model)")
++ return
++ with self.speaker_lock:
++ try:
++ subprocess.run(argv, input=data.encode("utf-8") if data is not None else None, timeout=120, check=False,
++ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
++ except (OSError, subprocess.SubprocessError) as e:
++ self.log("speaking failed: %s", e)
++
++ def _notify(self, text, url):
++ if self.cfg.get("notify", True) is False or not shutil.which("notify-send"):
++ return
++ body = text if not url else "%s\n%s" % (text, url)
++ try:
++ subprocess.run(["notify-send", "-a", "OpenVibe", "Vibe", body[:500]], timeout=5, check=False,
++ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
++ except (OSError, subprocess.SubprocessError):
++ pass
ddiff --git a/plugins/voice/openvibe_voice/__main__.py b/plugins/voice/openvibe_voice/__main__.py
nnew file mode 100644
iindex 0000000..d89004c
---- /dev/null
++++ b/plugins/voice/openvibe_voice/__main__.py
@@@ -0,0 +1,6 @@
++import sys
++
++from openvibe_plugin import run
++from openvibe_voice import Voice
++
++sys.exit(run(Voice()))
ddiff --git a/plugins/voice/pyproject.toml b/plugins/voice/pyproject.toml
nnew file mode 100644
iindex 0000000..8f48ced
---- /dev/null
++++ b/plugins/voice/pyproject.toml
@@@ -0,0 +1,17 @@
++[build-system]
++requires = ["setuptools>=61"]
++build-backend = "setuptools.build_meta"
++
++[project]
++name = "openvibe-voice"
++version = "0.1.0"
++description = "OpenVibe Node voice plugin: Vibe by voice on your own machine; a call word heard locally, only what follows goes to Vibe"
++requires-python = ">=3.9"
++license = { text = "AGPL-3.0-or-later" }
++dependencies = ["openvibe-plugin"]
++
++[project.optional-dependencies]
++vosk = ["vosk"]
++
++[tool.setuptools]
++packages = ["openvibe_voice"]
ddiff --git a/plugins/voice/tests/test_voice.py b/plugins/voice/tests/test_voice.py
nnew file mode 100644
iindex 0000000..432f48c
---- /dev/null
++++ b/plugins/voice/tests/test_voice.py
@@@ -0,0 +1,111 @@
++import io
++import json
++import os
++import sys
++import tempfile
++import time
++import unittest
++
++HERE = os.path.dirname(os.path.abspath(__file__))
++sys.path.insert(0, os.path.dirname(HERE))
++sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(HERE)), "sdk"))
++
++from openvibe_plugin import Runtime # noqa: E402
++from openvibe_voice import Voice, after_call_word # noqa: E402
++
++
++class CallWordTest(unittest.TestCase):
++ def test_the_words_after_the_call_word_whole_words_only(self):
++ self.assertEqual(after_call_word("Hey Vibe, what time is it?", "hey vibe"), "what time is it")
++ self.assertEqual(after_call_word("okay vibe turn the lights off", "hey vibe"), "turn the lights off", "a greeting written another way")
++ self.assertEqual(after_call_word("hey vibes play some jazz", "hey vibe"), "play some jazz", "the plural speech-to-text writes")
++ self.assertEqual(after_call_word("hey vibe", "hey vibe"), "")
++ self.assertIsNone(after_call_word("that's a vibe right there", "hey vibe"))
++ self.assertIsNone(after_call_word("hey vibration", "hey vibe"), "whole words only")
++ self.assertEqual(after_call_word("um hey desk open my mail", "hey desk"), "open my mail")
++
++
++class VoicePluginTest(unittest.TestCase):
++ def setUp(self):
++ self.dir = tempfile.mkdtemp()
++ self.heard = os.path.join(self.dir, "heard.txt")
++ self.said = os.path.join(self.dir, "said.txt")
++ self.out = io.StringIO()
++ self.p = Voice()
++ self.p.log = lambda *a: None
++ self.rt = Runtime(self.p, infile=io.StringIO(""), outfile=self.out)
++ self.send({"op": "hello", "config": {"engine": "file", "heard_file": self.heard, "lang": "en-US", "notify": False,
++ "speak": ["sh", "-c", 'cat >> "$0"', self.said]}})
++
++ def tearDown(self):
++ self.p.close()
++
++ def send(self, m):
++ self.rt.feed_line(json.dumps(m))
++
++ def lines(self):
++ return [json.loads(x) for x in self.out.getvalue().splitlines()]
++
++ def events(self, name):
++ return [m for m in self.lines() if m.get("op") == "event" and m.get("name") == name]
++
++ def test_describes_voice_and_asks_for_the_owners_setting(self):
++ d = [m for m in self.lines() if m.get("op") == "describe"][0]
++ self.assertEqual(d["driver"], "voice")
++ self.assertEqual(d["capabilities"]["voice"]["engine"], "file")
++ self.assertEqual(d["motion_kinds"], [])
++ self.assertEqual(len(self.events("voice_ready")), 1)
++
++ def test_off_nothing_is_sent_on_only_what_follows_the_call_word(self):
++ self.assertIsNone(self.p.heard("hey vibe what time is it"), "voice off: nothing")
++ self.send({"op": "voice", "on": True, "call_word": "Hey Vibe"})
++ self.assertIsNone(self.p.heard("so I told him about the weekend"), "not for Vibe: dropped")
++ self.assertEqual(self.p.heard("hey vibe what time is it"), "what time is it")
++ asks = self.events("voice_ask")
++ self.assertEqual(len(asks), 1)
++ self.assertEqual(asks[0]["text"], "what time is it")
++ self.assertEqual(asks[0]["lang"], "en-US")
++ self.assertRegex(asks[0]["id"], r"^va-[0-9a-f]{16}$")
++ self.assertNotIn("weekend", self.out.getvalue(), "what was not for Vibe never left the plugin")
++
++ def test_the_call_word_alone_then_the_question(self):
++ self.send({"op": "voice", "on": True, "call_word": "hey vibe"})
++ t = time.monotonic()
++ self.assertIsNone(self.p.heard("hey vibe", now=t))
++ self.assertEqual(self.p.heard("turn on the kitchen lights", now=t + 3), "turn on the kitchen lights")
++ self.assertIsNone(self.p.heard("and something else", now=t + 4), "the follow-up is one question")
++ self.assertIsNone(self.p.heard("hey vibe", now=t + 10))
++ self.assertIsNone(self.p.heard("too late now", now=t + 30), "after follow_up_s it is not a question")
++
++ def test_heard_from_the_file_engine(self):
++ self.send({"op": "voice", "on": True, "call_word": "hey vibe"})
++ with open(self.heard, "a") as f:
++ f.write("background noise\nhey vibe how far is the moon\n")
++ deadline = time.monotonic() + 5
++ while time.monotonic() < deadline and not self.events("voice_ask"):
++ time.sleep(0.05)
++ self.assertEqual([a["text"] for a in self.events("voice_ask")], ["how far is the moon"])
++ self.send({"op": "voice", "on": False, "call_word": "hey vibe"})
++ with open(self.heard, "a") as f:
++ f.write("hey vibe are you still there\n")
++ time.sleep(0.3)
++ self.assertEqual(len(self.events("voice_ask")), 1, "off: the listener stopped")
++
++ def test_the_answer_is_spoken_working_is_not(self):
++ self.send({"op": "voice_answer", "id": "va-1", "state": "working", "text": "On it."})
++ self.assertFalse(os.path.exists(self.said))
++ self.send({"op": "voice_answer", "id": "va-1", "state": "answered", "text": "It is about 384,000 kilometres away.", "url": "https://openvibe.chat/vibe/v-0123456789ab"})
++ with open(self.said) as f:
++ self.assertEqual(f.read(), "It is about 384,000 kilometres away.")
++ self.send({"op": "voice_answer", "id": "va-2", "state": "refused", "text": "Voice is off for this machine."})
++ with open(self.said) as f:
++ self.assertIn("Voice is off for this machine.", f.read())
++
++ def test_takes_no_commands(self):
++ self.send({"op": "command", "id": "c1", "kind": "say", "value": {"text": "x"}, "deadline_ms": 1000})
++ last = self.lines()[-1]
++ self.assertEqual((last["op"], last["fault_code"]), ("nack", "unsupported"))
++
++
++if __name__ == "__main__":
++ unittest.main()
ddiff --git a/scripts/dist.sh b/scripts/dist.sh
iindex a8fa1a4..b9919a7 100755
---- a/scripts/dist.sh
++++ b/scripts/dist.sh
@@@ -24,7 +24,7 @@ done
# The plugin bundle: the Python packages without their tests.
tar -czf dist/openvibe-node-plugins.tar.gz -C plugins \
--exclude=tests --exclude=__pycache__ --exclude='*.egg-info' --exclude=.pytest_cache --exclude=build \
-- sdk dryrun adeept_adr036 cozmo relay ros2 computer
++ sdk dryrun adeept_adr036 cozmo relay ros2 computer voice
# OpenVibe.Bot's GET /install route and BOT_INSTALLER_SOURCE_URL are documented to serve this asset name:
# https://github.com/OpenVibers/OpenVibe.Node/releases/latest/download/install.sh — do not rename it.
cp install/install.sh dist/install.sh