openvibers/openvibe.contracts Commits

bot.command@1: kind desk (start, stop, status a private desk on the machine) and desk: private on screen, input and browser (0.214.0) (#212)

OP OpenVibers committed 19 days ago · 814f79f664 · 9 files, +217 −6

Changes

ddiff --git a/CHANGELOG.md b/CHANGELOG.md
iindex 78d9827b..58a3006f 100644
---- a/CHANGELOG.md
++++ b/CHANGELOG.md
@@@ -6,6 +6,12 @@ the notes were in the tag and commit messages (`git tag -n1`).
  
  ## 0.214.0 — 2026-10-26
  
++- **A private desk for agents on a person's own machine** (plan track K, consumer plan step 2: Vibe's desk on Node).
++  `bot.command@1` takes kind `desk` (`action`: start | stop | status; `width`, `height` on start): a display of its own
++  on the machine (Xvfb, a window manager, its own session and browser profile) where an agent works without touching
++  the person's screen, which the person can watch and take over. `screen`, `input` and `browser` take `desk: private`
++  to act there instead of on the person's screen (`main`, the default). Starting and stopping act (refused at observe,
++  approved one by one at ask); status runs at every level. Fixtures for both.
  - **Support inboxes open to developer apps** (the owner, 2026-10-24/25: openvibe.chat "fully modular and SDK and
    API"). Three public, sensitive capabilities owned by `chat`, on the routes OpenVibe.Chat already serves; an app acts
    only for the person who authorized it (`on_behalf_of`), and that person's role in the inbox decides the rest:
ddiff --git a/contracts/bot/command.v1.json b/contracts/bot/command.v1.json
iindex 4e6cdd53..ecff69be 100644
---- a/contracts/bot/command.v1.json
++++ b/contracts/bot/command.v1.json
@@@ -2,7 +2,7 @@
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://openvibe.network/contracts/bot/command.v1.json",
    "title": "BotCommand",
--  "description": "bot.command@1: an operator's command frame on OpenVibe.Bot's operator WebSocket (wss://openvibe.bot/control, after join; server/realtime.js onOperatorCommand). id is the operator's idempotency key (at most 64 characters; Bot mints one when it is missing): a repeated id answers with the first result. Bot's gate stamps the robot, the operator and the role, builds the value from the robot profile's commands (clamped by the owner's limits) and sets the deadline; ms asks for a duration within max_command_ms. Answered by one command_result frame (bot.command-result@1). The kinds button (a profile button by name, with an optional state for a hold button) and point (a normalised x, y video click) were added in 0.109.0 (plan T15 R9). The computer kinds (exec, fs, screen, input, browser; OpenVibe.Node's computer plugin, Node docs/computer.md) are taken only for a person's own machine, from its owner or an agent the owner let use it (bot.computer.control); the machine's owner sets on the machine what they may do, and the answer comes back in bot.command-result@1's data. probe (a read-only check of something on the machine's own network) is the one computer kind OpenVibe.Watch may send, for owners who let their watches use the computer. input (added in 0.171.0) moves the mouse, clicks, scrolls, types or presses keys on the machine's screen, at coordinates in the space of the screen picture the caller last saw (its width and height; the Node scales them to the real screen): refused at level observe, approved one by one at ask. browser (added in 0.174.0) drives a Chromium-family browser the agent uses on the machine, in a profile of its own: open an http(s) URL, click a CSS selector or the button or link showing a text, fill a field, press a named key, go back, read the page, take a screenshot, or close it. read, screenshot and close run at every level; the rest act (refused at observe, approved one by one at ask), and below level full open refuses an address on the machine's own network.",
++  "description": "bot.command@1: an operator's command frame on OpenVibe.Bot's operator WebSocket (wss://openvibe.bot/control, after join; server/realtime.js onOperatorCommand). id is the operator's idempotency key (at most 64 characters; Bot mints one when it is missing): a repeated id answers with the first result. Bot's gate stamps the robot, the operator and the role, builds the value from the robot profile's commands (clamped by the owner's limits) and sets the deadline; ms asks for a duration within max_command_ms. Answered by one command_result frame (bot.command-result@1). The kinds button (a profile button by name, with an optional state for a hold button) and point (a normalised x, y video click) were added in 0.109.0 (plan T15 R9). The computer kinds (exec, fs, screen, input, browser; OpenVibe.Node's computer plugin, Node docs/computer.md) are taken only for a person's own machine, from its owner or an agent the owner let use it (bot.computer.control); the machine's owner sets on the machine what they may do, and the answer comes back in bot.command-result@1's data. probe (a read-only check of something on the machine's own network) is the one computer kind OpenVibe.Watch may send, for owners who let their watches use the computer. input (added in 0.171.0) moves the mouse, clicks, scrolls, types or presses keys on the machine's screen, at coordinates in the space of the screen picture the caller last saw (its width and height; the Node scales them to the real screen): refused at level observe, approved one by one at ask. browser (added in 0.174.0) drives a Chromium-family browser the agent uses on the machine, in a profile of its own: open an http(s) URL, click a CSS selector or the button or link showing a text, fill a field, press a named key, go back, read the page, take a screenshot, or close it. read, screenshot and close run at every level; the rest act (refused at observe, approved one by one at ask), and below level full open refuses an address on the machine's own network. desk (added in 0.214.0; plan track K, Vibe's desk) starts, stops or reports a private desk on the machine: a display of its own (Xvfb, a window manager, its own session and browser profile) where an agent works without touching the person's screen, which the person can watch and take over; screen, input and browser take desk: private to act there instead of on the person's screen (main, the default). Starting and stopping a desk act (refused at observe, approved one by one at ask); status runs at every level.",
    "type": "object",
    "required": [
      "type",
@@@ -44,7 +44,8 @@
          "screen",
          "probe",
          "input",
--        "browser"
++        "browser",
++        "desk"
        ]
      },
      "value": {
@@@ -280,6 +281,13 @@
                  "type": "integer",
                  "minimum": 320,
                  "maximum": 3840
++              },
++              "desk": {
++                "enum": [
++                  "main",
++                  "private"
++                ],
++                "description": "which display: the person's own screen (main, the default) or the machine's private desk (desk: start first)"
                }
              }
            }
@@@ -402,6 +410,13 @@
                  "maxLength": 64,
                  "pattern": "^[A-Za-z0-9_+\\-]+$",
                  "description": "one key or chord, for example Return, Escape, ctrl+s, alt+Tab"
++              },
++              "desk": {
++                "enum": [
++                  "main",
++                  "private"
++                ],
++                "description": "which display: the person's own screen (main, the default) or the machine's private desk (desk: start first)"
                }
              }
            }
@@@ -485,6 +500,13 @@
                  "type": "integer",
                  "minimum": 320,
                  "maximum": 2560
++              },
++              "desk": {
++                "enum": [
++                  "main",
++                  "private"
++                ],
++                "description": "which display: the person's own screen (main, the default) or the machine's private desk (desk: start first)"
                }
              },
              "allOf": [
@@@ -570,6 +592,54 @@
            }
          }
        }
++    },
++    {
++      "if": {
++        "properties": {
++          "kind": {
++            "const": "desk"
++          }
++        },
++        "required": [
++          "kind"
++        ]
++      },
++      "then": {
++        "required": [
++          "value"
++        ],
++        "properties": {
++          "value": {
++            "type": "object",
++            "required": [
++              "action"
++            ],
++            "additionalProperties": false,
++            "properties": {
++              "action": {
++                "enum": [
++                  "start",
++                  "stop",
++                  "status"
++                ],
++                "description": "start the private desk (a no-op when it runs), stop it, or report it"
++              },
++              "width": {
++                "type": "integer",
++                "minimum": 640,
++                "maximum": 3840,
++                "description": "start only: the desk's width in pixels (default 1280)"
++              },
++              "height": {
++                "type": "integer",
++                "minimum": 480,
++                "maximum": 2160,
++                "description": "start only: the desk's height in pixels (default 800)"
++              }
++            }
++          }
++        }
++      }
      }
    ]
  }
ddiff --git a/fixtures/bot.command/invalid/desk-bad-action.json b/fixtures/bot.command/invalid/desk-bad-action.json
nnew file mode 100644
iindex 00000000..cb65ada9
---- /dev/null
++++ b/fixtures/bot.command/invalid/desk-bad-action.json
@@@ -0,0 +1,12 @@
++{
++  "v": 1,
++  "seq": 73,
++  "ts": 1759600000000,
++  "type": "command",
++  "id": "op-73",
++  "kind": "desk",
++  "value": {
++    "action": "exec",
++    "width": 1280
++  }
++}
ddiff --git a/fixtures/bot.command/invalid/screen-unknown-desk.json b/fixtures/bot.command/invalid/screen-unknown-desk.json
nnew file mode 100644
iindex 00000000..a8352315
---- /dev/null
++++ b/fixtures/bot.command/invalid/screen-unknown-desk.json
@@@ -0,0 +1,11 @@
++{
++  "v": 1,
++  "seq": 74,
++  "ts": 1759600000000,
++  "type": "command",
++  "id": "op-74",
++  "kind": "screen",
++  "value": {
++    "desk": "neighbour"
++  }
++}
ddiff --git a/fixtures/bot.command/valid/desk-start.json b/fixtures/bot.command/valid/desk-start.json
nnew file mode 100644
iindex 00000000..147de952
---- /dev/null
++++ b/fixtures/bot.command/valid/desk-start.json
@@@ -0,0 +1,13 @@
++{
++  "v": 1,
++  "seq": 70,
++  "ts": 1759600000000,
++  "type": "command",
++  "id": "op-70",
++  "kind": "desk",
++  "value": {
++    "action": "start",
++    "width": 1280,
++    "height": 800
++  }
++}
ddiff --git a/fixtures/bot.command/valid/input-click-private-desk.json b/fixtures/bot.command/valid/input-click-private-desk.json
nnew file mode 100644
iindex 00000000..1a0ea572
---- /dev/null
++++ b/fixtures/bot.command/valid/input-click-private-desk.json
@@@ -0,0 +1,16 @@
++{
++  "v": 1,
++  "seq": 71,
++  "ts": 1759600000000,
++  "type": "command",
++  "id": "op-71",
++  "kind": "input",
++  "value": {
++    "action": "click",
++    "x": 640,
++    "y": 400,
++    "width": 1280,
++    "height": 800,
++    "desk": "private"
++  }
++}
ddiff --git a/fixtures/bot.command/valid/screen-private-desk.json b/fixtures/bot.command/valid/screen-private-desk.json
nnew file mode 100644
iindex 00000000..83ee5b86
---- /dev/null
++++ b/fixtures/bot.command/valid/screen-private-desk.json
@@@ -0,0 +1,12 @@
++{
++  "v": 1,
++  "seq": 72,
++  "ts": 1759600000000,
++  "type": "command",
++  "id": "op-72",
++  "kind": "screen",
++  "value": {
++    "max_width": 1280,
++    "desk": "private"
++  }
++}
ddiff --git a/generated/json-schema/bundle.json b/generated/json-schema/bundle.json
iindex bec242e2..f2285b63 100644
---- a/generated/json-schema/bundle.json
++++ b/generated/json-schema/bundle.json
@@@ -72484,7 +72484,7 @@
          "$schema": "https://json-schema.org/draft/2020-12/schema",
          "$id": "https://openvibe.network/contracts/bot/command.v1.json",
          "title": "BotCommand",
--        "description": "bot.command@1: an operator's command frame on OpenVibe.Bot's operator WebSocket (wss://openvibe.bot/control, after join; server/realtime.js onOperatorCommand). id is the operator's idempotency key (at most 64 characters; Bot mints one when it is missing): a repeated id answers with the first result. Bot's gate stamps the robot, the operator and the role, builds the value from the robot profile's commands (clamped by the owner's limits) and sets the deadline; ms asks for a duration within max_command_ms. Answered by one command_result frame (bot.command-result@1). The kinds button (a profile button by name, with an optional state for a hold button) and point (a normalised x, y video click) were added in 0.109.0 (plan T15 R9). The computer kinds (exec, fs, screen, input, browser; OpenVibe.Node's computer plugin, Node docs/computer.md) are taken only for a person's own machine, from its owner or an agent the owner let use it (bot.computer.control); the machine's owner sets on the machine what they may do, and the answer comes back in bot.command-result@1's data. probe (a read-only check of something on the machine's own network) is the one computer kind OpenVibe.Watch may send, for owners who let their watches use the computer. input (added in 0.171.0) moves the mouse, clicks, scrolls, types or presses keys on the machine's screen, at coordinates in the space of the screen picture the caller last saw (its width and height; the Node scales them to the real screen): refused at level observe, approved one by one at ask. browser (added in 0.174.0) drives a Chromium-family browser the agent uses on the machine, in a profile of its own: open an http(s) URL, click a CSS selector or the button or link showing a text, fill a field, press a named key, go back, read the page, take a screenshot, or close it. read, screenshot and close run at every level; the rest act (refused at observe, approved one by one at ask), and below level full open refuses an address on the machine's own network.",
++        "description": "bot.command@1: an operator's command frame on OpenVibe.Bot's operator WebSocket (wss://openvibe.bot/control, after join; server/realtime.js onOperatorCommand). id is the operator's idempotency key (at most 64 characters; Bot mints one when it is missing): a repeated id answers with the first result. Bot's gate stamps the robot, the operator and the role, builds the value from the robot profile's commands (clamped by the owner's limits) and sets the deadline; ms asks for a duration within max_command_ms. Answered by one command_result frame (bot.command-result@1). The kinds button (a profile button by name, with an optional state for a hold button) and point (a normalised x, y video click) were added in 0.109.0 (plan T15 R9). The computer kinds (exec, fs, screen, input, browser; OpenVibe.Node's computer plugin, Node docs/computer.md) are taken only for a person's own machine, from its owner or an agent the owner let use it (bot.computer.control); the machine's owner sets on the machine what they may do, and the answer comes back in bot.command-result@1's data. probe (a read-only check of something on the machine's own network) is the one computer kind OpenVibe.Watch may send, for owners who let their watches use the computer. input (added in 0.171.0) moves the mouse, clicks, scrolls, types or presses keys on the machine's screen, at coordinates in the space of the screen picture the caller last saw (its width and height; the Node scales them to the real screen): refused at level observe, approved one by one at ask. browser (added in 0.174.0) drives a Chromium-family browser the agent uses on the machine, in a profile of its own: open an http(s) URL, click a CSS selector or the button or link showing a text, fill a field, press a named key, go back, read the page, take a screenshot, or close it. read, screenshot and close run at every level; the rest act (refused at observe, approved one by one at ask), and below level full open refuses an address on the machine's own network. desk (added in 0.214.0; plan track K, Vibe's desk) starts, stops or reports a private desk on the machine: a display of its own (Xvfb, a window manager, its own session and browser profile) where an agent works without touching the person's screen, which the person can watch and take over; screen, input and browser take desk: private to act there instead of on the person's screen (main, the default). Starting and stopping a desk act (refused at observe, approved one by one at ask); status runs at every level.",
          "type": "object",
          "required": [
            "type",
@@@ -72526,7 +72526,8 @@
                "screen",
                "probe",
                "input",
--              "browser"
++              "browser",
++              "desk"
              ]
            },
            "value": {
@@@ -72762,6 +72763,13 @@
                        "type": "integer",
                        "minimum": 320,
                        "maximum": 3840
++                    },
++                    "desk": {
++                      "enum": [
++                        "main",
++                        "private"
++                      ],
++                      "description": "which display: the person's own screen (main, the default) or the machine's private desk (desk: start first)"
                      }
                    }
                  }
@@@ -72884,6 +72892,13 @@
                        "maxLength": 64,
                        "pattern": "^[A-Za-z0-9_+\\-]+$",
                        "description": "one key or chord, for example Return, Escape, ctrl+s, alt+Tab"
++                    },
++                    "desk": {
++                      "enum": [
++                        "main",
++                        "private"
++                      ],
++                      "description": "which display: the person's own screen (main, the default) or the machine's private desk (desk: start first)"
                      }
                    }
                  }
@@@ -72967,6 +72982,13 @@
                        "type": "integer",
                        "minimum": 320,
                        "maximum": 2560
++                    },
++                    "desk": {
++                      "enum": [
++                        "main",
++                        "private"
++                      ],
++                      "description": "which display: the person's own screen (main, the default) or the machine's private desk (desk: start first)"
                      }
                    },
                    "allOf": [
@@@ -73052,6 +73074,54 @@
                  }
                }
              }
++          },
++          {
++            "if": {
++              "properties": {
++                "kind": {
++                  "const": "desk"
++                }
++              },
++              "required": [
++                "kind"
++              ]
++            },
++            "then": {
++              "required": [
++                "value"
++              ],
++              "properties": {
++                "value": {
++                  "type": "object",
++                  "required": [
++                    "action"
++                  ],
++                  "additionalProperties": false,
++                  "properties": {
++                    "action": {
++                      "enum": [
++                        "start",
++                        "stop",
++                        "status"
++                      ],
++                      "description": "start the private desk (a no-op when it runs), stop it, or report it"
++                    },
++                    "width": {
++                      "type": "integer",
++                      "minimum": 640,
++                      "maximum": 3840,
++                      "description": "start only: the desk's width in pixels (default 1280)"
++                    },
++                    "height": {
++                      "type": "integer",
++                      "minimum": 480,
++                      "maximum": 2160,
++                      "description": "start only: the desk's height in pixels (default 800)"
++                    }
++                  }
++                }
++              }
++            }
            }
          ]
        }
ddiff --git a/generated/typescript/index.d.ts b/generated/typescript/index.d.ts
iindex 7ff135e9..20dd25de 100644
---- a/generated/typescript/index.d.ts
++++ b/generated/typescript/index.d.ts
@@@ -197037,7 +197037,7 @@ export type BotDeviceMessage =
  
  /** [email protected] (owner: bot) */
  /**
-- * bot.command@1: an operator's command frame on OpenVibe.Bot's operator WebSocket (wss://openvibe.bot/control, after join; server/realtime.js onOperatorCommand). id is the operator's idempotency key (at most 64 characters; Bot mints one when it is missing): a repeated id answers with the first result. Bot's gate stamps the robot, the operator and the role, builds the value from the robot profile's commands (clamped by the owner's limits) and sets the deadline; ms asks for a duration within max_command_ms. Answered by one command_result frame (bot.command-result@1). The kinds button (a profile button by name, with an optional state for a hold button) and point (a normalised x, y video click) were added in 0.109.0 (plan T15 R9). The computer kinds (exec, fs, screen, input, browser; OpenVibe.Node's computer plugin, Node docs/computer.md) are taken only for a person's own machine, from its owner or an agent the owner let use it (bot.computer.control); the machine's owner sets on the machine what they may do, and the answer comes back in bot.command-result@1's data. probe (a read-only check of something on the machine's own network) is the one computer kind OpenVibe.Watch may send, for owners who let their watches use the computer. input (added in 0.171.0) moves the mouse, clicks, scrolls, types or presses keys on the machine's screen, at coordinates in the space of the screen picture the caller last saw (its width and height; the Node scales them to the real screen): refused at level observe, approved one by one at ask. browser (added in 0.174.0) drives a Chromium-family browser the agent uses on the machine, in a profile of its own: open an http(s) URL, click a CSS selector or the button or link showing a text, fill a field, press a named key, go back, read the page, take a screenshot, or close it. read, screenshot and close run at every level; the rest act (refused at observe, approved one by one at ask), and below level full open refuses an address on the machine's own network.
++ * bot.command@1: an operator's command frame on OpenVibe.Bot's operator WebSocket (wss://openvibe.bot/control, after join; server/realtime.js onOperatorCommand). id is the operator's idempotency key (at most 64 characters; Bot mints one when it is missing): a repeated id answers with the first result. Bot's gate stamps the robot, the operator and the role, builds the value from the robot profile's commands (clamped by the owner's limits) and sets the deadline; ms asks for a duration within max_command_ms. Answered by one command_result frame (bot.command-result@1). The kinds button (a profile button by name, with an optional state for a hold button) and point (a normalised x, y video click) were added in 0.109.0 (plan T15 R9). The computer kinds (exec, fs, screen, input, browser; OpenVibe.Node's computer plugin, Node docs/computer.md) are taken only for a person's own machine, from its owner or an agent the owner let use it (bot.computer.control); the machine's owner sets on the machine what they may do, and the answer comes back in bot.command-result@1's data. probe (a read-only check of something on the machine's own network) is the one computer kind OpenVibe.Watch may send, for owners who let their watches use the computer. input (added in 0.171.0) moves the mouse, clicks, scrolls, types or presses keys on the machine's screen, at coordinates in the space of the screen picture the caller last saw (its width and height; the Node scales them to the real screen): refused at level observe, approved one by one at ask. browser (added in 0.174.0) drives a Chromium-family browser the agent uses on the machine, in a profile of its own: open an http(s) URL, click a CSS selector or the button or link showing a text, fill a field, press a named key, go back, read the page, take a screenshot, or close it. read, screenshot and close run at every level; the rest act (refused at observe, approved one by one at ask), and below level full open refuses an address on the machine's own network. desk (added in 0.214.0; plan track K, Vibe's desk) starts, stops or reports a private desk on the machine: a display of its own (Xvfb, a window manager, its own session and browser profile) where an agent works without touching the person's screen, which the person can watch and take over; screen, input and browser take desk: private to act there instead of on the person's screen (main, the default). Starting and stopping a desk act (refused at observe, approved one by one at ask); status runs at every level.
   */
  export type BotCommand = {
    [k: string]: unknown | undefined;
@@@ -197061,7 +197061,8 @@ export type BotCommand = {
      | "screen"
      | "probe"
      | "input"
--    | "browser";
++    | "browser"
++    | "desk";
    value?: {};
    ms?: number;
  };