openvibers/openvibe.link Commits

v0.23.0: browser desks (Desks plan D2): a headless browser alone, its page the desk's screen, input and live view, at half a desk's cost, on Linux or macOS; a desk host's screen and input reach its desks; a named desk's clicks scale to its own size (#69)

OP OpenVibers committed 17 days ago · 2e5f0e6e20 · 14 files, +691 −71

Changes

ddiff --git a/STATUS.json b/STATUS.json
iindex 75821fe..b01743f 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -5,8 +5,8 @@
"wave": "T15",
"plan": "Plan track T15 (Bot complete) and ADR-043 (Bot: devices, pairing, control and safety); the computer plugin and the job worker serve OpenVibe.Actor and the Codes job runner (tracks T14/T17)",
"kind": "client",
-- "version": "0.22.0",
-- "release": "v0.22.0, published 2026-10-27 as a GitHub release (desks on a desk host, found by the first production proof: a desk's temp folder is short (Chrome's SingletonSocket must fit a Unix socket path, and the desks folder on a desk host was too deep, so Chrome closed as it started), and a desk no command used for desk_idle_s (15 minutes) stops by itself, so a desk an agent left behind when it crashed or restarted ends; v0.21.0: element refs and receipts, the Desks plan's step D3: the agent's browser lists a page's interactive elements as compact refs (e7 button \"Sign in\"), kept on the element while it stays, and click, press and back answer with what changed (url, lines that appeared, lines gone), so agents act on the page's structure before screenshots; v0.20.0: desk hosts, the Desks plan's step D5: install.sh --desk-host turns an old laptop, a mini PC or a spare workstation into a host of desks for its owner's agents, run as its own openvibe-desk account with Xvfb, openbox, D-Bus, xdotool, ImageMagick and Chromium installed and max_desks sized to its memory; v0.19.0: several desks per machine, the Desks plan's step D1: the computer plugin runs named desks (desk-<name>, private kept), each with its own X display, D-Bus, browser profile, downloads and temp folder, up to max_desks (one per 2 GiB of memory by default), so two agents work at once without seeing or driving each other's screen; proven with two real Chrome desks that never share a cookie; v0.18.0: a motion command's validity also counts the time it waits in the queue to its plugin and in the pipe: the core fixes the instant it expires at receipt, writes what is left at the moment it writes the line with expires_at_mono_ns on the clock it shares with the plugin, and never writes one that has run out, and the Python runtime refuses one that ran out in the pipe before handle(); the clock bound is kept in nanoseconds and counts Bot's whole-millisecond stamps, rounded up from nanoseconds; a Node booting latched sets the e-stop on its plugins before starting them; the Python SDK package reports the Node release; v0.17.0: deadlines are checked against the latest the server's clock can be: the estimate plus its bound, the connection's fastest round trip, and motion waits for that bound on every new connection; every stop and estop names its cause, hello carries the newest halt epoch, and the Python SDK gives drivers on_stop(StopInfo) and on_resume to tell an ordinary stop from an emergency without re-arming anything; v0.16.1: the Python runtime refuses a motion command whose deadline passed while it waited behind a slow handler, so a driver never moves on a stale command; plugins.md answers where deadlines come from, re-stamping, order across a change of driver, stop versus estop, and that nothing is held past its validity; v0.16.0: openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS",
++ "version": "0.23.0",
++ "release": "v0.23.0, published 2026-10-27 as a GitHub release (browser desks, the Desks plan's step D2: desk start takes kind browser, a headless browser alone with no display at about half the cost of a whole desk, on Linux or macOS; its page is its screen, mouse and keyboard and live view over the browser's pipe, exactly the desk's size; a desk host, which has no screen or input of its own, gets screen, screen_stream and input on its desks; a click on a named desk scales to that desk's size; v0.22.0: desks on a desk host, found by the first production proof: a desk's temp folder is short (Chrome's SingletonSocket must fit a Unix socket path, and the desks folder on a desk host was too deep, so Chrome closed as it started), and a desk no command used for desk_idle_s (15 minutes) stops by itself, so a desk an agent left behind when it crashed or restarted ends; v0.21.0: element refs and receipts, the Desks plan's step D3: the agent's browser lists a page's interactive elements as compact refs (e7 button \"Sign in\"), kept on the element while it stays, and click, press and back answer with what changed (url, lines that appeared, lines gone), so agents act on the page's structure before screenshots; v0.20.0: desk hosts, the Desks plan's step D5: install.sh --desk-host turns an old laptop, a mini PC or a spare workstation into a host of desks for its owner's agents, run as its own openvibe-desk account with Xvfb, openbox, D-Bus, xdotool, ImageMagick and Chromium installed and max_desks sized to its memory; v0.19.0: several desks per machine, the Desks plan's step D1: the computer plugin runs named desks (desk-<name>, private kept), each with its own X display, D-Bus, browser profile, downloads and temp folder, up to max_desks (one per 2 GiB of memory by default), so two agents work at once without seeing or driving each other's screen; proven with two real Chrome desks that never share a cookie; v0.18.0: a motion command's validity also counts the time it waits in the queue to its plugin and in the pipe: the core fixes the instant it expires at receipt, writes what is left at the moment it writes the line with expires_at_mono_ns on the clock it shares with the plugin, and never writes one that has run out, and the Python runtime refuses one that ran out in the pipe before handle(); the clock bound is kept in nanoseconds and counts Bot's whole-millisecond stamps, rounded up from nanoseconds; a Node booting latched sets the e-stop on its plugins before starting them; the Python SDK package reports the Node release; v0.17.0: deadlines are checked against the latest the server's clock can be: the estimate plus its bound, the connection's fastest round trip, and motion waits for that bound on every new connection; every stop and estop names its cause, hello carries the newest halt epoch, and the Python SDK gives drivers on_stop(StopInfo) and on_resume to tell an ordinary stop from an emergency without re-arming anything; v0.16.1: the Python runtime refuses a motion command whose deadline passed while it waited behind a slow handler, so a driver never moves on a stale command; plugins.md answers where deadlines come from, re-stamping, order across a change of driver, stop versus estop, and that nothing is held past its validity; v0.16.0: openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS",
"runtime": "go 1.24 (one static binary per platform); plugins in Python 3; an ESP32 Arduino library",
"database": "none",
"deployed": false,
ddiff --git a/docs/computer.md b/docs/computer.md
iindex 4d2e5d3..66b62e2 100644
---- a/docs/computer.md
++++ b/docs/computer.md
@@@ -167,6 +167,25 @@ An agent can work on a desk of its own instead of your screen (plan track K: Vib
- **Needs** Linux with `Xvfb` (Debian/Ubuntu: `apt install xvfb`), `xdotool` for input and ImageMagick `import` or `scrot`
for screenshots. Not offered elsewhere; `allow_desk: false` turns it off.
++### Browser desks (`"kind": "browser"`, Node v0.23.0)
++
++Most agent work happens in a browser, and a browser does not need a screen. `{"action": "start", "desk": "desk-<name>",
++"kind": "browser"}` starts a **browser desk** (the Desks plan's step D2): a headless browser alone, with no display or
++window manager.
++- **About half the cost of a whole desk.** A browser desk counts as half a desk against `max_desks`, so a machine with
++ `max_desks: 2` runs four browser desks, or one whole desk and two browser desks.
++- **On a Mac too.** A browser desk needs only Chromium, Chrome, Edge or Brave, so it runs on macOS as well as Linux.
++ `describe` lists the kinds a machine runs in `desk.kinds` (`desktop`, `browser`).
++- **Its page is its screen.** `screen` takes a picture of the page. `input` is the page's own mouse and keyboard:
++ clicks, the wheel, keys and chords in xdotool's names (`Return`, `ctrl+a`), and text. A click answers with what it
++ changed (a receipt, as in the browser kind). `screen_stream` sends frames of the page, so you can watch a browser desk
++ live and take it over like any desk. All of it goes through the browser's pipe: no display, no capture tool, no
++ Pillow.
++- **Like any desk.** It has its own profile (`browser_profile` + `-<name>`), downloads and temp folder, and nothing of
++ your session (no `DISPLAY`, session bus or X cookie). `browser` drives its page, and its page is exactly the desk's
++ size (1280×800 by default).
++- `private` is always a whole desk; a whole desk is `"kind": "desktop"`, the default.
++
### Desk hosts (`--desk-host`, Node v0.20.0)
An old laptop, a mini PC or a spare workstation can host desks for your agents: no screen, keyboard or login needed.
@@@ -182,6 +201,9 @@ curl -fsSL https://openvibe.bot/install | sh -s -- --network … --pairing pair_
many. Each agent task gets a desk of its own (OpenVibe.Actor with OpenVibe.Bot, Node v0.19.0): its own screen,
browser profile, downloads and temp folder. A task beyond `max_desks` is told the machine is busy (its desk does not
start); it never shares another task's desk.
++- **No screen of its own.** A desk host offers no screen, mouse or keyboard of its own, only its desks'. The Node gives
++ `screen`, `screen_stream` and `input` to the plugin with desks, which acts on the desk named and refuses the host's
++ own screen (Node v0.23.0).
- **Then** turn on "Let my agent use this computer" for the machine on openvibe.bot/machines. Watch any desk live from
there, and take it over the same way.
@@@ -200,7 +222,7 @@ curl -fsSL https://openvibe.bot/install | sh -s -- --network … --pairing pair_
| `probe` | `true` | `false`: no probes (OpenVibe.Watch's `node` source) |
| `allow_input` | `true` | `false`: never use the mouse or keyboard (no `input` capability) |
| `allow_desk` | `true` | `false`: never start a private desk (no `desk` capability) |
--| `max_desks` | memory / 2 GiB, 1 to 8 | how many desks run at once (1 to 10) |
++| `max_desks` | memory / 2 GiB, 1 to 8 | how many whole desks run at once (1 to 10); a browser desk counts as half |
| `desk_idle_s` | `900` | a desk no command used this long is stopped (60 to 86400) |
| `desk_dir` | `~/.local/share/openvibe-node/desks` | each desk's own temp and downloads folders |
| `allow_browser` | `true` | `false`: no browser (no `browser` capability) |
ddiff --git a/internal/plugins/manager.go b/internal/plugins/manager.go
iindex 0e1237e..0326982 100644
---- a/internal/plugins/manager.go
++++ b/internal/plugins/manager.go
@@@ -27,6 +27,11 @@ var capabilityFor = map[string]string{
protocol.KindDesk: "desk",
}
++// deskKinds are the kinds a plugin's desks take too. A desk host (install.sh --desk-host) has no screen, mouse or
++// keyboard of its own, only its desks', so its computer plugin offers no "screen" or "input": these kinds then reach
++// the plugin that offers "desk", which acts on the desk a command names and refuses the machine's own screen itself.
++var deskKinds = map[string]bool{protocol.KindScreen: true, protocol.KindScreenStream: true, protocol.KindInput: true}
++
// Manager supervises every configured plugin.
type Manager struct {
log *slog.Logger
@@@ -141,5 +146,14 @@ func (m *Manager) Route(kind, target, actuator string) *Plugin {
fallback = p
}
}
++ if fallback == nil && deskKinds[kind] {
++ for _, p := range m.plugins {
++ if info := p.Info(); info.Describe != nil {
++ if _, ok := info.Describe.Capabilities["desk"]; ok {
++ return p
++ }
++ }
++ }
++ }
return fallback
}
ddiff --git a/internal/plugins/plugins_test.go b/internal/plugins/plugins_test.go
iindex 15e319e..3aa9f34 100644
---- a/internal/plugins/plugins_test.go
++++ b/internal/plugins/plugins_test.go
@@@ -66,8 +66,11 @@ func fakePlugin(logPath string, in io.Reader, out io.Writer) {
if mode == "mute" {
continue
}
-- w.Encode(map[string]any{"op": "describe", "driver": "fake", "version": "1",
-- "capabilities": map[string]any{"drive": map[string]any{}, "actuator": map[string]any{"names": []string{"lift"}}}})
++ caps := map[string]any{"drive": map[string]any{}, "actuator": map[string]any{"names": []string{"lift"}}}
++ if mode == "deskhost" { // a desk host's computer plugin: desks and files, no screen or input of its own
++ caps = map[string]any{"desk": map[string]any{"screen": true, "input": true}, "files": map[string]any{}}
++ }
++ w.Encode(map[string]any{"op": "describe", "driver": "fake", "version": "1", "capabilities": caps})
if m["probe"] == true {
continue
}
@@@ -265,6 +268,23 @@ func TestRouting(t *testing.T) {
}
}
++// A desk host offers no screen or input of its own, only its desks': screen, screen_stream and input still reach the
++// plugin with desks (it acts on the desk named, or refuses the machine's own screen); other kinds are not routed there.
++func TestRoutingOnADeskHost(t *testing.T) {
++ fx := start(t, "deskhost")
++ fx.waitState(StateReady)
++ for _, kind := range []string{"screen", "screen_stream", "input", "desk", "fs"} {
++ if fx.m.Route(kind, "", "") == nil {
++ t.Fatalf("%s not routed on a desk host", kind)
++ }
++ }
++ for _, kind := range []string{"exec", "browser", "drive"} {
++ if fx.m.Route(kind, "", "") != nil {
++ t.Fatalf("%s routed to a plugin that does not offer it", kind)
++ }
++ }
++}
++
func TestShutdownStopsAndClosesStdin(t *testing.T) {
fx := start(t, "")
fx.waitState(StateReady)
ddiff --git a/internal/protocol/protocol.go b/internal/protocol/protocol.go
iindex 7f46f75..32c690c 100644
---- a/internal/protocol/protocol.go
++++ b/internal/protocol/protocol.go
@@@ -180,7 +180,9 @@ const (
// core publishes them; a latched stop ends it and refuses a start.
KindScreenStream = "screen_stream"
// KindDesk starts, stops, reports or lists desks on the machine (contracts 0.214.0; named desks, Node v0.19.0):
-- // {"action": "start"|"stop"|"status"|"list", "desk"?: "private"|"desk-<name>", …}. Older form, kept:
++ // {"action": "start"|"stop"|"status"|"list", "desk"?: "private"|"desk-<name>", "kind"?: "desktop"|"browser", …}
++ // (kind browser, contracts 0.216.0 and Node v0.23.0: a headless browser alone, its page the desk's screen). Older
++ // form, kept:
// {"action": "start"|"stop"|
// "status", "width"?, "height"?}, an Xvfb display of its own where an agent works without touching the owner's
// screen; screen, screen_stream, input and browser take "desk": "private" to act there. Starting and stopping act.
ddiff --git a/plugins/computer/openvibe_computer/__init__.py b/plugins/computer/openvibe_computer/__init__.py
iindex fa7b7e5..7151b60 100644
---- a/plugins/computer/openvibe_computer/__init__.py
++++ b/plugins/computer/openvibe_computer/__init__.py
@@@ -48,9 +48,14 @@ Command kinds and their answers (the ack's ``data``):
its own display, bus, browser profile (``browser_profile`` + ``-<name>``), temp folder and downloads folder, so two
agents working at once never see or drive each other's screen. ``max_desks`` (default: one per 2 GiB of memory, 1 to
8) caps how many run at once; ``list`` reports the running ones. A desk no command used for ``desk_idle_s`` (15
-- minutes by default) is stopped, so a desk an agent left behind (it crashed or restarted mid-task) ends by itself. Starting and stopping
-- act (refused at observe, approved at ask), ``status`` runs at every level, and the kill switch stops the desk. Linux
-- with Xvfb; config ``allow_desk: false`` turns it off.
++ minutes by default) is stopped, so a desk an agent left behind (it crashed or restarted mid-task) ends by itself.
++ ``"kind": "browser"`` on start (Node v0.23, Desks plan D2) makes a **browser desk**: a headless browser alone, with no
++ display, about half the cost of a whole desk (``kind: desktop``, the default), on Linux or macOS. Its page is its
++ screen: ``screen``, ``input`` (mouse, wheel, keys and text, sent to the page) and ``screen_stream`` (frames of the page)
++ on it go through the browser's pipe, and ``browser`` drives that same page. A browser desk counts as half a desk
++ against ``max_desks``. ``describe`` lists the kinds this machine runs (``desk.kinds``). Starting and stopping act
++ (refused at observe, approved at ask), ``status`` runs at every level, and the kill switch stops the desk. Whole desks
++ need Linux with Xvfb, browser desks a Chromium-family browser; config ``allow_desk: false`` turns both off.
* ``probe`` ``{"target", "timeout_s"?}`` → for ``http(s)://…`` one GET, no redirect followed: ``{status, content_type,
etag, last_modified, body, truncated, ms}``; for ``tcp://host:port``: ``{open, ms, error?}``. Read-only, so it runs at
every level, and only towards the machine's own network: every address the name resolves to must be private,
@@@ -78,7 +83,7 @@ import urllib.parse
from openvibe_plugin import Deferred, Fault, Plugin
from .browser import KEY_CODES, Browser, BrowserError, find_binary
--from .desk import MAX_DESKS_CAP, Desks, PrivateDesk, default_max_desks
++from .desk import MAX_DESKS_CAP, BrowserDesk, Desks, PrivateDesk, default_max_desks
LEVELS = ("observe", "ask", "trusted", "full")
RANK = {name: i for i, name in enumerate(LEVELS)}
@@@ -144,7 +149,7 @@ def _int(v, name, lo, hi, default):
class Computer(Plugin):
driver = "computer"
-- version = "0.10.1"
++ version = "0.11.0"
motion_kinds = frozenset()
telemetry_interval_s = 2.0
@@@ -189,7 +194,8 @@ class Computer(Plugin):
self.browser_factory = browser_factory or (lambda binary, profile, headless, env=None: Browser(binary, profile, headless, env=env))
self.desk_factory = desk_factory or (lambda: PrivateDesk(which=which, popen=popen))
self.desk = desk or self.desk_factory() # the desk named private (the first one, as before v0.9.0)
-- self.desks = Desks(self.desk_factory, private=self.desk, max_desks=default_max_desks())
++ self.desks = Desks(self.desk_factory, private=self.desk, max_desks=default_max_desks(),
++ browser_factory=lambda: BrowserDesk(clock=time.time, base_env=self.env))
self.allow_desk = True
self.desk_browsers = {} # desk name -> the agent's browser on that desk (its own profile, shown there)
self.desk_used = {} # desk name -> when a command last used it (clock), for desk_idle_s
@@@ -248,11 +254,27 @@ class Computer(Plugin):
binary = self._browser_binary()
if binary:
caps["browser"] = {"level": self.level, "browser": os.path.basename(binary), "headless": self.browser_headless}
-- if self.allow_desk and self.platform.startswith("linux") and self.desks.available():
-- caps["desk"] = {"level": self.level, "window_manager": self.desks.window_manager(), "input": bool(self.which("xdotool")),
-- "screen": bool(self._x11_tool()), "named": True, "max_desks": self.desks.max_desks}
++ kinds = self._desk_kinds()
++ if kinds:
++ # A browser desk's screen and input are its page's own (the browser's pipe); a whole desk's need X tools.
++ whole, page = "desktop" in kinds, "browser" in kinds
++ caps["desk"] = {"level": self.level, "window_manager": self.desks.window_manager() if whole else None,
++ "input": page or (whole and bool(self.which("xdotool"))),
++ "screen": page or (whole and bool(self._x11_tool())),
++ "named": True, "max_desks": self.desks.max_desks, "kinds": kinds}
return caps
++ def _desk_kinds(self):
++ """The kinds of desk this machine runs: desktop (Linux with Xvfb) and browser (a Chromium-family browser)."""
++ if not self.allow_desk:
++ return []
++ kinds = []
++ if self.platform.startswith("linux") and self.desks.available():
++ kinds.append("desktop")
++ if self._browser_binary():
++ kinds.append("browser")
++ return kinds
++
def setup(self, config):
self._configure(config)
if hasattr(os, "geteuid") and os.geteuid() == 0:
@@@ -356,6 +378,10 @@ class Computer(Plugin):
desk_name = v.get("desk") if desk_env else None
if desk_name:
self.desk_used[desk_name] = self.clock()
++ page = bool(desk_name) and self.desks.kind(desk_name) == "browser" # a browser desk: its page is its screen
++ if cmd.kind == "screen" and page:
++ width = _int(v.get("max_width"), "max_width", 320, 3840, 1280)
++ return self._start(cmd, lambda: self._page_shot(desk_name, width))
if cmd.kind == "screen":
tool = self._x11_tool() if desk_env else self._screen_tool()
if not tool:
@@@ -363,7 +389,7 @@ class Computer(Plugin):
width = _int(v.get("max_width"), "max_width", 320, 3840, 1280)
return self._start(cmd, lambda: self._screen(tool, width, env=desk_env))
if cmd.kind == "screen_stream":
-- return self._screen_stream(v, env=desk_env, desk=desk_name)
++ return self._screen_stream(v, env=desk_env, desk=desk_name, page=page)
if cmd.kind == "fs":
op = v.get("op")
if op in READ_OPS:
@@@ -380,10 +406,13 @@ class Computer(Plugin):
if cmd.kind == "input":
if not self.allow_input:
raise Fault("unsupported", "this machine does not allow mouse and keyboard input")
-- tool = ("xdotool" if self.which("xdotool") else None) if desk_env else self._input_tool()
++ if page:
++ tool = "page"
++ else:
++ tool = ("xdotool" if self.which("xdotool") else None) if desk_env else self._input_tool()
if not tool:
raise Fault("unsupported", "no input tool on this machine (xdotool on X11, cliclick on macOS)")
-- run, summary = self._input_op(tool, v, env=desk_env)
++ run, summary = self._input_op(tool, v, env=desk_env, desk=desk_name)
return self._act(cmd, level, self._driving(run), summary, threaded=True)
if cmd.kind == "browser":
run, summary, acting = self._browser_op(v, level, env=desk_env, desk=desk_name)
@@@ -419,13 +448,23 @@ class Computer(Plugin):
if action not in ("start", "stop"):
raise Fault("bad_value", "action is start, stop, status or list")
self.desk_used[name] = self.clock()
-- if not self.allow_desk or not self.platform.startswith("linux"):
-- raise Fault("unsupported", "this machine does not offer a private desk (Linux with Xvfb, allow_desk on)")
++ if not self.allow_desk:
++ raise Fault("unsupported", "this machine does not offer desks (allow_desk is off)")
what = "a private desk" if name == "private" else "desk %s" % name
if action == "start":
++ kind = v.get("kind", "desktop")
++ if kind not in ("desktop", "browser"):
++ raise Fault("bad_value", "kind is desktop or browser")
++ kinds = self._desk_kinds()
++ if kind not in kinds:
++ if kind == "browser":
++ raise Fault("unsupported", "a browser desk needs Chromium, Chrome, Edge or Brave on this machine")
++ raise Fault("unsupported", "a whole desk needs Linux with Xvfb%s" % (": this machine runs browser desks (kind browser)" if kinds else ""))
width = _int(v.get("width"), "width", 640, 3840, 1280)
height = _int(v.get("height"), "height", 480, 2160, 800)
-- return self._act(cmd, level, lambda: self.desks.start(name, width, height),
++ if kind == "browser":
++ what = "a browser desk" if name == "private" else "browser desk %s" % name
++ return self._act(cmd, level, lambda: self.desks.start(name, width, height, kind),
"start %s (%d×%d) where your agent works without touching your screen" % (what, width, height),
threaded=True)
@@@ -434,6 +473,49 @@ class Computer(Plugin):
return self.desks.stop(name)
return self._act(cmd, level, stop, "stop %s (and close what runs on it)" % what, threaded=False)
++ def _desk_browser(self, desk, binary=None, env=None):
++ """The agent's browser on a desk, started on first use: shown on a whole desk's display, headless on a browser
++ desk (its window the desk's size: its page is the desk's screen). Its own profile, downloads and temp folder."""
++ with self.browser_mu:
++ b = self.desk_browsers.get(desk)
++ if b is not None and b.alive():
++ return b
++ binary = binary or self._browser_binary()
++ if not binary:
++ raise Fault("unsupported", "no browser on this machine (Chromium, Chrome, Edge or Brave), or this machine does not allow it")
++ env = env or self.desks.env(desk)
++ page = self.desks.kind(desk) == "browser"
++ profile = self.browser_profile + ("-desk" if desk == "private" else "-" + desk)
++ b = self.desk_browsers[desk] = self.browser_factory(binary, profile, page, env=env)
++ b.download_dir = self.desks.dir(desk, "downloads")
++ if page:
++ st = self.desks.status(desk)
++ b.window = b.page_size = (st.get("width") or 1280, st.get("height") or 800)
++ return b
++
++ def _desk_lock(self, desk):
++ with self.browser_mu:
++ return self.browser_locks.setdefault(desk, threading.Lock())
++
++ def _on_page(self, desk, act):
++ """act(browser) on a browser desk's page, one action at a time on that desk; a browser error is a fault."""
++ b = self._desk_browser(desk)
++ try:
++ with self._desk_lock(desk):
++ b.page()
++ return act(b)
++ except BrowserError as e:
++ raise Fault(e.code, e.message)
++
++ def _page_shot(self, desk, width):
++ """A picture of a browser desk's page (not under its action lock: a look never waits for a click)."""
++ b = self._desk_browser(desk)
++ try:
++ b.page()
++ return b.screenshot(width)
++ except BrowserError as e:
++ raise Fault(e.code, e.message)
++
def _close_desk_browser(self, name):
b = self.desk_browsers.pop(name, None)
if b is not None:
@@@ -956,14 +1038,9 @@ class Computer(Plugin):
lock_key = desk if env else "main"
def browser():
++ if env: # on a desk: its own browser (shown on a whole desk, headless on a browser desk)
++ return self._desk_browser(desk, binary, env)
with self.browser_mu:
-- if env: # on a desk: shown there (never headless), in a profile and a downloads folder of its own
-- b = self.desk_browsers.get(desk)
-- if b is None or not b.alive():
-- profile = self.browser_profile + ("-desk" if desk == "private" else "-" + desk)
-- b = self.desk_browsers[desk] = self.browser_factory(binary, profile, False, env=env)
-- b.download_dir = self.desks.dir(desk, "downloads")
-- return b
if self.browser is None or not self.browser.alive():
self.browser = self.browser_factory(binary, self.browser_profile, self.browser_headless)
return self.browser
@@@ -1018,9 +1095,10 @@ class Computer(Plugin):
return "xdotool"
return None
-- def _input_op(self, tool, v, env=None):
++ def _input_op(self, tool, v, env=None, desk=None):
"""(run, summary) for one input action; nothing touches the screen until run() (after approval at ask). With env,
-- on the private desk (its display; points scale to the desk's own size)."""
++ on a desk (its display; points scale to the desk's own size). tool "page": a browser desk's page, through the
++ browser's pipe (points scale to the page's size)."""
action = v.get("action")
if action not in INPUT_ACTIONS:
raise Fault("bad_value", "action is one of %s" % ", ".join(sorted(INPUT_ACTIONS)))
@@@ -1039,7 +1117,7 @@ class Computer(Plugin):
dy = _int(v.get("dy"), "dy", -20, 20, None)
if not dy:
raise Fault("bad_value", "scroll needs dy (clicks, -20 to 20, not 0)")
-- if tool != "xdotool":
++ if tool not in ("xdotool", "page"):
raise Fault("unsupported", "scrolling needs xdotool (Linux X11)")
if action == "type":
text = v.get("text")
@@@ -1051,7 +1129,9 @@ class Computer(Plugin):
raise Fault("bad_value", "keys is one key or chord, for example Return or ctrl+s")
def run():
-- at = (self._desk_scale(point) if env else self._scale(tool, point)) if point else None
++ if tool == "page":
++ return self._on_page(desk, lambda b: self._page_input(b, action, point, v))
++ at = (self._desk_scale(point, desk) if env else self._scale(tool, point)) if point else None
argv = self._input_argv(tool, action, at, v)
proc = self.popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, start_new_session=True,
**({"env": env} if env else {}))
@@@ -1067,7 +1147,7 @@ class Computer(Plugin):
out["x"], out["y"] = at
return out
-- where = " on the private desk" if env else ""
++ where = (" on the private desk" if desk == "private" else " on desk %s" % desk) if env else ""
if action == "type":
text = v["text"]
shown = text if len(text) <= 120 else text[:120] + "…"
@@@ -1081,13 +1161,39 @@ class Computer(Plugin):
summary = "%s at %d, %d%s%s" % (action.replace("_", " "), x, y, " of a %d×%d picture of the screen" % (w, h) if w else "", where)
return run, summary
-- def _desk_scale(self, point):
++ def _desk_scale(self, point, desk="private"):
++ """A point in a picture of the desk, on the desk's own display (the desk the command names, at its size)."""
x, y, w, h = point
-- st = self.desk.status()
-- if not w or not st.get("running"):
++ st = self.desks.status(desk or "private")
++ if not w or not st.get("running") or not st.get("width"):
return x, y
return min(st["width"] - 1, round(x * st["width"] / w)), min(st["height"] - 1, round(y * st["height"] / h))
++ @staticmethod
++ def _page_input(b, action, point, v):
++ """One input action on a browser desk's page. A point in a picture of the page scales to the page's own size;
++ a scroll without a point is at the page's middle."""
++ vw, vh = b.viewport()
++ if point:
++ x, y, w, h = point
++ if w:
++ x, y = round(x * vw / w), round(y * vh / h)
++ x, y = min(max(0, x), vw - 1), min(max(0, y), vh - 1)
++ else:
++ x, y = vw // 2, vh // 2
++ if action in POINT_ACTIONS:
++ out = b.mouse(action, x, y)
++ elif action == "scroll":
++ out = b.wheel(x, y, v["dy"])
++ elif action == "type":
++ out = b.insert(v["text"])
++ else:
++ out = b.keys(v["keys"])
++ out = dict(out, action=action)
++ if point:
++ out["x"], out["y"] = x, y
++ return out
++
def _screen_size(self, tool):
"""The screen's size in the units the input tool uses: pixels for xdotool, points for cliclick (a Retina
screen's picture is twice as many pixels)."""
@@@ -1212,7 +1318,7 @@ class Computer(Plugin):
def _stream_capable(self, tool):
return self._have_pillow() or self._native_jpeg(tool, "x.jpg") is not None
-- def _screen_stream(self, v, env=None, desk=None):
++ def _screen_stream(self, v, env=None, desk=None, page=False):
action = v.get("action")
if action == "stop":
return {"streaming": False, "stopped": self._end_stream("stopped by its owner")}
@@@ -1222,17 +1328,17 @@ class Computer(Plugin):
raise Fault("unsupported", "this machine does not show its screen live (allow_screen_stream is off)")
if self.runtime is not None and self.runtime.estopped:
raise Fault("local_stop", "stopped on the machine")
-- tool = self._x11_tool() if env else self._screen_tool()
++ tool = ("page", None) if page else (self._x11_tool() if env else self._screen_tool())
if not tool:
raise Fault("unsupported", "no screen capture tool on this machine")
-- if not self._stream_capable(tool):
++ if not page and not self._stream_capable(tool):
raise Fault("unsupported", "%s only writes PNG: install Pillow (pip install pillow) to show the screen live" % tool[0])
seconds = _int(v.get("seconds"), "seconds", 30, 1800, 600)
fps = _int(v.get("fps"), "fps", 1, 5, 2)
self._end_stream("replaced")
frames = tempfile.mkdtemp(prefix="openvibe-screen-stream-") # 0700, this account only
stream = {"sid": os.urandom(8).hex(), "until": self.clock() + seconds, "fps": fps, "tool": tool, "dir": frames,
-- "stop": threading.Event(), "env": env}
++ "stop": threading.Event(), "env": env, "desk": desk}
with self.stream_mu:
self.stream = stream
threading.Thread(target=self._stream_loop, args=(stream,), name="computer-screen-stream", daemon=True).start()
@@@ -1261,7 +1367,7 @@ class Computer(Plugin):
break
path = os.path.join(stream["dir"], "frame-%d.jpg" % (n % 3)) # the core reads each one at once
try:
-- self._stream_frame(stream["tool"], path, env=stream.get("env"))
++ self._stream_frame(stream["tool"], path, env=stream.get("env"), desk=stream.get("desk"))
n += 1
failures = 0
self.emit_video(path, stream="screen")
@@@ -1279,8 +1385,16 @@ class Computer(Plugin):
# The core ends the stream with the same id (a newer one that replaced it carries its own).
self.emit_event("screen_stream_ended", sid=stream["sid"], reason=stream.get("reason") or reason, frames=n)
-- def _stream_frame(self, tool, path, env=None):
-- """One JPEG frame of the screen at path, at most STREAM_WIDTH wide when Pillow can resize it."""
++ def _stream_frame(self, tool, path, env=None, desk=None):
++ """One JPEG frame of the screen at path, at most STREAM_WIDTH wide when Pillow can resize it. On a browser desk,
++ a picture of its page from the browser itself (JPEG and scaled there: no capture tool, no Pillow)."""
++ if tool[0] == "page":
++ # Not under the desk's action lock: the live view keeps moving while an action waits for a page to load.
++ try:
++ self._desk_browser(desk).frame(path, STREAM_WIDTH, STREAM_QUALITY)
++ except BrowserError as e:
++ raise Fault(e.code, e.message)
++ return
if self._have_pillow():
fd, png = tempfile.mkstemp(prefix="openvibe-screen-", suffix=".png")
os.close(fd)
ddiff --git a/plugins/computer/openvibe_computer/browser.py b/plugins/computer/openvibe_computer/browser.py
iindex 64b54fb..1172087 100644
---- a/plugins/computer/openvibe_computer/browser.py
++++ b/plugins/computer/openvibe_computer/browser.py
@@@ -9,6 +9,9 @@ that setting at one.
Browser(binary, profile, headless).open(url) / read() / click(selector|text) / fill(selector, text) / press(key)
/ back() / screenshot(max_width) / close()
++On a browser desk (Desks plan D2: a browser alone, no display) the page is also the desk's screen: ``mouse``,
++``wheel``, ``insert`` and ``keys`` are its mouse and keyboard, and ``frame`` writes one picture of it for the live view.
++
Every page answer is ``{url, title, text, links}``: the page's visible text (cut at ``MAX_TEXT``) and its first links.
"""
@@@ -29,6 +32,41 @@ KEY_CODES = {
"ArrowUp": (38, ""), "ArrowDown": (40, ""), "ArrowLeft": (37, ""), "ArrowRight": (39, ""),
"PageUp": (33, ""), "PageDown": (34, ""), "Home": (36, ""), "End": (35, ""), "Space": (32, " "),
}
++# xdotool's key names (what `input` takes) → the browser's keys: (key, code, Windows key code, text).
++PAGE_KEYS = {
++ "return": ("Enter", "Enter", 13, "\r"), "enter": ("Enter", "Enter", 13, "\r"), "kp_enter": ("Enter", "NumpadEnter", 13, "\r"),
++ "tab": ("Tab", "Tab", 9, ""), "escape": ("Escape", "Escape", 27, ""), "esc": ("Escape", "Escape", 27, ""),
++ "backspace": ("Backspace", "Backspace", 8, ""), "delete": ("Delete", "Delete", 46, ""), "space": (" ", "Space", 32, " "),
++ "up": ("ArrowUp", "ArrowUp", 38, ""), "down": ("ArrowDown", "ArrowDown", 40, ""), "left": ("ArrowLeft", "ArrowLeft", 37, ""),
++ "right": ("ArrowRight", "ArrowRight", 39, ""), "home": ("Home", "Home", 36, ""), "end": ("End", "End", 35, ""),
++ "page_up": ("PageUp", "PageUp", 33, ""), "prior": ("PageUp", "PageUp", 33, ""), "page_down": ("PageDown", "PageDown", 34, ""),
++ "next": ("PageDown", "PageDown", 34, ""), "insert": ("Insert", "Insert", 45, ""),
++}
++# Modifiers: the bit CDP's `modifiers` takes, and the key event that holds it down.
++PAGE_MODIFIERS = {
++ "alt": (1, ("Alt", "AltLeft", 18)), "option": (1, ("Alt", "AltLeft", 18)),
++ "ctrl": (2, ("Control", "ControlLeft", 17)), "control": (2, ("Control", "ControlLeft", 17)),
++ "super": (4, ("Meta", "MetaLeft", 91)), "meta": (4, ("Meta", "MetaLeft", 91)), "cmd": (4, ("Meta", "MetaLeft", 91)),
++ "command": (4, ("Meta", "MetaLeft", 91)), "win": (4, ("Meta", "MetaLeft", 91)),
++ "shift": (8, ("Shift", "ShiftLeft", 16)),
++}
++
++
++def page_key(name, shift=False):
++ """(key, code, Windows key code, text) for one xdotool key name on a page, or None."""
++ low = name.lower()
++ if low in PAGE_KEYS:
++ return PAGE_KEYS[low]
++ if len(name) == 1 and name.isalpha():
++ ch = name.upper() if shift else name.lower()
++ return (ch, "Key" + name.upper(), ord(name.upper()), ch)
++ if len(name) == 1 and name.isdigit():
++ return (name, "Digit" + name, ord(name), name)
++ if low.startswith("f") and low[1:].isdigit() and 1 <= int(low[1:]) <= 12:
++ return ("F" + low[1:], "F" + low[1:], 111 + int(low[1:]), "")
++ return None
++
++
CANDIDATES = ("chromium", "chromium-browser", "google-chrome", "google-chrome-stable", "microsoft-edge", "brave-browser")
MAC_APPS = ("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
"/Applications/Chromium.app/Contents/MacOS/Chromium",
@@@ -155,20 +193,28 @@ class Browser:
self.session = None
self.used_at = 0.0
self.download_dir = None # a desk's own downloads folder (set before the first action), else the browser's default
++ self.window = (1280, 900) # the window's size
++ self.page_size = None # on a browser desk, the page's exact size (the desk's: its page is its screen)
++ self.write_mu = threading.Lock() # one message at a time on the pipe: actions and the live view share it
++ self.start_mu = threading.Lock()
# ---- the process and the pipe ------------------------------------------------------------------------------------
def alive(self):
return self.proc is not None and self.proc.poll() is None
def start(self):
-- if self.alive():
-- return
++ # One start at a time: an action and a browser desk's live view may both find the browser closed at once.
++ with self.start_mu:
++ if not self.alive():
++ self._start()
++
++ def _start(self):
os.makedirs(self.profile, mode=0o700, exist_ok=True)
to_chrome_r, to_chrome_w = os.pipe()
from_chrome_r, from_chrome_w = os.pipe()
argv = [self.binary, "--remote-debugging-pipe", "--user-data-dir=%s" % self.profile, "--no-first-run",
"--no-default-browser-check", "--disable-background-networking", "--disable-sync",
-- "--disable-component-update", "--password-store=basic", "--window-size=1280,900", "about:blank"]
++ "--disable-component-update", "--password-store=basic", "--window-size=%d,%d" % self.window, "about:blank"]
if self.headless:
argv.insert(1, "--headless=new")
argv[1:1] = self.extra_args
@@@ -195,6 +241,9 @@ class Browser:
self.session = self.call("Target.attachToTarget", {"targetId": target, "flatten": True}, session=False)["sessionId"]
self.call("Page.enable")
self.call("Runtime.enable")
++ if self.page_size:
++ self.call("Emulation.setDeviceMetricsOverride", {"width": self.page_size[0], "height": self.page_size[1],
++ "deviceScaleFactor": 1, "mobile": False})
if self.download_dir:
self.call("Browser.setDownloadBehavior", {"behavior": "allow", "downloadPath": self.download_dir}, session=False)
@@@ -244,7 +293,8 @@ class Browser:
if session and self.session:
msg["sessionId"] = self.session
try:
-- self.out.write(json.dumps(msg).encode("utf-8") + b"\0")
++ with self.write_mu:
++ self.out.write(json.dumps(msg).encode("utf-8") + b"\0")
except (OSError, ValueError, AttributeError):
raise BrowserError("hardware", "the browser is not running")
ok = slot[0].wait(timeout_s or self.timeout_s)
@@@ -377,6 +427,98 @@ class Browser:
if not self.alive() or not self.session:
raise BrowserError("not_ready", "no page is open: open a URL first")
++ # ---- a browser desk: the page is the screen ---------------------------------------------------------------------
++ def page(self):
++ """Make sure the browser runs (a blank page until one is opened): a browser desk always has a screen."""
++ self.start()
++ self.used_at = time.monotonic()
++
++ def viewport(self):
++ """The page's size in CSS pixels, which mouse events use."""
++ metrics = self.call("Page.getLayoutMetrics")
++ view = metrics.get("cssVisualViewport") or metrics.get("visualViewport") or {}
++ return int(view.get("clientWidth") or self.window[0]), int(view.get("clientHeight") or self.window[1])
++
++ def mouse(self, action, x, y):
++ """move, click, double_click or right_click at (x, y) on the page, with what it changed (a receipt) for a click."""
++ self.page()
++ before = self._before() if action != "move" else None
++ self._mark()
++ self.call("Input.dispatchMouseEvent", {"type": "mouseMoved", "x": x, "y": y})
++ if action == "move":
++ return {"done": True}
++ button = "right" if action == "right_click" else "left"
++ for n in range(1, (2 if action == "double_click" else 1) + 1):
++ self.call("Input.dispatchMouseEvent", {"type": "mousePressed", "x": x, "y": y, "button": button, "clickCount": n})
++ self.call("Input.dispatchMouseEvent", {"type": "mouseReleased", "x": x, "y": y, "button": button, "clickCount": n})
++ self._settle(1.0)
++ return self._receipt_only(before)
++
++ def wheel(self, x, y, dy):
++ """Scroll at (x, y): dy clicks of the wheel (negative is up), about 100 pixels each."""
++ self.page()
++ self.call("Input.dispatchMouseEvent", {"type": "mouseWheel", "x": x, "y": y, "deltaX": 0, "deltaY": dy * 100})
++ time.sleep(0.2)
++ return {"done": True}
++
++ def insert(self, text):
++ """Type text where the page's focus is."""
++ self.page()
++ self.call("Input.insertText", {"text": text})
++ return {"done": True}
++
++ def keys(self, chord):
++ """One key or chord in xdotool's names (Return, ctrl+a, shift+Tab, F5), pressed on the page."""
++ parts = chord.split("+")
++ mods = [PAGE_MODIFIERS.get(p.lower()) for p in parts[:-1]]
++ if None in mods:
++ raise BrowserError("bad_value", "unknown modifier in %s" % chord)
++ bits = sum(m[0] for m in mods)
++ key = page_key(parts[-1], shift=bool(bits & 8))
++ if key is None:
++ raise BrowserError("unsupported", "the key %s is not supported on a browser desk" % parts[-1])
++ self.page()
++ before = self._before()
++ self._mark()
++ held = 0
++ for bit, (k, code, vk) in mods:
++ held |= bit
++ self.call("Input.dispatchKeyEvent", {"type": "rawKeyDown", "key": k, "code": code, "windowsVirtualKeyCode": vk, "modifiers": held})
++ k, code, vk, text = key
++ typed = text if text and not bits & 7 else "" # with ctrl, alt or meta held a key types nothing
++ down = {"type": "keyDown" if typed else "rawKeyDown", "key": k, "code": code, "windowsVirtualKeyCode": vk, "modifiers": bits}
++ if typed:
++ down["text"] = down["unmodifiedText"] = typed
++ self.call("Input.dispatchKeyEvent", down)
++ self.call("Input.dispatchKeyEvent", {"type": "keyUp", "key": k, "code": code, "windowsVirtualKeyCode": vk, "modifiers": bits})
++ for bit, (mk, mcode, mvk) in reversed(mods):
++ held &= ~bit
++ self.call("Input.dispatchKeyEvent", {"type": "keyUp", "key": mk, "code": mcode, "windowsVirtualKeyCode": mvk, "modifiers": held})
++ self._settle(1.0)
++ return self._receipt_only(before)
++
++ def _receipt_only(self, before):
++ out = {"done": True}
++ try:
++ r = receipt(before, self._eval(BEFORE_JS))
++ except BrowserError:
++ r = None
++ if r is not None:
++ out["receipt"] = r
++ return out
++
++ def frame(self, path, max_width, quality):
++ """One JPEG picture of the page at path (written whole, then moved into place), at most max_width wide."""
++ self.page()
++ width, height = self.viewport()
++ params = {"format": "jpeg", "quality": quality}
++ if width > max_width:
++ params["clip"] = {"x": 0, "y": 0, "width": width, "height": height, "scale": max_width / float(width)}
++ data = base64.b64decode(self.call("Page.captureScreenshot", params)["data"])
++ with open(path + ".part", "wb") as f:
++ f.write(data)
++ os.replace(path + ".part", path)
++
def close(self):
proc, self.proc, self.session = self.proc, None, None
for f in (self.out, self.inp):
ddiff --git a/plugins/computer/openvibe_computer/desk.py b/plugins/computer/openvibe_computer/desk.py
iindex 6a8abb7..fa3aedb 100644
---- a/plugins/computer/openvibe_computer/desk.py
++++ b/plugins/computer/openvibe_computer/desk.py
@@@ -29,6 +29,8 @@ START_TIMEOUT_S = 8.0
class PrivateDesk:
++ kind = "desktop"
++
def __init__(self, which=shutil.which, popen=subprocess.Popen, clock=time.time, sleep=time.sleep,
exists=os.path.exists, base_env=None, displays=DISPLAYS):
self.which = which
@@@ -179,6 +181,72 @@ class PrivateDesk:
pass
++# What a browser desk leaves out of the machine's environment: the owner's display, session bus and X cookie, so
++# nothing started for it can reach the owner's screen or their desktop apps.
++OWNER_SESSION = ("DISPLAY", "WAYLAND_DISPLAY", "DBUS_SESSION_BUS_ADDRESS", "XAUTHORITY")
++
++
++class BrowserDesk:
++ """A browser desk (Desks plan D2): a browser alone, with no display, on Linux or macOS.
++
++ Nothing runs here. The plugin starts the desk's headless browser (its own profile, downloads and temp folder) on
++ the desk's first command and closes it when the desk stops. Its screen, mouse and keyboard and its live view are
++ the page's own, over the browser's debugging pipe. The desk is its size, its start time and an environment
++ without the owner's session (OWNER_SESSION). It costs about half of a whole desk: no display, no window manager,
++ nothing drawn on a screen.
++ """
++ kind = "browser"
++
++ def __init__(self, clock=time.time, base_env=None):
++ self.clock = clock
++ self.base_env = dict(os.environ if base_env is None else base_env)
++ self.mu = threading.Lock()
++ self.on = False
++ self.width = self.height = self.since = None
++
++ def available(self):
++ return True # the plugin checks for a browser before it starts one
++
++ def window_manager(self):
++ return None
++
++ def running(self):
++ return self.on
++
++ def status(self):
++ with self.mu:
++ if not self.on:
++ return {"running": False, "available": True}
++ return {"running": True, "width": self.width, "height": self.height, "since": int(self.since)}
++
++ def start(self, width=1280, height=800):
++ with self.mu:
++ if not self.on:
++ self.on, self.width, self.height, self.since = True, width, height, self.clock()
++ return self.status()
++
++ def stop(self):
++ with self.mu:
++ was, self.on = self.on, False
++ self.width = self.height = self.since = None
++ return {"running": False, "stopped": was}
++
++ def env(self):
++ with self.mu:
++ if not self.on:
++ raise Fault("not_ready", "start the desk first (desk start)")
++ return {k: v for k, v in self.base_env.items() if k not in OWNER_SESSION}
++
++
++# A browser desk counts as half a desk against max_desks (a whole desk counts 2 of 2 * max_desks).
++DESK_COST = {"desktop": 2, "browser": 1}
++
++
++def kind_of(d):
++ """A desk's kind; a desk object without one is a whole desk (desktop)."""
++ return getattr(d, "kind", "desktop")
++
++
# A desk's name: `private` (the first desk, as before) or `desk-<name>` (Desks plan D1: several desks on one machine,
# one per agent task). Lower case, digits and dashes, so a name is also a safe path part.
DESK_NAME = re.compile(r"^(private|desk-[a-z0-9][a-z0-9-]{0,23})$")
@@@ -199,15 +267,18 @@ def default_max_desks(meminfo="/proc/meminfo"):
class Desks:
-- """The machine's desks by name, each a PrivateDesk with a display, a bus and (in the plugin) a browser of its own.
++ """The machine's desks by name: each a PrivateDesk (kind desktop: a display, a bus and, in the plugin, a browser
++ shown there) or a BrowserDesk (kind browser: a headless browser alone).
Two desks never share a display, a bus, a browser profile or a temp folder, so two agents working at once cannot
-- see or drive each other's screen. At most `max_desks` run at once; starting one more is refused, never queued.
++ see or drive each other's screen. At most `max_desks` whole desks run at once (a browser desk counts as half);
++ starting one more is refused, never queued.
Starts are serialized, so two desks starting together cannot pick the same display number.
"""
-- def __init__(self, factory, private=None, max_desks=2, base_dir=None):
++ def __init__(self, factory, private=None, max_desks=2, base_dir=None, browser_factory=None):
self.factory = factory
++ self.browser_factory = browser_factory or BrowserDesk
self.max_desks = max_desks
self.base_dir = base_dir
self.mu = threading.Lock()
@@@ -222,11 +293,11 @@ class Desks:
raise Fault("bad_value", "desk is main, private or desk-<name> (lower-case letters, digits and dashes, at most 29 characters)")
return name
-- def _desk(self, name, create=False):
-- d = self.desks.get(name)
-- if d is None and create:
-- d = self.desks[name] = self.factory()
-- return d
++ def kind(self, name):
++ """The kind of desk that name is now (desktop or browser), or None when there is no such desk."""
++ with self.mu:
++ d = self.desks.get(name)
++ return kind_of(d) if d is not None else None
def available(self):
d = self.desks.get("private") or self.factory()
@@@ -240,17 +311,25 @@ class Desks:
with self.mu:
return [n for n, d in self.desks.items() if d.running()]
-- def start(self, name, width=1280, height=800):
++ def start(self, name, width=1280, height=800, kind="desktop"):
self.check(name)
++ if kind not in DESK_COST:
++ raise Fault("bad_value", "kind is desktop or browser")
++ if name == "private" and kind != "desktop":
++ raise Fault("bad_value", "private is a whole desk: name a browser desk desk-<name>")
with self.mu:
-- d = self._desk(name, create=True)
++ d = self.desks.get(name)
++ if d is not None and d.running() and kind_of(d) != kind:
++ raise Fault("not_ready", "desk %s already runs as a %s desk: stop it first" % (name, kind_of(d)))
++ if d is None or kind_of(d) != kind:
++ d = self.desks[name] = self.factory() if kind == "desktop" else self.browser_factory()
if not d.running():
-- busy = [n for n, other in self.desks.items() if n != name and other.running()]
-- if len(busy) >= self.max_desks:
-- raise Fault("not_ready", "this machine already runs %d desks (%s), its limit (max_desks): stop one first"
-- % (len(busy), ", ".join(sorted(busy))))
++ busy = sorted((n, kind_of(other)) for n, other in self.desks.items() if n != name and other.running())
++ if sum(DESK_COST[k] for _, k in busy) + DESK_COST[kind] > 2 * self.max_desks:
++ raise Fault("not_ready", "this machine already runs %s, its limit (max_desks %d; a browser desk counts "
++ "as half): stop one first" % (", ".join("%s (%s)" % b for b in busy), self.max_desks))
st = d.start(width, height)
-- return dict(st, desk=name)
++ return dict(st, desk=name, kind=kind_of(d))
def stop(self, name):
self.check(name)
@@@ -276,12 +355,12 @@ class Desks:
d = self.desks.get(name)
if d is None:
return {"running": False, "available": self.available(), "desk": name}
-- return dict(d.status(), desk=name)
++ return dict(d.status(), desk=name, kind=kind_of(d))
def list(self):
with self.mu:
desks = sorted(self.desks.items())
-- running = [dict(d.status(), desk=n) for n, d in desks if d.running()]
++ running = [dict(d.status(), desk=n, kind=kind_of(d)) for n, d in desks if d.running()]
return {"desks": running, "max_desks": self.max_desks, "available": self.available()}
def env(self, name):
ddiff --git a/plugins/computer/pyproject.toml b/plugins/computer/pyproject.toml
iindex e30236c..2f36193 100644
---- a/plugins/computer/pyproject.toml
++++ b/plugins/computer/pyproject.toml
@@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "openvibe-computer"
--version = "0.10.1"
++version = "0.11.0"
description = "OpenVibe Node computer plugin: the machine's terminal, files and screen under the owner's control level"
requires-python = ">=3.9"
license = { text = "AGPL-3.0-or-later" }
ddiff --git a/plugins/computer/tests/test_browser.py b/plugins/computer/tests/test_browser.py
iindex 7f8a18a..0f73468 100644
---- a/plugins/computer/tests/test_browser.py
++++ b/plugins/computer/tests/test_browser.py
@@@ -1,4 +1,5 @@
import http.server
++import io
import os
import shutil
import socketserver
@@@ -12,7 +13,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(HERE)), "sdk"))
sys.path.insert(0, HERE)
from openvibe_computer import Computer # noqa: E402
--from openvibe_computer.browser import Browser, find_binary # noqa: E402
++from openvibe_computer.browser import Browser, BrowserError, find_binary # noqa: E402
from openvibe_plugin import Runtime # noqa: E402
from test_computer import Out, reply # noqa: E402
@@@ -172,6 +173,40 @@ class ReceiptTest(unittest.TestCase):
self.assertEqual(len(long["appeared"][0]), 160)
++class PageKeysTest(unittest.TestCase):
++ """A browser desk's keys (Desks plan D2): xdotool's names, pressed on the page as the browser's own key events."""
++
++ def browser(self):
++ b = Browser("chromium", "/nonexistent")
++ self.sent = []
++ b.call = lambda method, params=None, **kw: self.sent.append(params) or {}
++ b.start = lambda: None
++ b._eval = lambda expression: {"url": "https://a/", "text": ""}
++ b._settle = lambda *a: None
++ return b
++
++ def pressed(self):
++ return [(p["type"], p["key"], p["modifiers"], p.get("text")) for p in self.sent]
++
++ def test_keys_and_chords(self):
++ b = self.browser()
++ b.keys("ctrl+a")
++ self.assertEqual(self.pressed(), [("rawKeyDown", "Control", 2, None), ("rawKeyDown", "a", 2, None),
++ ("keyUp", "a", 2, None), ("keyUp", "Control", 0, None)], "a chord types nothing")
++ self.sent.clear()
++ b.keys("Return")
++ self.assertEqual(self.pressed(), [("keyDown", "Enter", 0, "\r"), ("keyUp", "Enter", 0, None)])
++ self.sent.clear()
++ b.keys("shift+a")
++ self.assertEqual(self.pressed()[1], ("keyDown", "A", 8, "A"))
++ self.sent.clear()
++ b.keys("Page_Down")
++ self.assertEqual(self.pressed()[0][:2], ("rawKeyDown", "PageDown"))
++ for bad in ("hyper+a", "Menu"):
++ with self.assertRaises(BrowserError):
++ b.keys(bad)
++
++
class Handler(http.server.SimpleHTTPRequestHandler):
PAGES = {
"/": b"<!doctype html><title>Home</title><h1>Hello</h1><a href='/two'>Second page</a>"
@@@ -235,6 +270,59 @@ class RealBrowserTest(unittest.TestCase):
b.close()
self.assertFalse(b.alive())
++ def test_a_browser_desk_its_page_is_the_screen(self):
++ # Desks plan D2 in a real browser: no display; the page is the desk's screen, mouse and keyboard, and live view.
++ srv = socketserver.TCPServer(("127.0.0.1", 0), Handler)
++ threading.Thread(target=srv.serve_forever, daemon=True).start()
++ self.addCleanup(srv.shutdown)
++ base = "http://127.0.0.1:%d" % srv.server_address[1]
++ tmp = __import__("tempfile").mkdtemp(prefix="ov-bdesk-")
++ self.addCleanup(lambda: shutil.rmtree(tmp, ignore_errors=True))
++ extra = ["--no-sandbox"] if os.environ.get("CI") else []
++ out = Out()
++ plugin = Computer(platform=sys.platform)
++ plugin.browser_factory = lambda binary, profile, headless, env=None: Browser(binary, profile, headless, env=env, extra_args=extra)
++ rt = Runtime(plugin, infile=io.StringIO(""), outfile=out)
++ # Level full: the test page is on 127.0.0.1, which only full may open.
++ rt.dispatch({"op": "hello", "config": {"level": "full", "roots": [tmp], "notify": False, "desk_dir": os.path.join(tmp, "desks"),
++ "browser_profile": os.path.join(tmp, "profile")}})
++ self.addCleanup(plugin.stop)
++ n = [0]
++
++ def cmd(kind, value):
++ n[0] += 1
++ rt.dispatch({"op": "command", "id": "b%d" % n[0], "kind": kind, "value": dict(value, desk="desk-t"), "deadline_ms": 60000})
++ cid = "b%d" % n[0]
++ r = out.wait_for(lambda m: m.get("id") == cid and m.get("op") in ("ack", "nack"), timeout=60) # a first start is slow
++ self.assertEqual(r["op"], "ack", r)
++ return r["data"]
++ self.assertEqual(cmd("desk", {"action": "start", "kind": "browser", "width": 1000, "height": 600})["kind"], "browser")
++ cmd("browser", {"action": "open", "url": base + "/app"})
++ shot = cmd("screen", {"max_width": 500})
++ self.assertEqual((shot["format"], shot["width"]), ("jpeg", 500), "a picture of the 1000-wide page, at most 500 wide")
++ b = plugin.desk_browsers["desk-t"]
++ self.assertEqual(b.viewport(), (1000, 600), "the page is the desk's size")
++ # Click the button where it is in the 500-wide picture: the page's own click, with what it changed.
++ r = b._eval("(() => { const r = document.getElementById('go').getBoundingClientRect(); return [r.x + r.width / 2, r.y + r.height / 2]; })()")
++ x, y = round(r[0] * shot["width"] / 1000), round(r[1] * shot["height"] / 600)
++ clicked = cmd("input", {"action": "click", "x": x, "y": y, "width": shot["width"], "height": shot["height"]})
++ self.assertEqual(clicked["receipt"]["appeared"], ["Row added"], clicked)
++ # Type into the field it focused, select all, delete.
++ b._eval("document.querySelector('input[type=email]').focus()")
++ cmd("input", {"action": "type", "text": "[email protected]"})
++ self.assertEqual(b._eval("document.querySelector('input[type=email]').value"), "[email protected]")
++ cmd("input", {"action": "key", "keys": "ctrl+a"})
++ cmd("input", {"action": "key", "keys": "BackSpace"})
++ self.assertEqual(b._eval("document.querySelector('input[type=email]').value"), "")
++ cmd("input", {"action": "scroll", "dy": 2})
++ # A frame of the live view: a JPEG from the browser itself.
++ frame = os.path.join(tmp, "frame.jpg")
++ plugin._stream_frame(("page", None), frame, desk="desk-t")
++ with open(frame, "rb") as f:
++ self.assertEqual(f.read(2), b"\xff\xd8")
++ cmd("desk", {"action": "stop"})
++ self.assertFalse(b.alive())
++
if __name__ == "__main__":
unittest.main()
ddiff --git a/plugins/computer/tests/test_desk.py b/plugins/computer/tests/test_desk.py
iindex 29f6e2e..3e3ebe6 100644
---- a/plugins/computer/tests/test_desk.py
++++ b/plugins/computer/tests/test_desk.py
@@@ -120,6 +120,58 @@ class FakeDesk:
return {"DISPLAY": ":%d" % self.display}
++class FakePageBrowser:
++ """A browser desk's browser, recorded: its page is the desk's screen, mouse, keyboard and live view."""
++
++ def __init__(self, binary, profile, headless, env=None):
++ self.profile, self.headless, self.env, self.download_dir = profile, headless, env, None
++ self.window = (1280, 900)
++ self.calls = []
++ self.on = True
++
++ def alive(self):
++ return self.on
++
++ def page(self):
++ self.calls.append(("page",))
++
++ def viewport(self):
++ return self.window
++
++ def open(self, url):
++ self.calls.append(("open", url))
++ return {"url": url, "title": "Example", "text": "", "links": []}
++
++ def mouse(self, action, x, y):
++ self.calls.append(("mouse", action, x, y))
++ return {"done": True, "receipt": {"url_changed": False, "appeared": ["Row added"], "gone": 0}}
++
++ def wheel(self, x, y, dy):
++ self.calls.append(("wheel", x, y, dy))
++ return {"done": True}
++
++ def insert(self, text):
++ self.calls.append(("insert", text))
++ return {"done": True}
++
++ def keys(self, chord):
++ self.calls.append(("keys", chord))
++ return {"done": True}
++
++ def screenshot(self, max_width):
++ self.calls.append(("screenshot", max_width))
++ return {"format": "jpeg", "width": 1000, "height": 500, "content_b64": "/9j/"}
++
++ def frame(self, path, max_width, quality):
++ self.calls.append(("frame", max_width, quality))
++ with open(path, "wb") as f:
++ f.write(b"\xff\xd8\xff")
++
++ def close(self):
++ self.on = False
++ self.calls.append(("close",))
++
++
class DeskPluginTest(unittest.TestCase):
def boot(self, level, **config):
self.out = Out()
@@@ -157,11 +209,11 @@ class DeskPluginTest(unittest.TestCase):
def test_describe_offers_the_desk(self):
caps = self.boot("trusted")[0]["capabilities"]
self.assertEqual(caps["desk"], {"level": "trusted", "window_manager": "metacity", "input": True, "screen": True,
-- "named": True, "max_desks": default_max_desks()})
++ "named": True, "max_desks": default_max_desks(), "kinds": ["desktop", "browser"]})
def test_observe_reports_but_does_not_start(self):
self.boot("observe")
-- self.assertEqual(reply(self.out, self.cmd("desk", {"action": "status"}))["data"], {"running": False, "available": True, "desk": "private"})
++ self.assertEqual(reply(self.out, self.cmd("desk", {"action": "status"}))["data"], {"running": False, "available": True, "desk": "private", "kind": "desktop"})
r = reply(self.out, self.cmd("desk", {"action": "start"}))
self.assertEqual((r["op"], r["fault_code"]), ("nack", "not_allowed"))
self.assertFalse(self.desk.on)
@@@ -269,6 +321,93 @@ class DeskPluginTest(unittest.TestCase):
self.assertNotEqual(made[0].download_dir, made[1].download_dir)
self.assertNotEqual(made[0].env["DISPLAY"], made[1].env["DISPLAY"])
++ def test_browser_desks_their_page_is_the_screen(self):
++ # Desks plan D2: a browser alone, with no display. Its page is its screen, mouse, keyboard and live view.
++ self.boot("trusted", max_desks=1)
++ made = []
++
++ def factory(binary, profile, headless, env=None):
++ b = FakePageBrowser(binary, profile, headless, env)
++ made.append(b)
++ return b
++ self.plugin.browser_factory = factory
++ before = len(self.calls)
++ r = reply(self.out, self.cmd("desk", {"action": "start", "desk": "desk-a", "kind": "browser", "width": 1000, "height": 500}))
++ self.assertEqual((r["op"], r["data"]["kind"], r["data"]["running"], r["data"]["width"]), ("ack", "browser", True, 1000), r)
++ r = reply(self.out, self.cmd("desk", {"action": "start", "desk": "desk-b", "kind": "browser"}))
++ self.assertEqual(r["op"], "ack", r)
++ self.assertEqual((self.made_desks, len(self.calls)), ([], before), "no display, no process started")
++ # Two browser desks fill max_desks 1 (each counts as half): a whole desk more is refused.
++ r = reply(self.out, self.cmd("desk", {"action": "start", "desk": "desk-c"}))
++ self.assertEqual(r["fault_code"], "not_ready")
++ self.assertIn("counts as half", r["message"])
++ listed = reply(self.out, self.cmd("desk", {"action": "list"}))["data"]["desks"]
++ self.assertEqual([(d["desk"], d["kind"]) for d in listed], [("desk-a", "browser"), ("desk-b", "browser")])
++ # Its browser: headless, the desk's size, a profile of its own, and never the owner's display.
++ r = reply(self.out, self.cmd("browser", {"action": "open", "url": "https://example.org/", "desk": "desk-a"}))
++ self.assertEqual(r["op"], "ack", r)
++ a = made[0]
++ self.assertEqual((a.headless, a.window), (True, (1000, 500)))
++ self.assertTrue(a.profile.endswith("-desk-a"))
++ self.assertNotIn("DISPLAY", a.env)
++ self.assertTrue(os.path.basename(a.env["TMPDIR"]).startswith("ov-desk-a-"))
++ # The mouse and keyboard are the page's: a point in a 100×100 picture lands on the 1000×500 page.
++ r = reply(self.out, self.cmd("input", {"action": "click", "x": 50, "y": 50, "width": 100, "height": 100, "desk": "desk-a"}))
++ self.assertEqual((r["op"], r["data"]["x"], r["data"]["y"], r["data"]["receipt"]["appeared"]), ("ack", 500, 250, ["Row added"]), r)
++ self.assertIn(("mouse", "click", 500, 250), a.calls)
++ reply(self.out, self.cmd("input", {"action": "scroll", "dy": 3, "desk": "desk-a"}))
++ reply(self.out, self.cmd("input", {"action": "type", "text": "hello", "desk": "desk-a"}))
++ reply(self.out, self.cmd("input", {"action": "key", "keys": "ctrl+a", "desk": "desk-a"}))
++ self.assertEqual([c for c in a.calls if c[0] in ("wheel", "insert", "keys")],
++ [("wheel", 500, 250, 3), ("insert", "hello"), ("keys", "ctrl+a")], "a scroll with no point is at the middle")
++ self.assertEqual(len(self.calls), before, "never xdotool")
++ # A look and the live view are pictures of the page.
++ shot = reply(self.out, self.cmd("screen", {"desk": "desk-a", "max_width": 800}))
++ self.assertEqual(shot["data"]["format"], "jpeg")
++ self.assertIn(("screenshot", 800), a.calls)
++ live = reply(self.out, self.cmd("screen_stream", {"action": "start", "seconds": 30, "desk": "desk-a"}))
++ self.assertEqual((live["data"]["streaming"], live["data"]["desk"], live["data"]["tool"]), (True, "desk-a", "page"), live)
++ reply(self.out, self.cmd("screen_stream", {"action": "stop"}))
++ frame = os.path.join(self.desk_dir, "frame.jpg")
++ self.plugin._stream_frame(("page", None), frame, desk="desk-a")
++ with open(frame, "rb") as f:
++ self.assertEqual(f.read(2), b"\xff\xd8")
++ # desk-b has a browser of its own.
++ reply(self.out, self.cmd("screen", {"desk": "desk-b"}))
++ self.assertEqual(len(made), 2)
++ self.assertTrue(made[1].profile.endswith("-desk-b"))
++ # Stopping a browser desk closes its browser. The private desk is always a whole desk; kinds are checked.
++ reply(self.out, self.cmd("desk", {"action": "stop", "desk": "desk-a"}))
++ self.assertFalse(a.alive())
++ self.assertEqual(reply(self.out, self.cmd("desk", {"action": "start", "kind": "browser"}))["fault_code"], "bad_value")
++ self.assertEqual(reply(self.out, self.cmd("desk", {"action": "start", "desk": "desk-z", "kind": "vm"}))["fault_code"], "bad_value")
++ self.plugin.stop()
++ self.assertFalse(made[1].alive())
++
++ def test_a_named_desk_scales_points_to_its_own_size(self):
++ # A click on desk-a scales to desk-a's size, not to the private desk's (which may not even run).
++ self.boot("trusted", max_desks=2)
++ reply(self.out, self.cmd("desk", {"action": "start", "desk": "desk-a", "width": 1000, "height": 500}))
++ reply(self.out, self.cmd("input", {"action": "click", "x": 50, "y": 50, "width": 100, "height": 100, "desk": "desk-a"}))
++ self.assertEqual(self.calls[-1][0][3:5], ["500", "250"])
++
++ def test_a_mac_runs_browser_desks(self):
++ out = Out()
++ plugin = Computer(env={}, platform="darwin", which=lambda n: "/usr/bin/chromium" if n == "chromium" else None,
++ screen_size=lambda tool: (100, 100))
++ rt = Runtime(plugin, infile=io.StringIO(""), outfile=out)
++ rt.dispatch({"op": "hello", "config": {"level": "trusted", "roots": [tempfile.gettempdir()], "notify": False,
++ "desk_dir": tempfile.mkdtemp(prefix="ov-desks-")}})
++ desk = out.msgs()[0]["capabilities"]["desk"]
++ self.assertEqual((desk["kinds"], desk["window_manager"], desk["screen"], desk["input"]), (["browser"], None, True, True))
++ rt.dispatch({"op": "command", "id": "m1", "kind": "desk", "value": {"action": "start", "desk": "desk-a"}, "deadline_ms": 30000})
++ r = reply(out, "m1")
++ self.assertEqual(r["fault_code"], "unsupported")
++ self.assertIn("browser desks", r["message"])
++ rt.dispatch({"op": "command", "id": "m2", "kind": "desk", "value": {"action": "start", "desk": "desk-a", "kind": "browser"}, "deadline_ms": 30000})
++ self.assertEqual(reply(out, "m2")["data"]["kind"], "browser")
++ plugin.stop()
++
def test_the_kill_switch_stops_the_desk(self):
self.boot("trusted")
reply(self.out, self.cmd("desk", {"action": "start"}))
ddiff --git a/plugins/sdk/openvibe_plugin/__init__.py b/plugins/sdk/openvibe_plugin/__init__.py
iindex d754e8e..4240caf 100644
---- a/plugins/sdk/openvibe_plugin/__init__.py
++++ b/plugins/sdk/openvibe_plugin/__init__.py
@@@ -24,7 +24,7 @@ __all__ = ["Plugin", "Fault", "Deferred", "Command", "StopInfo", "Runtime", "run
"RUNTIME_CAUSES", "SHARED_CLOCK", "__version__"]
# The Node release this runtime ships with (pyproject.toml carries the same).
--__version__ = "0.22.0"
++__version__ = "0.23.0"
PROTOCOL_VERSION = 1
HEARTBEAT_TIMEOUT_S = 1.0
DEFAULT_DEADLINE_MS = 300
ddiff --git a/plugins/sdk/pyproject.toml b/plugins/sdk/pyproject.toml
iindex 6709857..b73b656 100644
---- a/plugins/sdk/pyproject.toml
++++ b/plugins/sdk/pyproject.toml
@@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "openvibe-plugin"
--version = "0.22.0"
++version = "0.23.0"
description = "Runtime for OpenVibe Node plugins (JSON lines over stdin/stdout, deadlines, heartbeat, e-stop)"
requires-python = ">=3.9"
license = { text = "AGPL-3.0-or-later" }
ddiff --git a/plugins/sdk/tests/test_runtime.py b/plugins/sdk/tests/test_runtime.py
iindex 9c3173c..946bb4e 100644
---- a/plugins/sdk/tests/test_runtime.py
++++ b/plugins/sdk/tests/test_runtime.py
@@@ -439,7 +439,7 @@ class RuntimeTest(unittest.TestCase):
self.assertEqual(rt.stop_infos[-1].halt_epoch, 1792900000500)
def test_the_package_reports_the_node_release(self):
-- self.assertEqual(openvibe_plugin.__version__, "0.22.0")
++ self.assertEqual(openvibe_plugin.__version__, "0.23.0")
with open(os.path.join(os.path.dirname(HERE), "pyproject.toml")) as f:
self.assertIn('version = "%s"' % openvibe_plugin.__version__, f.read())