openvibers/openvibe.link Commits
Node: browser — the computer plugin drives a browser of its own over a pipe, at the level the owner set (#46)
Changes
@@@ -41,8 +41,8 @@ signature against the key Bot publishes (`--key-id` pins one) and names the plugon a community profile runs the same check and warns when it fails. Install a community plugin only when it passes([docs/plugins.md](docs/plugins.md#community-plugins)).--**Your computer for your agents:** `--computer observe|ask|trusted|full` adds the `computer` plugin (terminal, files--and screen for OpenVibe.Actor), running as your own account, at the level you set on the machine; `ask` waits for++**Your computer for your agents:** `--computer observe|ask|trusted|full` adds the `computer` plugin (terminal, files,++screen, mouse and keyboard, and a browser of its own for OpenVibe.Actor), running as your own account, at the level you set on the machine; `ask` waits for`openvibe-node approve <id>` ([docs/computer.md](docs/computer.md)).**A machine that runs your jobs:** add it on openvibe.bot/machines, which prints the same installer with `--worker`: the@@@ -48,7 +48,8 @@ When the Node runs in your graphical session, a waiting request also shows a desthat a request is waiting on the machine.**Kill switch:** `openvibe-node stop` (and an owner's e-stop) drops the plugin to `observe`: what runs is killed, what--waits is denied, and the core also refuses `exec`, file writes and `input` until `openvibe-node resume`. `halt` kills what runs.++waits is denied, the agent's browser is closed, and the core also refuses `exec`, file writes, `input` and the browser's++acting actions until `openvibe-node resume`. `halt` kills what runs.## What it does@@@ -59,6 +60,7 @@ waits is denied, and the core also refuses `exec`, file writes and `input` until| `probe` | `target`: `http(s)://…` (one GET, no redirect followed) or `tcp://host:port`; `timeout_s` | `status`, `content_type`, `etag`, `last_modified`, `body` (≤ 64 KB), `ms`; or `open`, `ms` || `screen` | `max_width` (default 1280) | `format` (`jpeg` with Pillow, else `png` ≤ 300 KB), `width`, `height`, `content_b64` || `input` | `action`: `move`, `click`, `double_click`, `right_click` (`x`, `y`), `scroll` (`dy` clicks, ±20), `type` (`text`, ≤ 2000), `key` (`keys`: one key or chord, `ctrl+s`, `Return`); `width`, `height`: the picture `x`, `y` are measured in | `done`, `action`, and the real `x`, `y` it used |++| `browser` | `action`: `open` (`url`, http or https), `click` (`selector`, or the `text` a button or link shows), `fill` (`selector`, `text` ≤ 2000), `press` (`key`: Enter, Tab, Escape, Backspace, Delete, arrows, PageUp, PageDown, Home, End, Space), `back`, `read`, `screenshot` (`max_width`), `close` | a page: `url`, `title`, `text` (its visible text, ≤ 20,000 characters), `truncated`, `links` (the first 60); `screenshot`: `format`, `width`, `height`, `content_b64`; `fill`: `done`, `chars` |A program runs in its own process group with the Node account's rights, at most 2 at a time, and is killed at`timeout_s` (the owner's `timeout_s` is the cap). A write with `overwrite` replaces the file atomically. Screens come@@@ -73,6 +75,14 @@ saw (`width` × `height`), scaled to your real screen. It uses `xdotool` on an Xtext and keys are passed to the tool as arguments, never through a shell, and `keys` takes only letters, digits, `+`,`-` and `_`.++**The browser** (`browser`) is a Chromium-family browser (Chromium, Chrome, Edge or Brave) the agent drives:++- **Its own profile.** It runs in a folder of its own (`browser_profile`, by default `~/.local/share/openvibe-node/browser`), so the agent never sees the sessions of your own browser unless you point it at one. Sign it in to a site yourself (`browser_headless: false` shows its window) if you want the agent to work there as you.++- **No port.** The Node starts it on the first command with `--remote-debugging-pipe`: it speaks the DevTools protocol over two file descriptors, so nothing else on the machine can drive it.++- **Levels.** `read`, `screenshot` and `close` run at every level. `open`, `click`, `fill`, `press` and `back` are acting commands: refused at `observe`, approved one by one at `ask`. The approval says exactly what would happen: "open https://… in the browser", "type 5 characters into #q in the browser: 'hello'".++- **Your own network.** Below level `full`, `open` refuses an address on the machine's own network (private, loopback or link-local, checked for every address the name resolves to).++- **Closing.** It closes after `browser_idle_s` without a command and at the kill switch.++- **Platforms.** Linux and macOS; not offered on Windows yet.++## Config| key | default | meaning |@@@ -87,8 +97,13 @@ text and keys are passed to the tool as arguments, never through a shell, and `k| `notify` | `true` | a desktop notification when a request waits (graphical session only) || `probe` | `true` | `false`: no probes (OpenVibe.Watch's `node` source) || `allow_input` | `true` | `false`: never use the mouse or keyboard (no `input` capability) |++| `allow_browser` | `true` | `false`: no browser (no `browser` capability) |++| `browser_path` | found on PATH | the browser to drive (Chromium, Chrome, Edge or Brave) |++| `browser_profile` | `~/.local/share/openvibe-node/browser` | its profile folder: the agent's own, unless you choose another |++| `browser_headless` | `true` | `false`: show its window |++| `browser_idle_s` | 600 | close it after this long without a command |--`describe` reports `terminal`, `files`, `screen` and `input` with the level and folders, so Bot shows what the machine allows;++`describe` reports `terminal`, `files`, `screen`, `input` and `browser` with the level and folders, so Bot shows what the machine allows;telemetry carries the sensors `computer_level` (observe while a stop is latched), `computer_waiting` and`computer_running`.Every request is logged by the Node (`journalctl -u openvibe-node`), with what it would do, approved or denied.@@@ -221,6 +221,9 @@ func TestComputerLocalStopLeavesObserveOnly(t *testing.T) {if nk := nackOf(t, e.computerCmd("s2i", protocol.KindInput, map[string]any{"action": "key", "keys": "Return"}, 5*time.Second)); nk.FaultCode != protocol.FaultLocalStop {t.Fatalf("input while stopped: %+v", nk)}++ if nk := nackOf(t, e.computerCmd("s2b", protocol.KindBrowser, map[string]any{"action": "open", "url": "https://example.org/"}, 5*time.Second)); nk.FaultCode != protocol.FaultLocalStop {++ t.Fatalf("browser open while stopped: %+v", nk)++ }if d := dataOf(t, e.computerCmd("s3", protocol.KindFS, map[string]any{"op": "stat", "path": work}, 5*time.Second)); d["type"] != "dir" {t.Fatalf("observing while stopped: %v", d)}@@@ -22,6 +22,7 @@ var capabilityFor = map[string]string{protocol.KindScreen: "screen",protocol.KindProbe: "probe",protocol.KindInput: "input",++ protocol.KindBrowser: "browser",}// Manager supervises every configured plugin.@@@ -167,13 +167,16 @@ const (// KindInput moves the mouse, clicks, scrolls, types or presses keys on the machine's screen (contracts 0.171.0):// an acting command, so the owner's level decides and a latched stop refuses it.KindInput = "input"++ // KindBrowser drives a Chromium the agent uses, in a profile of its own (contracts 0.174.0): open, click, fill,++ // press and back act (the owner's level decides, a latched stop refuses them); read, screenshot and close do not.++ KindBrowser = "browser")// Kinds lists every command kind the Node understands.--var Kinds = []string{KindDrive, KindActuator, KindPTZ, KindSay, KindDisplay, KindHalt, KindButton, KindPoint, KindExec, KindFS, KindScreen, KindProbe, KindInput}++var Kinds = []string{KindDrive, KindActuator, KindPTZ, KindSay, KindDisplay, KindHalt, KindButton, KindPoint, KindExec, KindFS, KindScreen, KindProbe, KindInput, KindBrowser}// ComputerKinds move nothing: they carry long deadlines (DefaultComputerMS, at most MaxComputerMS) and no clamping.--var ComputerKinds = map[string]bool{KindExec: true, KindFS: true, KindScreen: true, KindProbe: true, KindInput: true}++var ComputerKinds = map[string]bool{KindExec: true, KindFS: true, KindScreen: true, KindProbe: true, KindInput: true, KindBrowser: true}const (// DefaultComputerMS is a computer command's reply window when the server sends no deadline; MaxComputerMS caps@@@ -186,8 +189,8 @@ const ()// ComputerActs reports whether a computer command changes the machine (runs a program, writes a file, uses the mouse--// or keyboard): those are refused while the local stop or an e-stop is latched. Capturing the screen and reading files--// are not.++// or keyboard, drives the browser): those are refused while the local stop or an e-stop is latched. Capturing the++// screen, reading files and reading or closing the browser are not.func ComputerActs(kind string, value json.RawMessage) bool {switch kind {case KindExec, KindInput:@@@ -198,6 +201,12 @@ func ComputerActs(kind string, value json.RawMessage) bool {}_ = json.Unmarshal(value, &v)return v.Op != "list" && v.Op != "stat" && v.Op != "read"++ case KindBrowser:++ var v struct {++ Action string `json:"action"`++ }++ _ = json.Unmarshal(value, &v)++ return v.Action != "read" && v.Action != "screenshot" && v.Action != "close"}return false}@@@ -395,3 +395,15 @@ func TestJobUndecodableBody(t *testing.T) {t.Fatalf("numeric id: %q", reason)}}++++func TestComputerActsBrowser(t *testing.T) {++ want := map[string]bool{"open": true, "click": true, "fill": true, "press": true, "back": true, "read": false, "screenshot": false, "close": false}++ for action, acts := range want {++ if got := ComputerActs(KindBrowser, json.RawMessage(`{"action":"`+action+`"}`)); got != acts {++ t.Errorf("browser %s: acts=%v, want %v", action, got, acts)++ }++ }++ if !ComputerKinds[KindBrowser] {++ t.Error("browser is a computer kind")++ }++}@@@ -26,6 +26,13 @@ Command kinds and their answers (the ack's ``data``):``height`` (the ``screen`` answer) and are scaled to the real screen. An acting command: refused at observe, approvedone by one at ask. xdotool on X11, cliclick on macOS (no scroll there); none on Wayland yet. Config``allow_input: false`` turns it off.++* ``browser`` ``{"action", "url"?, "selector"?, "text"?, "key"?, "max_width"?}``: a Chromium the agent drives, in a profile++ of its own (``browser_profile``), over a pipe (no port is opened). ``open`` a ``http(s)`` URL, ``click`` a CSS++ ``selector`` or the button or link whose ``text`` it shows, ``fill`` a field (``selector`` and ``text``), ``press`` a++ ``key`` (Enter, Tab, Escape, arrows …), ``back`` → ``{url, title, text, truncated, links}``; ``read`` the page the same++ way; ``screenshot`` → ``{format, width, height, content_b64}``; ``close``. Reading, screenshots and closing run at every++ level; the rest are acting commands. Below level full, ``open`` refuses an address on the machine's own network.++ Config ``allow_browser: false`` turns it off; ``browser_path``, ``browser_headless`` (default true).* ``probe`` ``{"target", "timeout_s"?}`` → for ``http(s)://…`` one GET, no redirect followed: ``{status, content_type,etag, last_modified, body, truncated, ms}``; for ``tcp://host:port``: ``{open, ms, error?}``. Read-only, so it runs atevery level, and only towards the machine's own network: every address the name resolves to must be private,@@@ -50,6 +57,8 @@ import urllib.parsefrom openvibe_plugin import Deferred, Fault, Plugin++from .browser import KEY_CODES, Browser, BrowserError, find_binary++LEVELS = ("observe", "ask", "trusted", "full")RANK = {name: i for i, name in enumerate(LEVELS)}READ_OPS = frozenset({"list", "stat", "read"})@@@ -65,6 +74,10 @@ MAX_OUTPUT_CAP = 192 * 1024MAX_READ_CAP = 256 * 1024SCREEN_MAX_BYTES = 300 * 1024INPUT_ACTIONS = frozenset({"move", "click", "double_click", "right_click", "scroll", "type", "key"})++BROWSER_READS = frozenset({"read", "screenshot", "close"})++BROWSER_ACTS = frozenset({"open", "click", "fill", "press", "back"})++MAX_URL = 2048++MAX_SELECTOR = 500POINT_ACTIONS = frozenset({"move", "click", "double_click", "right_click"})MAX_TYPE = 2000KEYS_OK = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_+-")@@@ -87,11 +100,12 @@ def _int(v, name, lo, hi, default):class Computer(Plugin):driver = "computer"-- version = "0.4.0"++ version = "0.5.0"motion_kinds = frozenset()telemetry_interval_s = 2.0-- def __init__(self, which=shutil.which, popen=subprocess.Popen, clock=time.monotonic, env=None, platform=None, screen_size=None):++ def __init__(self, which=shutil.which, popen=subprocess.Popen, clock=time.monotonic, env=None, platform=None, screen_size=None,++ browser_factory=None, resolve=None):self.which = whichself.popen = popenself.screen_size = screen_size or self._screen_size@@@ -112,6 +126,16 @@ class Computer(Plugin):self.procs = set()self.running = 0self.mu = threading.Lock()++ self.allow_browser = True++ self.browser_path = None++ self.browser_profile = None++ self.browser_headless = True++ self.browser_idle_s = 600++ self.browser = None++ self.browser_mu = threading.Lock()++ self.browser_op_mu = threading.Lock()++ self.browser_factory = browser_factory or (lambda binary, profile, headless: Browser(binary, profile, headless))++ self.resolve = resolve or (lambda host: [ai[4][0] for ai in socket.getaddrinfo(host, None)])# ---- configuration -------------------------------------------------------------------------------------------def _configure(self, config):@@@ -133,6 +157,11 @@ class Computer(Plugin):self.allow_probe = config.get("probe", True) is not Falseself.allow_input = config.get("allow_input", True) is not Falseself.notify = config.get("notify", True) is not False++ self.allow_browser = config.get("allow_browser", True) is not False++ self.browser_path = config.get("browser_path") or None++ self.browser_profile = os.path.expanduser(config.get("browser_profile") or "~/.local/share/openvibe-node/browser")++ self.browser_headless = config.get("browser_headless", True) is not False++ self.browser_idle_s = _int(config.get("browser_idle_s"), "browser_idle_s", 60, 86400, 600)def describe(self, config):try:@@@ -150,6 +179,9 @@ class Computer(Plugin):itool = self._input_tool() if self.allow_input else Noneif itool:caps["input"] = {"level": self.level, "tool": itool, "scroll": itool == "xdotool"}++ binary = self._browser_binary()++ if binary:++ caps["browser"] = {"level": self.level, "browser": os.path.basename(binary), "headless": self.browser_headless}return capsdef setup(self, config):@@@ -172,6 +204,7 @@ class Computer(Plugin):self.pending.clear()for p in procs:self._kill(p)++ self._close_browser()for cid in waiting:self.emit_event("approval_done", id=cid, approved=False, reason="stopped")self.fail(cid, "local_stop", "stopped on the machine before it ran")@@@ -187,6 +220,9 @@ class Computer(Plugin):def poll(self):now = self.clock()++ b = self.browser++ if b is not None and b.alive() and b.used_at and time.monotonic() - b.used_at > self.browser_idle_s:++ self._close_browser()expired = []with self.mu:for cid, (_, at, _) in list(self.pending.items()):@@@ -253,6 +289,11 @@ class Computer(Plugin):raise Fault("unsupported", "no input tool on this machine (xdotool on X11, cliclick on macOS)")run, summary = self._input_op(tool, v)return self._act(cmd, level, run, summary, threaded=True)++ if cmd.kind == "browser":++ run, summary, acting = self._browser_op(v, level)++ if acting:++ return self._act(cmd, level, run, summary, threaded=True)++ return self._start(cmd, run)if cmd.kind == "exec":if not self.allow_exec:raise Fault("unsupported", "this machine does not allow running programs")@@@ -605,6 +646,112 @@ class Computer(Plugin):return run, "%s %s (%d bytes)" % (verb, path, len(data))# ---- input -----------------------------------------------------------------------------------------------------++ # ---- browser ---------------------------------------------------------------------------------------------------++ def _browser_binary(self):++ if not self.allow_browser or self.platform.startswith("win"):++ return None++ return find_binary(self.which, self.platform, self.browser_path)++++ def _close_browser(self):++ # Never waits for an action in flight: closing kills the process, and the action fails at once.++ b, self.browser = self.browser, None++ if b is not None:++ b.close()++++ def _local_host(self, host):++ """Does `host` name (or resolve to) an address on the machine's own network?"""++ try:++ addrs = [host] if self._is_ip(host) else self.resolve(host)++ except (OSError, UnicodeError):++ raise Fault("bad_value", "%s does not resolve" % host)++ return any(self._lan(a) for a in addrs) or not addrs++++ @staticmethod++ def _is_ip(host):++ try:++ ipaddress.ip_address(host)++ return True++ except ValueError:++ return False++++ def _browser_url(self, raw, level):++ if not isinstance(raw, str) or not raw or len(raw) > MAX_URL or any(c in raw for c in "\x00\r\n"):++ raise Fault("bad_value", "url is an http or https address")++ u = urllib.parse.urlsplit(raw)++ if u.scheme not in ("http", "https") or not u.hostname:++ raise Fault("bad_value", "url is an http or https address")++ if level != "full" and self._local_host(u.hostname.strip("[]")):++ raise Fault("not_allowed", "%s is on this machine's own network: only at level full" % u.hostname)++ return raw++++ def _browser_op(self, v, level):++ """(run, summary, acting) for one browser action; nothing is opened or clicked before run()."""++ binary = self._browser_binary()++ if not binary:++ raise Fault("unsupported", "no browser on this machine (Chromium, Chrome, Edge or Brave), or this machine does not allow it")++ action = v.get("action")++ if action not in BROWSER_READS and action not in BROWSER_ACTS:++ raise Fault("bad_value", "action is one of %s" % ", ".join(sorted(BROWSER_READS | BROWSER_ACTS)))++ selector, text, key, url = v.get("selector"), v.get("text"), v.get("key"), None++ if action == "open":++ url = self._browser_url(v.get("url"), level)++ if action in ("click", "fill") and selector is not None and (not isinstance(selector, str) or not selector or len(selector) > MAX_SELECTOR):++ raise Fault("bad_value", "selector is a CSS selector of at most %d characters" % MAX_SELECTOR)++ if action == "click" and selector is None and (not isinstance(text, str) or not text.strip() or len(text) > 200):++ raise Fault("bad_value", "click takes a selector, or the text of a button or link (at most 200 characters)")++ if action == "fill":++ if selector is None:++ raise Fault("bad_value", "fill takes the selector of the field")++ if not isinstance(text, str) or len(text) > MAX_TYPE or "\x00" in text:++ raise Fault("bad_value", "text is at most %d characters" % MAX_TYPE)++ if action == "press" and key not in KEY_CODES:++ raise Fault("bad_value", "key is one of %s" % ", ".join(sorted(KEY_CODES)))++ width = _int(v.get("max_width"), "max_width", 320, 2560, 1280)++++ def browser():++ with self.browser_mu:++ if self.browser is None or not self.browser.alive():++ self.browser = self.browser_factory(binary, self.browser_profile, self.browser_headless)++ return self.browser++++ def run():++ if action == "close":++ self._close_browser()++ return {"done": True}++ b = browser()++ try:++ with self.browser_op_mu: # one action at a time in the one browser++ if action == "open":++ return b.open(url)++ if action == "read":++ return b.read()++ if action == "screenshot":++ return b.screenshot(width)++ if action == "click":++ return b.click(selector=selector, text=None if selector else text)++ if action == "fill":++ return b.fill(selector, text)++ if action == "press":++ return b.press(key)++ return b.back()++ except BrowserError as e:++ raise Fault(e.code, e.message)++++ if action == "open":++ summary = "open %s in the browser" % url[:300]++ elif action == "click":++ summary = "click %s in the browser" % (selector if selector else "the %r button or link" % text[:120])++ elif action == "fill":++ shown = text if len(text) <= 120 else text[:120] + "…"++ summary = "type %d characters into %s in the browser: %r" % (len(text), selector[:200], shown)++ elif action == "press":++ summary = "press %s in the browser" % key++ elif action == "back":++ summary = "go back in the browser"++ else:++ summary = "%s the browser" % action++ return run, summary, action in BROWSER_ACTS++def _input_tool(self):"""xdotool on an X11 session, cliclick on macOS, else None (Wayland has no general input tool yet)."""if self.platform == "darwin" and self.which("cliclick"):nnew file mode 100644@@@ -0,0 +1,342 @@++"""The computer plugin's browser: one Chromium the agent drives, in a profile of its own (plan T14, the browser kind).++++Chromium is started on the first command with ``--remote-debugging-pipe``: it reads Chrome DevTools Protocol commands++from file descriptor 3 and writes answers and events to 4, each message JSON ended by a NUL byte. No port is opened,++so nothing else on the machine can drive it. The profile is a folder of its own (``browser_profile``, by default++``~/.local/share/openvibe-node/browser``): the agent never sees the owner's own browser sessions unless the owner points++that setting at one.++++ Browser(binary, profile, headless).open(url) / read() / click(selector|text) / fill(selector, text) / press(key)++ / back() / screenshot(max_width) / close()++++Every page answer is ``{url, title, text, links}``: the page's visible text (cut at ``MAX_TEXT``) and its first links.++"""++++import base64++import json++import os++import subprocess++import threading++import time++++MAX_TEXT = 20000++STARTUP_S = 45.0++MAX_LINKS = 60++KEY_CODES = {++ "Enter": (13, "\r"), "Tab": (9, "\t"), "Escape": (27, ""), "Backspace": (8, ""), "Delete": (46, ""),++ "ArrowUp": (38, ""), "ArrowDown": (40, ""), "ArrowLeft": (37, ""), "ArrowRight": (39, ""),++ "PageUp": (33, ""), "PageDown": (34, ""), "Home": (36, ""), "End": (35, ""), "Space": (32, " "),++}++CANDIDATES = ("chromium", "chromium-browser", "google-chrome", "google-chrome-stable", "microsoft-edge", "brave-browser")++MAC_APPS = ("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",++ "/Applications/Chromium.app/Contents/MacOS/Chromium",++ "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",++ "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser")++++SUMMARY_JS = """(() => {++ const text = (document.body ? document.body.innerText : '').replace(/\\n{3,}/g, '\\n\\n');++ const links = [...document.querySelectorAll('a[href]')]++ .filter((a) => /^https?:/.test(a.href))++ .slice(0, %d)++ .map((a) => ({ text: (a.innerText || a.getAttribute('aria-label') || '').trim().slice(0, 100), href: a.href }));++ return { url: location.href, title: document.title, text: text.slice(0, %d), truncated: text.length > %d, links };++})()""" % (MAX_LINKS, MAX_TEXT, MAX_TEXT)++++CLICK_JS = """((selector, text) => {++ let el = null;++ if (selector) el = document.querySelector(selector);++ else {++ const want = text.trim().toLowerCase();++ const items = [...document.querySelectorAll('a, button, [role=button], input[type=submit], input[type=button], summary, label')];++ el = items.find((e) => (e.innerText || e.value || e.getAttribute('aria-label') || '').trim().toLowerCase() === want)++ || items.find((e) => (e.innerText || e.value || e.getAttribute('aria-label') || '').trim().toLowerCase().includes(want));++ }++ if (!el) return false;++ el.scrollIntoView({ block: 'center' });++ el.click();++ return true;++})(%s, %s)"""++++FOCUS_JS = """((selector) => {++ const el = document.querySelector(selector);++ if (!el) return false;++ el.scrollIntoView({ block: 'center' });++ el.focus();++ if ('value' in el) { el.value = ''; el.dispatchEvent(new Event('input', { bubbles: true })); }++ else if (el.isContentEditable) el.textContent = '';++ return true;++})(%s)"""++++++class BrowserError(Exception):++ def __init__(self, code, message):++ super().__init__(message)++ self.code = code++ self.message = message++++++def find_binary(which, platform, configured=None):++ """The Chromium-family browser to drive: the configured path, else the first one on PATH (or a macOS app)."""++ if configured:++ return configured if os.path.isfile(configured) and os.access(configured, os.X_OK) else None++ for name in CANDIDATES:++ path = which(name)++ if path:++ return path++ if platform == "darwin":++ for path in MAC_APPS:++ if os.path.isfile(path):++ return path++ return None++++++class Browser:++ def __init__(self, binary, profile, headless=True, timeout_s=15.0, popen=subprocess.Popen, extra_args=()):++ self.binary = binary++ self.extra_args = list(extra_args)++ self.stderr_tail = []++ self.profile = profile++ self.headless = headless++ self.timeout_s = timeout_s++ self.popen = popen++ self.proc = None++ self.out = None++ self.inp = None++ self.next_id = 0++ self.waiting = {} # message id -> [event, result]++ self.events = [] # (method, params) seen since the last mark++ self.mu = threading.Lock()++ self.session = None++ self.used_at = 0.0++++ # ---- the process and the pipe ------------------------------------------------------------------------------------++ def alive(self):++ return self.proc is not None and self.proc.poll() is None++++ def start(self):++ if self.alive():++ return++ os.makedirs(self.profile, mode=0o700, exist_ok=True)++ to_chrome_r, to_chrome_w = os.pipe()++ from_chrome_r, from_chrome_w = os.pipe()++ argv = [self.binary, "--remote-debugging-pipe", "--user-data-dir=%s" % self.profile, "--no-first-run",++ "--no-default-browser-check", "--disable-background-networking", "--disable-sync",++ "--disable-component-update", "--password-store=basic", "--window-size=1280,900", "about:blank"]++ if self.headless:++ argv.insert(1, "--headless=new")++ argv[1:1] = self.extra_args++++ def fds():++ # In the child: Chromium reads commands on 3 and writes on 4. Both ends move above 10 first, so placing one++ # never closes the other; 3 and 4 are in pass_fds, so the close that follows this function keeps them.++ import fcntl++ r = fcntl.fcntl(to_chrome_r, fcntl.F_DUPFD, 10)++ w = fcntl.fcntl(from_chrome_w, fcntl.F_DUPFD, 10)++ os.dup2(r, 3)++ os.dup2(w, 4)++++ self.proc = self.popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE,++ pass_fds=(3, 4), preexec_fn=fds, start_new_session=True)++ threading.Thread(target=self._stderr, args=(self.proc.stderr,), name="computer-browser-err", daemon=True).start()++ os.close(to_chrome_r)++ os.close(from_chrome_w)++ self.out = os.fdopen(to_chrome_w, "wb", buffering=0)++ self.inp = os.fdopen(from_chrome_r, "rb", buffering=0)++ threading.Thread(target=self._reader, name="computer-browser", daemon=True).start()++ # A first start on a fresh profile can take a while on a slow machine: the first command gets a longer window.++ target = self.call("Target.createTarget", {"url": "about:blank"}, session=False, timeout_s=max(self.timeout_s, STARTUP_S))["targetId"]++ self.session = self.call("Target.attachToTarget", {"targetId": target, "flatten": True}, session=False)["sessionId"]++ self.call("Page.enable")++ self.call("Runtime.enable")++++ def _stderr(self, stream):++ """Keep the browser's last lines of stderr: why it closed, when it does."""++ for line in iter(stream.readline, b""):++ self.stderr_tail = (self.stderr_tail + [line.decode("utf-8", "replace").strip()])[-5:]++++ def _reader(self):++ buf = b""++ while True:++ try:++ chunk = self.inp.read(65536)++ except (OSError, ValueError):++ chunk = b""++ if not chunk:++ break++ buf += chunk++ while b"\0" in buf:++ raw, buf = buf.split(b"\0", 1)++ try:++ msg = json.loads(raw.decode("utf-8"))++ except ValueError:++ continue++ if "id" in msg:++ with self.mu:++ slot = self.waiting.get(msg["id"])++ if slot is not None:++ slot[1] = msg++ slot[0].set()++ elif "method" in msg:++ with self.mu:++ self.events.append(msg["method"])++ if len(self.events) > 500:++ del self.events[:250]++ with self.mu:++ for slot in self.waiting.values():++ slot[0].set()++++ def call(self, method, params=None, session=True, timeout_s=None):++ with self.mu:++ self.next_id += 1++ mid = self.next_id++ slot = [threading.Event(), None]++ self.waiting[mid] = slot++ msg = {"id": mid, "method": method, "params": params or {}}++ if session and self.session:++ msg["sessionId"] = self.session++ try:++ self.out.write(json.dumps(msg).encode("utf-8") + b"\0")++ except (OSError, ValueError, AttributeError):++ raise BrowserError("hardware", "the browser is not running")++ ok = slot[0].wait(timeout_s or self.timeout_s)++ with self.mu:++ self.waiting.pop(mid, None)++ if not ok:++ raise BrowserError("expired", "the browser did not answer %s in time" % method)++ if slot[1] is None:++ why = " | ".join(x for x in self.stderr_tail if x)[-400:]++ raise BrowserError("hardware", "the browser closed" + (": %s" % why if why else ""))++ if "error" in slot[1]:++ raise BrowserError("bad_value", "%s: %s" % (method, slot[1]["error"].get("message", "refused")))++ return slot[1].get("result", {})++++ def _mark(self):++ with self.mu:++ self.events = []++++ def _wait_for(self, methods, timeout_s):++ end = time.monotonic() + timeout_s++ while time.monotonic() < end:++ with self.mu:++ if any(m in methods for m in self.events):++ return True++ time.sleep(0.05)++ return False++++ def _eval(self, expression):++ r = self.call("Runtime.evaluate", {"expression": expression, "returnByValue": True, "awaitPromise": True})++ if r.get("exceptionDetails"):++ raise BrowserError("bad_value", "the page refused: %s" % r["exceptionDetails"].get("text", "an exception"))++ return r.get("result", {}).get("value")++++ def _settle(self, navigated_timeout=2.0):++ """After an action: if it started a navigation, wait for the load; then a short pause for scripts."""++ if self._wait_for(("Page.frameStartedLoading", "Page.frameNavigated"), navigated_timeout):++ self._wait_for(("Page.loadEventFired",), self.timeout_s)++ time.sleep(0.3)++++ # ---- the actions -----------------------------------------------------------------------------------------------++ def summary(self):++ self.used_at = time.monotonic()++ return self._eval(SUMMARY_JS)++++ def open(self, url):++ self.start()++ self._mark()++ r = self.call("Page.navigate", {"url": url})++ if r.get("errorText"):++ raise BrowserError("hardware", "could not open %s: %s" % (url, r["errorText"]))++ self._wait_for(("Page.loadEventFired",), self.timeout_s)++ time.sleep(0.3)++ return self.summary()++++ def read(self):++ self._need()++ return self.summary()++++ def click(self, selector=None, text=None):++ self._need()++ self._mark()++ if not self._eval(CLICK_JS % (json.dumps(selector), json.dumps(text or ""))):++ raise BrowserError("bad_value", "nothing on the page matches %s" % (selector or repr(text)))++ self._settle()++ return self.summary()++++ def fill(self, selector, text):++ self._need()++ if not self._eval(FOCUS_JS % json.dumps(selector)):++ raise BrowserError("bad_value", "nothing on the page matches %s" % selector)++ self.call("Input.insertText", {"text": text})++ return {"done": True, "selector": selector, "chars": len(text)}++++ def press(self, key):++ self._need()++ code, text = KEY_CODES[key]++ self._mark()++ base = {"key": key, "code": key if key != "Space" else "Space", "windowsVirtualKeyCode": code, "nativeVirtualKeyCode": code}++ self.call("Input.dispatchKeyEvent", dict(base, type="keyDown", **({"text": text} if text else {})))++ self.call("Input.dispatchKeyEvent", dict(base, type="keyUp"))++ self._settle()++ return self.summary()++++ def back(self):++ self._need()++ self._mark()++ self._eval("history.back()")++ self._settle()++ return self.summary()++++ def screenshot(self, max_width=1280):++ self._need()++ self.used_at = time.monotonic()++ r = self.call("Page.captureScreenshot", {"format": "jpeg", "quality": 70})++ data = base64.b64decode(r["data"])++ metrics = self.call("Page.getLayoutMetrics")++ view = metrics.get("cssVisualViewport") or metrics.get("visualViewport") or {}++ width, height = int(view.get("clientWidth", 1280)), int(view.get("clientHeight", 900))++ if width > max_width:++ try:++ from PIL import Image++ import io++ img = Image.open(io.BytesIO(data))++ ratio = max_width / float(img.width)++ img = img.resize((max_width, max(1, int(img.height * ratio))))++ out = io.BytesIO()++ img.convert("RGB").save(out, "JPEG", quality=70)++ data, width, height = out.getvalue(), img.width, img.height++ except Exception:++ pass++ return {"format": "jpeg", "width": width, "height": height, "content_b64": base64.b64encode(data).decode("ascii")}++++ def _need(self):++ if not self.alive() or not self.session:++ raise BrowserError("not_ready", "no page is open: open a URL first")++++ def close(self):++ proc, self.proc, self.session = self.proc, None, None++ for f in (self.out, self.inp):++ try:++ if f:++ f.close()++ except OSError:++ pass++ self.out = self.inp = None++ if proc is not None and proc.poll() is None:++ try:++ if hasattr(os, "killpg"):++ os.killpg(proc.pid, 9)++ else:++ proc.kill()++ except (OSError, ProcessLookupError):++ pass++ if proc is not None:++ try:++ proc.wait(timeout=5)++ except Exception:++ pass++ if proc.stderr:++ try:++ proc.stderr.close()++ except OSError:++ pass++ return {"done": True, "closed": proc is not None}nnew file mode 100644@@@ -0,0 +1,209 @@++import http.server++import os++import shutil++import socketserver++import sys++import threading++import unittest++++HERE = os.path.dirname(os.path.abspath(__file__))++sys.path.insert(0, os.path.dirname(HERE))++sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(HERE)), "sdk"))++sys.path.insert(0, HERE)++++from openvibe_computer import Computer # noqa: E402++from openvibe_computer.browser import Browser, find_binary # noqa: E402++from openvibe_plugin import Runtime # noqa: E402++from test_computer import Out, reply # noqa: E402++++++class FakeBrowser:++ """What the plugin asks of a browser, recorded; pages are made up."""++++ def __init__(self, binary, profile, headless):++ self.binary, self.profile, self.headless = binary, profile, headless++ self.calls = []++ self.open_ = True++ self.used_at = 0++++ def alive(self):++ return self.open_++++ def _page(self, url="https://example.org/"):++ return {"url": url, "title": "Example", "text": "Example page", "truncated": False, "links": []}++++ def open(self, url):++ self.calls.append(("open", url))++ return self._page(url)++++ def read(self):++ self.calls.append(("read",))++ return self._page()++++ def click(self, selector=None, text=None):++ self.calls.append(("click", selector, text))++ return self._page()++++ def fill(self, selector, text):++ self.calls.append(("fill", selector, text))++ return {"done": True, "selector": selector, "chars": len(text)}++++ def press(self, key):++ self.calls.append(("press", key))++ return self._page()++++ def back(self):++ self.calls.append(("back",))++ return self._page()++++ def screenshot(self, max_width):++ self.calls.append(("screenshot", max_width))++ return {"format": "jpeg", "width": max_width, "height": 10, "content_b64": ""}++++ def close(self):++ self.open_ = False++ self.calls.append(("close",))++ return {"done": True}++++++PUBLIC = {"example.org": ["93.184.215.14"], "router.lan": ["192.168.1.1"], "sneaky.example": ["93.184.215.14", "10.0.0.5"]}++++++class BrowserKindTest(unittest.TestCase):++ def boot(self, level, **config):++ self.made = []++++ def factory(binary, profile, headless):++ b = FakeBrowser(binary, profile, headless)++ self.made.append(b)++ return b++++ self.out = Out()++ self.plugin = Computer(env={}, platform="linux", which=lambda n: "/usr/bin/chromium" if n == "chromium" else None,++ browser_factory=factory, resolve=lambda h: PUBLIC.get(h, []))++ self.rt = Runtime(self.plugin, infile=__import__("io").StringIO(""), outfile=self.out)++ self.rt.dispatch({"op": "hello", "config": {"level": level, "roots": [HERE], **config}})++ self.n = 0++ return self.out.msgs()++++ def cmd(self, value):++ self.n += 1++ cid = "b%d" % self.n++ self.rt.dispatch({"op": "command", "id": cid, "kind": "browser", "value": value, "deadline_ms": 30000})++ return cid++++ def test_described_with_its_level(self):++ caps = self.boot("ask")[0]["capabilities"]++ self.assertEqual(caps["browser"], {"level": "ask", "browser": "chromium", "headless": True})++ caps = self.boot("ask", allow_browser=False)[0]["capabilities"]++ self.assertNotIn("browser", caps)++++ def test_reading_runs_at_observe_and_acting_does_not(self):++ self.boot("observe")++ r = reply(self.out, self.cmd({"action": "open", "url": "https://example.org/"}))++ self.assertEqual((r["op"], r["fault_code"]), ("nack", "not_allowed"))++ self.assertEqual(self.made, [], "nothing is started for a refused command")++ r = reply(self.out, self.cmd({"action": "read"}))++ self.assertEqual(r["op"], "ack", "reading the agent's page is observing")++ self.assertEqual(self.made[0].calls, [("read",)])++++ def test_ask_waits_for_the_owner_and_shows_what_is_typed(self):++ self.boot("ask")++ cid = self.cmd({"action": "fill", "selector": "#q", "text": "hello"})++ ev = self.out.wait_for(lambda m: m.get("op") == "event" and m.get("name") == "approval")++ self.assertEqual((ev["kind"], ev["summary"]), ("browser", "type 5 characters into #q in the browser: 'hello'"))++ self.assertEqual(self.made, [], "nothing happens before the owner approves")++ self.plugin.on_approval(cid, True)++ self.assertEqual(reply(self.out, cid)["data"], {"done": True, "selector": "#q", "chars": 5})++ cid = self.cmd({"action": "click", "text": "Sign in"})++ ev = self.out.wait_for(lambda m: m.get("op") == "event" and m.get("name") == "approval" and m.get("id") == cid)++ self.assertEqual(ev["summary"], "click the 'Sign in' button or link in the browser")++ self.plugin.on_approval(cid, False)++ self.assertEqual(reply(self.out, cid)["fault_code"], "not_allowed")++++ def test_trusted_drives_it_and_the_profile_is_its_own(self):++ self.boot("trusted")++ self.assertEqual(reply(self.out, self.cmd({"action": "open", "url": "https://example.org/a"}))["data"]["url"], "https://example.org/a")++ reply(self.out, self.cmd({"action": "click", "selector": "a.next"}))++ reply(self.out, self.cmd({"action": "press", "key": "Enter"}))++ reply(self.out, self.cmd({"action": "back"}))++ reply(self.out, self.cmd({"action": "screenshot", "max_width": 640}))++ b = self.made[0]++ self.assertEqual(len(self.made), 1, "one browser for the session")++ self.assertEqual(b.calls, [("open", "https://example.org/a"), ("click", "a.next", None), ("press", "Enter"), ("back",), ("screenshot", 640)])++ self.assertTrue(b.profile.endswith(os.path.join("openvibe-node", "browser")), b.profile)++++ def test_addresses_and_values_it_refuses(self):++ self.boot("trusted")++ for value, code in [++ ({"action": "open", "url": "file:///etc/passwd"}, "bad_value"),++ ({"action": "open", "url": "javascript:alert(1)"}, "bad_value"),++ ({"action": "open", "url": "http://192.168.1.1/"}, "not_allowed"),++ ({"action": "open", "url": "http://router.lan/"}, "not_allowed"),++ ({"action": "open", "url": "https://sneaky.example/"}, "not_allowed"),++ ({"action": "open", "url": "http://[::1]:8080/"}, "not_allowed"),++ ({"action": "press", "key": "ctrl+alt+Delete"}, "bad_value"),++ ({"action": "fill", "text": "x"}, "bad_value"),++ ({"action": "click"}, "bad_value"),++ ({"action": "eval", "script": "1"}, "bad_value"),++ ]:++ r = reply(self.out, self.cmd(value))++ self.assertEqual((r["op"], r["fault_code"]), ("nack", code), value)++ self.boot("full")++ self.assertEqual(reply(self.out, self.cmd({"action": "open", "url": "http://router.lan/"}))["op"], "ack", "full reaches the local network")++++ def test_the_kill_switch_closes_it(self):++ self.boot("trusted")++ reply(self.out, self.cmd({"action": "open", "url": "https://example.org/"}))++ self.plugin.stop()++ self.assertFalse(self.made[0].alive())++ self.assertEqual(reply(self.out, self.cmd({"action": "close"}))["data"], {"done": True})++++++class Handler(http.server.SimpleHTTPRequestHandler):++ PAGES = {++ "/": b"<!doctype html><title>Home</title><h1>Hello</h1><a href='/two'>Second page</a>"++ b"<form action='/two'><input id='q' name='q'><button>Search</button></form>",++ "/two": b"<!doctype html><title>Two</title><p>Arrived</p>",++ }++++ def do_GET(self):++ body = self.PAGES.get(self.path.split("?")[0], b"not found")++ self.send_response(200)++ self.send_header("Content-Type", "text/html")++ self.send_header("Content-Length", str(len(body)))++ self.end_headers()++ self.wfile.write(body)++++ def log_message(self, *a):++ pass+++++[email protected](find_binary(shutil.which, sys.platform), "no Chromium-family browser on this machine")++class RealBrowserTest(unittest.TestCase):++ def test_a_real_browser_over_the_pipe(self):++ srv = socketserver.TCPServer(("127.0.0.1", 0), Handler)++ threading.Thread(target=srv.serve_forever, daemon=True).start()++ self.addCleanup(srv.shutdown)++ base = "http://127.0.0.1:%d" % srv.server_address[1]++ profile = __import__("tempfile").mkdtemp(prefix="ov-browser-")++ self.addCleanup(lambda: shutil.rmtree(profile, ignore_errors=True))++ # CI runners (Ubuntu 24.04) refuse the unprivileged user namespaces Chrome's sandbox needs; a person's machine does++ # not, and the plugin never turns the sandbox off.++ b = Browser(find_binary(shutil.which, sys.platform), profile, extra_args=["--no-sandbox"] if os.environ.get("CI") else [])++ self.addCleanup(b.close)++ page = b.open(base + "/")++ self.assertEqual((page["title"], [l["text"] for l in page["links"]]), ("Home", ["Second page"]))++ self.assertEqual(b.click(text="Second page")["title"], "Two")++ self.assertEqual(b.back()["title"], "Home")++ self.assertEqual(b.fill("#q", "two words")["chars"], 9)++ self.assertTrue(b.press("Enter")["url"].endswith("/two?q=two+words"))++ shot = b.screenshot(640)++ self.assertEqual((shot["format"], shot["width"] <= 1280), ("jpeg", True))++ self.assertTrue(shot["content_b64"])++ b.close()++ self.assertFalse(b.alive())++++++if __name__ == "__main__":++ unittest.main()
@@ -41,8 +41,8 @@ signature against the key Bot publishes (`--key-id` pins one) and names the plug
@@ -41,8 +41,8 @@ signature against the key Bot publishes (`--key-id` pins one) and names the plug
on a community profile runs the same check and warns when it fails. Install a community plugin only when it passes
on a community profile runs the same check and warns when it fails. Install a community plugin only when it passes
([docs/plugins.md](docs/plugins.md#community-plugins)).
([docs/plugins.md](docs/plugins.md#community-plugins)).
-**Your computer for your agents:** `--computer observe|ask|trusted|full` adds the `computer` plugin (terminal, files
-and screen for OpenVibe.Actor), running as your own account, at the level you set on the machine; `ask` waits for
+**Your computer for your agents:** `--computer observe|ask|trusted|full` adds the `computer` plugin (terminal, files,
+screen, mouse and keyboard, and a browser of its own for OpenVibe.Actor), running as your own account, at the level you set on the machine; `ask` waits for
`openvibe-node approve <id>` ([docs/computer.md](docs/computer.md)).
`openvibe-node approve <id>` ([docs/computer.md](docs/computer.md)).
**A machine that runs your jobs:** add it on openvibe.bot/machines, which prints the same installer with `--worker`: the
**A machine that runs your jobs:** add it on openvibe.bot/machines, which prints the same installer with `--worker`: the
@@ -48,7 +48,8 @@ When the Node runs in your graphical session, a waiting request also shows a des
@@ -48,7 +48,8 @@ When the Node runs in your graphical session, a waiting request also shows a des
that a request is waiting on the machine.
that a request is waiting on the machine.
**Kill switch:** `openvibe-node stop` (and an owner's e-stop) drops the plugin to `observe`: what runs is killed, what
**Kill switch:** `openvibe-node stop` (and an owner's e-stop) drops the plugin to `observe`: what runs is killed, what
-waits is denied, and the core also refuses `exec`, file writes and `input` until `openvibe-node resume`. `halt` kills what runs.
+waits is denied, the agent's browser is closed, and the core also refuses `exec`, file writes, `input` and the browser's
+acting actions until `openvibe-node resume`. `halt` kills what runs.
## What it does
## What it does
@@ -59,6 +60,7 @@ waits is denied, and the core also refuses `exec`, file writes and `input` until
@@ -59,6 +60,7 @@ waits is denied, and the core also refuses `exec`, file writes and `input` until
| `probe` | `target`: `http(s)://…` (one GET, no redirect followed) or `tcp://host:port`; `timeout_s` | `status`, `content_type`, `etag`, `last_modified`, `body` (≤ 64 KB), `ms`; or `open`, `ms` |
| `probe` | `target`: `http(s)://…` (one GET, no redirect followed) or `tcp://host:port`; `timeout_s` | `status`, `content_type`, `etag`, `last_modified`, `body` (≤ 64 KB), `ms`; or `open`, `ms` |
| `screen` | `max_width` (default 1280) | `format` (`jpeg` with Pillow, else `png` ≤ 300 KB), `width`, `height`, `content_b64` |
| `screen` | `max_width` (default 1280) | `format` (`jpeg` with Pillow, else `png` ≤ 300 KB), `width`, `height`, `content_b64` |
| `input` | `action`: `move`, `click`, `double_click`, `right_click` (`x`, `y`), `scroll` (`dy` clicks, ±20), `type` (`text`, ≤ 2000), `key` (`keys`: one key or chord, `ctrl+s`, `Return`); `width`, `height`: the picture `x`, `y` are measured in | `done`, `action`, and the real `x`, `y` it used |
| `input` | `action`: `move`, `click`, `double_click`, `right_click` (`x`, `y`), `scroll` (`dy` clicks, ±20), `type` (`text`, ≤ 2000), `key` (`keys`: one key or chord, `ctrl+s`, `Return`); `width`, `height`: the picture `x`, `y` are measured in | `done`, `action`, and the real `x`, `y` it used |
+| `browser` | `action`: `open` (`url`, http or https), `click` (`selector`, or the `text` a button or link shows), `fill` (`selector`, `text` ≤ 2000), `press` (`key`: Enter, Tab, Escape, Backspace, Delete, arrows, PageUp, PageDown, Home, End, Space), `back`, `read`, `screenshot` (`max_width`), `close` | a page: `url`, `title`, `text` (its visible text, ≤ 20,000 characters), `truncated`, `links` (the first 60); `screenshot`: `format`, `width`, `height`, `content_b64`; `fill`: `done`, `chars` |
A program runs in its own process group with the Node account's rights, at most 2 at a time, and is killed at
A program runs in its own process group with the Node account's rights, at most 2 at a time, and is killed at
`timeout_s` (the owner's `timeout_s` is the cap). A write with `overwrite` replaces the file atomically. Screens come
`timeout_s` (the owner's `timeout_s` is the cap). A write with `overwrite` replaces the file atomically. Screens come
@@ -73,6 +75,14 @@ saw (`width` × `height`), scaled to your real screen. It uses `xdotool` on an X
@@ -73,6 +75,14 @@ saw (`width` × `height`), scaled to your real screen. It uses `xdotool` on an X
text and keys are passed to the tool as arguments, never through a shell, and `keys` takes only letters, digits, `+`,
text and keys are passed to the tool as arguments, never through a shell, and `keys` takes only letters, digits, `+`,
`-` and `_`.
`-` and `_`.
+**The browser** (`browser`) is a Chromium-family browser (Chromium, Chrome, Edge or Brave) the agent drives:
+- **Its own profile.** It runs in a folder of its own (`browser_profile`, by default `~/.local/share/openvibe-node/browser`), so the agent never sees the sessions of your own browser unless you point it at one. Sign it in to a site yourself (`browser_headless: false` shows its window) if you want the agent to work there as you.
+- **No port.** The Node starts it on the first command with `--remote-debugging-pipe`: it speaks the DevTools protocol over two file descriptors, so nothing else on the machine can drive it.
+- **Levels.** `read`, `screenshot` and `close` run at every level. `open`, `click`, `fill`, `press` and `back` are acting commands: refused at `observe`, approved one by one at `ask`. The approval says exactly what would happen: "open https://… in the browser", "type 5 characters into #q in the browser: 'hello'".
+- **Your own network.** Below level `full`, `open` refuses an address on the machine's own network (private, loopback or link-local, checked for every address the name resolves to).
+- **Closing.** It closes after `browser_idle_s` without a command and at the kill switch.
+- **Platforms.** Linux and macOS; not offered on Windows yet.
+
## Config
## Config
| key | default | meaning |
| key | default | meaning |
@@ -87,8 +97,13 @@ text and keys are passed to the tool as arguments, never through a shell, and `k
@@ -87,8 +97,13 @@ text and keys are passed to the tool as arguments, never through a shell, and `k
| `notify` | `true` | a desktop notification when a request waits (graphical session only) |
| `notify` | `true` | a desktop notification when a request waits (graphical session only) |
| `probe` | `true` | `false`: no probes (OpenVibe.Watch's `node` source) |
| `probe` | `true` | `false`: no probes (OpenVibe.Watch's `node` source) |
| `allow_input` | `true` | `false`: never use the mouse or keyboard (no `input` capability) |
| `allow_input` | `true` | `false`: never use the mouse or keyboard (no `input` capability) |
+| `allow_browser` | `true` | `false`: no browser (no `browser` capability) |
+| `browser_path` | found on PATH | the browser to drive (Chromium, Chrome, Edge or Brave) |
+| `browser_profile` | `~/.local/share/openvibe-node/browser` | its profile folder: the agent's own, unless you choose another |
+| `browser_headless` | `true` | `false`: show its window |
+| `browser_idle_s` | 600 | close it after this long without a command |
-`describe` reports `terminal`, `files`, `screen` and `input` with the level and folders, so Bot shows what the machine allows;
+`describe` reports `terminal`, `files`, `screen`, `input` and `browser` with the level and folders, so Bot shows what the machine allows;
telemetry carries the sensors `computer_level` (observe while a stop is latched), `computer_waiting` and
telemetry carries the sensors `computer_level` (observe while a stop is latched), `computer_waiting` and
`computer_running`.
`computer_running`.
Every request is logged by the Node (`journalctl -u openvibe-node`), with what it would do, approved or denied.
Every request is logged by the Node (`journalctl -u openvibe-node`), with what it would do, approved or denied.
@@ -221,6 +221,9 @@ func TestComputerLocalStopLeavesObserveOnly(t *testing.T) {
@@ -221,6 +221,9 @@ func TestComputerLocalStopLeavesObserveOnly(t *testing.T) {
if nk := nackOf(t, e.computerCmd("s2i", protocol.KindInput, map[string]any{"action": "key", "keys": "Return"}, 5*time.Second)); nk.FaultCode != protocol.FaultLocalStop {
if nk := nackOf(t, e.computerCmd("s2i", protocol.KindInput, map[string]any{"action": "key", "keys": "Return"}, 5*time.Second)); nk.FaultCode != protocol.FaultLocalStop {
t.Fatalf("input while stopped: %+v", nk)
t.Fatalf("input while stopped: %+v", nk)
}
}
+ if nk := nackOf(t, e.computerCmd("s2b", protocol.KindBrowser, map[string]any{"action": "open", "url": "https://example.org/"}, 5*time.Second)); nk.FaultCode != protocol.FaultLocalStop {
+ t.Fatalf("browser open while stopped: %+v", nk)
+ }
if d := dataOf(t, e.computerCmd("s3", protocol.KindFS, map[string]any{"op": "stat", "path": work}, 5*time.Second)); d["type"] != "dir" {
if d := dataOf(t, e.computerCmd("s3", protocol.KindFS, map[string]any{"op": "stat", "path": work}, 5*time.Second)); d["type"] != "dir" {
t.Fatalf("observing while stopped: %v", d)
t.Fatalf("observing while stopped: %v", d)
}
}
@@ -22,6 +22,7 @@ var capabilityFor = map[string]string{
@@ -22,6 +22,7 @@ var capabilityFor = map[string]string{
protocol.KindScreen: "screen",
protocol.KindScreen: "screen",
protocol.KindProbe: "probe",
protocol.KindProbe: "probe",
protocol.KindInput: "input",
protocol.KindInput: "input",
+ protocol.KindBrowser: "browser",
}
}
// Manager supervises every configured plugin.
// Manager supervises every configured plugin.
@@ -167,13 +167,16 @@ const (
@@ -167,13 +167,16 @@ const (
// KindInput moves the mouse, clicks, scrolls, types or presses keys on the machine's screen (contracts 0.171.0):
// KindInput moves the mouse, clicks, scrolls, types or presses keys on the machine's screen (contracts 0.171.0):
// an acting command, so the owner's level decides and a latched stop refuses it.
// an acting command, so the owner's level decides and a latched stop refuses it.
KindInput = "input"
KindInput = "input"
+ // KindBrowser drives a Chromium the agent uses, in a profile of its own (contracts 0.174.0): open, click, fill,
+ // press and back act (the owner's level decides, a latched stop refuses them); read, screenshot and close do not.
+ KindBrowser = "browser"
)
)
// Kinds lists every command kind the Node understands.
// Kinds lists every command kind the Node understands.
-var Kinds = []string{KindDrive, KindActuator, KindPTZ, KindSay, KindDisplay, KindHalt, KindButton, KindPoint, KindExec, KindFS, KindScreen, KindProbe, KindInput}
+var Kinds = []string{KindDrive, KindActuator, KindPTZ, KindSay, KindDisplay, KindHalt, KindButton, KindPoint, KindExec, KindFS, KindScreen, KindProbe, KindInput, KindBrowser}
// ComputerKinds move nothing: they carry long deadlines (DefaultComputerMS, at most MaxComputerMS) and no clamping.
// ComputerKinds move nothing: they carry long deadlines (DefaultComputerMS, at most MaxComputerMS) and no clamping.
-var ComputerKinds = map[string]bool{KindExec: true, KindFS: true, KindScreen: true, KindProbe: true, KindInput: true}
+var ComputerKinds = map[string]bool{KindExec: true, KindFS: true, KindScreen: true, KindProbe: true, KindInput: true, KindBrowser: true}
const (
const (
// DefaultComputerMS is a computer command's reply window when the server sends no deadline; MaxComputerMS caps
// DefaultComputerMS is a computer command's reply window when the server sends no deadline; MaxComputerMS caps
@@ -186,8 +189,8 @@ const (
@@ -186,8 +189,8 @@ const (
)
)
// ComputerActs reports whether a computer command changes the machine (runs a program, writes a file, uses the mouse
// ComputerActs reports whether a computer command changes the machine (runs a program, writes a file, uses the mouse
-// or keyboard): those are refused while the local stop or an e-stop is latched. Capturing the screen and reading files
-// are not.
+// or keyboard, drives the browser): those are refused while the local stop or an e-stop is latched. Capturing the
+// screen, reading files and reading or closing the browser are not.
func ComputerActs(kind string, value json.RawMessage) bool {
func ComputerActs(kind string, value json.RawMessage) bool {
switch kind {
switch kind {
case KindExec, KindInput:
case KindExec, KindInput:
@@ -198,6 +201,12 @@ func ComputerActs(kind string, value json.RawMessage) bool {
@@ -198,6 +201,12 @@ func ComputerActs(kind string, value json.RawMessage) bool {
}
}
_ = json.Unmarshal(value, &v)
_ = json.Unmarshal(value, &v)
return v.Op != "list" && v.Op != "stat" && v.Op != "read"
return v.Op != "list" && v.Op != "stat" && v.Op != "read"
+ case KindBrowser:
+ var v struct {
+ Action string `json:"action"`
+ }
+ _ = json.Unmarshal(value, &v)
+ return v.Action != "read" && v.Action != "screenshot" && v.Action != "close"
}
}
return false
return false
}
}
@@ -395,3 +395,15 @@ func TestJobUndecodableBody(t *testing.T) {
@@ -395,3 +395,15 @@ func TestJobUndecodableBody(t *testing.T) {
t.Fatalf("numeric id: %q", reason)
t.Fatalf("numeric id: %q", reason)
}
}
}
}
+
+func TestComputerActsBrowser(t *testing.T) {
+ want := map[string]bool{"open": true, "click": true, "fill": true, "press": true, "back": true, "read": false, "screenshot": false, "close": false}
+ for action, acts := range want {
+ if got := ComputerActs(KindBrowser, json.RawMessage(`{"action":"`+action+`"}`)); got != acts {
+ t.Errorf("browser %s: acts=%v, want %v", action, got, acts)
+ }
+ }
+ if !ComputerKinds[KindBrowser] {
+ t.Error("browser is a computer kind")
+ }
+}
@@ -26,6 +26,13 @@ Command kinds and their answers (the ack's ``data``):
@@ -26,6 +26,13 @@ Command kinds and their answers (the ack's ``data``):
``height`` (the ``screen`` answer) and are scaled to the real screen. An acting command: refused at observe, approved
``height`` (the ``screen`` answer) and are scaled to the real screen. An acting command: refused at observe, approved
one by one at ask. xdotool on X11, cliclick on macOS (no scroll there); none on Wayland yet. Config
one by one at ask. xdotool on X11, cliclick on macOS (no scroll there); none on Wayland yet. Config
``allow_input: false`` turns it off.
``allow_input: false`` turns it off.
+* ``browser`` ``{"action", "url"?, "selector"?, "text"?, "key"?, "max_width"?}``: a Chromium the agent drives, in a profile
+ of its own (``browser_profile``), over a pipe (no port is opened). ``open`` a ``http(s)`` URL, ``click`` a CSS
+ ``selector`` or the button or link whose ``text`` it shows, ``fill`` a field (``selector`` and ``text``), ``press`` a
+ ``key`` (Enter, Tab, Escape, arrows …), ``back`` → ``{url, title, text, truncated, links}``; ``read`` the page the same
+ way; ``screenshot`` → ``{format, width, height, content_b64}``; ``close``. Reading, screenshots and closing run at every
+ level; the rest are acting commands. Below level full, ``open`` refuses an address on the machine's own network.
+ Config ``allow_browser: false`` turns it off; ``browser_path``, ``browser_headless`` (default true).
* ``probe`` ``{"target", "timeout_s"?}`` → for ``http(s)://…`` one GET, no redirect followed: ``{status, content_type,
* ``probe`` ``{"target", "timeout_s"?}`` → for ``http(s)://…`` one GET, no redirect followed: ``{status, content_type,
etag, last_modified, body, truncated, ms}``; for ``tcp://host:port``: ``{open, ms, error?}``. Read-only, so it runs at
etag, last_modified, body, truncated, ms}``; for ``tcp://host:port``: ``{open, ms, error?}``. Read-only, so it runs at
every level, and only towards the machine's own network: every address the name resolves to must be private,
every level, and only towards the machine's own network: every address the name resolves to must be private,
@@ -50,6 +57,8 @@ import urllib.parse
@@ -50,6 +57,8 @@ import urllib.parse
from openvibe_plugin import Deferred, Fault, Plugin
from openvibe_plugin import Deferred, Fault, Plugin
+from .browser import KEY_CODES, Browser, BrowserError, find_binary
+
LEVELS = ("observe", "ask", "trusted", "full")
LEVELS = ("observe", "ask", "trusted", "full")
RANK = {name: i for i, name in enumerate(LEVELS)}
RANK = {name: i for i, name in enumerate(LEVELS)}
READ_OPS = frozenset({"list", "stat", "read"})
READ_OPS = frozenset({"list", "stat", "read"})
@@ -65,6 +74,10 @@ MAX_OUTPUT_CAP = 192 * 1024
@@ -65,6 +74,10 @@ MAX_OUTPUT_CAP = 192 * 1024
MAX_READ_CAP = 256 * 1024
MAX_READ_CAP = 256 * 1024
SCREEN_MAX_BYTES = 300 * 1024
SCREEN_MAX_BYTES = 300 * 1024
INPUT_ACTIONS = frozenset({"move", "click", "double_click", "right_click", "scroll", "type", "key"})
INPUT_ACTIONS = frozenset({"move", "click", "double_click", "right_click", "scroll", "type", "key"})
+BROWSER_READS = frozenset({"read", "screenshot", "close"})
+BROWSER_ACTS = frozenset({"open", "click", "fill", "press", "back"})
+MAX_URL = 2048
+MAX_SELECTOR = 500
POINT_ACTIONS = frozenset({"move", "click", "double_click", "right_click"})
POINT_ACTIONS = frozenset({"move", "click", "double_click", "right_click"})
MAX_TYPE = 2000
MAX_TYPE = 2000
KEYS_OK = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_+-")
KEYS_OK = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_+-")
@@ -87,11 +100,12 @@ def _int(v, name, lo, hi, default):
@@ -87,11 +100,12 @@ def _int(v, name, lo, hi, default):
class Computer(Plugin):
class Computer(Plugin):
driver = "computer"
driver = "computer"
- version = "0.4.0"
+ version = "0.5.0"
motion_kinds = frozenset()
motion_kinds = frozenset()
telemetry_interval_s = 2.0
telemetry_interval_s = 2.0
- def __init__(self, which=shutil.which, popen=subprocess.Popen, clock=time.monotonic, env=None, platform=None, screen_size=None):
+ def __init__(self, which=shutil.which, popen=subprocess.Popen, clock=time.monotonic, env=None, platform=None, screen_size=None,
+ browser_factory=None, resolve=None):
self.which = which
self.which = which
self.popen = popen
self.popen = popen
self.screen_size = screen_size or self._screen_size
self.screen_size = screen_size or self._screen_size
@@ -112,6 +126,16 @@ class Computer(Plugin):
@@ -112,6 +126,16 @@ class Computer(Plugin):
self.procs = set()
self.procs = set()
self.running = 0
self.running = 0
self.mu = threading.Lock()
self.mu = threading.Lock()
+ self.allow_browser = True
+ self.browser_path = None
+ self.browser_profile = None
+ self.browser_headless = True
+ self.browser_idle_s = 600
+ self.browser = None
+ self.browser_mu = threading.Lock()
+ self.browser_op_mu = threading.Lock()
+ self.browser_factory = browser_factory or (lambda binary, profile, headless: Browser(binary, profile, headless))
+ self.resolve = resolve or (lambda host: [ai[4][0] for ai in socket.getaddrinfo(host, None)])
# ---- configuration -------------------------------------------------------------------------------------------
# ---- configuration -------------------------------------------------------------------------------------------
def _configure(self, config):
def _configure(self, config):
@@ -133,6 +157,11 @@ class Computer(Plugin):
@@ -133,6 +157,11 @@ class Computer(Plugin):
self.allow_probe = config.get("probe", True) is not False
self.allow_probe = config.get("probe", True) is not False
self.allow_input = config.get("allow_input", True) is not False
self.allow_input = config.get("allow_input", True) is not False
self.notify = config.get("notify", True) is not False
self.notify = config.get("notify", True) is not False
+ self.allow_browser = config.get("allow_browser", True) is not False
+ self.browser_path = config.get("browser_path") or None
+ self.browser_profile = os.path.expanduser(config.get("browser_profile") or "~/.local/share/openvibe-node/browser")
+ self.browser_headless = config.get("browser_headless", True) is not False
+ self.browser_idle_s = _int(config.get("browser_idle_s"), "browser_idle_s", 60, 86400, 600)
def describe(self, config):
def describe(self, config):
try:
try:
@@ -150,6 +179,9 @@ class Computer(Plugin):
@@ -150,6 +179,9 @@ class Computer(Plugin):
itool = self._input_tool() if self.allow_input else None
itool = self._input_tool() if self.allow_input else None
if itool:
if itool:
caps["input"] = {"level": self.level, "tool": itool, "scroll": itool == "xdotool"}
caps["input"] = {"level": self.level, "tool": itool, "scroll": itool == "xdotool"}
+ binary = self._browser_binary()
+ if binary:
+ caps["browser"] = {"level": self.level, "browser": os.path.basename(binary), "headless": self.browser_headless}
return caps
return caps
def setup(self, config):
def setup(self, config):
@@ -172,6 +204,7 @@ class Computer(Plugin):
@@ -172,6 +204,7 @@ class Computer(Plugin):
self.pending.clear()
self.pending.clear()
for p in procs:
for p in procs:
self._kill(p)
self._kill(p)
+ self._close_browser()
for cid in waiting:
for cid in waiting:
self.emit_event("approval_done", id=cid, approved=False, reason="stopped")
self.emit_event("approval_done", id=cid, approved=False, reason="stopped")
self.fail(cid, "local_stop", "stopped on the machine before it ran")
self.fail(cid, "local_stop", "stopped on the machine before it ran")
@@ -187,6 +220,9 @@ class Computer(Plugin):
@@ -187,6 +220,9 @@ class Computer(Plugin):
def poll(self):
def poll(self):
now = self.clock()
now = self.clock()
+ b = self.browser
+ if b is not None and b.alive() and b.used_at and time.monotonic() - b.used_at > self.browser_idle_s:
+ self._close_browser()
expired = []
expired = []
with self.mu:
with self.mu:
for cid, (_, at, _) in list(self.pending.items()):
for cid, (_, at, _) in list(self.pending.items()):
@@ -253,6 +289,11 @@ class Computer(Plugin):
@@ -253,6 +289,11 @@ class Computer(Plugin):
raise Fault("unsupported", "no input tool on this machine (xdotool on X11, cliclick on macOS)")
raise Fault("unsupported", "no input tool on this machine (xdotool on X11, cliclick on macOS)")
run, summary = self._input_op(tool, v)
run, summary = self._input_op(tool, v)
return self._act(cmd, level, run, summary, threaded=True)
return self._act(cmd, level, run, summary, threaded=True)
+ if cmd.kind == "browser":
+ run, summary, acting = self._browser_op(v, level)
+ if acting:
+ return self._act(cmd, level, run, summary, threaded=True)
+ return self._start(cmd, run)
if cmd.kind == "exec":
if cmd.kind == "exec":
if not self.allow_exec:
if not self.allow_exec:
raise Fault("unsupported", "this machine does not allow running programs")
raise Fault("unsupported", "this machine does not allow running programs")
@@ -605,6 +646,112 @@ class Computer(Plugin):
@@ -605,6 +646,112 @@ class Computer(Plugin):
return run, "%s %s (%d bytes)" % (verb, path, len(data))
return run, "%s %s (%d bytes)" % (verb, path, len(data))
# ---- input -----------------------------------------------------------------------------------------------------
# ---- input -----------------------------------------------------------------------------------------------------
+ # ---- browser ---------------------------------------------------------------------------------------------------
+ def _browser_binary(self):
+ if not self.allow_browser or self.platform.startswith("win"):
+ return None
+ return find_binary(self.which, self.platform, self.browser_path)
+
+ def _close_browser(self):
+ # Never waits for an action in flight: closing kills the process, and the action fails at once.
+ b, self.browser = self.browser, None
+ if b is not None:
+ b.close()
+
+ def _local_host(self, host):
+ """Does `host` name (or resolve to) an address on the machine's own network?"""
+ try:
+ addrs = [host] if self._is_ip(host) else self.resolve(host)
+ except (OSError, UnicodeError):
+ raise Fault("bad_value", "%s does not resolve" % host)
+ return any(self._lan(a) for a in addrs) or not addrs
+
+ @staticmethod
+ def _is_ip(host):
+ try:
+ ipaddress.ip_address(host)
+ return True
+ except ValueError:
+ return False
+
+ def _browser_url(self, raw, level):
+ if not isinstance(raw, str) or not raw or len(raw) > MAX_URL or any(c in raw for c in "\x00\r\n"):
+ raise Fault("bad_value", "url is an http or https address")
+ u = urllib.parse.urlsplit(raw)
+ if u.scheme not in ("http", "https") or not u.hostname:
+ raise Fault("bad_value", "url is an http or https address")
+ if level != "full" and self._local_host(u.hostname.strip("[]")):
+ raise Fault("not_allowed", "%s is on this machine's own network: only at level full" % u.hostname)
+ return raw
+
+ def _browser_op(self, v, level):
+ """(run, summary, acting) for one browser action; nothing is opened or clicked before run()."""
+ binary = self._browser_binary()
+ if not binary:
+ raise Fault("unsupported", "no browser on this machine (Chromium, Chrome, Edge or Brave), or this machine does not allow it")
+ action = v.get("action")
+ if action not in BROWSER_READS and action not in BROWSER_ACTS:
+ raise Fault("bad_value", "action is one of %s" % ", ".join(sorted(BROWSER_READS | BROWSER_ACTS)))
+ selector, text, key, url = v.get("selector"), v.get("text"), v.get("key"), None
+ if action == "open":
+ url = self._browser_url(v.get("url"), level)
+ if action in ("click", "fill") and selector is not None and (not isinstance(selector, str) or not selector or len(selector) > MAX_SELECTOR):
+ raise Fault("bad_value", "selector is a CSS selector of at most %d characters" % MAX_SELECTOR)
+ if action == "click" and selector is None and (not isinstance(text, str) or not text.strip() or len(text) > 200):
+ raise Fault("bad_value", "click takes a selector, or the text of a button or link (at most 200 characters)")
+ if action == "fill":
+ if selector is None:
+ raise Fault("bad_value", "fill takes the selector of the field")
+ if not isinstance(text, str) or len(text) > MAX_TYPE or "\x00" in text:
+ raise Fault("bad_value", "text is at most %d characters" % MAX_TYPE)
+ if action == "press" and key not in KEY_CODES:
+ raise Fault("bad_value", "key is one of %s" % ", ".join(sorted(KEY_CODES)))
+ width = _int(v.get("max_width"), "max_width", 320, 2560, 1280)
+
+ def browser():
+ with self.browser_mu:
+ if self.browser is None or not self.browser.alive():
+ self.browser = self.browser_factory(binary, self.browser_profile, self.browser_headless)
+ return self.browser
+
+ def run():
+ if action == "close":
+ self._close_browser()
+ return {"done": True}
+ b = browser()
+ try:
+ with self.browser_op_mu: # one action at a time in the one browser
+ if action == "open":
+ return b.open(url)
+ if action == "read":
+ return b.read()
+ if action == "screenshot":
+ return b.screenshot(width)
+ if action == "click":
+ return b.click(selector=selector, text=None if selector else text)
+ if action == "fill":
+ return b.fill(selector, text)
+ if action == "press":
+ return b.press(key)
+ return b.back()
+ except BrowserError as e:
+ raise Fault(e.code, e.message)
+
+ if action == "open":
+ summary = "open %s in the browser" % url[:300]
+ elif action == "click":
+ summary = "click %s in the browser" % (selector if selector else "the %r button or link" % text[:120])
+ elif action == "fill":
+ shown = text if len(text) <= 120 else text[:120] + "…"
+ summary = "type %d characters into %s in the browser: %r" % (len(text), selector[:200], shown)
+ elif action == "press":
+ summary = "press %s in the browser" % key
+ elif action == "back":
+ summary = "go back in the browser"
+ else:
+ summary = "%s the browser" % action
+ return run, summary, action in BROWSER_ACTS
+
def _input_tool(self):
def _input_tool(self):
"""xdotool on an X11 session, cliclick on macOS, else None (Wayland has no general input tool yet)."""
"""xdotool on an X11 session, cliclick on macOS, else None (Wayland has no general input tool yet)."""
if self.platform == "darwin" and self.which("cliclick"):
if self.platform == "darwin" and self.which("cliclick"):
new file mode 100644
new file mode 100644
@@ -0,0 +1,342 @@
@@ -0,0 +1,342 @@
+"""The computer plugin's browser: one Chromium the agent drives, in a profile of its own (plan T14, the browser kind).
+
+Chromium is started on the first command with ``--remote-debugging-pipe``: it reads Chrome DevTools Protocol commands
+from file descriptor 3 and writes answers and events to 4, each message JSON ended by a NUL byte. No port is opened,
+so nothing else on the machine can drive it. The profile is a folder of its own (``browser_profile``, by default
+``~/.local/share/openvibe-node/browser``): the agent never sees the owner's own browser sessions unless the owner points
+that setting at one.
+
+ Browser(binary, profile, headless).open(url) / read() / click(selector|text) / fill(selector, text) / press(key)
+ / back() / screenshot(max_width) / close()
+
+Every page answer is ``{url, title, text, links}``: the page's visible text (cut at ``MAX_TEXT``) and its first links.
+"""
+
+import base64
+import json
+import os
+import subprocess
+import threading
+import time
+
+MAX_TEXT = 20000
+STARTUP_S = 45.0
+MAX_LINKS = 60
+KEY_CODES = {
+ "Enter": (13, "\r"), "Tab": (9, "\t"), "Escape": (27, ""), "Backspace": (8, ""), "Delete": (46, ""),
+ "ArrowUp": (38, ""), "ArrowDown": (40, ""), "ArrowLeft": (37, ""), "ArrowRight": (39, ""),
+ "PageUp": (33, ""), "PageDown": (34, ""), "Home": (36, ""), "End": (35, ""), "Space": (32, " "),
+}
+CANDIDATES = ("chromium", "chromium-browser", "google-chrome", "google-chrome-stable", "microsoft-edge", "brave-browser")
+MAC_APPS = ("/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
+ "/Applications/Chromium.app/Contents/MacOS/Chromium",
+ "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge",
+ "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser")
+
+SUMMARY_JS = """(() => {
+ const text = (document.body ? document.body.innerText : '').replace(/\\n{3,}/g, '\\n\\n');
+ const links = [...document.querySelectorAll('a[href]')]
+ .filter((a) => /^https?:/.test(a.href))
+ .slice(0, %d)
+ .map((a) => ({ text: (a.innerText || a.getAttribute('aria-label') || '').trim().slice(0, 100), href: a.href }));
+ return { url: location.href, title: document.title, text: text.slice(0, %d), truncated: text.length > %d, links };
+})()""" % (MAX_LINKS, MAX_TEXT, MAX_TEXT)
+
+CLICK_JS = """((selector, text) => {
+ let el = null;
+ if (selector) el = document.querySelector(selector);
+ else {
+ const want = text.trim().toLowerCase();
+ const items = [...document.querySelectorAll('a, button, [role=button], input[type=submit], input[type=button], summary, label')];
+ el = items.find((e) => (e.innerText || e.value || e.getAttribute('aria-label') || '').trim().toLowerCase() === want)
+ || items.find((e) => (e.innerText || e.value || e.getAttribute('aria-label') || '').trim().toLowerCase().includes(want));
+ }
+ if (!el) return false;
+ el.scrollIntoView({ block: 'center' });
+ el.click();
+ return true;
+})(%s, %s)"""
+
+FOCUS_JS = """((selector) => {
+ const el = document.querySelector(selector);
+ if (!el) return false;
+ el.scrollIntoView({ block: 'center' });
+ el.focus();
+ if ('value' in el) { el.value = ''; el.dispatchEvent(new Event('input', { bubbles: true })); }
+ else if (el.isContentEditable) el.textContent = '';
+ return true;
+})(%s)"""
+
+
+class BrowserError(Exception):
+ def __init__(self, code, message):
+ super().__init__(message)
+ self.code = code
+ self.message = message
+
+
+def find_binary(which, platform, configured=None):
+ """The Chromium-family browser to drive: the configured path, else the first one on PATH (or a macOS app)."""
+ if configured:
+ return configured if os.path.isfile(configured) and os.access(configured, os.X_OK) else None
+ for name in CANDIDATES:
+ path = which(name)
+ if path:
+ return path
+ if platform == "darwin":
+ for path in MAC_APPS:
+ if os.path.isfile(path):
+ return path
+ return None
+
+
+class Browser:
+ def __init__(self, binary, profile, headless=True, timeout_s=15.0, popen=subprocess.Popen, extra_args=()):
+ self.binary = binary
+ self.extra_args = list(extra_args)
+ self.stderr_tail = []
+ self.profile = profile
+ self.headless = headless
+ self.timeout_s = timeout_s
+ self.popen = popen
+ self.proc = None
+ self.out = None
+ self.inp = None
+ self.next_id = 0
+ self.waiting = {} # message id -> [event, result]
+ self.events = [] # (method, params) seen since the last mark
+ self.mu = threading.Lock()
+ self.session = None
+ self.used_at = 0.0
+
+ # ---- the process and the pipe ------------------------------------------------------------------------------------
+ def alive(self):
+ return self.proc is not None and self.proc.poll() is None
+
+ def start(self):
+ if self.alive():
+ return
+ os.makedirs(self.profile, mode=0o700, exist_ok=True)
+ to_chrome_r, to_chrome_w = os.pipe()
+ from_chrome_r, from_chrome_w = os.pipe()
+ argv = [self.binary, "--remote-debugging-pipe", "--user-data-dir=%s" % self.profile, "--no-first-run",
+ "--no-default-browser-check", "--disable-background-networking", "--disable-sync",
+ "--disable-component-update", "--password-store=basic", "--window-size=1280,900", "about:blank"]
+ if self.headless:
+ argv.insert(1, "--headless=new")
+ argv[1:1] = self.extra_args
+
+ def fds():
+ # In the child: Chromium reads commands on 3 and writes on 4. Both ends move above 10 first, so placing one
+ # never closes the other; 3 and 4 are in pass_fds, so the close that follows this function keeps them.
+ import fcntl
+ r = fcntl.fcntl(to_chrome_r, fcntl.F_DUPFD, 10)
+ w = fcntl.fcntl(from_chrome_w, fcntl.F_DUPFD, 10)
+ os.dup2(r, 3)
+ os.dup2(w, 4)
+
+ self.proc = self.popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE,
+ pass_fds=(3, 4), preexec_fn=fds, start_new_session=True)
+ threading.Thread(target=self._stderr, args=(self.proc.stderr,), name="computer-browser-err", daemon=True).start()
+ os.close(to_chrome_r)
+ os.close(from_chrome_w)
+ self.out = os.fdopen(to_chrome_w, "wb", buffering=0)
+ self.inp = os.fdopen(from_chrome_r, "rb", buffering=0)
+ threading.Thread(target=self._reader, name="computer-browser", daemon=True).start()
+ # A first start on a fresh profile can take a while on a slow machine: the first command gets a longer window.
+ target = self.call("Target.createTarget", {"url": "about:blank"}, session=False, timeout_s=max(self.timeout_s, STARTUP_S))["targetId"]
+ self.session = self.call("Target.attachToTarget", {"targetId": target, "flatten": True}, session=False)["sessionId"]
+ self.call("Page.enable")
+ self.call("Runtime.enable")
+
+ def _stderr(self, stream):
+ """Keep the browser's last lines of stderr: why it closed, when it does."""
+ for line in iter(stream.readline, b""):
+ self.stderr_tail = (self.stderr_tail + [line.decode("utf-8", "replace").strip()])[-5:]
+
+ def _reader(self):
+ buf = b""
+ while True:
+ try:
+ chunk = self.inp.read(65536)
+ except (OSError, ValueError):
+ chunk = b""
+ if not chunk:
+ break
+ buf += chunk
+ while b"\0" in buf:
+ raw, buf = buf.split(b"\0", 1)
+ try:
+ msg = json.loads(raw.decode("utf-8"))
+ except ValueError:
+ continue
+ if "id" in msg:
+ with self.mu:
+ slot = self.waiting.get(msg["id"])
+ if slot is not None:
+ slot[1] = msg
+ slot[0].set()
+ elif "method" in msg:
+ with self.mu:
+ self.events.append(msg["method"])
+ if len(self.events) > 500:
+ del self.events[:250]
+ with self.mu:
+ for slot in self.waiting.values():
+ slot[0].set()
+
+ def call(self, method, params=None, session=True, timeout_s=None):
+ with self.mu:
+ self.next_id += 1
+ mid = self.next_id
+ slot = [threading.Event(), None]
+ self.waiting[mid] = slot
+ msg = {"id": mid, "method": method, "params": params or {}}
+ if session and self.session:
+ msg["sessionId"] = self.session
+ try:
+ self.out.write(json.dumps(msg).encode("utf-8") + b"\0")
+ except (OSError, ValueError, AttributeError):
+ raise BrowserError("hardware", "the browser is not running")
+ ok = slot[0].wait(timeout_s or self.timeout_s)
+ with self.mu:
+ self.waiting.pop(mid, None)
+ if not ok:
+ raise BrowserError("expired", "the browser did not answer %s in time" % method)
+ if slot[1] is None:
+ why = " | ".join(x for x in self.stderr_tail if x)[-400:]
+ raise BrowserError("hardware", "the browser closed" + (": %s" % why if why else ""))
+ if "error" in slot[1]:
+ raise BrowserError("bad_value", "%s: %s" % (method, slot[1]["error"].get("message", "refused")))
+ return slot[1].get("result", {})
+
+ def _mark(self):
+ with self.mu:
+ self.events = []
+
+ def _wait_for(self, methods, timeout_s):
+ end = time.monotonic() + timeout_s
+ while time.monotonic() < end:
+ with self.mu:
+ if any(m in methods for m in self.events):
+ return True
+ time.sleep(0.05)
+ return False
+
+ def _eval(self, expression):
+ r = self.call("Runtime.evaluate", {"expression": expression, "returnByValue": True, "awaitPromise": True})
+ if r.get("exceptionDetails"):
+ raise BrowserError("bad_value", "the page refused: %s" % r["exceptionDetails"].get("text", "an exception"))
+ return r.get("result", {}).get("value")
+
+ def _settle(self, navigated_timeout=2.0):
+ """After an action: if it started a navigation, wait for the load; then a short pause for scripts."""
+ if self._wait_for(("Page.frameStartedLoading", "Page.frameNavigated"), navigated_timeout):
+ self._wait_for(("Page.loadEventFired",), self.timeout_s)
+ time.sleep(0.3)
+
+ # ---- the actions -----------------------------------------------------------------------------------------------
+ def summary(self):
+ self.used_at = time.monotonic()
+ return self._eval(SUMMARY_JS)
+
+ def open(self, url):
+ self.start()
+ self._mark()
+ r = self.call("Page.navigate", {"url": url})
+ if r.get("errorText"):
+ raise BrowserError("hardware", "could not open %s: %s" % (url, r["errorText"]))
+ self._wait_for(("Page.loadEventFired",), self.timeout_s)
+ time.sleep(0.3)
+ return self.summary()
+
+ def read(self):
+ self._need()
+ return self.summary()
+
+ def click(self, selector=None, text=None):
+ self._need()
+ self._mark()
+ if not self._eval(CLICK_JS % (json.dumps(selector), json.dumps(text or ""))):
+ raise BrowserError("bad_value", "nothing on the page matches %s" % (selector or repr(text)))
+ self._settle()
+ return self.summary()
+
+ def fill(self, selector, text):
+ self._need()
+ if not self._eval(FOCUS_JS % json.dumps(selector)):
+ raise BrowserError("bad_value", "nothing on the page matches %s" % selector)
+ self.call("Input.insertText", {"text": text})
+ return {"done": True, "selector": selector, "chars": len(text)}
+
+ def press(self, key):
+ self._need()
+ code, text = KEY_CODES[key]
+ self._mark()
+ base = {"key": key, "code": key if key != "Space" else "Space", "windowsVirtualKeyCode": code, "nativeVirtualKeyCode": code}
+ self.call("Input.dispatchKeyEvent", dict(base, type="keyDown", **({"text": text} if text else {})))
+ self.call("Input.dispatchKeyEvent", dict(base, type="keyUp"))
+ self._settle()
+ return self.summary()
+
+ def back(self):
+ self._need()
+ self._mark()
+ self._eval("history.back()")
+ self._settle()
+ return self.summary()
+
+ def screenshot(self, max_width=1280):
+ self._need()
+ self.used_at = time.monotonic()
+ r = self.call("Page.captureScreenshot", {"format": "jpeg", "quality": 70})
+ data = base64.b64decode(r["data"])
+ metrics = self.call("Page.getLayoutMetrics")
+ view = metrics.get("cssVisualViewport") or metrics.get("visualViewport") or {}
+ width, height = int(view.get("clientWidth", 1280)), int(view.get("clientHeight", 900))
+ if width > max_width:
+ try:
+ from PIL import Image
+ import io
+ img = Image.open(io.BytesIO(data))
+ ratio = max_width / float(img.width)
+ img = img.resize((max_width, max(1, int(img.height * ratio))))
+ out = io.BytesIO()
+ img.convert("RGB").save(out, "JPEG", quality=70)
+ data, width, height = out.getvalue(), img.width, img.height
+ except Exception:
+ pass
+ return {"format": "jpeg", "width": width, "height": height, "content_b64": base64.b64encode(data).decode("ascii")}
+
+ def _need(self):
+ if not self.alive() or not self.session:
+ raise BrowserError("not_ready", "no page is open: open a URL first")
+
+ def close(self):
+ proc, self.proc, self.session = self.proc, None, None
+ for f in (self.out, self.inp):
+ try:
+ if f:
+ f.close()
+ except OSError:
+ pass
+ self.out = self.inp = None
+ if proc is not None and proc.poll() is None:
+ try:
+ if hasattr(os, "killpg"):
+ os.killpg(proc.pid, 9)
+ else:
+ proc.kill()
+ except (OSError, ProcessLookupError):
+ pass
+ if proc is not None:
+ try:
+ proc.wait(timeout=5)
+ except Exception:
+ pass
+ if proc.stderr:
+ try:
+ proc.stderr.close()
+ except OSError:
+ pass
+ return {"done": True, "closed": proc is not None}
new file mode 100644
new file mode 100644
@@ -0,0 +1,209 @@
@@ -0,0 +1,209 @@
+import http.server
+import os
+import shutil
+import socketserver
+import sys
+import threading
+import unittest
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(HERE))
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(HERE)), "sdk"))
+sys.path.insert(0, HERE)
+
+from openvibe_computer import Computer # noqa: E402
+from openvibe_computer.browser import Browser, find_binary # noqa: E402
+from openvibe_plugin import Runtime # noqa: E402
+from test_computer import Out, reply # noqa: E402
+
+
+class FakeBrowser:
+ """What the plugin asks of a browser, recorded; pages are made up."""
+
+ def __init__(self, binary, profile, headless):
+ self.binary, self.profile, self.headless = binary, profile, headless
+ self.calls = []
+ self.open_ = True
+ self.used_at = 0
+
+ def alive(self):
+ return self.open_
+
+ def _page(self, url="https://example.org/"):
+ return {"url": url, "title": "Example", "text": "Example page", "truncated": False, "links": []}
+
+ def open(self, url):
+ self.calls.append(("open", url))
+ return self._page(url)
+
+ def read(self):
+ self.calls.append(("read",))
+ return self._page()
+
+ def click(self, selector=None, text=None):
+ self.calls.append(("click", selector, text))
+ return self._page()
+
+ def fill(self, selector, text):
+ self.calls.append(("fill", selector, text))
+ return {"done": True, "selector": selector, "chars": len(text)}
+
+ def press(self, key):
+ self.calls.append(("press", key))
+ return self._page()
+
+ def back(self):
+ self.calls.append(("back",))
+ return self._page()
+
+ def screenshot(self, max_width):
+ self.calls.append(("screenshot", max_width))
+ return {"format": "jpeg", "width": max_width, "height": 10, "content_b64": ""}
+
+ def close(self):
+ self.open_ = False
+ self.calls.append(("close",))
+ return {"done": True}
+
+
+PUBLIC = {"example.org": ["93.184.215.14"], "router.lan": ["192.168.1.1"], "sneaky.example": ["93.184.215.14", "10.0.0.5"]}
+
+
+class BrowserKindTest(unittest.TestCase):
+ def boot(self, level, **config):
+ self.made = []
+
+ def factory(binary, profile, headless):
+ b = FakeBrowser(binary, profile, headless)
+ self.made.append(b)
+ return b
+
+ self.out = Out()
+ self.plugin = Computer(env={}, platform="linux", which=lambda n: "/usr/bin/chromium" if n == "chromium" else None,
+ browser_factory=factory, resolve=lambda h: PUBLIC.get(h, []))
+ self.rt = Runtime(self.plugin, infile=__import__("io").StringIO(""), outfile=self.out)
+ self.rt.dispatch({"op": "hello", "config": {"level": level, "roots": [HERE], **config}})
+ self.n = 0
+ return self.out.msgs()
+
+ def cmd(self, value):
+ self.n += 1
+ cid = "b%d" % self.n
+ self.rt.dispatch({"op": "command", "id": cid, "kind": "browser", "value": value, "deadline_ms": 30000})
+ return cid
+
+ def test_described_with_its_level(self):
+ caps = self.boot("ask")[0]["capabilities"]
+ self.assertEqual(caps["browser"], {"level": "ask", "browser": "chromium", "headless": True})
+ caps = self.boot("ask", allow_browser=False)[0]["capabilities"]
+ self.assertNotIn("browser", caps)
+
+ def test_reading_runs_at_observe_and_acting_does_not(self):
+ self.boot("observe")
+ r = reply(self.out, self.cmd({"action": "open", "url": "https://example.org/"}))
+ self.assertEqual((r["op"], r["fault_code"]), ("nack", "not_allowed"))
+ self.assertEqual(self.made, [], "nothing is started for a refused command")
+ r = reply(self.out, self.cmd({"action": "read"}))
+ self.assertEqual(r["op"], "ack", "reading the agent's page is observing")
+ self.assertEqual(self.made[0].calls, [("read",)])
+
+ def test_ask_waits_for_the_owner_and_shows_what_is_typed(self):
+ self.boot("ask")
+ cid = self.cmd({"action": "fill", "selector": "#q", "text": "hello"})
+ ev = self.out.wait_for(lambda m: m.get("op") == "event" and m.get("name") == "approval")
+ self.assertEqual((ev["kind"], ev["summary"]), ("browser", "type 5 characters into #q in the browser: 'hello'"))
+ self.assertEqual(self.made, [], "nothing happens before the owner approves")
+ self.plugin.on_approval(cid, True)
+ self.assertEqual(reply(self.out, cid)["data"], {"done": True, "selector": "#q", "chars": 5})
+ cid = self.cmd({"action": "click", "text": "Sign in"})
+ ev = self.out.wait_for(lambda m: m.get("op") == "event" and m.get("name") == "approval" and m.get("id") == cid)
+ self.assertEqual(ev["summary"], "click the 'Sign in' button or link in the browser")
+ self.plugin.on_approval(cid, False)
+ self.assertEqual(reply(self.out, cid)["fault_code"], "not_allowed")
+
+ def test_trusted_drives_it_and_the_profile_is_its_own(self):
+ self.boot("trusted")
+ self.assertEqual(reply(self.out, self.cmd({"action": "open", "url": "https://example.org/a"}))["data"]["url"], "https://example.org/a")
+ reply(self.out, self.cmd({"action": "click", "selector": "a.next"}))
+ reply(self.out, self.cmd({"action": "press", "key": "Enter"}))
+ reply(self.out, self.cmd({"action": "back"}))
+ reply(self.out, self.cmd({"action": "screenshot", "max_width": 640}))
+ b = self.made[0]
+ self.assertEqual(len(self.made), 1, "one browser for the session")
+ self.assertEqual(b.calls, [("open", "https://example.org/a"), ("click", "a.next", None), ("press", "Enter"), ("back",), ("screenshot", 640)])
+ self.assertTrue(b.profile.endswith(os.path.join("openvibe-node", "browser")), b.profile)
+
+ def test_addresses_and_values_it_refuses(self):
+ self.boot("trusted")
+ for value, code in [
+ ({"action": "open", "url": "file:///etc/passwd"}, "bad_value"),
+ ({"action": "open", "url": "javascript:alert(1)"}, "bad_value"),
+ ({"action": "open", "url": "http://192.168.1.1/"}, "not_allowed"),
+ ({"action": "open", "url": "http://router.lan/"}, "not_allowed"),
+ ({"action": "open", "url": "https://sneaky.example/"}, "not_allowed"),
+ ({"action": "open", "url": "http://[::1]:8080/"}, "not_allowed"),
+ ({"action": "press", "key": "ctrl+alt+Delete"}, "bad_value"),
+ ({"action": "fill", "text": "x"}, "bad_value"),
+ ({"action": "click"}, "bad_value"),
+ ({"action": "eval", "script": "1"}, "bad_value"),
+ ]:
+ r = reply(self.out, self.cmd(value))
+ self.assertEqual((r["op"], r["fault_code"]), ("nack", code), value)
+ self.boot("full")
+ self.assertEqual(reply(self.out, self.cmd({"action": "open", "url": "http://router.lan/"}))["op"], "ack", "full reaches the local network")
+
+ def test_the_kill_switch_closes_it(self):
+ self.boot("trusted")
+ reply(self.out, self.cmd({"action": "open", "url": "https://example.org/"}))
+ self.plugin.stop()
+ self.assertFalse(self.made[0].alive())
+ self.assertEqual(reply(self.out, self.cmd({"action": "close"}))["data"], {"done": True})
+
+
+class Handler(http.server.SimpleHTTPRequestHandler):
+ PAGES = {
+ "/": b"<!doctype html><title>Home</title><h1>Hello</h1><a href='/two'>Second page</a>"
+ b"<form action='/two'><input id='q' name='q'><button>Search</button></form>",
+ "/two": b"<!doctype html><title>Two</title><p>Arrived</p>",
+ }
+
+ def do_GET(self):
+ body = self.PAGES.get(self.path.split("?")[0], b"not found")
+ self.send_response(200)
+ self.send_header("Content-Type", "text/html")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def log_message(self, *a):
+ pass
+
+
[email protected](find_binary(shutil.which, sys.platform), "no Chromium-family browser on this machine")
+class RealBrowserTest(unittest.TestCase):
+ def test_a_real_browser_over_the_pipe(self):
+ srv = socketserver.TCPServer(("127.0.0.1", 0), Handler)
+ threading.Thread(target=srv.serve_forever, daemon=True).start()
+ self.addCleanup(srv.shutdown)
+ base = "http://127.0.0.1:%d" % srv.server_address[1]
+ profile = __import__("tempfile").mkdtemp(prefix="ov-browser-")
+ self.addCleanup(lambda: shutil.rmtree(profile, ignore_errors=True))
+ # CI runners (Ubuntu 24.04) refuse the unprivileged user namespaces Chrome's sandbox needs; a person's machine does
+ # not, and the plugin never turns the sandbox off.
+ b = Browser(find_binary(shutil.which, sys.platform), profile, extra_args=["--no-sandbox"] if os.environ.get("CI") else [])
+ self.addCleanup(b.close)
+ page = b.open(base + "/")
+ self.assertEqual((page["title"], [l["text"] for l in page["links"]]), ("Home", ["Second page"]))
+ self.assertEqual(b.click(text="Second page")["title"], "Two")
+ self.assertEqual(b.back()["title"], "Home")
+ self.assertEqual(b.fill("#q", "two words")["chars"], 9)
+ self.assertTrue(b.press("Enter")["url"].endswith("/two?q=two+words"))
+ shot = b.screenshot(640)
+ self.assertEqual((shot["format"], shot["width"] <= 1280), ("jpeg", True))
+ self.assertTrue(shot["content_b64"])
+ b.close()
+ self.assertFalse(b.alive())
+
+
+if __name__ == "__main__":
+ unittest.main()