openvibers/openvibe.link Commits

v0.24.0: kept desk profiles (sign in once: a desk starts from the person's kept browser profile and saves it back when it stops), and the browser follows a popup or new tab its page opens and comes back when it closes; the agent's clicks count as a person's (#70)

OP OpenVibers committed 17 days ago · db68fb741c · 11 files, +393 −24

Changes

ddiff --git a/STATUS.json b/STATUS.json
iindex b01743f..c707128 100644
---- a/STATUS.json
++++ b/STATUS.json
@@@ -5,8 +5,8 @@
"wave": "T15",
"plan": "Plan track T15 (Bot complete) and ADR-043 (Bot: devices, pairing, control and safety); the computer plugin and the job worker serve OpenVibe.Actor and the Codes job runner (tracks T14/T17)",
"kind": "client",
-- "version": "0.23.0",
-- "release": "v0.23.0, published 2026-10-27 as a GitHub release (browser desks, the Desks plan's step D2: desk start takes kind browser, a headless browser alone with no display at about half the cost of a whole desk, on Linux or macOS; its page is its screen, mouse and keyboard and live view over the browser's pipe, exactly the desk's size; a desk host, which has no screen or input of its own, gets screen, screen_stream and input on its desks; a click on a named desk scales to that desk's size; v0.22.0: desks on a desk host, found by the first production proof: a desk's temp folder is short (Chrome's SingletonSocket must fit a Unix socket path, and the desks folder on a desk host was too deep, so Chrome closed as it started), and a desk no command used for desk_idle_s (15 minutes) stops by itself, so a desk an agent left behind when it crashed or restarted ends; v0.21.0: element refs and receipts, the Desks plan's step D3: the agent's browser lists a page's interactive elements as compact refs (e7 button \"Sign in\"), kept on the element while it stays, and click, press and back answer with what changed (url, lines that appeared, lines gone), so agents act on the page's structure before screenshots; v0.20.0: desk hosts, the Desks plan's step D5: install.sh --desk-host turns an old laptop, a mini PC or a spare workstation into a host of desks for its owner's agents, run as its own openvibe-desk account with Xvfb, openbox, D-Bus, xdotool, ImageMagick and Chromium installed and max_desks sized to its memory; v0.19.0: several desks per machine, the Desks plan's step D1: the computer plugin runs named desks (desk-<name>, private kept), each with its own X display, D-Bus, browser profile, downloads and temp folder, up to max_desks (one per 2 GiB of memory by default), so two agents work at once without seeing or driving each other's screen; proven with two real Chrome desks that never share a cookie; v0.18.0: a motion command's validity also counts the time it waits in the queue to its plugin and in the pipe: the core fixes the instant it expires at receipt, writes what is left at the moment it writes the line with expires_at_mono_ns on the clock it shares with the plugin, and never writes one that has run out, and the Python runtime refuses one that ran out in the pipe before handle(); the clock bound is kept in nanoseconds and counts Bot's whole-millisecond stamps, rounded up from nanoseconds; a Node booting latched sets the e-stop on its plugins before starting them; the Python SDK package reports the Node release; v0.17.0: deadlines are checked against the latest the server's clock can be: the estimate plus its bound, the connection's fastest round trip, and motion waits for that bound on every new connection; every stop and estop names its cause, hello carries the newest halt epoch, and the Python SDK gives drivers on_stop(StopInfo) and on_resume to tell an ordinary stop from an emergency without re-arming anything; v0.16.1: the Python runtime refuses a motion command whose deadline passed while it waited behind a slow handler, so a driver never moves on a stale command; plugins.md answers where deadlines come from, re-stamping, order across a change of driver, stop versus estop, and that nothing is held past its validity; v0.16.0: openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS",
++ "version": "0.24.0",
++ "release": "v0.24.0, published 2026-10-27 as a GitHub release (kept desk profiles: desk start takes profile, so a desk's browser starts from the person's kept profile (sign in once, on any desk) and saves it back when it stops, closing the browser gracefully so its cookies are written, the last desk to stop wins, caches never kept, and each desk's own copy removed; the browser follows a page its page opens (a sign-in popup, a new tab) and comes back when it closes; the agent's clicks count as a person's, so sites open their sign-in popups; v0.23.0: browser desks, the Desks plan's step D2: desk start takes kind browser, a headless browser alone with no display at about half the cost of a whole desk, on Linux or macOS; its page is its screen, mouse and keyboard and live view over the browser's pipe, exactly the desk's size; a desk host, which has no screen or input of its own, gets screen, screen_stream and input on its desks; a click on a named desk scales to that desk's size; v0.22.0: desks on a desk host, found by the first production proof: a desk's temp folder is short (Chrome's SingletonSocket must fit a Unix socket path, and the desks folder on a desk host was too deep, so Chrome closed as it started), and a desk no command used for desk_idle_s (15 minutes) stops by itself, so a desk an agent left behind when it crashed or restarted ends; v0.21.0: element refs and receipts, the Desks plan's step D3: the agent's browser lists a page's interactive elements as compact refs (e7 button \"Sign in\"), kept on the element while it stays, and click, press and back answer with what changed (url, lines that appeared, lines gone), so agents act on the page's structure before screenshots; v0.20.0: desk hosts, the Desks plan's step D5: install.sh --desk-host turns an old laptop, a mini PC or a spare workstation into a host of desks for its owner's agents, run as its own openvibe-desk account with Xvfb, openbox, D-Bus, xdotool, ImageMagick and Chromium installed and max_desks sized to its memory; v0.19.0: several desks per machine, the Desks plan's step D1: the computer plugin runs named desks (desk-<name>, private kept), each with its own X display, D-Bus, browser profile, downloads and temp folder, up to max_desks (one per 2 GiB of memory by default), so two agents work at once without seeing or driving each other's screen; proven with two real Chrome desks that never share a cookie; v0.18.0: a motion command's validity also counts the time it waits in the queue to its plugin and in the pipe: the core fixes the instant it expires at receipt, writes what is left at the moment it writes the line with expires_at_mono_ns on the clock it shares with the plugin, and never writes one that has run out, and the Python runtime refuses one that ran out in the pipe before handle(); the clock bound is kept in nanoseconds and counts Bot's whole-millisecond stamps, rounded up from nanoseconds; a Node booting latched sets the e-stop on its plugins before starting them; the Python SDK package reports the Node release; v0.17.0: deadlines are checked against the latest the server's clock can be: the estimate plus its bound, the connection's fastest round trip, and motion waits for that bound on every new connection; every stop and estop names its cause, hello carries the newest halt epoch, and the Python SDK gives drivers on_stop(StopInfo) and on_resume to tell an ordinary stop from an emergency without re-arming anything; v0.16.1: the Python runtime refuses a motion command whose deadline passed while it waited behind a slow handler, so a driver never moves on a stale command; plugins.md answers where deadlines come from, re-stamping, order across a change of driver, stop versus estop, and that nothing is held past its validity; v0.16.0: openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS",
"runtime": "go 1.24 (one static binary per platform); plugins in Python 3; an ESP32 Arduino library",
"database": "none",
"deployed": false,
ddiff --git a/docs/computer.md b/docs/computer.md
iindex 66b62e2..8f67b46 100644
---- a/docs/computer.md
++++ b/docs/computer.md
@@@ -130,6 +130,11 @@ text and keys are passed to the tool as arguments, never through a shell, and `k
- **Receipts.** `click`, `press` and `back` answer with a `receipt` of what they changed: `url_changed`, the first lines
that `appeared` (up to 8) and how many lines are `gone`. An agent knows at once whether its click did anything,
without another read.
++- **Popups and new tabs** (Node v0.24.0). A page the page opens, such as a "Sign in with Google" popup or a link to a
++ new tab, comes to the front. The agent reads and acts on it, and a browser desk shows it, with `"page": "opened"` in
++ the receipt. When it closes (a sign-in popup closes itself), the page it came from is in front again
++ (`"page": "closed"`). Pages are followed at most 5 deep, and only pages opened from the followed ones. The agent's
++ clicks count as a person's (user activation), so a site may open its sign-in popup from them.
- **Closing.** It closes after `browser_idle_s` without a command and at the kill switch.
- **Platforms.** Linux and macOS; not offered on Windows yet.
@@@ -186,6 +191,22 @@ window manager.
size (1280×800 by default).
- `private` is always a whole desk; a whole desk is `"kind": "desktop"`, the default.
++### Kept profiles (`"profile": "<name>"`, Node v0.24.0)
++
++Each agent task gets a desk of its own, and with it a fresh browser. A **kept profile** carries the person's sign-ins
++from desk to desk: `{"action": "start", "desk": "desk-<name>", "profile": "agent"}`. OpenVibe.Actor starts every
++task's desk from the profile `agent`.
++- **Sign in once.** The desk's browser starts from a copy of the kept profile: its cookies and site storage. When the
++ person signs in on an agent's desk (a "Needs you" step), the next task is already signed in.
++- **Saved when the desk stops.** The browser is asked to quit first, so it writes its cookies down. The desk's profile
++ is then saved back (`profile_saved` in the answer), and the desk's own copy is removed, so a desk per task leaves
++ nothing behind. When two desks with the same profile stop, the last one wins.
++- **Not kept:** caches, crash reports, locks and the last session's tabs. A profile larger than `desk_profile_max_mb`
++ (256 MB) is not saved, and an event `desk_profile_too_big` says so. The kill switch saves nothing.
++- **Where.** `browser_profile` + `-kept-<name>`, 0700, on this machine only. Profiles that move between machines are
++ the Desks plan's step D7, later.
++- The private desk keeps its own profile, as before. `describe` says `desk.profiles: true`.
++
### Desk hosts (`--desk-host`, Node v0.20.0)
An old laptop, a mini PC or a spare workstation can host desks for your agents: no screen, keyboard or login needed.
@@@ -225,6 +246,7 @@ curl -fsSL https://openvibe.bot/install | sh -s -- --network … --pairing pair_
| `max_desks` | memory / 2 GiB, 1 to 8 | how many whole desks run at once (1 to 10); a browser desk counts as half |
| `desk_idle_s` | `900` | a desk no command used this long is stopped (60 to 86400) |
| `desk_dir` | `~/.local/share/openvibe-node/desks` | each desk's own temp and downloads folders |
++| `desk_profile_max_mb` | `256` | a kept profile larger than this is not saved (16 to 4096) |
| `allow_browser` | `true` | `false`: no browser (no `browser` capability) |
| `allow_screen_stream` | `true` | `false`: never show the screen live (`screen.stream` is false) |
| `browser_path` | found on PATH | the browser to drive (Chromium, Chrome, Edge or Brave) |
ddiff --git a/internal/protocol/protocol.go b/internal/protocol/protocol.go
iindex 32c690c..91fc3d6 100644
---- a/internal/protocol/protocol.go
++++ b/internal/protocol/protocol.go
@@@ -181,7 +181,8 @@ const (
KindScreenStream = "screen_stream"
// KindDesk starts, stops, reports or lists desks on the machine (contracts 0.214.0; named desks, Node v0.19.0):
// {"action": "start"|"stop"|"status"|"list", "desk"?: "private"|"desk-<name>", "kind"?: "desktop"|"browser", …}
-- // (kind browser, contracts 0.216.0 and Node v0.23.0: a headless browser alone, its page the desk's screen). Older
++ // (kind browser, contracts 0.216.0 and Node v0.23.0: a headless browser alone, its page the desk's screen;
++ // "profile"?, contracts 0.217.0 and Node v0.24.0: the kept browser profile it starts from and saves back). Older
// form, kept:
// {"action": "start"|"stop"|
// "status", "width"?, "height"?}, an Xvfb display of its own where an agent works without touching the owner's
ddiff --git a/plugins/computer/openvibe_computer/__init__.py b/plugins/computer/openvibe_computer/__init__.py
iindex 7151b60..62a171f 100644
---- a/plugins/computer/openvibe_computer/__init__.py
++++ b/plugins/computer/openvibe_computer/__init__.py
@@@ -49,6 +49,10 @@ Command kinds and their answers (the ack's ``data``):
agents working at once never see or drive each other's screen. ``max_desks`` (default: one per 2 GiB of memory, 1 to
8) caps how many run at once; ``list`` reports the running ones. A desk no command used for ``desk_idle_s`` (15
minutes by default) is stopped, so a desk an agent left behind (it crashed or restarted mid-task) ends by itself.
++ ``"profile": "<name>"`` on start (Node v0.24, Desks plan D7 on one machine) keeps the desk's browser profile by
++ name: the desk's browser starts from that profile (its cookies and site storage: sign in once, on any desk) and the
++ desk saves it back when it stops (the last desk to stop wins), then removes its own copy. Caches are never kept, a
++ profile larger than ``desk_profile_max_mb`` is not saved, and the kill switch saves nothing.
``"kind": "browser"`` on start (Node v0.23, Desks plan D2) makes a **browser desk**: a headless browser alone, with no
display, about half the cost of a whole desk (``kind: desktop``, the default), on Linux or macOS. Its page is its
screen: ``screen``, ``input`` (mouse, wheel, keys and text, sent to the page) and ``screen_stream`` (frames of the page)
@@@ -67,6 +71,7 @@ import contextlib
import http.client
import ipaddress
import os
++import re
import socket
import ssl
import shlex
@@@ -111,6 +116,12 @@ INPUT_ACTIONS = frozenset({"move", "click", "double_click", "right_click", "scro
BROWSER_READS = frozenset({"read", "screenshot", "close"})
BROWSER_ACTS = frozenset({"open", "click", "fill", "press", "back"})
MAX_URL = 2048
++PROFILE_NAME = re.compile(r"^[a-z0-9][a-z0-9-]{0,23}$")
++# What a kept profile leaves out: caches, crash reports, locks and the last session's tabs (a desk opens on a blank page).
++PROFILE_SKIP = shutil.ignore_patterns("Cache", "Code Cache", "GPUCache", "GrShaderCache", "GraphiteDawnCache", "ShaderCache",
++ "DawnCache", "DawnGraphiteCache", "DawnWebGPUCache", "CacheStorage", "ScriptCache",
++ "component_crx_cache", "Crashpad", "BrowserMetrics*", "Safe Browsing",
++ "optimization_guide_*", "Singleton*", "Sessions", "blob_storage", "*.tmp")
MAX_SELECTOR = 500
POINT_ACTIONS = frozenset({"move", "click", "double_click", "right_click"})
MAX_TYPE = 2000
@@@ -139,6 +150,21 @@ def _parse_dialog(out):
return None
return True if "button returned:Approve" in out else False if "button returned:Deny" in out else None
++def _tree_size(path, ignore):
++ """The bytes a copy of path with that ignore would hold."""
++ total = 0
++ for root, dirs, files in os.walk(path):
++ skip = ignore(root, dirs + files)
++ dirs[:] = [d for d in dirs if d not in skip]
++ for f in files:
++ if f not in skip:
++ try:
++ total += os.lstat(os.path.join(root, f)).st_size
++ except OSError:
++ pass
++ return total
++
++
def _int(v, name, lo, hi, default):
if v is None:
return default
@@@ -149,7 +175,7 @@ def _int(v, name, lo, hi, default):
class Computer(Plugin):
driver = "computer"
-- version = "0.11.0"
++ version = "0.12.0"
motion_kinds = frozenset()
telemetry_interval_s = 2.0
@@@ -199,6 +225,10 @@ class Computer(Plugin):
self.allow_desk = True
self.desk_browsers = {} # desk name -> the agent's browser on that desk (its own profile, shown there)
self.desk_used = {} # desk name -> when a command last used it (clock), for desk_idle_s
++ self.desk_profiles = {} # desk name -> the kept profile it started with (Node v0.24.0)
++ self.desk_restored = set() # desks whose browser started from their kept profile (so it is saved back on stop)
++ self.profile_locks = {} # profile name -> one copy in or out at a time
++ self.desk_profile_max_mb = 256
self.desk_idle_s = 900
self.browser_locks = {} # "main" or a desk name -> one action at a time in that browser
self.resolve = resolve or (lambda host: [ai[4][0] for ai in socket.getaddrinfo(host, None)])
@@@ -228,6 +258,7 @@ class Computer(Plugin):
self.allow_desk = config.get("allow_desk", True) is not False
self.desks.max_desks = _int(config.get("max_desks"), "max_desks", 1, MAX_DESKS_CAP, default_max_desks())
self.desk_idle_s = _int(config.get("desk_idle_s"), "desk_idle_s", 60, 86400, 900)
++ self.desk_profile_max_mb = _int(config.get("desk_profile_max_mb"), "desk_profile_max_mb", 16, 4096, 256)
self.desks.base_dir = os.path.expanduser(config.get("desk_dir") or "~/.local/share/openvibe-node/desks")
self.browser_path = config.get("browser_path") or None
self.browser_profile = os.path.expanduser(config.get("browser_profile") or "~/.local/share/openvibe-node/browser")
@@@ -261,7 +292,7 @@ class Computer(Plugin):
caps["desk"] = {"level": self.level, "window_manager": self.desks.window_manager() if whole else None,
"input": page or (whole and bool(self.which("xdotool"))),
"screen": page or (whole and bool(self._x11_tool())),
-- "named": True, "max_desks": self.desks.max_desks, "kinds": kinds}
++ "named": True, "max_desks": self.desks.max_desks, "kinds": kinds, "profiles": True}
return caps
def _desk_kinds(self):
@@@ -299,6 +330,7 @@ class Computer(Plugin):
self._end_stream("stopped on the machine")
self._close_desk_browsers()
self.desks.stop_all()
++ self._drop_desk_profiles()
for cid in waiting:
self._close_prompt(cid)
self.emit_event("approval_done", id=cid, approved=False, reason="stopped")
@@@ -322,7 +354,7 @@ class Computer(Plugin):
# own stop, and a desk left running would hold its share of the machine and its browser's sessions.
for name in self.desks.running_names():
if now - self.desk_used.get(name, now) > self.desk_idle_s:
-- self._close_desk_browser(name)
++ self._end_desk_browser(name)
self.desks.stop(name)
self.desk_used.pop(name, None)
self.emit_event("desk_idle_stopped", desk=name, idle_s=self.desk_idle_s)
@@@ -462,15 +494,29 @@ class Computer(Plugin):
raise Fault("unsupported", "a whole desk needs Linux with Xvfb%s" % (": this machine runs browser desks (kind browser)" if kinds else ""))
width = _int(v.get("width"), "width", 640, 3840, 1280)
height = _int(v.get("height"), "height", 480, 2160, 800)
++ profile = v.get("profile")
++ if profile is not None and (not isinstance(profile, str) or not PROFILE_NAME.match(profile)):
++ raise Fault("bad_value", "profile is a name: lower-case letters, digits and dashes, at most 24 characters")
++ if profile is not None and name == "private":
++ raise Fault("bad_value", "the private desk keeps its own profile: name a desk desk-<name> to start one from a kept profile")
if kind == "browser":
what = "a browser desk" if name == "private" else "browser desk %s" % name
-- return self._act(cmd, level, lambda: self.desks.start(name, width, height, kind),
-- "start %s (%d×%d) where your agent works without touching your screen" % (what, width, height),
++
++ def start():
++ st = self.desks.start(name, width, height, kind)
++ if profile and self.desk_profiles.get(name) != profile:
++ self.desk_profiles[name] = profile
++ self.desk_restored.discard(name)
++ return dict(st, profile=profile) if profile else st
++ return self._act(cmd, level, start,
++ "start %s (%d×%d) where your agent works without touching your screen%s"
++ % (what, width, height, ", in your kept browser profile %s" % profile if profile else ""),
threaded=True)
def stop():
-- self._close_desk_browser(name)
-- return self.desks.stop(name)
++ saved = self._end_desk_browser(name)
++ out = self.desks.stop(name)
++ return dict(out, profile_saved=saved) if saved is not None else out
return self._act(cmd, level, stop, "stop %s (and close what runs on it)" % what, threaded=False)
def _desk_browser(self, desk, binary=None, env=None):
@@@ -485,7 +531,8 @@ class Computer(Plugin):
raise Fault("unsupported", "no browser on this machine (Chromium, Chrome, Edge or Brave), or this machine does not allow it")
env = env or self.desks.env(desk)
page = self.desks.kind(desk) == "browser"
-- profile = self.browser_profile + ("-desk" if desk == "private" else "-" + desk)
++ profile = self._desk_profile_dir(desk)
++ self._restore_profile(desk)
b = self.desk_browsers[desk] = self.browser_factory(binary, profile, page, env=env)
b.download_dir = self.desks.dir(desk, "downloads")
if page:
@@@ -516,6 +563,73 @@ class Computer(Plugin):
except BrowserError as e:
raise Fault(e.code, e.message)
++ def _desk_profile_dir(self, desk):
++ return self.browser_profile + ("-desk" if desk == "private" else "-" + desk)
++
++ def _kept_profile_dir(self, profile):
++ return self.browser_profile + "-kept-" + profile
++
++ def _profile_lock(self, profile):
++ with self.mu:
++ return self.profile_locks.setdefault(profile, threading.Lock())
++
++ def _restore_profile(self, desk):
++ """Before a desk's browser first starts in this run of the desk: its profile is a fresh copy of the kept one."""
++ profile = self.desk_profiles.get(desk)
++ if not profile or desk in self.desk_restored:
++ return
++ self.desk_restored.add(desk)
++ own, kept = self._desk_profile_dir(desk), self._kept_profile_dir(profile)
++ shutil.rmtree(own, ignore_errors=True)
++ with self._profile_lock(profile):
++ if os.path.isdir(kept):
++ shutil.copytree(kept, own, symlinks=True, ignore=PROFILE_SKIP)
++ os.chmod(own, 0o700)
++
++ def _end_desk_browser(self, desk):
++ """Close a desk's browser so it writes everything down, save its profile back when it started from a kept one
++ (the last desk to stop wins), and remove the desk's own copy. → True when saved, False when not (too large or
++ a failed copy), None for a desk without a kept profile."""
++ b = self.desk_browsers.pop(desk, None)
++ if b is not None:
++ b.close(graceful=True)
++ profile = self.desk_profiles.pop(desk, None)
++ used = desk in self.desk_restored
++ self.desk_restored.discard(desk)
++ if not profile:
++ return None
++ own, kept = self._desk_profile_dir(desk), self._kept_profile_dir(profile)
++ try:
++ if not used or not os.path.isdir(own):
++ return False
++ size = _tree_size(own, PROFILE_SKIP)
++ if size > self.desk_profile_max_mb * 1024 * 1024:
++ self.emit_event("desk_profile_too_big", desk=desk, profile=profile, mb=size // (1024 * 1024))
++ return False
++ with self._profile_lock(profile):
++ new, old = kept + ".new", kept + ".old"
++ shutil.rmtree(new, ignore_errors=True)
++ shutil.rmtree(old, ignore_errors=True)
++ shutil.copytree(own, new, symlinks=True, ignore=PROFILE_SKIP)
++ os.chmod(new, 0o700)
++ if os.path.isdir(kept):
++ os.rename(kept, old)
++ os.rename(new, kept)
++ shutil.rmtree(old, ignore_errors=True)
++ return True
++ except OSError as e:
++ self.log("desk %s: its profile %s was not saved: %s", desk, profile, e)
++ return False
++ finally:
++ shutil.rmtree(own, ignore_errors=True) # a copy for one run of the desk
++
++ def _drop_desk_profiles(self):
++ """The kill switch: nothing is saved, and the desks' own copies go."""
++ for desk in list(self.desk_profiles):
++ self.desk_profiles.pop(desk, None)
++ self.desk_restored.discard(desk)
++ shutil.rmtree(self._desk_profile_dir(desk), ignore_errors=True)
++
def _close_desk_browser(self, name):
b = self.desk_browsers.pop(name, None)
if b is not None:
ddiff --git a/plugins/computer/openvibe_computer/browser.py b/plugins/computer/openvibe_computer/browser.py
iindex 1172087..d1a140c 100644
---- a/plugins/computer/openvibe_computer/browser.py
++++ b/plugins/computer/openvibe_computer/browser.py
@@@ -27,6 +27,7 @@ STARTUP_S = 45.0
MAX_LINKS = 60
MAX_ELEMENTS = 120
MAX_APPEARED = 8
++MAX_FOLLOWED = 5 # pages opened from pages (a sign-in popup, a new tab) followed at most this deep
KEY_CODES = {
"Enter": (13, "\r"), "Tab": (9, "\t"), "Escape": (27, ""), "Backspace": (8, ""), "Delete": (46, ""),
"ArrowUp": (38, ""), "ArrowDown": (40, ""), "ArrowLeft": (37, ""), "ArrowRight": (39, ""),
@@@ -197,6 +198,11 @@ class Browser:
self.page_size = None # on a browser desk, the page's exact size (the desk's: its page is its screen)
self.write_mu = threading.Lock() # one message at a time on the pipe: actions and the live view share it
self.start_mu = threading.Lock()
++ self.target = None # the page in front: the agent reads and acts on it, a browser desk shows it
++ self.behind = [] # (target, session) of the pages it was opened from, newest last
++ self.tab_events = [] # Target.targetCreated / targetDestroyed seen since the last _follow
++ self.tab_change = None # "opened" or "closed" since the last _mark: said in the action's receipt
++ self.follow_mu = threading.Lock()
# ---- the process and the pipe ------------------------------------------------------------------------------------
def alive(self):
@@@ -238,14 +244,76 @@ class Browser:
threading.Thread(target=self._reader, name="computer-browser", daemon=True).start()
# A first start on a fresh profile can take a while on a slow machine: the first command gets a longer window.
target = self.call("Target.createTarget", {"url": "about:blank"}, session=False, timeout_s=max(self.timeout_s, STARTUP_S))["targetId"]
++ self.target, self.behind = target, []
++ self._attach(target)
++ # Pages this page opens (a sign-in popup, a link to a new tab) are followed: see _follow.
++ self.call("Target.setDiscoverTargets", {"discover": True}, session=False)
++ if self.download_dir:
++ self.call("Browser.setDownloadBehavior", {"behavior": "allow", "downloadPath": self.download_dir}, session=False)
++
++ def _attach(self, target):
++ """Drive that page: a session of its own, its events on, and a browser desk's exact size."""
self.session = self.call("Target.attachToTarget", {"targetId": target, "flatten": True}, session=False)["sessionId"]
self.call("Page.enable")
self.call("Runtime.enable")
if self.page_size:
self.call("Emulation.setDeviceMetricsOverride", {"width": self.page_size[0], "height": self.page_size[1],
"deviceScaleFactor": 1, "mobile": False})
-- if self.download_dir:
-- self.call("Browser.setDownloadBehavior", {"behavior": "allow", "downloadPath": self.download_dir}, session=False)
++
++ def _follow(self):
++ """Keep the page in front in front: a page this page opened (a sign-in popup, a link that opens a new tab)
++ becomes the page the agent reads and acts on, and a browser desk shows; when it closes (a popup closes itself
++ after a sign-in), the page it came from is in front again. Only pages opened from the followed pages, at most
++ MAX_FOLLOWED deep; anything else in the browser is left alone."""
++ with self.follow_mu:
++ with self.mu:
++ events, self.tab_events = self.tab_events, []
++ for method, params in events:
++ info = params.get("targetInfo") or {}
++ ours = [self.target] + [t for t, _ in self.behind]
++ if method == "Target.targetCreated" and info.get("type") == "page" and info.get("openerId") in ours:
++ if len(self.behind) >= MAX_FOLLOWED:
++ continue
++ was = (self.target, self.session)
++ try:
++ self.target = info["targetId"]
++ self._attach(self.target)
++ except BrowserError:
++ self.target, self.session = was
++ continue
++ self.behind.append(was)
++ self.tab_change = "opened"
++ self._ready()
++ elif method == "Target.targetDestroyed":
++ gone = params.get("targetId")
++ if gone == self.target and self.behind:
++ self.target, self.session = self.behind.pop()
++ self.tab_change = "closed"
++ else:
++ self.behind = [(t, sess) for t, sess in self.behind if t != gone]
++
++ def _front_changed(self, session, timeout_s=1.0):
++ """Whether the page in front changed from that session's (its page closed), waiting up to timeout_s for the
++ browser to say so."""
++ end = time.monotonic() + timeout_s
++ while True:
++ self._follow()
++ if self.session != session:
++ return True
++ if time.monotonic() >= end:
++ return False
++ time.sleep(0.05)
++
++ def _ready(self, timeout_s=10.0):
++ """Wait (a little) for a page just followed to finish loading."""
++ end = time.monotonic() + timeout_s
++ while time.monotonic() < end:
++ try:
++ if self._eval("document.readyState") == "complete":
++ return
++ except BrowserError:
++ pass
++ time.sleep(0.1)
def _stderr(self, stream):
"""Keep the browser's last lines of stderr: why it closed, when it does."""
@@@ -277,6 +345,8 @@ class Browser:
elif "method" in msg:
with self.mu:
self.events.append(msg["method"])
++ if msg["method"] in ("Target.targetCreated", "Target.targetDestroyed"):
++ self.tab_events = (self.tab_events + [(msg["method"], msg.get("params") or {})])[-50:]
if len(self.events) > 500:
del self.events[:250]
with self.mu:
@@@ -312,6 +382,7 @@ class Browser:
def _mark(self):
with self.mu:
self.events = []
++ self.tab_change = None
def _wait_for(self, methods, timeout_s):
end = time.monotonic() + timeout_s
@@@ -322,8 +393,20 @@ class Browser:
time.sleep(0.05)
return False
-- def _eval(self, expression):
-- r = self.call("Runtime.evaluate", {"expression": expression, "returnByValue": True, "awaitPromise": True})
++ def _eval(self, expression, gesture=False):
++ # gesture: as a person's click (user activation), so a page may open its sign-in popup from the agent's click.
++ params = {"expression": expression, "returnByValue": True, "awaitPromise": True}
++ if gesture:
++ params["userGesture"] = True
++ session = self.session
++ try:
++ r = self.call("Runtime.evaluate", params)
++ except BrowserError as e:
++ # The page in front just closed (a popup closes itself after a sign-in) before the browser said so: wait for
++ # it to say so (a second at most), then once more on the page in front again.
++ if "session" not in e.message.lower() or not self._front_changed(session):
++ raise
++ r = self.call("Runtime.evaluate", params)
if r.get("exceptionDetails"):
raise BrowserError("bad_value", "the page refused: %s" % r["exceptionDetails"].get("text", "an exception"))
return r.get("result", {}).get("value")
@@@ -337,6 +420,7 @@ class Browser:
# ---- the actions -----------------------------------------------------------------------------------------------
def summary(self):
self.used_at = time.monotonic()
++ self._follow()
return self._eval(SUMMARY_JS)
def open(self, url):
@@@ -363,6 +447,8 @@ class Browser:
after = self.summary()
r = receipt(before, after)
if r is not None:
++ if self.tab_change:
++ r["page"] = self.tab_change # a page it opened is now in front ("opened"), or that page closed ("closed")
after["receipt"] = r
return after
@@@ -370,7 +456,7 @@ class Browser:
self._need()
before = self._before()
self._mark()
-- if not self._eval(CLICK_JS % (json.dumps(selector), json.dumps(text or ""))):
++ if not self._eval(CLICK_JS % (json.dumps(selector), json.dumps(text or "")), gesture=True):
raise BrowserError("bad_value", "nothing on the page matches %s" % (selector or repr(text)))
self._settle()
return self._with_receipt(before)
@@@ -426,12 +512,14 @@ class Browser:
def _need(self):
if not self.alive() or not self.session:
raise BrowserError("not_ready", "no page is open: open a URL first")
++ self._follow()
# ---- a browser desk: the page is the screen ---------------------------------------------------------------------
def page(self):
"""Make sure the browser runs (a blank page until one is opened): a browser desk always has a screen."""
self.start()
self.used_at = time.monotonic()
++ self._follow()
def viewport(self):
"""The page's size in CSS pixels, which mouse events use."""
@@@ -500,10 +588,13 @@ class Browser:
def _receipt_only(self, before):
out = {"done": True}
try:
++ self._follow()
r = receipt(before, self._eval(BEFORE_JS))
except BrowserError:
r = None
if r is not None:
++ if self.tab_change:
++ r["page"] = self.tab_change
out["receipt"] = r
return out
@@@ -519,8 +610,20 @@ class Browser:
f.write(data)
os.replace(path + ".part", path)
-- def close(self):
++ def close(self, graceful=False):
++ """End the browser. graceful: ask it to quit first (a few seconds), so it writes its cookies and storage to the
++ profile (a desk's profile is kept, Node v0.24.0); else, and when it does not quit, it is killed."""
++ if graceful and self.alive():
++ try:
++ self.call("Browser.close", session=False, timeout_s=5.0)
++ except BrowserError:
++ pass
++ try:
++ self.proc.wait(timeout=5)
++ except Exception:
++ pass
proc, self.proc, self.session = self.proc, None, None
++ self.target, self.behind = None, []
for f in (self.out, self.inp):
try:
if f:
ddiff --git a/plugins/computer/pyproject.toml b/plugins/computer/pyproject.toml
iindex 2f36193..826c07e 100644
---- a/plugins/computer/pyproject.toml
++++ b/plugins/computer/pyproject.toml
@@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "openvibe-computer"
--version = "0.11.0"
++version = "0.12.0"
description = "OpenVibe Node computer plugin: the machine's terminal, files and screen under the owner's control level"
requires-python = ">=3.9"
license = { text = "AGPL-3.0-or-later" }
ddiff --git a/plugins/computer/tests/test_browser.py b/plugins/computer/tests/test_browser.py
iindex 0f73468..88f9586 100644
---- a/plugins/computer/tests/test_browser.py
++++ b/plugins/computer/tests/test_browser.py
@@@ -212,6 +212,9 @@ class Handler(http.server.SimpleHTTPRequestHandler):
"/": b"<!doctype html><title>Home</title><h1>Hello</h1><a href='/two'>Second page</a>"
b"<form action='/two'><input id='q' name='q'><button>Search</button></form>",
"/two": b"<!doctype html><title>Two</title><p>Arrived</p>",
++ # Node v0.24.0: a page that opens another (a sign-in popup), and one that sets a cookie.
++ "/pop": b"<!doctype html><title>Pop</title><button id='open' onclick=\"window.open('/two', 'signin', 'popup')\">Sign in</button>",
++ "/cookie": b"<!doctype html><title>Cookie</title><script>if (!document.cookie.includes('kept=yes')) document.cookie = 'kept=yes; max-age=3600; path=/';</script>",
# Desks plan D3: a page whose button changes the page without going anywhere, and things that are not elements.
"/app": b"<!doctype html><title>App</title><p>Ready</p><button id='go' aria-label='Add a row'>+</button>"
b"<input type='email' placeholder='Email'><input type='checkbox' checked><a href='/two'>Second page</a>"
@@@ -270,6 +273,32 @@ class RealBrowserTest(unittest.TestCase):
b.close()
self.assertFalse(b.alive())
++ def test_a_popup_is_followed_and_a_cookie_kept(self):
++ # Node v0.24.0: a page this page opens comes to the front (the agent and a person watching see it), the page
++ # it came from when it closes; and a browser closed gracefully keeps the cookie it was given.
++ srv = socketserver.TCPServer(("127.0.0.1", 0), Handler)
++ threading.Thread(target=srv.serve_forever, daemon=True).start()
++ self.addCleanup(srv.shutdown)
++ base = "http://127.0.0.1:%d" % srv.server_address[1]
++ profile = __import__("tempfile").mkdtemp(prefix="ov-browser-")
++ self.addCleanup(lambda: shutil.rmtree(profile, ignore_errors=True))
++ extra = ["--no-sandbox"] if os.environ.get("CI") else []
++ b = Browser(find_binary(shutil.which, sys.platform), profile, extra_args=extra)
++ self.addCleanup(b.close)
++ self.assertEqual(b.open(base + "/pop")["title"], "Pop")
++ opened = b.click(selector="#open")
++ self.assertEqual((opened["title"], opened["receipt"]["page"]), ("Two", "opened"), opened.get("receipt"))
++ b._eval("window.close()")
++ back = b.read()
++ self.assertEqual(back["title"], "Pop", "the page it came from is in front again")
++ # A cookie survives a graceful close.
++ b.open(base + "/cookie")
++ b.close(graceful=True)
++ again = Browser(find_binary(shutil.which, sys.platform), profile, extra_args=extra)
++ self.addCleanup(again.close)
++ again.open(base + "/cookie")
++ self.assertIn("kept=yes", again._eval("document.cookie"))
++
def test_a_browser_desk_its_page_is_the_screen(self):
# Desks plan D2 in a real browser: no display; the page is the desk's screen, mouse and keyboard, and live view.
srv = socketserver.TCPServer(("127.0.0.1", 0), Handler)
ddiff --git a/plugins/computer/tests/test_desk.py b/plugins/computer/tests/test_desk.py
iindex 3e3ebe6..5d3c771 100644
---- a/plugins/computer/tests/test_desk.py
++++ b/plugins/computer/tests/test_desk.py
@@@ -167,11 +167,39 @@ class FakePageBrowser:
with open(path, "wb") as f:
f.write(b"\xff\xd8\xff")
-- def close(self):
++ def close(self, graceful=False):
self.on = False
self.calls.append(("close",))
++class ProfileBrowser:
++ """A desk's browser that keeps one cookie in its profile folder, like a site's sign-in."""
++ made = []
++
++ def __init__(self, binary, profile, headless, env=None):
++ self.profile, self.download_dir, self.window, self.page_size = profile, None, (1280, 900), None
++ self.on, self.graceful = True, None
++ path = os.path.join(profile, "Default", "Cookies")
++ self.found = open(path).read() if os.path.exists(path) else None # what the desk started with
++ ProfileBrowser.made.append(self)
++
++ def alive(self):
++ return self.on
++
++ def sign_in(self, who, cache_mb=0):
++ os.makedirs(os.path.join(self.profile, "Default", "Cache"), exist_ok=True)
++ with open(os.path.join(self.profile, "Default", "Cookies"), "w") as f:
++ f.write(who)
++ with open(os.path.join(self.profile, "Default", "Cache", "data"), "wb") as f:
++ f.write(b"x" * (cache_mb * 1024 * 1024 or 10))
++
++ def open(self, url):
++ return {"url": url}
++
++ def close(self, graceful=False):
++ self.on, self.graceful = False, graceful
++
++
class DeskPluginTest(unittest.TestCase):
def boot(self, level, **config):
self.out = Out()
@@@ -209,7 +237,7 @@ class DeskPluginTest(unittest.TestCase):
def test_describe_offers_the_desk(self):
caps = self.boot("trusted")[0]["capabilities"]
self.assertEqual(caps["desk"], {"level": "trusted", "window_manager": "metacity", "input": True, "screen": True,
-- "named": True, "max_desks": default_max_desks(), "kinds": ["desktop", "browser"]})
++ "named": True, "max_desks": default_max_desks(), "kinds": ["desktop", "browser"], "profiles": True})
def test_observe_reports_but_does_not_start(self):
self.boot("observe")
@@@ -305,7 +333,7 @@ class DeskPluginTest(unittest.TestCase):
def open(self, url):
return {"url": url}
-- def close(self):
++ def close(self, graceful=False):
pass
self.boot("trusted", max_desks=3)
self.plugin.browser_factory = FakeBrowser
@@@ -408,6 +436,78 @@ class DeskPluginTest(unittest.TestCase):
self.assertEqual(reply(out, "m2")["data"]["kind"], "browser")
plugin.stop()
++ def test_a_kept_profile_follows_the_agent_from_desk_to_desk(self):
++ # Node v0.24.0: sign in once. A desk started with profile "agent" starts from the kept profile and saves it back
++ # when it stops (the last to stop wins); its own copy is removed; caches are never kept.
++ home = tempfile.mkdtemp(prefix="ov-prof-")
++ self.addCleanup(lambda: shutil.rmtree(home, ignore_errors=True))
++ self.boot("trusted", max_desks=4, browser_profile=os.path.join(home, "browser"), desk_profile_max_mb=16)
++ ProfileBrowser.made = []
++ self.plugin.browser_factory = ProfileBrowser
++ kept = os.path.join(home, "browser-kept-agent", "Default")
++
++ def start(desk, **extra):
++ return reply(self.out, self.cmd("desk", {"action": "start", "desk": desk, "kind": "browser", **extra}))
++
++ def use(desk):
++ self.assertEqual(reply(self.out, self.cmd("browser", {"action": "open", "url": "https://example.org/", "desk": desk}))["op"], "ack")
++ return ProfileBrowser.made[-1]
++
++ def stop(desk):
++ return reply(self.out, self.cmd("desk", {"action": "stop", "desk": desk}))["data"]
++ self.assertEqual(start("desk-a", profile="agent")["data"]["profile"], "agent")
++ a = use("desk-a")
++ self.assertIsNone(a.found, "nothing kept yet")
++ self.assertEqual(a.profile, os.path.join(home, "browser-desk-a"))
++ a.sign_in("one")
++ self.assertEqual(stop("desk-a")["profile_saved"], True)
++ self.assertTrue(a.graceful, "closed so it writes its cookies down")
++ with open(os.path.join(kept, "Cookies")) as f:
++ self.assertEqual(f.read(), "one")
++ self.assertFalse(os.path.exists(os.path.join(kept, "Cache")), "caches are never kept")
++ self.assertFalse(os.path.exists(a.profile), "the desk's own copy is removed")
++ self.assertEqual(os.stat(os.path.dirname(kept)).st_mode & 0o777, 0o700)
++ # Two desks at once start from the same kept profile; the last to stop wins.
++ start("desk-b", profile="agent")
++ start("desk-c", profile="agent")
++ b, c = use("desk-b"), use("desk-c")
++ self.assertEqual((b.found, c.found), ("one", "one"), "signed in on both")
++ c.sign_in("three")
++ b.sign_in("two")
++ stop("desk-c")
++ stop("desk-b")
++ with open(os.path.join(kept, "Cookies")) as f:
++ self.assertEqual(f.read(), "two")
++ # A desk whose browser never opened leaves the kept profile alone; a desk without a profile keeps its own folder.
++ start("desk-d", profile="agent")
++ self.assertEqual(stop("desk-d")["profile_saved"], False)
++ start("desk-e")
++ use("desk-e").sign_in("mine")
++ self.assertNotIn("profile_saved", stop("desk-e"))
++ self.assertTrue(os.path.exists(os.path.join(home, "browser-desk-e")), "as before v0.24.0")
++ with open(os.path.join(kept, "Cookies")) as f:
++ self.assertEqual(f.read(), "two")
++ # Too large to keep (desk_profile_max_mb 16): not saved, and said.
++ start("desk-f", profile="agent")
++ use("desk-f").sign_in("big", cache_mb=0)
++ with open(os.path.join(home, "browser-desk-f", "Default", "big"), "wb") as f:
++ f.write(b"x" * (17 * 1024 * 1024))
++ self.assertEqual(stop("desk-f")["profile_saved"], False)
++ self.assertTrue(any(m.get("name") == "desk_profile_too_big" for m in self.out.msgs()))
++ # The kill switch saves nothing and removes the desks' copies.
++ start("desk-g", profile="agent")
++ use("desk-g").sign_in("stopped")
++ self.plugin.stop()
++ self.assertFalse(os.path.exists(os.path.join(home, "browser-desk-g")))
++ with open(os.path.join(kept, "Cookies")) as f:
++ self.assertEqual(f.read(), "two")
++ # Names are checked; the private desk keeps its own profile.
++ self.assertEqual(start("desk-h", profile="../x")["fault_code"], "bad_value")
++ r = reply(self.out, self.cmd("desk", {"action": "start", "profile": "agent"}))
++ self.assertEqual(r["fault_code"], "bad_value")
++ caps = self.plugin.describe({"level": "trusted"})
++ self.assertTrue(caps["desk"]["profiles"])
++
def test_the_kill_switch_stops_the_desk(self):
self.boot("trusted")
reply(self.out, self.cmd("desk", {"action": "start"}))
ddiff --git a/plugins/sdk/openvibe_plugin/__init__.py b/plugins/sdk/openvibe_plugin/__init__.py
iindex 4240caf..c6b199f 100644
---- a/plugins/sdk/openvibe_plugin/__init__.py
++++ b/plugins/sdk/openvibe_plugin/__init__.py
@@@ -24,7 +24,7 @@ __all__ = ["Plugin", "Fault", "Deferred", "Command", "StopInfo", "Runtime", "run
"RUNTIME_CAUSES", "SHARED_CLOCK", "__version__"]
# The Node release this runtime ships with (pyproject.toml carries the same).
--__version__ = "0.23.0"
++__version__ = "0.24.0"
PROTOCOL_VERSION = 1
HEARTBEAT_TIMEOUT_S = 1.0
DEFAULT_DEADLINE_MS = 300
ddiff --git a/plugins/sdk/pyproject.toml b/plugins/sdk/pyproject.toml
iindex b73b656..1af4ee8 100644
---- a/plugins/sdk/pyproject.toml
++++ b/plugins/sdk/pyproject.toml
@@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "openvibe-plugin"
--version = "0.23.0"
++version = "0.24.0"
description = "Runtime for OpenVibe Node plugins (JSON lines over stdin/stdout, deadlines, heartbeat, e-stop)"
requires-python = ">=3.9"
license = { text = "AGPL-3.0-or-later" }
ddiff --git a/plugins/sdk/tests/test_runtime.py b/plugins/sdk/tests/test_runtime.py
iindex 946bb4e..11fbaef 100644
---- a/plugins/sdk/tests/test_runtime.py
++++ b/plugins/sdk/tests/test_runtime.py
@@@ -439,7 +439,7 @@ class RuntimeTest(unittest.TestCase):
self.assertEqual(rt.stop_infos[-1].halt_epoch, 1792900000500)
def test_the_package_reports_the_node_release(self):
-- self.assertEqual(openvibe_plugin.__version__, "0.23.0")
++ self.assertEqual(openvibe_plugin.__version__, "0.24.0")
with open(os.path.join(os.path.dirname(HERE), "pyproject.toml")) as f:
self.assertIn('version = "%s"' % openvibe_plugin.__version__, f.read())