openvibers/openvibe.actor Commits
See it work: a running task's desk tile and viewer, take over and hand back, and a step-by-step replay (#449)
Changes
@@@ -2,6 +2,22 @@What changed in OpenVibe.Actor, newest first. Each site also publishes its patch notes at /updates.++## Unreleased — See it work (W1): the desk, the viewer and the replay++++- **The desk tile** in a running task's card: the desk's private Live stream (started with the task's desk through Link's++ `desk.stream` and stopped with the task), with the still at 1/s when no stream is up. Under it the latest step in++ plain words, a ring where it last clicked, the browser page's address and where it is ("Your PC · its own desk",++ "OpenVibe computer"). The tile pauses its stream while off screen.++- **The viewer** (a tap on the tile): the desk fills a phone and sits beside the steps from 900 px, with Take over,++ Pause, Stop and Hand back. Take over takes a Run desk's live lease or makes Bot take the person's own input on a Link++ desk (their token, mapped to desk coordinates); the agent's input is refused while they drive. Hand back takes an++ optional note that reaches the run as the person's message. A `hand_to_person` step is an Ask that opens the viewer++ in control.++- **Replay**: every computer step keeps its still and its action. A finished task's card has a timeline to scrub, and++ each step opens the picture it saw. Typed text is never stored in a replay; a click keeps only its x,y.++- Migration 0070 (task stills); `GET /api/v1/tasks/:id/desks`, `/desk-still`, `/replay`, the desk take-over/hand-back++ and the person's input. Tests: `test/see-it-work.test.js`; screenshots: `test/see-it-work-shots.js`.++## Unreleased — the chat is the one place you use Actor (one agent system, stage 1)- **Composer v2** (openvibe-shared 4.215.0, the `agent` preset): the chat mounts `OpenVibeComposer.mount` with Attach,@@@ -121,6 +121,7 @@ The resource index is for OpenVibe.Services on loopback and is blocked at the pu- **OpenVibe.Search** (`ACTOR_SEARCH_URL`), through the Search Fabric (`/api/v1/query`): content across the network for the runtime's `search_openvibe` tool (anonymous), and the web for `search_web` (cited results; asked as Actor's service principal with the grant `search.query.delegate`, naming the task's person in `X-OV-Subject`, so the paid steps past the open index count against them).- **OpenVibe.Tools** (`ACTOR_TOOLS_URL`): the catalog, the run API and the page reader, for `find_tools`, `run_tool` and `read_page`. Called as any anonymous caller would.- **OpenVibe.Bot** (`ACTOR_BOT_URL`, default `http://127.0.0.1:4630`; empty turns it off): a signed-in person's own robots, for `list_my_robots` and `robot_steps`. Actor calls Bot with its own client-credentials token (audience `openvibe.bot`, `bot.robot.read` + `bot.robot.control`) and `X-OV-Subject` naming the person. Bot answers only for the robot's owner, and only on robots where that owner turned on "Let my OpenVibe.Actor agent drive this robot". Bot's gate, limits and e-stop judge every step. App, agent and service tasks never see these tools.++- **See it work** (one-agent system, W1): a running task's card shows its desk — the private Live stream Link publishes for the task's desk (or the still at 1/s), the latest step in plain words, a ring where it last clicked, the page address and where it is. A tap opens the viewer: Take over (Run's lease, or Bot input as the person on a Link desk), Pause, Stop and Hand back with a note. A sign-in step is an Ask that opens it in control. A finished run has a replay of each step's still and action; typed text is never kept. The SDK calls are `desks`, `takeOver`, `handBack` and `replay` in `openvibe-sdk/agents` (PR open, released by the main session).- **The chat is the one page** (one-agent system, stage 1): the Console is folded into it. `/console` answers 301 to `/chat`; the running tasks the Console listed are the "Now" strip at the top of the chat (one line each, Stop on each, hidden when nothing runs), and a running task is a card in its conversation that updates in place. The run settings ("New runs: Only me … Apply to all my existing runs") live in `/chat/settings`; the composer is openvibe-shared's composer v2, agent preset (Attach, Speak, the Fast/Balanced/Best mode chip, the round send, progress on each file). The empty chat shows four compact suggestion chips. `/me/spend` stays (`server/http/console-pages.js`, `server/http/chat-pages.js`, `test/one-page.test.js`).- **A person's own computers, also through OpenVibe.Bot** (plan T14/T17, contracts 0.166.0): `list_my_computers`, `computer_list`, `computer_read`, `computer_run` (argv, never a shell line) and `computer_write`, on machines installed with OpenVibe Link's `--computer` plugin. Actor calls Bot's machine API with its own token (`bot.computer.control`) and `X-OV-Subject` naming the person. Bot answers only for the machine's owner, and only where that owner turned on "Let my agent use this computer". The level the owner set on the machine decides what may happen there; at `ask`, each run or write waits up to two minutes for the owner's approval on the machine. Only a signed-in person's task sees these tools (`server/agents/computers.js`, `test/computers.test.js`). `computer_screen` (when an OpenAI key is set) takes a picture of the machine's screen through Bot (1280 px wide) and has OpenAI's vision model answer the agent's question about it. The runtime model gets that answer, never the picture; the picture is not stored; secrets visible on screen are named, not repeated; and each look is metered on the task like any model call (refused once the task's budget is spent). `computer_input` (offered with `computer_screen`; contracts 0.171.0) uses the mouse and keyboard: a click, double or right click or move at a position in the last screen picture (its width and height are passed, and the Link scales them to the real screen), a scroll, typed text, or one key or chord. It is an acting command, so at level `ask` each one waits for the person's approval on the machine; the runtime is told to look first, act in small steps, look again, and never type a secret the task did not give it.- `computer_browser` (contracts 0.174.0, Link v0.10.0) drives a browser of the agent's own on the machine. It is annew file mode 100644@@@ -0,0 +1,15 @@++-- phase: expand++-- See it work (W1): the picture each step saw, for a finished run's replay. The bytes are Actor's own (30-day task++-- retention); secret fields are blanked by the desk's curtain before the picture ever leaves it.++CREATE TABLE IF NOT EXISTS actor_task_stills (++ task_id text NOT NULL,++ seq integer NOT NULL,++ tool text,++ mime text NOT NULL,++ width integer,++ height integer,++ bytes bytea NOT NULL,++ at timestamptz NOT NULL,++ PRIMARY KEY (task_id, seq)++);++CREATE INDEX IF NOT EXISTS actor_task_stills_task_at ON actor_task_stills (task_id, at);@@@ -254,3 +254,74 @@ body:has(.page-app) > footer, body:has(.page-app) #ov-footer, body:has(.page-app.ct-share-defaults input[type=checkbox] { width: 18px; height: 18px; }.ct-share-defaults .sc-btn { min-height: 44px; }.ct-share-defaults .muted { flex-basis: 100%; font-size: 12.5px; line-height: 1.45; }++++/* ── See it work (W1): the desk tile in a running card, the viewer, and the finished run's replay. ─────────────── */++.ct-desk-tile { margin: 0; display: grid; gap: 6px; }++.ct-desk-frame { position: relative; aspect-ratio: 16 / 9; overflow: hidden; border-radius: 14px; border: 1px solid var(--line); background: #05070d; }++.ct-desk-frame img, .ct-desk-frame iframe { position: absolute; inset: 0; width: 100%; height: 100%; border: 0; object-fit: contain; background: #05070d; }++.ct-desk-open { position: absolute; inset: 0; z-index: 2; display: flex; align-items: flex-end; justify-content: flex-end; padding: 8px; border: 0; background: transparent; cursor: pointer; }++.ct-desk-open span { display: inline-flex; align-items: center; min-height: 44px; padding: 0 12px; border-radius: 999px; background: rgba(2, 6, 14, .72); color: #fff; font: inherit; font-size: 12.5px; font-weight: 700; opacity: .9; }++.ct-desk-open:hover span, .ct-desk-open:focus-visible span { opacity: 1; background: color-mix(in srgb, var(--accent, #f97316) 70%, #05070d); }++.ct-desk-live { position: absolute; z-index: 1; top: 8px; left: 8px; display: inline-flex; align-items: center; gap: 5px; padding: 2px 8px; border-radius: 999px; background: rgba(0, 0, 0, .6); color: #fff; font-size: 11px; font-weight: 750; letter-spacing: .04em; text-transform: uppercase; }++.ct-desk-live::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: #ef4444; animation: ct-breathe 1.4s ease-in-out infinite; }++.ct-desk-ring { position: absolute; z-index: 1; width: 22px; height: 22px; margin: -11px 0 0 -11px; border-radius: 50%; border: 2px solid #ef4444; box-shadow: 0 0 0 3px rgba(239, 68, 68, .35); pointer-events: none; }++.ct-desk-tile figcaption { display: flex; flex-wrap: wrap; align-items: center; gap: 3px 10px; color: var(--text-muted, #7386a3); font-size: 12.5px; line-height: 1.4; }++.ct-desk-tile figcaption [data-ct-desk-caption] { color: var(--text-secondary, #a8b3c7); }++.ct-desk-place { font-weight: 650; color: var(--text-muted, #7386a3); }++.ct-desk-tile figcaption a { color: var(--accent, #f97316); text-decoration: none; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 100%; }++.ct-desk-ask { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 8px; padding: 10px 12px; border-radius: 12px; border: 1px solid color-mix(in srgb, var(--accent, #f97316) 45%, var(--line)); background: color-mix(in srgb, var(--accent, #f97316) 8%, var(--bg-card, #131c2e)); }++.ct-desk-ask p { margin: 0; flex: 1 1 220px; font-size: 13.5px; color: var(--text-primary, #e6edf7); }++.ct-desk-ask button { min-height: 44px; padding: 0 14px; border: 0; border-radius: 12px; background: var(--accent, #f97316); color: var(--on-accent, #fff); font: inherit; font-weight: 700; }++++/* The viewer: a phone gets the desk filling the screen, a computer gets it beside the steps. */++.ct-viewer { position: fixed; inset: 0; z-index: 60; display: grid; grid-template-rows: minmax(0, 1fr) auto; background: #05070d; color: #e6edf7; }++.ct-viewer[hidden] { display: none; }++.ct-viewer-stage { position: relative; min-height: 0; overflow: hidden; background: #05070d; }++.ct-viewer-stage img, .ct-viewer-stage iframe { position: absolute; inset: 0; width: 100%; height: 100%; border: 0; object-fit: contain; background: #05070d; }++.ct-viewer-input { position: absolute; inset: 0; z-index: 3; display: none; touch-action: none; cursor: crosshair; outline: 0; }++.ct-viewer[data-control="1"] .ct-viewer-input { display: block; }++.ct-viewer-ring { position: absolute; z-index: 2; width: 24px; height: 24px; margin: -12px 0 0 -12px; border-radius: 50%; border: 2px solid #ef4444; box-shadow: 0 0 0 4px rgba(239, 68, 68, .35); pointer-events: none; }++.ct-viewer-badge { position: absolute; z-index: 4; top: 10px; left: 10px; display: inline-flex; align-items: center; gap: 6px; padding: 3px 9px; border-radius: 999px; background: rgba(0, 0, 0, .65); color: #fff; font-size: 11px; font-weight: 750; letter-spacing: .04em; text-transform: uppercase; }++.ct-viewer[data-state="control"] .ct-viewer-badge { background: color-mix(in srgb, var(--accent, #f97316) 75%, #05070d); }++.ct-viewer-caption { position: absolute; z-index: 4; left: 10px; right: 10px; bottom: 10px; display: flex; flex-wrap: wrap; align-items: center; gap: 4px 10px; padding: 8px 10px; border-radius: 12px; background: rgba(2, 6, 14, .72); color: #e6edf7; font-size: 12.5px; }++.ct-viewer-caption a { color: #fdba74; text-decoration: none; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 100%; }++.ct-viewer-side { display: none; }++.ct-viewer-bar { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; padding: 8px 10px calc(72px + env(safe-area-inset-bottom, 0px)); background: linear-gradient(180deg, transparent, #05070d 40%); }++.ct-viewer-bar button, .ct-viewer-note button { min-height: 44px; padding: 0 14px; border-radius: 12px; border: 1px solid var(--line); background: #0f1729; color: #e6edf7; font: inherit; font-weight: 700; cursor: pointer; }++.ct-viewer-bar button.is-primary, .ct-viewer-note button { border: 0; background: var(--accent, #f97316); color: var(--on-accent, #fff); }++.ct-viewer-bar button:disabled { opacity: .5; cursor: default; }++.ct-viewer-hint { flex: 1 1 200px; margin: 0; color: #a8b3c7; font-size: 12.5px; line-height: 1.35; }++.ct-viewer-note { display: flex; flex-wrap: wrap; gap: 8px; width: 100%; }++.ct-viewer-note input { flex: 1 1 180px; min-height: 44px; padding: 0 12px; border-radius: 12px; border: 1px solid var(--line); background: #0b1220; color: inherit; font: inherit; }++.ct-viewer[data-control="0"] .ct-viewer-note { display: none; }++@media (min-width: 900px) {++ .ct-viewer { grid-template-columns: minmax(0, 1.6fr) minmax(280px, .8fr); grid-template-rows: minmax(0, 1fr) auto; }++ .ct-viewer-stage { grid-column: 1; grid-row: 1 / span 2; }++ .ct-viewer-side { grid-column: 2; grid-row: 1; display: block; overflow: auto; padding: 16px; border-left: 1px solid var(--line); }++ .ct-viewer-bar { grid-column: 2; grid-row: 2; display: grid; grid-template-columns: 1fr 1fr; padding: 10px 16px 16px; background: #05070d; }++ .ct-viewer-hint, .ct-viewer-note { grid-column: 1 / -1; }++ .ct-viewer-side h2 { margin: 0 0 8px; font-size: 15px; }++ .ct-viewer-side .ovt-w-list { max-height: none; }++}++@media (max-width: 899px) {++ .ct-viewer-bar { position: fixed; left: 0; right: 0; bottom: 0; z-index: 61; }++ .ct-viewer { padding-bottom: 0; }++}++.ct-replay { margin-top: 10px; border: 1px solid var(--line); border-radius: 14px; padding: 0 12px; background: var(--bg-card, #131c2e); }++.ct-replay > summary { display: flex; align-items: center; min-height: 44px; cursor: pointer; font-size: 13.5px; font-weight: 650; color: var(--text-secondary, #a8b3c7); list-style: none; }++.ct-replay > summary::-webkit-details-marker { display: none; }++.ct-replay-body { display: grid; gap: 8px; padding: 0 0 12px; }++.ct-replay-body input[type=range] { width: 100%; min-height: 44px; accent-color: var(--accent, #f97316); }++.ct-replay-shot { margin: 0; display: grid; gap: 6px; }++.ct-replay-shot img { width: 100%; aspect-ratio: 16 / 9; object-fit: contain; border-radius: 12px; background: #05070d; }++.ct-replay-none { display: grid; place-items: center; min-height: 120px; border-radius: 12px; background: #05070d; color: var(--text-muted, #7386a3); font-size: 13px; }++.ct-replay-shot figcaption { display: flex; flex-wrap: wrap; justify-content: space-between; gap: 4px 10px; font-size: 12.5px; color: var(--text-muted, #7386a3); }++.ct-replay-shot figcaption a { color: var(--accent, #f97316); }++.ct-replay-steps { list-style: none; margin: 0; padding: 0; display: grid; gap: 4px; max-height: 220px; overflow: auto; }++.ct-replay-steps button { display: flex; align-items: center; gap: 8px; width: 100%; min-height: 44px; padding: 0 8px; border: 0; border-radius: 9px; background: transparent; color: var(--text-secondary, #a8b3c7); font: inherit; font-size: 13px; text-align: left; cursor: pointer; }++.ct-replay-steps button:hover, .ct-replay-steps button:focus-visible, .ct-replay-steps button.is-on { background: color-mix(in srgb, var(--accent, #f97316) 12%, transparent); color: var(--text-primary, #e6edf7); }++.ct-replay-steps button span { flex: none; display: grid; place-items: center; width: 22px; height: 22px; border-radius: 50%; background: color-mix(in srgb, var(--accent, #f97316) 16%, transparent); color: var(--accent, #f97316); font-size: 11px; font-weight: 750; }++@media (max-width: 640px) {++ .ct-desk-open span { min-height: 44px; }++ .ct-viewer-bar button, .ct-viewer-note button { min-height: 44px; }++}@@@ -500,6 +500,7 @@liveLine(turn);draft(turn);syncWait(turn, next);++ if (window.OVDesks) window.OVDesks.sync(turn, next);} else {stopScreen(turn);// An answer the person already watched being written is not typed in again: it only takes its@@@ -507,6 +508,7 @@if (turn.querySelector('[data-ovt-draft]')) turn.removeAttribute('data-open');turn._draft = null;if (window.OVThread) window.OVThread.replace(id, h); else turn.replaceWith(next);++ if (window.OVDesks) window.OVDesks.scan();var dot = chat.querySelector('.ovt-list a[aria-current] .ovt-dot'); if (dot) dot.remove();if (note) note.textContent = '';syncSend();nnew file mode 100644@@@ -0,0 +1,429 @@++/* See it work (W1): the desk tile in a running task's card, the viewer with Take over / Pause / Stop, and a finished++ * run's replay. The calls are the SDK's names (desks, takeOver, handBack, replay) over Actor's own routes until++ * openvibe-sdk/agents carries them; a page can do what a script can.++ */++(function () {++ 'use strict';++ var api = {++ async json(url, opts) {++ var r = await fetch(url, Object.assign({ credentials: 'same-origin', headers: { Accept: 'application/json' } }, opts || {}));++ var data = await r.json().catch(function () { return {}; });++ if (!r.ok) throw new Error((data && (data.detail || data.error)) || 'That did not work.');++ return data;++ },++ desks(run) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks'); },++ takeOver(run, desk) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks/' + encodeURIComponent(desk) + '/take-over', { method: 'POST' }); },++ handBack(run, desk, note) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks/' + encodeURIComponent(desk) + '/hand-back', { method: 'POST', headers: { Accept: 'application/json', 'Content-Type': 'application/json' }, body: JSON.stringify({ note: note || '' }) }); },++ input(run, desk, value) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks/' + encodeURIComponent(desk) + '/input', { method: 'POST', headers: { Accept: 'application/json', 'Content-Type': 'application/json' }, body: JSON.stringify(value) }); },++ replay(run) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/replay'); },++ };++ window.OVAgents = api;++++ var seen = new WeakSet();++ var streams = new WeakMap(); // tile → { live, frame, timer, visible, paused, blob }++ var viewer = null;++++ function taskOf(el) { return el.getAttribute('data-task'); }++ function deskIdOf(el) { return el.getAttribute('data-desk'); }++++ function stopTimer(st) { if (st && st.timer) { clearInterval(st.timer); st.timer = 0; } }++ function stopBlob(st) { if (st && st.blob) { URL.revokeObjectURL(st.blob); st.blob = null; } }++++ /** The stream is loaded only while its tile is on screen; Live's encoder stops when nobody watches. */++ function visible(el) {++ var r = el.getBoundingClientRect();++ return r.bottom > 0 && r.top < (window.innerHeight || 0) && r.right > 0 && r.left < (window.innerWidth || 0);++ }++++ function show(img, src) {++ var st = streams.get(img);++ if (!st) return;++ fetch(src, { credentials: 'same-origin', cache: 'no-store' })++ .then(function (r) { return r.ok ? r.blob() : null; })++ .then(function (blob) {++ if (!blob || !st.visible || st.live || st.paused) return;++ stopBlob(st);++ st.blob = URL.createObjectURL(blob);++ img.src = st.blob;++ img.hidden = false;++ })++ .catch(function () {});++ }++++ function startStill(tile, st) {++ stopTimer(st);++ var img = tile.querySelector('[data-ct-desk-still]');++ var src = tile.getAttribute('data-still');++ if (!img || !src) return;++ if (!st.visible || st.live || st.paused) return;++ show(img, src);++ st.timer = setInterval(function () { show(img, tile.getAttribute('data-still')); }, 1000);++ }++++ /** Load the Live player while the tile is visible, and remove it when the tile leaves the screen. */++ function startStream(tile, st) {++ if (st.frame || !st.live || !st.live.embed_url || !st.visible || st.paused) return;++ var box = tile.querySelector('[data-ct-desk-frame]');++ if (!box) return;++ var frame = document.createElement('iframe');++ frame.src = st.live.embed_url;++ frame.title = 'The desk Actor works on, live';++ frame.setAttribute('allow', 'autoplay; fullscreen; clipboard-write');++ frame.setAttribute('referrerpolicy', 'no-referrer');++ frame.setAttribute('sandbox', 'allow-scripts allow-same-origin allow-forms allow-pointer-lock');++ box.appendChild(frame);++ st.frame = frame;++ var img = tile.querySelector('[data-ct-desk-still]');++ if (img) img.hidden = true;++ var badge = tile.querySelector('[data-ct-desk-live]');++ if (badge) badge.hidden = false;++ stopTimer(st);++ }++ function stopStream(st) {++ if (st.frame) { st.frame.remove(); st.frame = null; }++ var tile = st.tile;++ if (tile) {++ var badge = tile.querySelector('[data-ct-desk-live]');++ if (badge) badge.hidden = true;++ var img = tile.querySelector('[data-ct-desk-still]');++ if (img) img.hidden = false;++ }++ }++++ function syncMeta(tile, fresh) {++ if (!tile || !fresh) return;++ for (var key of ['data-desk', 'data-where', 'data-kind', 'data-label', 'data-still']) {++ if (fresh.hasAttribute(key)) tile.setAttribute(key, fresh.getAttribute(key));++ }++ var cap = tile.querySelector('[data-ct-desk-caption]'), freshCap = fresh.querySelector('[data-ct-desk-caption]');++ if (cap && freshCap) cap.textContent = freshCap.textContent;++ var place = tile.querySelector('[data-ct-desk-place]'), freshPlace = fresh.querySelector('[data-ct-desk-place]');++ if (place && freshPlace) place.textContent = freshPlace.textContent;++ var ring = tile.querySelector('[data-ct-desk-ring]'), freshRing = fresh.querySelector('[data-ct-desk-ring]');++ if (ring && freshRing) ring.outerHTML = freshRing.outerHTML;++ else if (ring) ring.remove();++ else if (freshRing) tile.querySelector('[data-ct-desk-frame]').appendChild(freshRing);++ var url = tile.querySelector('[data-ct-desk-url]'), freshUrl = fresh.querySelector('[data-ct-desk-url]');++ if (url && freshUrl) { url.href = freshUrl.href; url.textContent = freshUrl.textContent; url.hidden = false; }++ else if (url) url.hidden = true;++ else if (freshUrl) tile.querySelector('figcaption').appendChild(freshUrl);++ var ask = tile.querySelector('[data-ct-desk-ask]'), freshAsk = fresh.querySelector('[data-ct-desk-ask]');++ if (ask && freshAsk) ask.outerHTML = freshAsk.outerHTML;++ else if (ask) ask.remove();++ else if (freshAsk) tile.appendChild(freshAsk);++ }++++ function scan() {++ document.querySelectorAll('[data-ct-desk]').forEach(function (tile) {++ if (seen.has(tile)) return;++ seen.add(tile);++ var st = { tile: tile, live: null, frame: null, timer: 0, visible: false, paused: false, blob: null };++ streams.set(tile, st);++ var task = taskOf(tile);++ if (task) {++ api.desks(task).then(function (j) {++ var found = (j.desks || []).find(function (d) { return d.id === deskIdOf(tile); }) || (j.desks || [])[0];++ if (!found) return;++ st.live = found.live || null;++ st.width = found.width || 1280; st.height = found.height || 800;++ tile.setAttribute('data-width', String(st.width)); tile.setAttribute('data-height', String(st.height));++ if (found.label) tile.setAttribute('data-label', found.label);++ if (found.browser_url) {++ var a = tile.querySelector('[data-ct-desk-url]');++ if (a) { a.href = found.browser_url; a.textContent = found.browser_url.replace(/^https?:\/\//, ''); a.hidden = false; }++ }++ paint(tile, st);++ }).catch(function () {});++ }++ paint(tile, st);++ var io = typeof IntersectionObserver === 'function' ? new IntersectionObserver(function (entries) {++ entries.forEach(function (entry) {++ st.visible = entry.isIntersecting;++ if (st.visible) { startStream(tile, st); startStill(tile, st); }++ else { stopStream(st); stopTimer(st); }++ });++ }, { rootMargin: '80px' }) : null;++ if (io) io.observe(tile); else { st.visible = true; paint(tile, st); }++ });++ scanReplays();++ }++++ function paint(tile, st) {++ st.visible = st.visible || visible(tile);++ if (st.live && st.live.embed_url) startStream(tile, st);++ else startStill(tile, st);++ }++++ function openTile(tile, control) {++ var task = taskOf(tile);++ var desk = deskIdOf(tile);++ if (!task || !desk) return;++ var v = ensureViewer();++ v.hidden = false;++ v.dataset.task = task; v.dataset.desk = desk;++ v.dataset.control = '0'; v.dataset.state = 'watch';++ v._size = { width: Number(tile.getAttribute('data-width')) || 1280, height: Number(tile.getAttribute('data-height')) || 800 };++ v.querySelector('[data-ct-viewer-badge]').textContent = 'Watching';++ var badge = v.querySelector('[data-ct-viewer-badge]');++ badge.textContent = tile.getAttribute('data-label') || 'The desk';++ v.querySelector('[data-ct-viewer-caption-text]').textContent = (tile.querySelector('[data-ct-desk-caption]') || {}).textContent || '';++ var page = tile.querySelector('[data-ct-desk-url]');++ var capLink = v.querySelector('[data-ct-viewer-page]');++ if (page && page.href) { capLink.href = page.href; capLink.textContent = page.textContent; capLink.hidden = false; } else capLink.hidden = true;++ var stage = v.querySelector('[data-ct-viewer-stage]');++ stage.querySelectorAll('iframe,img').forEach(function (n) { n.remove(); });++ stage.querySelector('[data-ct-viewer-input]').hidden = false;++ var img = document.createElement('img');++ img.alt = 'The desk Actor works on';++ img.hidden = false;++ stage.insertBefore(img, stage.firstChild);++ var same = streams.get(tile);++ if (same) { stopStream(same); stopTimer(same); }++ v._live = same && same.live && same.live.embed_url ? same.live : null;++ v._img = img;++ if (v._live) {++ var frame = document.createElement('iframe');++ frame.src = v._live.embed_url;++ frame.title = 'The desk Actor works on, live';++ frame.setAttribute('allow', 'autoplay; fullscreen; clipboard-write');++ frame.setAttribute('referrerpolicy', 'no-referrer');++ frame.setAttribute('sandbox', 'allow-scripts allow-same-origin allow-forms allow-pointer-lock');++ stage.insertBefore(frame, stage.firstChild);++ img.hidden = true;++ }++ // The picture polls at 1/s only when there is no stream (or the stream is paused).++ v._still = function () {++ if (v.hidden || v.dataset.paused === '1') return;++ var src = tile.getAttribute('data-still');++ if (!src || (same && same.live && !v.dataset.paused)) return;++ fetch(src, { credentials: 'same-origin', cache: 'no-store' }).then(function (r) { return r.ok ? r.blob() : null; }).then(function (b) {++ if (b && !v.hidden && v.dataset.paused !== '1') img.src = URL.createObjectURL(b);++ }).catch(function () {});++ };++ v._still();++ clearInterval(v._timer);++ v._timer = setInterval(v._still, 1000);++ var side = v.querySelector('[data-ct-viewer-steps]');++ side.replaceChildren();++ var turn = document.querySelector('.ovt-turn[data-task="' + task + '"]');++ var list = turn && turn.querySelector('[data-ovt-work]');++ if (list) side.appendChild(list.cloneNode(true));++ var hint = v.querySelector('[data-ct-viewer-hint]');++ hint.textContent = 'Watch Actor work, or take over to use the mouse and keyboard yourself.';++ v.querySelector('[data-ct-viewer-take]').hidden = false;++ v.querySelector('[data-ct-viewer-hand]').hidden = true;++ v.querySelector('[data-ct-viewer-note]').hidden = true;++ v.querySelector('[data-ct-viewer-pause]').textContent = 'Pause';++ v.dataset.paused = '0';++ v.querySelector('[data-ct-viewer-input]').focus();++ if (control) takeOver(v);++ else v.querySelector('[data-ct-viewer-take]').focus();++ }++++ function ensureViewer() {++ if (viewer) return viewer;++ viewer = document.createElement('div');++ viewer.className = 'ct-viewer';++ viewer.setAttribute('role', 'dialog');++ viewer.setAttribute('aria-modal', 'true');++ viewer.setAttribute('aria-label', 'The desk Actor works on');++ viewer.hidden = true;++ viewer.innerHTML = '<div class="ct-viewer-stage" data-ct-viewer-stage><span class="ct-viewer-badge" data-ct-viewer-badge>Watching</span><div class="ct-viewer-input" data-ct-viewer-input tabindex="0" aria-label="Use the desk"></div><div class="ct-viewer-caption"><span data-ct-viewer-caption-text></span><a data-ct-viewer-page hidden></a></div></div>'++ + '<aside class="ct-viewer-side"><h2>What Actor is doing</h2><div data-ct-viewer-steps></div></aside>'++ + '<div class="ct-viewer-bar"><p class="ct-viewer-hint" data-ct-viewer-hint></p><button type="button" class="is-primary" data-ct-viewer-take>Take over</button><button type="button" data-ct-viewer-pause>Pause</button><button type="button" data-ct-viewer-hand hidden>Hand back</button><button type="button" data-ct-viewer-stop>Stop</button><button type="button" data-ct-viewer-close>Close</button><div class="ct-viewer-note" data-ct-viewer-note hidden><input type="text" maxlength="1000" placeholder="A note for Actor, optional" data-ct-viewer-note-input aria-label="A note for Actor"><button type="button" data-ct-viewer-note-send>Hand back with note</button></div></div>';++ document.body.appendChild(viewer);++ viewer.querySelector('[data-ct-viewer-close]').addEventListener('click', closeViewer);++ viewer.querySelector('[data-ct-viewer-take]').addEventListener('click', function () { takeOver(viewer); });++ viewer.querySelector('[data-ct-viewer-pause]').addEventListener('click', function () { pauseViewer(viewer); });++ viewer.querySelector('[data-ct-viewer-stop]').addEventListener('click', function () { stopViewer(viewer); });++ viewer.querySelector('[data-ct-viewer-hand]').addEventListener('click', function () { handBack(viewer); });++ viewer.querySelector('[data-ct-viewer-note-send]').addEventListener('click', function () { handBack(viewer); });++ viewer.querySelector('[data-ct-viewer-input]').addEventListener('pointerdown', onPointer);++ viewer.querySelector('[data-ct-viewer-input]').addEventListener('wheel', onWheel, { passive: false });++ viewer.querySelector('[data-ct-viewer-input]').addEventListener('keydown', onKey);++ document.addEventListener('keydown', function (e) { if (e.key === 'Escape' && viewer && !viewer.hidden) closeViewer(); });++ return viewer;++ }++++ function closeViewer() {++ if (!viewer) return;++ viewer.hidden = true;++ clearInterval(viewer._timer);++ viewer.querySelectorAll('iframe').forEach(function (f) { f.remove(); });++ document.querySelectorAll('[data-ct-desk]').forEach(function (tile) { var st = streams.get(tile); if (st) paint(tile, st); });++ }++++ function stageSize(v) {++ var stage = v.querySelector('[data-ct-viewer-stage]');++ var img = stage.querySelector('img');++ var size = v._size || {};++ return { width: (img && img.naturalWidth) || size.width || 1280, height: (img && img.naturalHeight) || size.height || 800, rect: stage.getBoundingClientRect() };++ }++++ function sendInput(v, value) {++ var task = v.dataset.task, desk = v.dataset.desk;++ if (!task || !desk) return;++ api.input(task, desk, value).catch(function (err) {++ var hint = v.querySelector('[data-ct-viewer-hint]');++ if (hint) hint.textContent = err.message || 'The desk did not take that.';++ });++ }++++ function point(v, e) {++ var s = stageSize(v);++ var x = Math.round((e.clientX - s.rect.left) / Math.max(1, s.rect.width) * s.width);++ var y = Math.round((e.clientY - s.rect.top) / Math.max(1, s.rect.height) * s.height);++ var ring = v.querySelector('.ct-viewer-ring');++ if (!ring) { ring = document.createElement('span'); ring.className = 'ct-viewer-ring'; v.querySelector('[data-ct-viewer-stage]').appendChild(ring); }++ ring.style.left = ((x / Math.max(1, s.width)) * 100).toFixed(2) + '%';++ ring.style.top = ((y / Math.max(1, s.height)) * 100).toFixed(2) + '%';++ return { x: x, y: y, width: s.width, height: s.height };++ }++ function onPointer(e) {++ if (!viewer || viewer.dataset.control !== '1') return;++ e.preventDefault();++ var p = point(viewer, e);++ sendInput(viewer, { action: e.button === 2 ? 'right_click' : 'click', x: p.x, y: p.y, width: p.width, height: p.height });++ viewer.querySelector('[data-ct-viewer-input]').focus();++ }++ function onWheel(e) {++ if (!viewer || viewer.dataset.control !== '1') return;++ e.preventDefault();++ var p = point(viewer, e);++ sendInput(viewer, { action: 'scroll', x: p.x, y: p.y, width: p.width, height: p.height, dy: e.deltaY > 0 ? 3 : -3 });++ }++ function onKey(e) {++ if (!viewer || viewer.dataset.control !== '1') return;++ var keys = { Enter: 'Return', Backspace: 'BackSpace', Tab: 'Tab', Escape: 'Escape', ArrowUp: 'Up', ArrowDown: 'Down', ArrowLeft: 'Left', ArrowRight: 'Right', Delete: 'Delete', ' ': 'space' };++ if (keys[e.key]) { e.preventDefault(); sendInput(viewer, { action: 'key', keys: keys[e.key] }); return; }++ if (e.key.length === 1 && !e.ctrlKey && !e.metaKey && !e.altKey) { e.preventDefault(); sendInput(viewer, { action: 'type', text: e.key }); }++ }++++ async function takeOver(v) {++ var hint = v.querySelector('[data-ct-viewer-hint]');++ try {++ await api.takeOver(v.dataset.task, v.dataset.desk);++ v.dataset.control = '1';++ v.dataset.state = 'control';++ v.querySelector('[data-ct-viewer-badge]').textContent = 'You are driving';++ v.querySelector('[data-ct-viewer-take]').hidden = true;++ v.querySelector('[data-ct-viewer-hand]').hidden = false;++ v.querySelector('[data-ct-viewer-note]').hidden = false;++ hint.textContent = 'You are driving. Actor waits; click and type on the picture.';++ v.querySelector('[data-ct-viewer-input]').focus();++ } catch (err) {++ hint.textContent = err.message || 'Actor could not hand over the desk.';++ }++ }++ async function handBack(v) {++ var note = (v.querySelector('[data-ct-viewer-note-input]') || {}).value || '';++ try {++ await api.handBack(v.dataset.task, v.dataset.desk, note);++ v.dataset.control = '0';++ v.dataset.state = 'watch';++ v.querySelector('[data-ct-viewer-badge]').textContent = 'Actor is working';++ v.querySelector('[data-ct-viewer-take]').hidden = false;++ v.querySelector('[data-ct-viewer-hand]').hidden = true;++ v.querySelector('[data-ct-viewer-note]').hidden = true;++ v.querySelector('[data-ct-viewer-note-input]').value = '';++ v.querySelector('[data-ct-viewer-hint]').textContent = note.trim() ? 'Handed back with your note. Actor goes on.' : 'Handed back. Actor goes on.';++ } catch (err) {++ v.querySelector('[data-ct-viewer-hint]').textContent = err.message || 'Actor could not take the desk back.';++ }++ }++ function pauseViewer(v) {++ var paused = v.dataset.paused === '1';++ v.dataset.paused = paused ? '0' : '1';++ v.querySelector('[data-ct-viewer-pause]').textContent = paused ? 'Pause' : 'Resume';++ var stage = v.querySelector('[data-ct-viewer-stage]');++ if (!paused) {++ // Pause the picture: the Live player leaves, so the desk's encoder stops when nobody else watches.++ stage.querySelectorAll('iframe').forEach(function (f) { f.remove(); });++ if (v._img) v._img.hidden = false;++ v._still();++ } else if (v._live) {++ var frame = document.createElement('iframe');++ frame.src = v._live.embed_url;++ frame.title = 'The desk Actor works on, live';++ frame.setAttribute('allow', 'autoplay; fullscreen; clipboard-write');++ frame.setAttribute('referrerpolicy', 'no-referrer');++ frame.setAttribute('sandbox', 'allow-scripts allow-same-origin allow-forms allow-pointer-lock');++ stage.insertBefore(frame, stage.firstChild);++ if (v._img) v._img.hidden = true;++ }++ }++ function stopViewer(v) {++ fetch('/tasks/' + encodeURIComponent(v.dataset.task) + '/cancel', { method: 'POST', credentials: 'same-origin', redirect: 'manual' })++ .catch(function () {}).then(function () { closeViewer(); });++ }++++ // ── Replay ──────────────────────────────────────────────────────────────────────────────────────────────────────++ function scanReplays() {++ document.querySelectorAll('[data-ct-replay]').forEach(function (box) {++ if (seen.has(box)) return;++ seen.add(box);++ var data = null;++ try { data = JSON.parse((box.querySelector('[data-ct-replay-data]') || {}).textContent || '[]'); } catch { data = []; }++ var range = box.querySelector('[data-ct-replay-range]');++ var img = box.querySelector('[data-ct-replay-img]');++ var caption = box.querySelector('[data-ct-replay-caption]');++ var link = box.querySelector('[data-ct-replay-link]');++ function show(i) {++ var st = data[i] || null;++ if (!st) return;++ range.value = String(i);++ caption.textContent = captionOf(st);++ if (st.still && img) { img.src = st.still.url; img.hidden = false; if (link) { link.href = st.still.url; link.hidden = false; } }++ else if (img) { img.hidden = true; if (link) link.hidden = true; }++ box.querySelectorAll('[data-ct-replay-step]').forEach(function (b) { b.classList.toggle('is-on', Number(b.getAttribute('data-ct-replay-step')) === i); });++ }++ function captionOf(st) {++ if (!st) return '';++ if (st.tool === 'computer_input') {++ if (st.action === 'click' || st.action === 'double_click' || st.action === 'right_click') return 'Clicked at ' + Math.round(st.x || 0) + ', ' + Math.round(st.y || 0);++ if (st.action === 'scroll') return 'Scrolled';++ if (st.action === 'key') return 'Pressed a key';++ if (st.action === 'move') return 'Moved the mouse';++ return 'Typed';++ }++ if (st.tool === 'computer_browser') {++ if (st.action === 'open') return 'Opened a page' + (st.url ? ': ' + st.url : '');++ if (st.action === 'read') return 'Read the page';++ if (st.action === 'click') return 'Clicked on the page';++ if (st.action === 'fill') return 'Filled a field';++ if (st.action === 'press') return 'Pressed a key';++ if (st.action === 'back') return 'Went back';++ return 'Used the browser';++ }++ if (st.tool === 'computer_screen') return 'Looked at the screen';++ return st.action === 'start' ? 'Started the desk' : st.action === 'stop' ? 'Stopped the desk' : 'Checked the desk';++ }++ if (range) range.addEventListener('input', function () { show(Number(range.value) || 0); });++ box.querySelectorAll('[data-ct-replay-step]').forEach(function (b) {++ b.addEventListener('click', function () { show(Number(b.getAttribute('data-ct-replay-step')) || 0); });++ });++ if (data.length) show(0);++ });++ }++++ function syncDesk(turn, next) {++ var now = turn.querySelector('[data-ct-desk]'), fresh = next.querySelector('[data-ct-desk]');++ if (now && fresh) syncMeta(now, fresh);++ else if (now && !fresh) now.remove();++ else if (!now && fresh) {++ var reply = turn.querySelector('.ovt-reply');++ if (reply) reply.insertBefore(fresh, reply.firstChild);++ scan();++ }++ }++++ document.addEventListener('click', function (e) {++ var take = e.target.closest && e.target.closest('[data-ct-takeover]');++ if (take) { var t = take.closest('[data-ct-desk]'); if (t) openTile(t, true); return; }++ var open = e.target.closest && e.target.closest('[data-ct-desk-open]');++ if (open) { var tile = open.closest('[data-ct-desk]'); if (tile) openTile(tile, false); }++ });++ window.OVDesks = { scan: scan, sync: syncDesk, open: openTile };++ if (document.readyState === 'loading') document.addEventListener('DOMContentLoaded', scan); else scan();++ document.addEventListener('ov:boost:load', scan);++})();@@@ -316,9 +316,8 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensO* Mouse, scroll, typing or one key on the machine (contracts 0.171.0 `input`): an acting command, so at level "ask"* it waits for the owner's approval there. x, y are measured in a screen picture of width × height (computer_screen's).*/-- async function input(subject, machine, args, { signal } = {}) {-- const bad = guard(subject, machine);-- if (bad) return bad;++ /** A validated `input` value for a person's own mouse and keyboard, or { error }. */++ function inputValue(args) {const a = args && typeof args === 'object' ? args : {};if (!INPUT_ACTIONS.has(a.action)) return { error: `action is one of ${[...INPUT_ACTIONS].join(', ')}` };const value = { action: a.action, ...onDesk(a) };@@@ -343,9 +342,69 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensOif (typeof a.keys !== 'string' || !/^[A-Za-z0-9_+-]{1,64}$/.test(a.keys) || a.keys.startsWith('-')) return { error: 'key needs keys: one key or chord, for example Return, Escape, ctrl+s' };value.keys = a.keys;}-- const r = await command(subject, machine, 'input', value, { signal, windowMs: ACT_WINDOW_MS });++ return { value };++ }++ async function input(subject, machine, args, { signal } = {}) {++ const bad = guard(subject, machine);++ if (bad) return bad;++ const made = inputValue(args);++ if (made.error) return made;++ const r = await command(subject, machine, 'input', made.value, { signal, windowMs: ACT_WINDOW_MS });++ if (r.error) return r;++ return { done: !!r.data.done, action: r.data.action || made.value.action, ...(r.data.x != null ? { x: r.data.x, y: r.data.y } : {}) };++ }++++ /**++ * The person's own mouse and keyboard on their Link desk (See it work, W1: "their taps and keys on the picture map++ * to desk coordinates"). Actor passes the person's own Network token to Bot, never its service token: it is their++ * input, at their own level, so Bot's "ask" does not ask them to approve themselves. A Run desk uses the session's++ * person-commands route instead.++ */++ async function personInput(subject, token, machine, args, { signal } = {}) {++ const bad = guard(subject, machine);++ if (bad) return bad;++ if (SESSION_MACHINE.test(String(machine || ''))) return { error: 'a Run desk takes the person\'s input through its own session' };++ if (!token) return { error: 'sign in again: your own session is needed to use the desk' };++ const made = inputValue(args);++ if (made.error) return made;++ const sig = signal ? AbortSignal.any([signal, AbortSignal.timeout(ACT_WINDOW_MS + 10_000)]) : AbortSignal.timeout(ACT_WINDOW_MS + 10_000);++ let res;++ try {++ res = await fetchImpl(`${base}/api/v1/machines/${encodeURIComponent(machine)}/commands`, {++ method: 'POST',++ headers: { Accept: 'application/json', 'Content-Type': 'application/json', Authorization: `Bearer ${token}` },++ body: JSON.stringify({ kind: 'input', value: made.value, ms: ACT_WINDOW_MS }), signal: sig,++ });++ } catch { return { error: 'Bot did not answer' }; }++ const data = await res.json().catch(() => null);++ if (!res.ok) return problem({ status: res.status, data }, 'your input');++ if (!data || data.result !== 'ack') return { error: (data && (data.reason || (data.result === 'expired' ? 'the machine did not answer in time' : null))) || 'the machine refused it' };++ return { done: true, action: (data.data && data.data.action) || made.value.action };++ }++++ /** A private agent stream for the task's desk on Link (See it work, W1; Link docs/stream.md). */++ const DESK_STREAM_ACTIONS = new Set(['start', 'stop', 'status']);++ async function deskStream(subject, machine, args, { signal } = {}) {++ const bad = guard(subject, machine);++ if (bad) return bad;++ if (SESSION_MACHINE.test(String(machine || ''))) return { error: 'a Run desk has no Link stream: use its still' };++ const a = args && typeof args === 'object' ? args : {};++ if (!DESK_STREAM_ACTIONS.has(a.action)) return { error: 'action is start, stop or status' };++ const value = { action: a.action };++ if (a.desk != null) value.desk = String(a.desk).slice(0, 40);++ if (a.action === 'start') {++ if (a.title != null) value.title = String(a.title).replace(/\s+/g, ' ').trim().slice(0, 120);++ if (a.task && typeof a.task === 'object') value.task = { source: ['codes', 'actor', 'desk', 'node'].includes(a.task.source) ? a.task.source : 'actor', ref: String(a.task.ref || '').slice(0, 100), title: String(a.task.title || '').slice(0, 120), ...(typeof a.task.url === 'string' && /^https:\/\//.test(a.task.url) ? { url: a.task.url.slice(0, 300) } : {}) };++ value.visibility = ['private', 'unlisted', 'public'].includes(a.visibility) ? a.visibility : 'private';++ if (Array.isArray(a.allow_sites)) value.allow_sites = a.allow_sites.filter((u) => typeof u === 'string' && /^https:\/\//.test(u)).slice(0, 20);++ }++ const r = await command(subject, machine, 'desk.stream', value, { signal, windowMs: a.action === 'status' ? READ_WINDOW_MS : 120_000 });if (r.error) return r;-- return { done: !!r.data.done, action: r.data.action || a.action, ...(r.data.x != null ? { x: r.data.x, y: r.data.y } : {}) };++ const d = r.data || {};++ if (a.action === 'stop') return { stopped: !!d.stopped, ...(d.id ? { id: d.id } : {}) };++ if (a.action === 'status') return { streaming: !!d.streaming, ...(d.id ? { id: d.id } : {}), ...(d.page_url ? { page_url: d.page_url } : {}) };++ if (!d.id || !/^ags_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(d.id))) return { error: 'the machine did not start a stream' };++ return { streaming: !!d.streaming, id: d.id, page_url: d.page_url || null, visibility: d.visibility || 'private', expires_at: d.expires_at || null, capture: d.capture || null };}/**@@@ -476,7 +535,7 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensOreturn { url: data.url, expires_at: data.expires_at || null, control: !!data.control, desk: data.desk || desk };}-- return { enabled, mine, list, read, run, write, screen, input, browser, desk, deskTurn, watchUrl, watchLink, sessionList, profileSites, personSession, openSession, endSession, fillLogin, still, sessionsOn, isSession: (m) => SESSION_MACHINE.test(String(m || '')) };++ return { enabled, mine, list, read, run, write, screen, input, personInput, browser, desk, deskStream, deskTurn, watchUrl, watchLink, sessionList, profileSites, personSession, openSession, endSession, fillLogin, still, sessionsOn, isSession: (m) => SESSION_MACHINE.test(String(m || '')) };}module.exports = { createComputers };nnew file mode 100644@@@ -0,0 +1,49 @@++'use strict';++++/**++ * OpenVibe.Live, the small part Actor needs for "See it work" (Agent Streams S1): a view ticket for an agent stream,++ * and the embed address that carries it. The desk publishes the stream (Link's desk.stream creates it with the machine's++ * own node token); Actor only mints the person's ten-minute view ticket with the person's own token, the way Live's docs++ * say a first-party page does, and puts it in `/embed/agent/<ags_id>?vt=…`.++ *++ * Actor never keeps the ticket: it is returned to the page request that asked, lives ten minutes, and renews there.++ * Without a person token there is no ticket, and the page falls back to the still.++ */++const STREAM_RE = /^ags_[0-9A-HJKMNP-TV-Z]{26}$/;++++function createLive({ config, fetchImpl = globalThis.fetch } = {}) {++ const base = config && config.services && config.services.liveUrl;++ const publicBase = ((config && config.services && config.services.livePublicUrl) || base || '').replace(/\/+$/, '');++ const enabled = !!base;++++ async function viewTicket(token, streamId, { signal } = {}) {++ if (!enabled) return { error: 'OpenVibe.Live is not connected here' };++ if (!token) return { error: 'a view ticket needs the person\'s own session' };++ if (!STREAM_RE.test(String(streamId || ''))) return { error: 'not an agent stream' };++ const sig = signal ? AbortSignal.any([signal, AbortSignal.timeout(8000)]) : AbortSignal.timeout(8000);++ let res;++ try {++ res = await fetchImpl(`${base}/api/streams/agent/${encodeURIComponent(streamId)}/view-ticket`, {++ method: 'POST',++ headers: { Accept: 'application/json', 'Content-Type': 'application/json', Authorization: `Bearer ${token}` },++ body: '{}', signal: sig,++ });++ } catch {++ return { error: 'OpenVibe.Live did not answer' };++ }++ const data = await res.json().catch(() => null);++ if (!res.ok || !data || !data.view_ticket) {++ return { error: (data && (data.detail || data.error)) || `OpenVibe.Live answered ${res.status}`, status: res.status || 502 };++ }++ return {++ stream_id: streamId,++ view_ticket: data.view_ticket,++ expires_at: data.expires_at || null,++ page_url: `${publicBase}/embed/agent/${encodeURIComponent(streamId)}?vt=${encodeURIComponent(data.view_ticket)}`,++ };++ }++++ return { enabled, viewTicket, base, publicBase };++}++++module.exports = { createLive, STREAM_RE };@@@ -23,6 +23,7 @@ const { createKeyStore } = require('./auth/keys');const { createSso } = require('./auth/sso');const catalog = require('./agents/catalog');const { createAdapters } = require('./agents/adapters');++const { createLive } = require('./agents/live');const { createModels } = require('./runtime/models');const createAiCatalogConfig = require('./ai-catalog-config').create;const { createRegistry } = require('./runtime/tools');@@@ -163,9 +164,11 @@ async function createApp(opts = {}) {const computers = (opts.bot && opts.bot.computers) || require('./agents/computers').createComputers({ config, fetchImpl: providerFetch, log });// An agent's money (plan HO7): the client that pays a tip through OpenVibe.Tips with the person's own token.const tips = require('./agents/spend').createTips({ config, fetchImpl });++ // OpenVibe.Live: a view ticket for an agent desk's private stream, minted with the person's own token.++ const live = opts.live || createLive({ config, fetchImpl });// An agent answers the mail sent to its own address (track HO6): the consumer below takes the event, the engine// reports each run's end back here, and the approvals decide() asks it to send an approved draft.-- const ctx = { config, s, keys, aiCatalogConfig, sso, stream, adapters, engine, webhooks, principal, projects, log, chatAsks, vibeDelivery, models, registry, loop, notifier, stats, checkpoints, connectors, waits, sharing, computers, triggers, skillRun, media, tips };++ const ctx = { config, s, keys, aiCatalogConfig, sso, stream, adapters, engine, webhooks, principal, projects, log, chatAsks, vibeDelivery, models, registry, loop, notifier, stats, checkpoints, connectors, waits, sharing, computers, live, triggers, skillRun, media, tips };const fabricMail = require('./mail/client').createMailClient({ config, fetchImpl: opts.mailFetch || fetchImpl, tokens: opts.mailTokens || null });ctx.fabric = require('./fabric/service').createFabric({ s, enabled: !!config.fabric?.enabled, readMail: async (args) => {const out = await fabricMail.getMessage(args);@@@ -247,7 +250,7 @@ async function createApp(opts = {}) {// The navbar's Messages count asks Chat (openvibe-shared 4.130.0).connectSrc: ["'self'", 'https://openvibe.chat', 'https://openvibe.network', 'https://cloudflareinsights.com', 'https://openvibe.events'],// Bot's watch links: a chat turn frames the desk the agent works on, live (plan L55 C1b).-- frameSrc: ["'self'", 'https://openvibe.network', ...(() => { try { return [new URL(config.services.botPublicUrl).origin]; } catch { return []; } })()],++ frameSrc: ["'self'", 'https://openvibe.network', ...(() => { try { return [new URL(config.services.botPublicUrl).origin]; } catch { return []; } })(), ...(() => { try { return [new URL(config.services.livePublicUrl).origin]; } catch { return []; } })()],frameAncestors: ["'none'"],objectSrc: ["'none'"],baseUri: ["'self'"],@@@ -177,6 +177,10 @@ function load(env = process.env) {// OpenVibe.Run: a browser session on a desk host when none of the person's own machines has a desk for the// task (Desks plan D6). It is also where a skill's code runs (ADR-060). Empty turns it off.runUrl: trim(env.ACTOR_RUN_URL != null ? env.ACTOR_RUN_URL : (selfHosted ? '' : 'http://127.0.0.1:4920')),++ // OpenVibe.Live: an agent desk's private stream and its embed (Agent Streams S1). Empty turns the live++ // tile off (the still fallback stays). The person's own token mints the view ticket; Actor never stores it.++ liveUrl: trim(env.ACTOR_LIVE_URL != null ? env.ACTOR_LIVE_URL : 'https://openvibe.live'),++ livePublicUrl: trim(env.ACTOR_LIVE_PUBLIC_URL != null ? env.ACTOR_LIVE_PUBLIC_URL : 'https://openvibe.live'),// OpenVibe.Media: where a sandbox skill's code bytes are stored, private to Actor's tenant (ADR-060).// Empty means a sandbox skill cannot be published (a prompt skill is unaffected).mediaUrl: trim(env.ACTOR_MEDIA_URL != null ? env.ACTOR_MEDIA_URL : (selfHosted ? '' : 'http://127.0.0.1:4100')),@@@ -46,6 +46,8 @@ const timeline = require('../agents/timeline');const assignments = require('../agents/assignments');const modelPrefs = require('../runtime/model-prefs');const driving = require('../agents/driving');++const stills = require('../tasks/stills');++const replay = require('../tasks/replay');const canvases = require('../canvases');const skills = require('../skills');@@@ -68,6 +70,13 @@ function createApi(ctx) {};const triggerFail = (req, res, out) => problem(req, res, (out.error && out.error.status) || 502, (out.error && out.error.code) || 'actor.trigger.failed', (out.error && out.error.detail) || 'The trigger could not be saved.');const triggerOwner = (req) => String(req.principal.requester || '').replace(/^user:/, '');++ // The person's own token, for the calls only they may make: Live's view ticket and Bot's own input route.++ const personToken = (req) => {++ if (!req.principal || req.principal.kind !== 'user') return null;++ const h = String(req.headers.authorization || '');++ if (h.startsWith('Bearer ')) return h.slice(7).trim();++ return req.token || (req.viewer && req.viewer.token) || null;++ };r.use(express.json({ limit: '64kb' }));r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });@@@ -720,10 +729,141 @@ function createApi(ctx) {} catch { ctx.log.warn('[Actor] task view issue failed', req.params.id); res.sendStatus(503); }});-- // Take over / Give it back (plan 2026-11-07), built on the hand_to_person machinery: the person drives the desk-- // they were handed, the agent's input is refused while they do, and the task's timeline gets one hand-off step.-- // Only the task's own people (load()), and only while it runs on a Computer.-- async function handoff(req, res, { take }) {++ // A still is its task's person's, and nobody else's: a signed-out visitor and an app token get the same 404 as a++ // task that does not exist (never a 401 that would confirm one).++ const stillPerson = (req, res, next) => ((req.principal && req.principal.kind === 'user') ? next() : problem(req, res, 404, 'actor.task.not_found', 'No such task.'));++++ // ── See it work (W1): the task's desk, its private Live stream, the person's control, and the replay ─────────++ // One description of where a task's desk is, from its own steps: a Link desk on the person's machine, or a Run++ // session on a desk host. Nothing here trusts the model; the desk and the Computer are read from tool events.++ async function deskTarget(row) {++ const computer = await store.taskComputer(s, row);++ const desk = await store.taskDesk(s, row);++ if (computer && computer.session_id) {++ return { id: `run:${computer.session_id}`, where: 'run', session_id: computer.session_id,++ machine: null, desk: null, kind: 'browser', computer, label: computer.where === 'yours' ? `${computer.machine || 'Your computer'} · a desk host` : 'OpenVibe computer' };++ }++ const place = desk && store.deskPlace(desk.url);++ if (!place) return null;++ return { id: `link:${place.machine}:${place.desk}`, where: 'link', session_id: null, machine: place.machine,++ desk: place.desk, kind: desk.kind || 'desktop', computer, label: `${(computer && computer.machine) || 'Your PC'} · ${place.desk === 'main' ? 'your screen' : 'its own desk'}` };++ }++ async function browserUrlOf(row) {++ const events = await store.eventsAfter(s, row.id, 0);++ for (let i = events.length - 1; i >= 0; i--) {++ const e = events[i];++ if (e && e.kind === 'output' && e.step === 'tool_result' && e.tool === 'computer_browser' && !e.is_error) {++ try { const d = JSON.parse(e.text || '{}'); if (d && typeof d.url === 'string' && /^https?:\/\//.test(d.url)) return d.url.slice(0, 500); } catch { /* not JSON */ }++ }++ }++ return null;++ }++ async function deskView(row, req, target) {++ const token = personToken(req);++ const desk = await store.taskDesk(s, row);++ let live = null;++ if (target.where === 'link' && desk && desk.stream && desk.stream.id && token && ctx.live && ctx.live.enabled) {++ const view = await ctx.live.viewTicket(token, desk.stream.id).catch(() => null);++ // The embed must be Live's own, never an address another service convinced it to return.++ let embed = null;++ try {++ const u = new URL(String(view && view.page_url || ''));++ const expected = new URL(String(ctx.live.publicBase || 'https://openvibe.live'));++ if (/^https?:$/.test(u.protocol) && u.origin === expected.origin) embed = u.href;++ } catch { embed = null; }++ if (view && !view.error && embed) {++ live = { stream_id: desk.stream.id, embed_url: embed, page_url: embed, visibility: desk.stream.visibility || 'private', expires_at: view.expires_at || null };++ }++ }++ const pictures = await stills.list(s, row.id).catch(() => []);++ const last = pictures.length ? pictures[pictures.length - 1] : null;++ return {++ id: target.id, where: target.where, machine: target.machine, desk: target.desk, kind: target.kind,++ label: target.label, browser_url: target.where === 'link' ? await browserUrlOf(row) : null,++ width: (last && last.width) || 1280, height: (last && last.height) || 800,++ live,++ still: target.where === 'run' ? `${config.baseUrl}/api/v1/tasks/${row.id}/still` : `${config.baseUrl}/api/v1/tasks/${row.id}/desk-still?desk=${encodeURIComponent(target.desk || 'private')}`,++ driving: driving.is(row.id, s.now()),++ };++ }++ r.get('/tasks/:id/desks', principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {++ const row = await load(req, res);++ if (!row) return;++ const target = await deskTarget(row);++ res.json({ desks: target ? [await deskView(row, req, target)] : [] });++ });++++ // The still fallback: Link's screen through Bot (at most once a second, served from a one-frame cache), or Run's++ // session still. Only the task's own person; a picture must never confirm a task or a desk exists.++ const stillFrame = new Map(); // task id -> { at, type, bytes }++ r.get('/tasks/:id/desk-still', stillPerson, principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {++ const row = await load(req, res);++ if (!row) return;++ const target = await deskTarget(row);++ if (!target) return problem(req, res, 404, 'actor.task.no_still', 'This task has no desk picture.');++ if (target.where === 'run') return problem(req, res, 404, 'actor.task.no_still', 'Use the session still.');++ const key = `${row.id}:${target.desk}`;++ const kept = stillFrame.get(key);++ const at = Date.now();++ if (kept && at - kept.at < 1000) {++ res.set('Cache-Control', 'private, no-store'); res.set('X-Content-Type-Options', 'nosniff'); res.type(kept.type).send(kept.bytes); return;++ }++ const shot = await ctx.computers.screen(row.requester.slice(5), target.machine, { desk: target.desk, maxWidth: 640 });++ if (!shot || shot.error) return problem(req, res, 502, 'actor.task.no_still', (shot && shot.error) || 'The desk sent no picture.');++ const type = shot.format === 'png' ? 'image/png' : 'image/jpeg';++ const bytes = Buffer.from(String(shot.content_b64 || ''), 'base64');++ if (!bytes.length) return problem(req, res, 502, 'actor.task.no_still', 'The desk sent no picture.');++ stillFrame.set(key, { at, type, bytes });++ while (stillFrame.size > 500) stillFrame.delete(stillFrame.keys().next().value);++ res.set('Cache-Control', 'private, no-store'); res.set('X-Content-Type-Options', 'nosniff'); res.type(type).send(bytes);++ });++++ // The person's own input on the picture (Link: their token at Bot; Run: the session's person-commands route).++ r.post('/tasks/:id/desks/:desk/input', principal.requireCapability('actor.task.create'), async (req, res) => {++ const row = await load(req, res);++ if (!row) return;++ if (!store.OPEN.includes(row.state)) return problem(req, res, 409, 'actor.task.ended', 'This task has ended: its desk is gone.');++ if (!driving.is(row.id, s.now())) return problem(req, res, 409, 'actor.task.not_driving', 'Take over the desk before using it.');++ const target = await deskTarget(row);++ if (!target || target.id !== String(req.params.desk || '')) return problem(req, res, 404, 'actor.task.no_computer', 'This task has no such desk.');++ const subject = row.requester.slice(5);++ const body = req.body && typeof req.body === 'object' ? req.body : {};++ let out;++ if (target.where === 'run') out = await ctx.computers.personSession(subject, target.session_id, 'input', { kind: 'input', value: body }).catch((err) => ({ error: (err && err.message) || 'the session refused it' }));++ else {++ const token = personToken(req);++ if (!token) return problem(req, res, 401, 'actor.task.sign_in', 'Sign in again: your own session is needed to use the desk.');++ out = await ctx.computers.personInput(subject, token, target.machine, { ...body, desk: target.desk });++ }++ if (!out || out.error) return problem(req, res, 502, 'actor.task.input_failed', (out && out.error) || 'The desk did not take that.');++ return res.json({ done: true, action: out.action || body.action || null });++ });++++ r.get('/tasks/:id/replay', principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {++ const row = await load(req, res);++ if (!row) return;++ res.json({ task_id: row.id, steps: await replay.steps(s, row, { baseUrl: config.baseUrl }) });++ });++ r.get('/tasks/:id/replay/:seq/still', stillPerson, principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {++ const row = await load(req, res);++ if (!row) return;++ const seq = Number.parseInt(req.params.seq, 10);++ if (!Number.isInteger(seq) || seq < 1) return problem(req, res, 404, 'actor.task.no_still', 'No such picture.');++ const shot = await stills.get(s, row.id, seq);++ if (!shot || !shot.bytes) return problem(req, res, 404, 'actor.task.no_still', 'No such picture.');++ const bytes = Buffer.isBuffer(shot.bytes) ? shot.bytes : Buffer.from(shot.bytes);++ res.set('Cache-Control', 'private, no-store');++ res.set('X-Content-Type-Options', 'nosniff');++ res.set('Content-Type', shot.mime);++ res.set('Content-Length', String(bytes.length));++ return res.status(200).end(bytes);++ });++++ // Take over / Give it back (plan 2026-11-07; See it work, W1): the person drives the desk they were handed, the++ // agent's input is refused while they do, and the task's timeline gets one hand-off step. A Run desk's lease is++ // really taken and given back; a Link desk's input is the person's own (their token at Bot). A note handed back is++ // the person's own message to the run.++ async function handoff(req, res, { take, deskId = null, note = null }) {const row = await load(req, res);if (!row) return;if (!store.OPEN.includes(row.state)) return problem(req, res, 409, 'actor.task.ended', 'This task has ended: its Computer is gone.');@@@ -731,16 +871,26 @@ function createApi(ctx) {const desk = await store.taskDesk(s, row);if (!computer && !desk) return problem(req, res, 409, 'actor.task.no_computer', 'This task has no Computer to hand over.');if (!String(row.requester).startsWith('user:')) return problem(req, res, 409, 'actor.task.no_person', 'A Computer is a person\'s: sign in as the task\'s own person to take it.');++ const target = await deskTarget(row);++ if (deskId && (!target || target.id !== deskId)) return problem(req, res, 404, 'actor.task.no_computer', 'This task has no such desk.');++ const subject = row.requester.slice(5);++ if (target && target.where === 'run' && ctx.computers.personSession) {++ const lease = take++ ? await ctx.computers.personSession(subject, target.session_id, 'take').catch((err) => ({ error: err.message }))++ : await ctx.computers.personSession(subject, target.session_id, 'back').catch((err) => ({ error: err.message }));++ if (lease && lease.error) return problem(req, res, 502, 'actor.session.control_failed', lease.error);++ }const at = s.iso();const now = take ? (driving.take(row.id, s.now()), true) : (driving.give(row.id), false);-- // The step in the task's own timeline (the same events /events streams), carrying the Computer.++ if (!take && typeof note === 'string' && note.trim()) {++ await engine.steer(row.id, note.trim().slice(0, 1000)).catch(() => null);++ }const event = await store.appendEvent(s, row.id, {at, kind: 'output', step: 'handoff',text: now ? 'You took over the desk. The agent waits until you give it back.' : 'You gave the desk back. The agent goes on.',...(computer ? { computer } : {}),});stream.publish(row.id, event);-- // And one step in the one timeline behind the Dock (agent.timeline-entry@1), the same Computer.try {await timeline.upsert(s, row.requester, {step_id: `${row.id}:takeover`, mode: 'browse', product: 'actor', ref: row.id,@@@ -756,14 +906,15 @@ function createApi(ctx) {res.json(out);}r.post('/tasks/:id/takeover', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: true }));-- r.post('/tasks/:id/giveback', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: false }));++ r.post('/tasks/:id/giveback', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: false, note: req.body && typeof req.body.note === 'string' ? req.body.note : null }));++ r.post('/tasks/:id/desks/:desk/take-over', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: true, deskId: String(req.params.desk || '') }));++ r.post('/tasks/:id/desks/:desk/hand-back', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: false, deskId: String(req.params.desk || ''), note: req.body && typeof req.body.note === 'string' ? req.body.note : null }));// The Computer's newest frame, fetched from the session's own service (Run) as Actor, acting for the task's// person: the person's own token would have to cross an origin, and the picture is theirs. Owner/project only// (load()); a task with no Run session has no picture, and neither has anyone asking for another person's task.// A still is its task's person's, and nobody else's: a signed-out visitor and an app token get the same 404 as a// task that does not exist (never a 401 that would confirm one).-- const stillPerson = (req, res, next) => ((req.principal && req.principal.kind === 'user') ? next() : problem(req, res, 404, 'actor.task.not_found', 'No such task.'));r.get('/tasks/:id/still', stillPerson, principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {const row = await load(req, res);if (!row) return;@@@ -34,6 +34,7 @@ const inputs = require('../tasks/inputs');const modelPrefs = require('../runtime/model-prefs');const aiCatalog = require('../runtime/ai-catalog');const canvases = require('../canvases');++const replay = require('../tasks/replay');// The model a person's tasks run on (plan L55 C3; the owner, 11-01: "use your own API keys, or OpenVibe's"). Auto is// OpenVibe's own; "own:<provider>" is the person's key, sealed in OpenVibe.AI, which Actor never sees.@@@ -240,6 +241,131 @@ ${from ? html`<input type="hidden" name="from" value="${from}">` : ''}<label forreturn raw(thread.work({ open, live: LIVE_TEXT[row.state] || 'Working', took: tookOf(row) || null, rows: workRows(lines) }));}++ /**++ * Where a running task's desk is (See it work, W1): Link's desk on the person's machine or a Run session on a desk++ * host, read from the task's own tool results. → { where, id, machine, desk, kind, label, still } or null.++ */++ function deskContext(events) {++ const desk = store.deskFrom(events);++ const computer = store.computerFrom(events);++ if (computer && computer.session_id) {++ return { where: 'run', id: `run:${computer.session_id}`, session_id: computer.session_id, machine: null, desk: null,++ kind: 'browser', label: computer.where === 'yours' ? `${computer.machine || 'Your computer'} · a desk host` : 'OpenVibe computer',++ still: `/api/v1/tasks/__TASK__/still` };++ }++ const place = desk && store.deskPlace(desk.url);++ // The person's own screen is never framed: an agent's desk is private or desk-<name>.++ if (!place || place.desk === 'main') return null;++ return { where: 'link', id: `link:${place.machine}:${place.desk}`, machine: place.machine, desk: place.desk,++ kind: desk.kind || 'desktop', label: `${(computer && computer.machine) || 'Your PC'} · ${place.desk === 'main' ? 'your screen' : 'its own desk'}`,++ still: `/api/v1/tasks/__TASK__/desk-still?desk=${encodeURIComponent(place.desk)}` };++ }++++ /** The latest browser address the desk's browser reported (a page address, never model text). */++ function browserUrlOf(events) {++ for (let i = events.length - 1; i >= 0; i--) {++ const e = events[i];++ if (!e || e.kind !== 'output' || e.step !== 'tool_result' || e.tool !== 'computer_browser' || e.is_error) continue;++ try { const d = JSON.parse(e.text || '{}'); if (d && typeof d.url === 'string' && /^https?:\/\//.test(d.url)) return d.url.slice(0, 500); } catch { /* not JSON */ }++ }++ return null;++ }++++ /** The last click or move, for the ring: x,y in the screen picture's pixels, and that picture's size. */++ function lastClick(events) {++ for (let i = events.length - 1; i >= 0; i--) {++ const e = events[i];++ if (!e || e.kind !== 'output' || e.step !== 'tool_call' || e.tool !== 'computer_input') continue;++ const a = e.args && typeof e.args === 'object' ? e.args : {};++ if (!['click', 'double_click', 'right_click', 'move'].includes(String(a.action))) continue;++ if (!Number.isFinite(a.x) || !Number.isFinite(a.y)) continue;++ return { x: Number(a.x), y: Number(a.y), width: Number(a.width) || 1280, height: Number(a.height) || 800 };++ }++ return null;++ }++++ /** The latest computer step in plain words, for the tile's one caption. Never the typed text. */++ function deskCaption(events) {++ for (let i = events.length - 1; i >= 0; i--) {++ const e = events[i];++ if (!e || e.kind !== 'output' || e.step !== 'tool_call') continue;++ const a = e.args && typeof e.args === 'object' ? e.args : {};++ if (e.tool === 'computer_input') {++ if (a.action === 'type') return 'Typing';++ if (a.action === 'click' || a.action === 'double_click' || a.action === 'right_click') return `Clicking at ${Math.round(Number(a.x) || 0)}, ${Math.round(Number(a.y) || 0)}`;++ if (a.action === 'scroll') return 'Scrolling';++ if (a.action === 'key') return 'Pressing a key';++ if (a.action === 'move') return 'Moving the mouse';++ return 'Using the computer';++ }++ if (e.tool === 'computer_browser') {++ if (a.action === 'open') return 'Opening a page';++ if (a.action === 'read') return 'Reading the page';++ if (a.action === 'click') return 'Clicking on the page';++ if (a.action === 'fill') return 'Typing in a field';++ if (a.action === 'press') return 'Pressing a key';++ if (a.action === 'back') return 'Going back';++ return 'Using the browser';++ }++ if (e.tool === 'computer_screen') return 'Looking at the screen';++ if (e.tool === 'computer_desk') return a.action === 'start' ? 'Starting its desk' : a.action === 'stop' ? 'Stopping its desk' : 'Checking its desk';++ }++ return 'Getting ready';++ }++++ /** A step waiting for the person (hand_to_person) becomes an Ask whose button opens the viewer in control. */++ function pendingAsk(events) {++ // The public body keeps the tool but not call_id; a hand_to_person stays open until its result arrives.++ let open = null;++ for (const e of events) {++ if (!e || e.kind !== 'output') continue;++ if (e.step === 'tool_call' && e.tool === 'hand_to_person') open = String((e.args && e.args.what) || 'Actor needs you on its desk').slice(0, 300);++ if (e.step === 'tool_result' && e.tool === 'hand_to_person') open = null;++ }++ return open;++ }++++ function deskTile(row, events, readOnly = false) {++ const d = deskContext(events);++ if (!d || readOnly) return '';++ const still = d.still.replace('__TASK__', row.id);++ const click = lastClick(events);++ const ring = click ? (() => {++ const left = Math.max(0, Math.min(100, (click.x / Math.max(1, click.width)) * 100));++ const top = Math.max(0, Math.min(100, (click.y / Math.max(1, click.height)) * 100));++ return `<span class="ct-desk-ring" data-ct-desk-ring aria-hidden="true" style="left:${left.toFixed(2)}%;top:${top.toFixed(2)}%"></span>`;++ })() : '';++ const page = browserUrlOf(events);++ const ask = pendingAsk(events);++ return html`<figure class="ct-desk-tile" data-ct-desk data-task="${row.id}" data-desk="${d.id}" data-where="${d.where}" data-kind="${d.kind}" data-label="${d.label}" data-still="${still}">++<div class="ct-desk-frame" data-ct-desk-frame>++<img data-ct-desk-still src="${still}" alt="Actor's desk" loading="lazy" decoding="async">++<span class="ct-desk-live" data-ct-desk-live hidden>Live</span>++<button type="button" class="ct-desk-open" data-ct-desk-open aria-label="Open the desk viewer" title="Open the desk viewer"><span>Watch</span></button>++${raw(ring)}++</div>++<figcaption><span data-ct-desk-caption>${deskCaption(events)}</span><span class="ct-desk-place">${d.label}</span>${page ? html`<a data-ct-desk-url href="${page}" target="_blank" rel="noopener nofollow" hidden>${String(page).replace(/^https?:\/\//, '').slice(0, 80)}</a>` : ''}</figcaption>++${ask ? html`<div class="ct-desk-ask" data-ct-desk-ask><p><b>Actor needs you:</b> ${ask}</p><button type="button" class="ct-desk-take" data-ct-takeover>Take over</button></div>` : ''}++</figure>`;++ }++++ /** A finished task's replay: the steps with stills, and the first picture until the person scrubs. */++ async function replayBlock(row) {++ const steps = await replay.steps(s, row);++ if (!steps.length) return '';++ const first = steps.find((x) => x.still) || null;++ const json = raw(JSON.stringify(steps).replace(/</g, '\\u003c').replace(/&/g, '\\u0026'));++ return html`<details class="ct-replay" data-ct-replay data-task="${row.id}">++<summary>Replay · ${String(steps.length)} step${steps.length === 1 ? '' : 's'}</summary>++<div class="ct-replay-body">++<input type="range" min="0" max="${String(Math.max(0, steps.length - 1))}" value="0" step="1" data-ct-replay-range aria-label="Scrub the replay">++<figure class="ct-replay-shot">${first ? html`<img data-ct-replay-img src="${first.still.url}" alt="What the step saw" loading="lazy">` : html`<span class="ct-replay-none">No pictures were kept for this run.</span>`}++<figcaption><span data-ct-replay-caption>${replay.caption(steps[0])}</span>${first ? html`<a data-ct-replay-link href="${first.still.url}" target="_blank" rel="noopener">Open the picture</a>` : ''}</figcaption></figure>++<ol class="ct-replay-steps">${steps.map((st, i) => html`<li><button type="button" data-ct-replay-step="${String(i)}"><span>${String(i + 1)}</span>${replay.caption(st)}${st.still ? '' : ' · no picture'}</button></li>`)}</ol>++<script type="application/json" data-ct-replay-data>${json}</script>++</div></details>`;++ }++/*** A turn that is running is a card in the conversation (one-agent system, stage 1: the owner, 11-12, "a more* modular system… reused in a shared modular way"), not a separate console panel: its title, its state, the last@@@ -262,6 +388,7 @@ ${from ? html`<input type="hidden" name="from" value="${from}">` : ''}<label forreturn `<li class="ovt-w-step"><span>${esc(clip(r.text, 200))}</span></li>`;};return html`<article class="ovt-card ct-run" data-ct-run="${row.id}">++${deskTile(row, events, readOnly)}<p class="ovt-card-h"><span class="ct-run-title">${clip(row.task, 160)}</span><span class="ct-run-state" data-ct-run-state>${LIVE_TEXT[row.state] || 'Working'}…</span></p><p class="ct-run-live" data-ovt-live>${live}</p><ol class="ct-run-steps" data-ovt-work>${raw(rows.map(item).join(''))}</ol>@@@ -523,7 +650,7 @@ ${shown.length ? html`<div class="ct-trig-runs">${shown}</div>` : ''}`;async function turnHtml(row, { events = null, files = null, readOnly = false } = {}) {const attached = files || (await inputs.list(s, [row.id])).get(row.id) || [];const t = store.toWire(row, { baseUrl: config.baseUrl });-- const evs = events || await store.eventsAfter(s, row.id, 0);++ const evs = events || await store.eventsDetailed(s, row.id, 0);const open = OPEN.includes(row.state);let reply;if (row.state === 'succeeded') reply = html`${workBlock(row, evs, false)}${screenBlock(row, evs, false)}${answerBlock(row, t)}`;@@@ -534,7 +661,14 @@ ${shown.length ? html`<div class="ct-trig-runs">${shown}</div>` : ''}`;// A running turn is a card in the conversation (one-agent system, stage 1). A turn parked on a wait still// says what it waits for (plan A2).const parked = ctx.waits && typeof ctx.waits.listForTask === 'function' ? (await ctx.waits.listForTask(row.id).catch(() => [])).find((w) => w.status === 'waiting') : null;-- reply = html`${runCard(row, evs, readOnly)}${screenBlock(row, evs, true)}${waitBlock(row, parked, readOnly)}`;++ // The desk's own tile is the live picture (See it work, W1): the old separate screen panel stays only for a++ // run that has no desk to stream (a single look), so there is never a second, empty Live box under the card.++ reply = html`${runCard(row, evs, readOnly)}${deskContext(evs) ? '' : screenBlock(row, evs, true)}${waitBlock(row, parked, readOnly)}`;++ }++ // See it work (W1): a finished run keeps a replay of the pictures its steps saw.++ if (!readOnly && ['succeeded', 'failed', 'cancelled'].includes(row.state)) {++ const shown = await replayBlock(row).catch(() => '');++ if (shown) reply = html`${reply}${shown}`;}// A routine the turn made is its card, after whatever the turn said (plan §12 A-4).const routines = await triggerBlocks(row, evs, readOnly);@@@ -674,7 +808,7 @@ ${parent ? html`<input type="hidden" name="parent" value="${parent}">` : ''}<texviewer: req.viewer, config, path: req.originalUrl, title, app: true, noReferrer: true,// The network's sidebar (kit.css, kit-web.js) and its conversation UI (thread.css, ov-thread.js); composer// v2 draws the box (ov-composer.js, 4.215.0).-- css: ['css/chat.css', 'css/skill-draft.css'], scripts: ['js/chat.js'], styles: ['kit.css', 'thread.css', 'ov-share.css', 'ov-upload.css'],++ css: ['css/chat.css', 'css/skill-draft.css'], scripts: ['js/desks.js', 'js/chat.js'], styles: ['kit.css', 'thread.css', 'ov-share.css', 'ov-upload.css'],sharedScripts: ['kit-web.js', 'ov-thread.js', 'ov-composer.js', 'ov-share.js', 'ov-canvas.js', 'ov-upload.js'],body: html`${busy && !shared ? raw('<noscript><meta http-equiv="refresh" content="5"></noscript>') : ''}<div class="chat" data-chat="${first ? first.id : ''}">${raw(thread.page({@@@ -20,7 +20,7 @@ const PERSON_QUIET_MS = 20_000;const AGENT_PROFILE = 'agent';function deskRun(taskId = null) {-- return { name: `desk-${crypto.randomBytes(5).toString('hex')}`, taskId, named: null, kinds: new Map(), keeps: new Set(), used: new Set(), placed: null, names: new Map() };++ return { name: `desk-${crypto.randomBytes(5).toString('hex')}`, taskId, named: null, kinds: new Map(), keeps: new Set(), used: new Set(), placed: null, names: new Map(), streams: new Map() };}/** The person's own word for their machine (their name for it), remembered from list_my_computers. */@@@ -72,11 +72,46 @@ async function closeDesks(run, requester, core) {await core.pcs.endSession(person, run.placed).catch(() => null); // the task's desk on a desk host ends with itrun.placed = null;}-- if (!run || !person || !run.used.size) return;++ if (!run || !person) return;++ // The Live stream goes before the desk it shows (See it work, W1): the desk's WHIP session ends with it, and the++ // private stream is ended, not left publishing a screen nobody owns.++ if (run.streams && run.streams.size && typeof core.pcs.deskStream === 'function') {++ await Promise.all([...run.streams.entries()].map(([machine, s]) => core.pcs.deskStream(person, machine, { action: 'stop', desk: s.desk }).catch(() => null)));++ run.streams.clear();++ }++ if (!run.used.size) return;await Promise.all([...run.used].map((machine) => core.pcs.desk(person, machine, { action: 'stop', desk: run.name }).catch(() => null)));run.used.clear();}++/**++ * Start the task's private Live stream for the desk it just started (See it work, W1). The stream is the same desk the++ * agent works on, private to the person, and its id/page travel in the computer_desk result; closeDesks ends it when the++ * task does. A refusal leaves the desk working and the tile falls back to the still.++ */++async function startDeskStream(run, person, core, ctx, machine, desk, out) {++ if (!run || !out || out.error || !person || !ctx || !ctx.taskId) return null;++ if (core.pcs.isSession && core.pcs.isSession(machine)) return null; // a Run desk has its still, no Link stream++ if (typeof core.pcs.deskStream !== 'function') return null;++ const title = String(ctx.personText || 'Actor task').replace(/\s+/g, ' ').trim().slice(0, 120) || 'Actor task';++ const stream = await core.pcs.deskStream(person, machine, {++ action: 'start', desk, title, visibility: 'private',++ task: { source: 'actor', ref: ctx.taskId, title, url: `${(ctx.config && ctx.config.baseUrl) || 'https://openvibe.actor'}/chat/${ctx.taskId}` },++ signal: ctx.signal,++ }).catch((err) => ({ error: (err && err.message) || 'the stream did not start' }));++ if (!stream || stream.error) return null;++ run.streams.set(machine, { desk, id: stream.id });++ return stream;++}++++async function stopDeskStream(run, person, core, machine, desk) {++ if (!run || !person || !core.pcs || typeof core.pcs.deskStream !== 'function') return null;++ const kept = run.streams && run.streams.get(machine);++ if (!kept || (desk && kept.desk !== desk)) return null;++ run.streams.delete(machine);++ return core.pcs.deskStream(person, machine, { action: 'stop', desk: kept.desk }).catch(() => null);++}++/** A look at a computer's screen: the picture to the vision tier with the question, its answer back. */async function lookAtScreen(person, a, ctx, core) {if (!ctx.models) return { error: 'no vision model is configured on this server, so the screen cannot be read' };@@@ -158,6 +193,6 @@ async function prepareDesk(tool, a, person, core, ctx) {}module.exports = {-- personOf, deskRun, deskFor, deskKind, placementOrder, closeDesks, lookAtScreen, handToPerson, prepareDesk, SCREEN_SYSTEM,++ personOf, deskRun, deskFor, deskKind, placementOrder, closeDesks, startDeskStream, stopDeskStream, lookAtScreen, handToPerson, prepareDesk, SCREEN_SYSTEM,PERSON_WAIT_MS, PERSON_POLL_MS, PERSON_QUIET_MS, AGENT_PROFILE, DESK_TOOLS, rememberMachines, machineWord,};@@@ -1,7 +1,7 @@'use strict';const { serviceFor, json, personOf } = require('./_openvibe');--const { prepareDesk, deskKind, placementOrder, AGENT_PROFILE, machineWord } = require('./_person');++const { prepareDesk, deskKind, placementOrder, AGENT_PROFILE, machineWord, startDeskStream, stopDeskStream } = require('./_person');/*** The Computer this task opened, as platform.task@1 carries it (plan 2026-11-07): the one id every surface reads.@@@ -39,7 +39,8 @@ async function startDesk(a, person, core, ctx) {// kept for their agent on that machine, and saves it back when it stops. Never the model's choice.delete body.profile;if (body.desk !== 'private' && desks && desks.keeps && desks.keeps.has(a.machine)) body.profile = AGENT_PROFILE;-- return core.pcs.desk(person, body.machine, body, { signal: ctx.signal });++ const out = await core.pcs.desk(person, body.machine, body, { signal: ctx.signal });++ return out && !out.error ? { ...out, desk: body.desk || 'private' } : out;}module.exports = {@@@ -78,8 +79,10 @@ module.exports = {out = await startDesk({ ...a, machine: id }, person, core, ctx);if (!(out && out.error && /limit|max_desks|already runs/i.test(out.error))) {if (out && !out.error && desks && !desks.placed) desks.placed = id;++ const stream = out && !out.error ? await startDeskStream(desks, person, core, ctx, id, out.desk || a.desk, out) : null;const computer = out && !out.error ? computerOf({ run: desks, machine: id }) : undefined;-- return { text: json(out && !out.error ? { machine: id, ...out } : out), data: out, ...(computer ? { computer } : {}) };++ const watch = out && !out.error ? core.pcs.watchUrl(id, out.desk || a.desk) : null;++ return { text: json(out && !out.error ? { watch, machine: id, ...out, ...(stream ? { stream } : {}) } : out), data: out, ...(computer ? { computer } : {}) };}}// None of the person's own machines has room (or none runs desks): a browser desk on a desk host,@@@ -103,9 +106,11 @@ module.exports = {if (a.action === 'start') {const out = await startDesk(a, person, core, ctx);if (out && !out.error && desks && !desks.placed) desks.placed = a.machine;++ const stream = out && !out.error ? await startDeskStream(desks, person, core, ctx, a.machine, out.desk || a.desk, out) : null;const computer = out && !out.error ? computerOf({ run: desks, machine: a.machine }) : undefined;-- return { text: json(out && out.running && !(core.pcs.isSession && core.pcs.isSession(a.machine)) ? { watch: core.pcs.watchUrl(a.machine, a.desk), ...out } : out), data: out, ...(computer ? { computer } : {}) };++ return { text: json(out && out.running && !(core.pcs.isSession && core.pcs.isSession(a.machine)) ? { watch: core.pcs.watchUrl(a.machine, a.desk), ...out, ...(stream ? { stream } : {}) } : out), data: out, ...(computer ? { computer } : {}) };}++ if (a.action === 'stop' && desks) await stopDeskStream(desks, person, core, a.machine, a.desk);const out = await core.pcs.desk(person, a.machine, a, { signal: ctx.signal });if (desks && out && !out.error) {if (a.action === 'stop' && desks.placed === a.machine) desks.placed = null;@@@ -1,7 +1,8 @@'use strict';const { serviceFor, json, personOf } = require('./_openvibe');--const { handToPerson } = require('./_person');++const { handToPerson, PERSON_WAIT_MS, PERSON_POLL_MS } = require('./_person');++const driving = require('../../agents/driving');const machine = { type: 'string', description: 'the machine id (dev_…) from list_my_computers' };@@@ -24,7 +25,21 @@ module.exports = {if (ctx.day && core.pcs.isSession && core.pcs.isSession(target)) {return ctx.day.ask(ctx.taskId, { kind: 'sign_in', text: input.what, options: ['Sign in on its screen', 'Done'], session_id: target.slice(4) });}-- if (core.pcs.isSession && core.pcs.isSession(target)) return { text: json({ done: false, note: 'Your desk is on a desk host, which the person cannot take over from here: tell them in your answer what is left for them to do, and where.' }), data: null };++ if (core.pcs.isSession && core.pcs.isSession(target)) {++ // A Run desk's Ask: the viewer takes the session's lease and says so in Actor's driving marker. Wait for the++ // person to take over and hand back, then let the agent look again. No note is passed in a tool result.++ const what = String(input.what || '').replace(/\s+/g, ' ').trim().slice(0, 300);++ if (ctx.emit) await ctx.emit({ step: 'text', text: `Needs you: ${what}` });++ const end = Date.now() + PERSON_WAIT_MS;++ let took = false;++ const pause = (ms) => new Promise((resolve) => { const timer = setTimeout(resolve, ms); if (ctx.signal) ctx.signal.addEventListener('abort', () => { clearTimeout(timer); resolve(); }, { once: true }); });++ while (Date.now() < end && !(ctx.signal && ctx.signal.aborted)) {++ if (driving.is(ctx.taskId)) took = true;++ else if (took) return { text: json({ done: true, note: 'The person used the desk and handed it back. Look at it again before you go on.', session_id: target.slice(4) }), data: { done: true } };++ await pause(PERSON_POLL_MS);++ }++ return { text: json({ done: false, note: took ? 'The person was still using the desk when the wait ended: tell them what is left.' : 'The person did not take the desk in time: tell them in your answer what is left for them to do.' }), data: { done: false } };++ }const data = await handToPerson(person, input, ctx, core);return { text: json(data), data };},@@@ -19,6 +19,8 @@ const PERSON_WAIT_MAX_MS = 15 * 60_000;* At most config.maxRunning tasks run at once; the rest wait queued in order.*/const store = require('./store');++const stills = require('./stills');++const looks = require('../agents/looks');const modelPrefs = require('../runtime/model-prefs');// The person's own model choice rides with the models layer (plan L55 C3). A stand-in models object (a test's)// does not carry it, and a run then keeps today's behaviour.@@@ -188,6 +190,12 @@ function createEngine({ config, s, adapters, stream, webhooks = null, now = () =const saved = await store.appendEvent(s, id, event);if (day) await day.event(id, event);stream.publish(id, saved);++ // See it work (W1): every computer step keeps the picture it saw, for the finished run's replay. The desk's++ // own curtain has already blanked a secret screen before the shot reached Actor; typed text is not a picture.++ if (event.kind === 'output' && event.step === 'tool_result' && /^computer_/.test(String(event.tool || '')) && saved.seq) {++ const shot = looks.get(id);++ if (shot) await stills.save(s, id, saved.seq, { ...shot, tool: event.tool }).catch(() => null);++ }return saved;}/**nnew file mode 100644@@@ -0,0 +1,79 @@++'use strict';++++/**++ * A finished run's replay (See it work, W1): every computer step, the action it took and the still it saw. Built from++ * the task's own events and the pictures kept by tasks/stills; the model never supplies an action or a picture.++ *++ * Typed text is never returned: a step may say that something was typed or filled, never what. A click keeps its x,y in++ * the picture's pixels so a page can put a ring where it happened.++ */++const store = require('./store');++const stills = require('./stills');++++const COMPUTER = /^computer_(input|browser|screen|desk)$/;++++async function steps(s, row, { baseUrl = '' } = {}) {++ const events = await store.eventsDetailed(s, row.id, 0);++ const pictures = await stills.list(s, row.id);++ const bySeq = new Map(pictures.map((p) => [p.seq, p]));++ const results = new Map();++ for (const e of events) if (e && e.kind === 'output' && e.step === 'tool_result' && e.call_id) results.set(e.call_id, e);++ const out = [];++ for (const e of events) {++ if (!e || e.kind !== 'output' || e.step !== 'tool_call' || !COMPUTER.test(String(e.tool || ''))) continue;++ const result = results.get(e.call_id) || null;++ const args = e.args && typeof e.args === 'object' ? e.args : {};++ let action = args.action || null, x = null, y = null, width = null, height = null, url = null, secret = false;++ if (e.tool === 'computer_input') {++ action = String(args.action || 'input');++ for (const [k, v] of [['x', args.x], ['y', args.y], ['width', args.width], ['height', args.height]]) {++ if (Number.isFinite(v)) { if (k === 'x') x = Number(v); else if (k === 'y') y = Number(v); else if (k === 'width') width = Number(v); else height = Number(v); }++ }++ if (action === 'type') secret = true;++ } else if (e.tool === 'computer_browser') {++ action = String(args.action || 'browser');++ if (action === 'open' && typeof args.url === 'string') url = args.url.slice(0, 400);++ if (action === 'fill') secret = true;++ } else if (e.tool === 'computer_screen') action = 'look';++ else action = String(args.action || 'desk');++ if (result) {++ try { const d = JSON.parse(result.text || '{}'); if (d && typeof d.url === 'string' && /^https?:\/\//.test(d.url)) url = d.url.slice(0, 400); } catch { /* not JSON */ }++ }++ const seq = result ? result.seq : e.seq;++ let picture = bySeq.get(seq);++ for (let p = seq; p >= 1 && !picture; p--) picture = bySeq.get(p);++ out.push({++ seq, at: result && result.at ? result.at : e.at || null, tool: e.tool, action: secret ? 'secret' : action,++ x, y, width, height, url, secret,++ text: result ? String(result.text || '').slice(0, 400) : null,++ error: !!(result && result.is_error),++ still: picture ? { seq: picture.seq, url: `${baseUrl}/api/v1/tasks/${row.id}/replay/${picture.seq}/still`, width: picture.width, height: picture.height, mime: picture.mime } : null,++ });++ }++ return out.slice(-200);++}++++/** One line for the replay's caption, in plain words. */++function caption(step) {++ if (!step) return '';++ if (step.tool === 'computer_input') {++ if (step.action === 'click' || step.action === 'double_click' || step.action === 'right_click') return `Clicked at ${Math.round(step.x || 0)}, ${Math.round(step.y || 0)}`;++ if (step.action === 'move') return `Moved to ${Math.round(step.x || 0)}, ${Math.round(step.y || 0)}`;++ if (step.action === 'scroll') return 'Scrolled';++ if (step.action === 'key') return 'Pressed a key';++ return 'Typed';++ }++ if (step.tool === 'computer_browser') {++ if (step.action === 'open') return 'Opened a page';++ if (step.action === 'read') return 'Read the page';++ if (step.action === 'click') return 'Clicked on the page';++ if (step.action === 'fill') return 'Filled a field';++ if (step.action === 'press') return 'Pressed a key';++ if (step.action === 'back') return 'Went back';++ return 'Used the browser';++ }++ if (step.tool === 'computer_screen') return 'Looked at the screen';++ return step.action === 'start' ? 'Started the desk' : step.action === 'stop' ? 'Stopped the desk' : 'Checked the desk';++}++++module.exports = { steps, caption, COMPUTER };nnew file mode 100644@@@ -0,0 +1,63 @@++'use strict';++++/**++ * The pictures a task's steps saw (See it work, W1): one still per computer step, kept with its event seq so a finished++ * run's replay can scrub through them. The desk's own curtain blanks a secret screen before this ever sees it; typed++ * text is never part of a still.++ *++ * In Actor's own database, gone with the task (store.prune deletes the orphans; tasks are kept 30 days). At most++ * MAX_PER_TASK per task, 640 px wide, WebP where sharp is available, otherwise the JPEG/PNG the machine sent.++ */++const MAX_PER_TASK = 120;++const MAX_BYTES = 400 * 1024;++const MAX_KEEP = 640;++++const sharp = (() => { try { return require('sharp'); } catch { return null; } })();++++/** Shrink to a replay picture and encode it. → { mime, bytes, width, height } */++async function shrink(shot) {++ const bytes = Buffer.isBuffer(shot.data) ? shot.data : Buffer.from(String(shot.data || ''), 'base64');++ if (!bytes.length) return null;++ const format = shot.format === 'png' ? 'png' : 'jpeg';++ if (sharp) {++ try {++ const out = await sharp(bytes).rotate().resize({ width: MAX_KEEP, withoutEnlargement: true }).webp({ quality: 72 }).toBuffer({ resolveWithObject: true });++ if (out.data.length && out.data.length <= MAX_BYTES) {++ return { mime: 'image/webp', bytes: out.data, width: out.info.width || null, height: out.info.height || null };++ }++ } catch { /* fall through to the original */ }++ }++ if (bytes.length > MAX_BYTES) return null;++ return { mime: `image/${format}`, bytes, width: Number(shot.width) || null, height: Number(shot.height) || null };++}++++/** Remember the still a computer step saw at its result event's seq. Best effort: a picture never fails the run. */++async function save(s, taskId, seq, shot) {++ if (!s || !taskId || !Number.isInteger(seq) || seq < 1 || !shot) return false;++ const made = await shrink(shot);++ if (!made) return false;++ const count = await s.db.value('SELECT count(*)::int FROM actor_task_stills WHERE task_id = $1', [taskId]).catch(() => 0);++ if (Number(count) >= MAX_PER_TASK) return false;++ await s.db.query(++ `INSERT INTO actor_task_stills (task_id, seq, tool, mime, width, height, bytes, at)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8)++ ON CONFLICT (task_id, seq) DO NOTHING`,++ [taskId, seq, shot.tool || null, made.mime, made.width, made.height, made.bytes, s.iso()],++ );++ return true;++}++++async function list(s, taskId) {++ return s.db.many('SELECT seq, tool, mime, width, height, at FROM actor_task_stills WHERE task_id = $1 ORDER BY seq', [taskId]);++}++++async function get(s, taskId, seq) {++ return s.db.maybe('SELECT seq, tool, mime, width, height, bytes, at FROM actor_task_stills WHERE task_id = $1 AND seq = $2', [taskId, seq]);++}++++/** Drop the pictures of tasks that are gone (called by store.prune). */++async function prune(s) {++ await s.db.query('DELETE FROM actor_task_stills WHERE task_id NOT IN (SELECT id FROM tasks)');++}++++module.exports = { save, list, get, prune, MAX_PER_TASK };@@@ -134,8 +134,13 @@ function deskFrom(events) {if (!e || e.step !== 'tool_result' || e.tool !== 'computer_desk' || e.is_error) continue;const t = String(e.text || '');const m = DESK_WATCH.exec(t);-- if (m && /"running":true/.test(t)) desk = { url: m[1], kind: (/"kind":"(browser|desktop)"/.exec(t) || [])[1] || null };-- else if (/"running":false/.test(t)) desk = null;++ if (m && /"running":true/.test(t)) {++ let parsed = null;++ try { parsed = JSON.parse(t); } catch { parsed = null; }++ desk = { url: m[1], kind: (/"kind":"(browser|desktop)"/.exec(t) || [])[1] || null };++ // See it work: the private Live stream the desk started for this task (Link's desk.stream), if any.++ if (parsed && parsed.stream && typeof parsed.stream === 'object') desk.stream = parsed.stream;++ } else if (/"running":false/.test(t)) desk = null;}return desk;}@@@ -181,6 +186,20 @@ async function eventsAfter(s, taskId, after = 0) {return rows.map((r) => parse(r.body));}++/**++ * The events with the detail a replay needs (See it work, W1): the tool args that appendEvent keeps out of the public++ * body (a click's x,y; a browser action). They were stripped of credential-shaped fields before they were stored, and++ * the replay blanks typed text again. Only the task's own replay reads this.++ */++async function eventsDetailed(s, taskId, after = 0) {++ const rows = await s.db.many('SELECT body, ui_detail FROM task_events WHERE task_id = $1 AND seq > $2 ORDER BY seq', [taskId, after]);++ return rows.map((r) => {++ const body = parse(r.body) || {};++ const detail = parse(r.ui_detail) || {};++ return { ...body, ...(detail.args !== undefined ? { args: detail.args } : {}), ...(detail.call_id ? { call_id: detail.call_id } : {}) };++ });++}++/** Add to a requester's and everyone's spend today (tasks counts only the requester's). */async function addSpend(s, requester, usd, { tasks = 0 } = {}) {const day = dayOf(s.now());@@@ -256,6 +275,8 @@ async function prune(s) {const cutoff = new Date(s.now() - RETENTION_DAYS * 86_400_000).toISOString();const n = await s.db.exec('DELETE FROM tasks WHERE created_at < $1', [cutoff]);await s.db.query('DELETE FROM spend_daily WHERE day < $1', [cutoff.slice(0, 10)]);++ // See it work: a replay's pictures go with the task that saw them.++ await require('./stills').prune(s).catch(() => null);return n;}@@@ -290,4 +311,4 @@ function toWire(row, { baseUrl }) {return t;}--module.exports = { insertTask, getTask, byIdempotency, listTasks, updateTask, appendEvent, eventsAfter, lastEvent, needsYou, deskFrom, deskPlace, taskDesk, computerFrom, taskComputer, addSpend, spendToday, spendDays, lockSpendToday, failInterrupted, openTasks, prune, toWire, subjectRef, dayOf, parseJson: parse, OPEN, RETENTION_DAYS, followContext, followUps, forks, recentSucceeded };++module.exports = { insertTask, getTask, byIdempotency, listTasks, updateTask, appendEvent, eventsAfter, eventsDetailed, lastEvent, needsYou, deskFrom, deskPlace, taskDesk, computerFrom, taskComputer, addSpend, spendToday, spendDays, lockSpendToday, failInterrupted, openTasks, prune, toWire, subjectRef, dayOf, parseJson: parse, OPEN, RETENTION_DAYS, followContext, followUps, forks, recentSucceeded };@@@ -37,10 +37,11 @@ const MACHINE = 'dev_01JZ0000000000000000000000';await s.db.query("UPDATE tasks SET state = 'running' WHERE id = $1", [id]);await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`);let page = await t.get(`/chat/${id}`, { as: kim });-- assert.match(page.text, new RegExp(`data-watch="/chat/${id}/watch"`), 'the figure says where to ask');++ assert.match(page.text, new RegExp(`<figure class="ct-desk-tile" data-ct-desk data-task="${id}" data-desk="link:${MACHINE}:desk-trains"`), 'the desk tile says where the desk is and opens the viewer');++ assert.match(page.text, /data-ct-desk-open/, 'the tile opens the viewer');await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen`);page = await t.get(`/chat/${id}`, { as: kim });-- assert.ok(!page.text.includes('data-watch='), 'the owner\'s own screen is never framed');++ assert.ok(!page.text.includes('data-ct-desk'), 'the owner\'s own screen is never framed');await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`);});@@@ -30,6 +30,15 @@ const ROUTE_EXTENSIONS = new Map([['GET /api/v1/agent/timeline/:id', 'actor.timeline.read'],['POST /api/v1/agent/approvals/:product/:id/decide', 'actor.approval.decide'],['POST /api/v1/agent/kill-switch', 'actor.resource.control'],++ // See it work (W1): the desk, its still, the person's control and the replay. They are read/create on the task++ // the person already holds; Contracts lists them after this stage (the same pattern as the trigger routes).++ ['GET /api/v1/tasks/:id/desks', 'actor.task.read'],++ ['GET /api/v1/tasks/:id/desk-still', 'actor.task.read'],++ ['POST /api/v1/tasks/:id/desks/:desk/input', 'actor.task.create'],++ ['POST /api/v1/tasks/:id/desks/:desk/take-over', 'actor.task.create'],++ ['POST /api/v1/tasks/:id/desks/:desk/hand-back', 'actor.task.create'],++ ['GET /api/v1/tasks/:id/replay', 'actor.task.read'],++ ['GET /api/v1/tasks/:id/replay/:seq/still', 'actor.task.read'],// Triggers made in a conversation (plan §12 A-4). The routes exist before Contracts lists actor.trigger.*; a// trigger is the person's own work record, so the task capabilities the person already holds guard it.['GET /api/v1/triggers', 'actor.task.read'],@@@ -31,6 +31,9 @@ async function boot(opts = {}) {// A test boot makes many runs in one "day"; the free allowance is measured in free-allowance.test.js, which// sets its own. This keeps the other suites about what they are about.ACTOR_FREE_PER_DAY_USD: '1000',++ // Tests stand Live in when they need a view ticket; no test calls openvibe.live.++ ACTOR_LIVE_URL: '',++ ACTOR_LIVE_PUBLIC_URL: 'https://openvibe.live',OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,OV_OAUTH_CLIENT_ID: 'actor', OV_OAUTH_CLIENT_SECRET: 'actor-secret', COOKIE_SECURE: 'false',...providers.env,@@@ -45,7 +48,7 @@ async function boot(opts = {}) {const { createStore } = require('../../server/db');const testdb = await require('./db').testDb();const store = createStore(testdb.db, { now: opts.now });-- const built = await createApp({ config, env, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, adapters: opts.adapters, accountSend: opts.accountSend, webhookPost: opts.webhookPost, chatClient: opts.chatClient, secretSigner: opts.secretSigner, bot: opts.bot, projects: opts.projects, mailFetch: opts.mailFetch, mailTokens: opts.mailTokens, notifier: opts.notifier, models: opts.models, registry: opts.registry, loop: opts.loop, connectors: opts.connectors, guard: opts.guard });++ const built = await createApp({ config, env, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, adapters: opts.adapters, accountSend: opts.accountSend, webhookPost: opts.webhookPost, chatClient: opts.chatClient, secretSigner: opts.secretSigner, bot: opts.bot, live: opts.live, projects: opts.projects, mailFetch: opts.mailFetch, mailTokens: opts.mailTokens, notifier: opts.notifier, models: opts.models, registry: opts.registry, loop: opts.loop, connectors: opts.connectors, guard: opts.guard });await built.ctx.keys.ensure();const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });const base = `http://127.0.0.1:${server.address().port}`;@@@ -109,6 +109,7 @@ const SAME = { 'sec-fetch-site': 'same-origin' };'server/agents/providers.js': 'the configured provider base URLs (DeepSeek, OpenAI, the open model) with the operator\'s keys','server/agents/codes.js': 'the configured OpenVibe.Codes base (ACTOR_CODES_URL) with Actor\'s own service token; the subject must be usr_<ULID> and every task id tsk_<ULID> before it goes in a path; the repository named in the task is data for the person\'s own machine, never fetched here (test/codes.test.js)','server/agents/computers.js': 'the configured OpenVibe.Bot base (ACTOR_BOT_URL) with Actor\'s own service token (bot.computer.control); the machine id must be dev_<ULID> and the subject usr_<ULID> before anything is sent; a path or program is data for the person\'s own machine, never fetched here (test/computers.test.js)',++ 'server/agents/live.js': "the configured OpenVibe.Live base (ACTOR_LIVE_URL) for one POST /api/streams/agent/<ags_ULID>/view-ticket, with the person's own bearer token; the stream id must be ags_<ULID> and is URL-encoded, and the returned embed is checked to Live's configured origin before the page gets it (test/see-it-work.test.js)",'server/agents/timeline.js': 'the approval proxy answers the mode that owns an approval: the product base is the configured ACTOR_<PRODUCT>_URL (or the config map) and the approval id is URL-encoded; the person\'s own bearer token is forwarded, and a stop uses the same base with the step\'s own ref (codes cancel, run cancel, live end); no URL from a step\'s content (test/timeline.test.js)','server/chat/client.js': 'the configured OpenVibe.Chat base (ACTOR_CHAT_URL, loopback) with Actor\'s own service token; the room and the message it answers come from a signed chat.message.created, never from the task (test/chat-asks.test.js)','server/mail/client.js': 'the configured OpenVibe.Mail base (ACTOR_MAIL_URL, loopback) with Actor\'s own service token (mail.agent.read, mail.agent.reply, mail.agent.answering); the person is the task\'s owner in X-OV-Subject, the agent name is slugified and URL-encoded, and the message id is Mail\'s own and URL-encoded; no address or URL from the letter is ever fetched here (test/agent-mail.test.js)',nnew file mode 100644@@@ -0,0 +1,111 @@++'use strict';++// Screenshots for See it work (W1), not run by test/run.js: a task with a live desk tile, the viewer in control, and a++// finished run's replay, at 390 and 1280. The Live embed is a local stand-in (the test page never calls openvibe.live).++// Usage: node test/see-it-work-shots.js <outdir>++const fs = require('fs');++const http = require('http');++const path = require('path');++const h = require('openvibe-shared/browser-harness');++const { boot } = require('./helpers/boot');++const store = require('../server/tasks/store');++const stills = require('../server/tasks/stills');++++const MACHINE = 'dev_01JZ0000000000000000000000';++const STREAM = 'ags_01JZ0000000000000000000000';++const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==';++++(async () => {++ const out = process.argv[2] || path.join(__dirname, '..', '..', 'research-2026-11-12', 'actor-shots', 'see-it-work');++ fs.mkdirSync(out, { recursive: true });++ // A stand-in Live embed: what a viewer sees on openvibe.live, without leaving this machine.++ const live = http.createServer((req, res) => {++ res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });++ res.end(`<!doctype html><meta charset="utf-8"><style>html,body{margin:0;height:100%;background:#05070d;color:#e6edf7;font:14px system-ui;overflow:hidden}++.window{position:absolute;inset:10px;border-radius:10px;background:#0f1729;border:1px solid #334155;overflow:hidden}++.bar{height:34px;display:flex;align-items:center;gap:6px;padding:0 10px;background:#1e293b;border-bottom:1px solid #334155}++.dot{width:7px;height:7px;border-radius:50%;background:#64748b}.url{flex:1;height:20px;border-radius:6px;background:#0b1220;border:1px solid #334155;color:#94a3b8;font-size:11px;line-height:20px;padding:0 8px}++.page{padding:22px}.page h1{margin:0 0 8px;font-size:22px}.page p{color:#94a3b8;margin:0 0 18px}.card{height:86px;border-radius:10px;background:#162033;border:1px solid #334155;margin:8px 0}++.tag{position:absolute;left:18px;bottom:16px;display:inline-flex;align-items:center;gap:7px;padding:3px 10px;border-radius:999px;background:#0b1220cc;border:1px solid #334155;font-size:11px;font-weight:700;letter-spacing:.06em;text-transform:uppercase}++.tag::before{content:"";width:7px;height:7px;border-radius:50%;background:#ef4444;box-shadow:0 0 0 3px #ef444455}</style>++<div class="window"><div class="bar"><span class="dot"></span><span class="dot"></span><span class="dot"></span><span class="url">example.com/trains</span></div>++<div class="page"><h1>Trains to Porto</h1><p>Tuesday, 14:10 · from Lisbon</p><div class="card"></div><div class="card"></div><div class="card"></div></div></div><span class="tag">AI agent · live</span>`);++ });++ await new Promise((r) => live.listen(0, '127.0.0.1', r));++ const liveBase = `http://127.0.0.1:${live.address().port}`;++ const t = await boot({ env: { ACTOR_LIVE_URL: '', ACTOR_LIVE_PUBLIC_URL: liveBase } });++ // The page's own view ticket call: the same shape Live answers, pointing at the stand-in embed.++ t.ctx.live.enabled = true;++ t.ctx.live.viewTicket = async (token, id) => ({ stream_id: id, view_ticket: 'wvt_test', expires_at: new Date(Date.now() + 600e3).toISOString(), page_url: `${liveBase}/embed/agent/${id}?vt=wvt_test` });++ const kim = t.network.addUser('kim');++ const s = t.ctx.s;++ const owner = `user:${kim.subject}`;++ const event = (id, body) => store.appendEvent(s, id, { kind: 'output', at: s.iso(), ...body });++ const make = async (state) => {++ const id = s.newId('tsk');++ await store.insertTask(s, { id, requester: owner, task: 'Book the cheapest train to Porto', mode: 'balanced', budget_task: 0.05, budget_day: 0.5, created_at: s.iso() });++ await store.updateTask(s, id, { state });++ return id;++ };++ const makeLive = async () => {++ const id = await make('running');++ await event(id, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ watch: `https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`, running: true, kind: 'browser', stream: { id: STREAM, page_url: `${liveBase}/embed/agent/${STREAM}`, visibility: 'private' } }) });++ await event(id, { step: 'tool_call', tool: 'computer_input', call_id: 'c1', args: { action: 'click', x: 470, y: 250, width: 1280, height: 800 }, input: 'click' });++ await event(id, { step: 'tool_result', tool: 'computer_input', call_id: 'c1', text: '{"done":true}' });++ await event(id, { step: 'tool_call', tool: 'computer_browser', call_id: 'c2', args: { action: 'open', url: 'https://example.com/trains' }, input: 'open' });++ await event(id, { step: 'tool_result', tool: 'computer_browser', call_id: 'c2', text: '{"url":"https://example.com/trains","title":"Trains"}' });++ return id;++ };++++ const replayId = await make('succeeded');++ await store.updateTask(s, replayId, { finished_at: s.iso(), result: { answer: 'The 14:10 from Lisbon is the cheapest: €29.50.' } });++ const type = await event(replayId, { step: 'tool_call', tool: 'computer_input', call_id: 'r1', args: { action: 'type', text: 'Porto', desk: 'private' }, input: 'type' });++ await stills.save(s, replayId, type.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 1280, height: 800 });++ await event(replayId, { step: 'tool_result', tool: 'computer_input', call_id: 'r1', text: '{"done":true}' });++ const click = await event(replayId, { step: 'tool_call', tool: 'computer_input', call_id: 'r2', args: { action: 'click', x: 320, y: 180, width: 1280, height: 800 }, input: 'click' });++ await stills.save(s, replayId, click.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 1280, height: 800 });++ await event(replayId, { step: 'tool_result', tool: 'computer_input', call_id: 'r2', text: '{"done":true}' });++++ const browser = await h.launch({});++ try {++ for (const width of [390, 1280]) {++ const liveId = await makeLive();++ const cases = [++ ['tile', false, async () => {}, liveId],++ ['viewer', true, async (page) => {++ await page.evaluate(`document.querySelector('[data-ct-desk-open]').click()`);++ await new Promise((r) => setTimeout(r, 800));++ await page.evaluate(`document.querySelector('[data-ct-viewer-take]').click()`);++ await new Promise((r) => setTimeout(r, 1000));++ }, liveId],++ ['replay', false, async (page) => {++ await page.evaluate(`(() => { const d = document.querySelector('[data-ct-replay]'); if (d) { d.open = true; const b = d.querySelector('[data-ct-replay-step="1"]'); if (b) b.click(); } })()`);++ await new Promise((r) => setTimeout(r, 700));++ }, replayId],++ ];++ for (const [name, viewer, act, id] of cases) {++ const page = await h.openPage(browser, await browser.newContext(), { width, height: 1500 });++ await page.send('Network.setCookie', { name: 'actor_at', value: t.signIn(kim).replace(/^actor_at=/, ''), url: t.base, path: '/' });++ await page.send('Network.setExtraHTTPHeaders', { headers: { Authorization: `Bearer ${t.network.userToken(kim)}` } });++ await page.goto(`${t.base}/chat/${id}`, { maxMs: 10000 });++ await new Promise((r) => setTimeout(r, 900));++ await act(page);++ const shot = path.join(out, `${name}-${width}.png`);++ const png = await page.send('Page.captureScreenshot', { format: 'png' });++ fs.writeFileSync(shot, Buffer.from(png.data, 'base64'));++ const read = await page.evaluate(`(() => {++ const wide = [...document.querySelectorAll('main *')].filter((e) => { const r = e.getBoundingClientRect(); return r.width > 0 && (r.right > innerWidth + 1 || r.left < -1); }).slice(0, 5).map((e) => e.tagName.toLowerCase());++ const small = [...document.querySelectorAll('.ct-desk-tile button, .ct-viewer button, .ct-viewer input, .ct-replay button, .ct-replay input')].filter((e) => { const r = e.getBoundingClientRect(); return !e.hidden && r.width > 0 && r.height > 0 && (r.width < 44 || r.height < 44); }).map((e) => e.tagName.toLowerCase());++ return { sw: document.documentElement.scrollWidth, iw: innerWidth, wide, small, live: !!document.querySelector('[data-ct-desk-frame] iframe'), viewer: !!document.querySelector('.ct-viewer:not([hidden])'), replay: !!document.querySelector('[data-ct-replay][open]') };++ })()`);++ process.stdout.write(`${shot} scrollWidth=${read.sw}/${read.iw} live=${read.live} viewer=${read.viewer} replay=${read.replay}${read.wide.length ? ` WIDE ${read.wide.join(',')}` : ''}${read.small.length ? ` SMALL ${read.small.join(',')}` : ''}\n`);++ await page.close();++ }++ await store.updateTask(s, liveId, { state: 'cancelled', finished_at: s.iso() }).catch(() => null);++ }++ } finally {++ await browser.close();++ live.close();++ await t.close();++ }++ process.exit(0);++})().catch((e) => { process.stderr.write(String(e && e.stack)); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,217 @@++'use strict';++// See it work (W1): a running task's desk tile, the private Live stream's view ticket, the still fallback, the person's++// take-over and hand-back (Run lease / Link input as the person), the Ask that opens the viewer in control, and a++// finished run's replay with the still each step saw. Typed text is never in a replay.++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const store = require('../server/tasks/store');++const stills = require('../server/tasks/stills');++++const SAME = { 'sec-fetch-site': 'same-origin' };++const MACHINE = 'dev_01JZ0000000000000000000000';++const SESSION = 'ses_01JZ0000000000000000000000';++const STREAM = 'ags_01JZ0000000000000000000000';++const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==';++++(async () => {++ const calls = [];++ const computers = {++ async still(subject, id) { calls.push(['still', subject, id]); return { bytes: Buffer.from(PNG, 'base64'), type: 'image/png' }; },++ async screen(subject, machine, opts) { calls.push(['screen', subject, machine, opts]); return { format: 'png', width: 640, height: 400, content_b64: PNG }; },++ async personSession(subject, id, action, body) { calls.push(['session', subject, id, action, body]); return { lease: { holder: action === 'take' ? 'person' : 'ai' }, done: true }; },++ async personInput(subject, token, machine, args) { calls.push(['input', subject, token, machine, args]); return { done: true, action: args.action }; },++ async deskStream() { return { error: 'not used in this test' }; },++ };++ const live = {++ enabled: true,++ async viewTicket(token, id) { calls.push(['ticket', token, id]); return { view_ticket: 'wvt_private', expires_at: new Date(Date.now() + 600e3).toISOString(), page_url: `https://openvibe.live/embed/agent/${id}?vt=wvt_private` }; },++ };++ const t = await boot({ bot: { computers }, live });++ const kim = t.network.addUser('kim');++ const bo = t.network.addUser('bo');++ const s = t.ctx.s;++ const owner = `user:${kim.subject}`;++ const token = t.network.userToken(kim);++ const event = (id, body) => store.appendEvent(s, id, { kind: 'output', at: s.iso(), ...body });++ const make = async (state = 'running') => {++ const id = s.newId('tsk');++ await store.insertTask(s, { id, requester: owner, task: 'Book the cheapest train to Porto', mode: 'balanced', budget_task: 0.05, budget_day: 0.5, created_at: s.iso() });++ if (state !== 'queued') await store.updateTask(s, id, { state });++ return id;++ };++ try {++ const id = await make();++ await event(id, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ watch: `https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`, running: true, kind: 'browser', stream: { id: STREAM, page_url: `https://openvibe.live/embed/agent/${STREAM}`, visibility: 'private', expires_at: new Date(Date.now() + 4 * 3600e3).toISOString() } }) });++ await event(id, { step: 'tool_call', tool: 'computer_input', call_id: 'c1', args: { action: 'click', x: 320, y: 200, width: 1280, height: 800 }, input: 'click' });++ await event(id, { step: 'tool_result', tool: 'computer_input', call_id: 'c1', text: '{"done":true}' });++ await event(id, { step: 'tool_call', tool: 'computer_browser', call_id: 'c2', args: { action: 'open', url: 'https://example.com/trains' }, input: 'open' });++ await event(id, { step: 'tool_result', tool: 'computer_browser', call_id: 'c2', text: '{"url":"https://example.com/trains","title":"Trains"}' });++++ await check('the running card has the desk tile: label, caption, ring, page address and the watch button', async () => {++ const page = await t.get(`/chat/${id}`, { as: kim });++ assert.strictEqual(page.status, 200);++ assert.match(page.text, new RegExp(`<figure class="ct-desk-tile" data-ct-desk data-task="${id}" data-desk="link:${MACHINE}:desk-trains" data-where="link" data-kind="browser"`));++ assert.match(page.text, /Opening a page/);++ assert.match(page.text, /style="left:25\.00%;top:25\.00%"/, 'the ring is where it clicked');++ assert.match(page.text, /example\.com\/trains/, 'the browser address');++ assert.match(page.text, /Your PC · its own desk/);++ assert.match(page.text, /data-ct-desk-open/);++ assert.match(page.text, new RegExp(`src="/api/v1/tasks/${id}/desk-still\\?desk=desk-trains"`), 'the still fallback');++ });++++ await check('the desks call names the private Live stream with a view ticket; the still is Bot\'s picture at 640', async () => {++ const r = await t.get(`/api/v1/tasks/${id}/desks`, { as: kim, bearer: token, headers: SAME });++ assert.strictEqual(r.status, 200, r.text);++ const d = r.json().desks[0];++ assert.strictEqual(d.id, `link:${MACHINE}:desk-trains`);++ assert.strictEqual(d.label, 'Your PC · its own desk');++ assert.strictEqual(d.live.stream_id, STREAM);++ assert.match(d.live.embed_url, new RegExp(`/embed/agent/${STREAM}\\?vt=wvt_private`));++ assert.ok(calls.some((c) => c[0] === 'ticket' && c[1] === token && c[2] === STREAM), 'Live got the person\'s own token');++ const still = await t.get(`/api/v1/tasks/${id}/desk-still?desk=desk-trains`, { as: kim });++ assert.strictEqual(still.status, 200);++ assert.match(still.headers.get('content-type'), /^image\/png/);++ assert.deepStrictEqual(still.buffer, Buffer.from(PNG, 'base64'));++ assert.strictEqual((await t.get(`/api/v1/tasks/${id}/desks`, { as: bo })).status, 404, 'another person: not there');++ });++++ await check('take over, person input, hand back with a note (Link: the person\'s own token to Bot)', async () => {++ const take = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/take-over`, { as: kim, method: 'POST', headers: SAME });++ assert.strictEqual(take.status, 200, take.text);++ assert.ok(take.json().desk, 'the desk is in the answer');++ assert.strictEqual((await t.get(`/api/v1/tasks/${id}/desks`, { as: kim, bearer: token })).json().desks[0].driving, true);++ const input = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/input`, { as: kim, bearer: token, json: { action: 'click', x: 100, y: 120, width: 640, height: 400 }, headers: SAME });++ assert.strictEqual(input.status, 200, input.text);++ const got = calls.find((c) => c[0] === 'input');++ assert.deepStrictEqual([got[1], got[2], got[3], got[4].action, got[4].x, got[4].y, got[4].width, got[4].height], [kim.subject, token, MACHINE, 'click', 100, 120, 640, 400]);++ let steered = null;++ const before = t.ctx.engine.steer;++ t.ctx.engine.steer = async (taskId, note) => { steered = [taskId, note]; return { ok: true }; };++ try {++ const back = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/hand-back`, { as: kim, json: { note: 'signed in, carry on' }, headers: SAME });++ assert.strictEqual(back.status, 200, back.text);++ assert.strictEqual((await t.get(`/api/v1/tasks/${id}/desks`, { as: kim, bearer: token })).json().desks[0].driving, false);++ assert.deepStrictEqual(steered, [id, 'signed in, carry on'], 'the note goes to the task as the person\'s message');++ } finally { t.ctx.engine.steer = before; }++ const again = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/input`, { as: kim, bearer: token, json: { action: 'click', x: 1, y: 1 }, headers: SAME });++ assert.strictEqual(again.status, 409, 'not driving: no input');++ });++++ await check('a step that meets a sign-in becomes an Ask whose button opens the viewer in control', async () => {++ const other = await make();++ await event(other, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ watch: `https://openvibe.link/machines/${MACHINE}/screen?desk=desk-signin`, running: true, stream: { id: STREAM, visibility: 'private' } }) });++ await event(other, { step: 'tool_call', tool: 'hand_to_person', call_id: 'h1', args: { machine: MACHINE, what: 'Sign in to your train account.' } });++ const page = await t.get(`/chat/${other}`, { as: kim });++ assert.match(page.text, /data-ct-desk-ask/);++ assert.match(page.text, /Actor needs you:/);++ assert.match(page.text, /Sign in to your train account\./);++ assert.match(page.text, /data-ct-takeover>Take over</);++ });++++ await check('a Run desk uses its session still and the Run lease; take over, input and hand back go through the session', async () => {++ const runId = await make();++ await event(runId, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ machine: `run:${SESSION}`, running: true, kind: 'browser', where: 'an OpenVibe computer', computer: { session_id: SESSION, machine: 'an OpenVibe desk host', where: 'network' } }), computer: { session_id: SESSION, machine: 'an OpenVibe desk host', where: 'network' } });++ const listed = (await t.get(`/api/v1/tasks/${runId}/desks`, { as: kim, bearer: token })).json();++ assert.strictEqual(listed.desks[0].id, `run:${SESSION}`);++ assert.strictEqual(listed.desks[0].live, null);++ assert.match(listed.desks[0].still, new RegExp(`/api/v1/tasks/${runId}/still$`));++ calls.length = 0;++ const take = await t.get(`/api/v1/tasks/${runId}/desks/${encodeURIComponent(`run:${SESSION}`)}/take-over`, { as: kim, method: 'POST', headers: SAME });++ assert.strictEqual(take.status, 200, take.text);++ assert.deepStrictEqual(calls.find((c) => c[0] === 'session').slice(0, 4), ['session', kim.subject, SESSION, 'take']);++ const input = await t.get(`/api/v1/tasks/${runId}/desks/${encodeURIComponent(`run:${SESSION}`)}/input`, { as: kim, bearer: token, json: { action: 'key', keys: 'Return' }, headers: SAME });++ assert.strictEqual(input.status, 200, input.text);++ const sent = calls.filter((c) => c[0] === 'session').at(-1);++ assert.deepStrictEqual(sent[4], { kind: 'input', value: { action: 'key', keys: 'Return' } });++ const back = await t.get(`/api/v1/tasks/${runId}/desks/${encodeURIComponent(`run:${SESSION}`)}/hand-back`, { as: kim, json: {}, headers: SAME });++ assert.strictEqual(back.status, 200, back.text);++ assert.deepStrictEqual(calls.filter((c) => c[0] === 'session').at(-1).slice(0, 4), ['session', kim.subject, SESSION, 'back']);++ });++++ await check('a finished run has a replay: each step keeps its still, and typed text is never returned', async () => {++ const doneId = await make('succeeded');++ await store.updateTask(s, doneId, { finished_at: s.iso(), result: { answer: 'Booked.' } });++ const typeCall = await event(doneId, { step: 'tool_call', tool: 'computer_input', call_id: 't1', args: { action: 'type', text: 'hunter2', desk: 'private' }, input: 'type' });++ await stills.save(s, doneId, typeCall.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 1280, height: 800, tool: 'computer_input' });++ await event(doneId, { step: 'tool_result', tool: 'computer_input', call_id: 't1', text: '{"done":true}' });++ const clickCall = await event(doneId, { step: 'tool_call', tool: 'computer_input', call_id: 't2', args: { action: 'click', x: 64, y: 32, width: 640, height: 400 }, input: 'click' });++ await stills.save(s, doneId, clickCall.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 640, height: 400, tool: 'computer_input' });++ await event(doneId, { step: 'tool_result', tool: 'computer_input', call_id: 't2', text: '{"done":true}' });++ const r = await t.get(`/api/v1/tasks/${doneId}/replay`, { as: kim });++ assert.strictEqual(r.status, 200, r.text);++ const steps = r.json().steps;++ assert.strictEqual(steps.length, 2);++ assert.strictEqual(steps[0].secret, true, 'typed text is marked secret');++ assert.strictEqual(steps[0].action, 'secret');++ assert.ok(!JSON.stringify(steps).includes('hunter2'), 'the typed text is never returned');++ assert.strictEqual(steps[1].action, 'click');++ assert.deepStrictEqual([steps[1].x, steps[1].y, steps[1].width, steps[1].height], [64, 32, 640, 400]);++ assert.ok(steps[0].still && steps[1].still, 'both steps keep a picture');++ const shot = await t.get(new URL(steps[0].still.url).pathname, { as: kim });++ assert.strictEqual(shot.status, 200);++ assert.match(shot.headers.get('content-type'), /^image\/webp/);++ assert.strictEqual((await t.get(new URL(steps[0].still.url).pathname, { as: bo })).status, 404, 'another person: no picture');++ const page = await t.get(`/chat/${doneId}`, { as: kim });++ assert.match(page.text, /data-ct-replay/);++ assert.match(page.text, /Replay · 2 steps/);++ assert.match(page.text, /Clicked at 64, 32/);++ assert.ok(!page.text.includes('hunter2'), 'the page does not carry the typed text either');++ });++ await check('the Live client only mints a person\'s view ticket for an ags_ stream', async () => {++ const { createLive } = require('../server/agents/live');++ const asked = [];++ const live = createLive({ config: { services: { liveUrl: 'https://live.test', livePublicUrl: 'https://openvibe.live' } }, fetchImpl: async (url, init) => {++ asked.push([url, init.headers.Authorization]);++ return new Response(JSON.stringify({ view_ticket: 'wvt_x', expires_at: '2030-01-01T00:00:00Z' }), { status: 200, headers: { 'content-type': 'application/json' } });++ } });++ const view = await live.viewTicket('person-token', STREAM);++ assert.match(view.page_url, new RegExp(`https://openvibe\\.live/embed/agent/${STREAM}\\?vt=wvt_x`));++ assert.deepStrictEqual(asked, [[`https://live.test/api/streams/agent/${STREAM}/view-ticket`, 'Bearer person-token']]);++ assert.match((await live.viewTicket('person-token', 'not-a-stream')).error, /not an agent stream/);++ assert.strictEqual(asked.length, 1, 'a malformed id never reaches Live');++ assert.match((await live.viewTicket(null, STREAM)).error, /own session/);++ const replayLib = require('../server/tasks/replay');++ assert.strictEqual(typeof replayLib.steps, 'function');++ assert.strictEqual(replayLib.caption({ tool: 'computer_screen' }), 'Looked at the screen');++ assert.strictEqual(require('../server/runtime/tools/computer_desk').name, 'computer_desk');++ assert.ok(require('fs').existsSync(require('path').join(__dirname, '..', 'public', 'js', 'desks.js')));++ });++++ await check('the page uses the SDK names, pauses a stream off screen, and polls a still at 1/s', async () => {++ const js = require('fs').readFileSync(require('path').join(__dirname, '..', 'public', 'js', 'desks.js'), 'utf8');++ assert.match(js, /desks\(run\)/);++ assert.match(js, /takeOver\(run, desk\)/);++ assert.match(js, /handBack\(run, desk, note\)/);++ assert.match(js, /replay\(run\)/);++ assert.match(js, /IntersectionObserver/);++ assert.match(js, /stopStream\(st\)/);++ assert.match(js, /setInterval\(function \(\) \{ show\(img, tile\.getAttribute\('data-still'\)\); \}, 1000\)/);++ assert.match(js, /data-ct-replay-step/);++ const css = require('fs').readFileSync(require('path').join(__dirname, '..', 'public', 'css', 'chat.css'), 'utf8');++ assert.match(css, /\.ct-desk-frame \{[^}]*aspect-ratio: 16 \/ 9/);++ assert.match(css, /@media \(min-width: 900px\)[\s\S]*\.ct-viewer-side \{/);++ assert.match(css, /\.ct-viewer-bar button[^}]*min-height: 44px/);++ });++++ await check('the desk stream starts with the desk and stops with the task (Link desk.stream)', async () => {++ const { startDeskStream, closeDesks } = require('../server/runtime/tools/_person');++ const core = { pcs: {++ isSession: () => false,++ async deskStream(subject, machine, args) { calls.push(['stream', subject, machine, args]); return args.action === 'start' ? { id: STREAM, page_url: 'https://openvibe.live/embed/agent/' + STREAM } : { stopped: true }; },++ async desk(subject, machine, args) { calls.push(['desk', subject, machine, args]); return { running: false }; },++ } };++ const run = { streams: new Map(), used: new Set([MACHINE]), placed: null, name: 'desk-stream-test' };++ await startDeskStream(run, kim.subject, core, { taskId: id, personText: 'Book the cheapest train to Porto', config: { baseUrl: 'https://openvibe.actor' } }, MACHINE, 'desk-trains', { running: true });++ const start = calls.find((c) => c[0] === 'stream' && c[3].action === 'start');++ assert.ok(start, 'the stream started');++ assert.deepStrictEqual([start[3].desk, start[3].visibility, start[3].task.ref], ['desk-trains', 'private', id]);++ assert.ok(run.streams.has(MACHINE), 'the run remembers the stream for its end');++ await closeDesks(run, owner, core);++ const stop = calls.find((c) => c[0] === 'stream' && c[3].action === 'stop');++ assert.ok(stop, 'the stream stopped with the task');++ assert.strictEqual(run.streams.size, 0);++ });++++ } finally { await t.close(); }++ done();++})().catch((e) => { console.error(e); process.exit(1); });
@@ -2,6 +2,22 @@
@@ -2,6 +2,22 @@
What changed in OpenVibe.Actor, newest first. Each site also publishes its patch notes at /updates.
What changed in OpenVibe.Actor, newest first. Each site also publishes its patch notes at /updates.
+## Unreleased — See it work (W1): the desk, the viewer and the replay
+
+- **The desk tile** in a running task's card: the desk's private Live stream (started with the task's desk through Link's
+ `desk.stream` and stopped with the task), with the still at 1/s when no stream is up. Under it the latest step in
+ plain words, a ring where it last clicked, the browser page's address and where it is ("Your PC · its own desk",
+ "OpenVibe computer"). The tile pauses its stream while off screen.
+- **The viewer** (a tap on the tile): the desk fills a phone and sits beside the steps from 900 px, with Take over,
+ Pause, Stop and Hand back. Take over takes a Run desk's live lease or makes Bot take the person's own input on a Link
+ desk (their token, mapped to desk coordinates); the agent's input is refused while they drive. Hand back takes an
+ optional note that reaches the run as the person's message. A `hand_to_person` step is an Ask that opens the viewer
+ in control.
+- **Replay**: every computer step keeps its still and its action. A finished task's card has a timeline to scrub, and
+ each step opens the picture it saw. Typed text is never stored in a replay; a click keeps only its x,y.
+- Migration 0070 (task stills); `GET /api/v1/tasks/:id/desks`, `/desk-still`, `/replay`, the desk take-over/hand-back
+ and the person's input. Tests: `test/see-it-work.test.js`; screenshots: `test/see-it-work-shots.js`.
+
## Unreleased — the chat is the one place you use Actor (one agent system, stage 1)
## Unreleased — the chat is the one place you use Actor (one agent system, stage 1)
- **Composer v2** (openvibe-shared 4.215.0, the `agent` preset): the chat mounts `OpenVibeComposer.mount` with Attach,
- **Composer v2** (openvibe-shared 4.215.0, the `agent` preset): the chat mounts `OpenVibeComposer.mount` with Attach,
@@ -121,6 +121,7 @@ The resource index is for OpenVibe.Services on loopback and is blocked at the pu
@@ -121,6 +121,7 @@ The resource index is for OpenVibe.Services on loopback and is blocked at the pu
- **OpenVibe.Search** (`ACTOR_SEARCH_URL`), through the Search Fabric (`/api/v1/query`): content across the network for the runtime's `search_openvibe` tool (anonymous), and the web for `search_web` (cited results; asked as Actor's service principal with the grant `search.query.delegate`, naming the task's person in `X-OV-Subject`, so the paid steps past the open index count against them).
- **OpenVibe.Search** (`ACTOR_SEARCH_URL`), through the Search Fabric (`/api/v1/query`): content across the network for the runtime's `search_openvibe` tool (anonymous), and the web for `search_web` (cited results; asked as Actor's service principal with the grant `search.query.delegate`, naming the task's person in `X-OV-Subject`, so the paid steps past the open index count against them).
- **OpenVibe.Tools** (`ACTOR_TOOLS_URL`): the catalog, the run API and the page reader, for `find_tools`, `run_tool` and `read_page`. Called as any anonymous caller would.
- **OpenVibe.Tools** (`ACTOR_TOOLS_URL`): the catalog, the run API and the page reader, for `find_tools`, `run_tool` and `read_page`. Called as any anonymous caller would.
- **OpenVibe.Bot** (`ACTOR_BOT_URL`, default `http://127.0.0.1:4630`; empty turns it off): a signed-in person's own robots, for `list_my_robots` and `robot_steps`. Actor calls Bot with its own client-credentials token (audience `openvibe.bot`, `bot.robot.read` + `bot.robot.control`) and `X-OV-Subject` naming the person. Bot answers only for the robot's owner, and only on robots where that owner turned on "Let my OpenVibe.Actor agent drive this robot". Bot's gate, limits and e-stop judge every step. App, agent and service tasks never see these tools.
- **OpenVibe.Bot** (`ACTOR_BOT_URL`, default `http://127.0.0.1:4630`; empty turns it off): a signed-in person's own robots, for `list_my_robots` and `robot_steps`. Actor calls Bot with its own client-credentials token (audience `openvibe.bot`, `bot.robot.read` + `bot.robot.control`) and `X-OV-Subject` naming the person. Bot answers only for the robot's owner, and only on robots where that owner turned on "Let my OpenVibe.Actor agent drive this robot". Bot's gate, limits and e-stop judge every step. App, agent and service tasks never see these tools.
+- **See it work** (one-agent system, W1): a running task's card shows its desk — the private Live stream Link publishes for the task's desk (or the still at 1/s), the latest step in plain words, a ring where it last clicked, the page address and where it is. A tap opens the viewer: Take over (Run's lease, or Bot input as the person on a Link desk), Pause, Stop and Hand back with a note. A sign-in step is an Ask that opens it in control. A finished run has a replay of each step's still and action; typed text is never kept. The SDK calls are `desks`, `takeOver`, `handBack` and `replay` in `openvibe-sdk/agents` (PR open, released by the main session).
- **The chat is the one page** (one-agent system, stage 1): the Console is folded into it. `/console` answers 301 to `/chat`; the running tasks the Console listed are the "Now" strip at the top of the chat (one line each, Stop on each, hidden when nothing runs), and a running task is a card in its conversation that updates in place. The run settings ("New runs: Only me … Apply to all my existing runs") live in `/chat/settings`; the composer is openvibe-shared's composer v2, agent preset (Attach, Speak, the Fast/Balanced/Best mode chip, the round send, progress on each file). The empty chat shows four compact suggestion chips. `/me/spend` stays (`server/http/console-pages.js`, `server/http/chat-pages.js`, `test/one-page.test.js`).
- **The chat is the one page** (one-agent system, stage 1): the Console is folded into it. `/console` answers 301 to `/chat`; the running tasks the Console listed are the "Now" strip at the top of the chat (one line each, Stop on each, hidden when nothing runs), and a running task is a card in its conversation that updates in place. The run settings ("New runs: Only me … Apply to all my existing runs") live in `/chat/settings`; the composer is openvibe-shared's composer v2, agent preset (Attach, Speak, the Fast/Balanced/Best mode chip, the round send, progress on each file). The empty chat shows four compact suggestion chips. `/me/spend` stays (`server/http/console-pages.js`, `server/http/chat-pages.js`, `test/one-page.test.js`).
- **A person's own computers, also through OpenVibe.Bot** (plan T14/T17, contracts 0.166.0): `list_my_computers`, `computer_list`, `computer_read`, `computer_run` (argv, never a shell line) and `computer_write`, on machines installed with OpenVibe Link's `--computer` plugin. Actor calls Bot's machine API with its own token (`bot.computer.control`) and `X-OV-Subject` naming the person. Bot answers only for the machine's owner, and only where that owner turned on "Let my agent use this computer". The level the owner set on the machine decides what may happen there; at `ask`, each run or write waits up to two minutes for the owner's approval on the machine. Only a signed-in person's task sees these tools (`server/agents/computers.js`, `test/computers.test.js`). `computer_screen` (when an OpenAI key is set) takes a picture of the machine's screen through Bot (1280 px wide) and has OpenAI's vision model answer the agent's question about it. The runtime model gets that answer, never the picture; the picture is not stored; secrets visible on screen are named, not repeated; and each look is metered on the task like any model call (refused once the task's budget is spent). `computer_input` (offered with `computer_screen`; contracts 0.171.0) uses the mouse and keyboard: a click, double or right click or move at a position in the last screen picture (its width and height are passed, and the Link scales them to the real screen), a scroll, typed text, or one key or chord. It is an acting command, so at level `ask` each one waits for the person's approval on the machine; the runtime is told to look first, act in small steps, look again, and never type a secret the task did not give it.
- **A person's own computers, also through OpenVibe.Bot** (plan T14/T17, contracts 0.166.0): `list_my_computers`, `computer_list`, `computer_read`, `computer_run` (argv, never a shell line) and `computer_write`, on machines installed with OpenVibe Link's `--computer` plugin. Actor calls Bot's machine API with its own token (`bot.computer.control`) and `X-OV-Subject` naming the person. Bot answers only for the machine's owner, and only where that owner turned on "Let my agent use this computer". The level the owner set on the machine decides what may happen there; at `ask`, each run or write waits up to two minutes for the owner's approval on the machine. Only a signed-in person's task sees these tools (`server/agents/computers.js`, `test/computers.test.js`). `computer_screen` (when an OpenAI key is set) takes a picture of the machine's screen through Bot (1280 px wide) and has OpenAI's vision model answer the agent's question about it. The runtime model gets that answer, never the picture; the picture is not stored; secrets visible on screen are named, not repeated; and each look is metered on the task like any model call (refused once the task's budget is spent). `computer_input` (offered with `computer_screen`; contracts 0.171.0) uses the mouse and keyboard: a click, double or right click or move at a position in the last screen picture (its width and height are passed, and the Link scales them to the real screen), a scroll, typed text, or one key or chord. It is an acting command, so at level `ask` each one waits for the person's approval on the machine; the runtime is told to look first, act in small steps, look again, and never type a secret the task did not give it.
- `computer_browser` (contracts 0.174.0, Link v0.10.0) drives a browser of the agent's own on the machine. It is a
- `computer_browser` (contracts 0.174.0, Link v0.10.0) drives a browser of the agent's own on the machine. It is a
new file mode 100644
new file mode 100644
@@ -0,0 +1,15 @@
@@ -0,0 +1,15 @@
+-- phase: expand
+-- See it work (W1): the picture each step saw, for a finished run's replay. The bytes are Actor's own (30-day task
+-- retention); secret fields are blanked by the desk's curtain before the picture ever leaves it.
+CREATE TABLE IF NOT EXISTS actor_task_stills (
+ task_id text NOT NULL,
+ seq integer NOT NULL,
+ tool text,
+ mime text NOT NULL,
+ width integer,
+ height integer,
+ bytes bytea NOT NULL,
+ at timestamptz NOT NULL,
+ PRIMARY KEY (task_id, seq)
+);
+CREATE INDEX IF NOT EXISTS actor_task_stills_task_at ON actor_task_stills (task_id, at);
@@ -254,3 +254,74 @@ body:has(.page-app) > footer, body:has(.page-app) #ov-footer, body:has(.page-app
@@ -254,3 +254,74 @@ body:has(.page-app) > footer, body:has(.page-app) #ov-footer, body:has(.page-app
.ct-share-defaults input[type=checkbox] { width: 18px; height: 18px; }
.ct-share-defaults input[type=checkbox] { width: 18px; height: 18px; }
.ct-share-defaults .sc-btn { min-height: 44px; }
.ct-share-defaults .sc-btn { min-height: 44px; }
.ct-share-defaults .muted { flex-basis: 100%; font-size: 12.5px; line-height: 1.45; }
.ct-share-defaults .muted { flex-basis: 100%; font-size: 12.5px; line-height: 1.45; }
+
+/* ── See it work (W1): the desk tile in a running card, the viewer, and the finished run's replay. ─────────────── */
+.ct-desk-tile { margin: 0; display: grid; gap: 6px; }
+.ct-desk-frame { position: relative; aspect-ratio: 16 / 9; overflow: hidden; border-radius: 14px; border: 1px solid var(--line); background: #05070d; }
+.ct-desk-frame img, .ct-desk-frame iframe { position: absolute; inset: 0; width: 100%; height: 100%; border: 0; object-fit: contain; background: #05070d; }
+.ct-desk-open { position: absolute; inset: 0; z-index: 2; display: flex; align-items: flex-end; justify-content: flex-end; padding: 8px; border: 0; background: transparent; cursor: pointer; }
+.ct-desk-open span { display: inline-flex; align-items: center; min-height: 44px; padding: 0 12px; border-radius: 999px; background: rgba(2, 6, 14, .72); color: #fff; font: inherit; font-size: 12.5px; font-weight: 700; opacity: .9; }
+.ct-desk-open:hover span, .ct-desk-open:focus-visible span { opacity: 1; background: color-mix(in srgb, var(--accent, #f97316) 70%, #05070d); }
+.ct-desk-live { position: absolute; z-index: 1; top: 8px; left: 8px; display: inline-flex; align-items: center; gap: 5px; padding: 2px 8px; border-radius: 999px; background: rgba(0, 0, 0, .6); color: #fff; font-size: 11px; font-weight: 750; letter-spacing: .04em; text-transform: uppercase; }
+.ct-desk-live::before { content: ""; width: 7px; height: 7px; border-radius: 50%; background: #ef4444; animation: ct-breathe 1.4s ease-in-out infinite; }
+.ct-desk-ring { position: absolute; z-index: 1; width: 22px; height: 22px; margin: -11px 0 0 -11px; border-radius: 50%; border: 2px solid #ef4444; box-shadow: 0 0 0 3px rgba(239, 68, 68, .35); pointer-events: none; }
+.ct-desk-tile figcaption { display: flex; flex-wrap: wrap; align-items: center; gap: 3px 10px; color: var(--text-muted, #7386a3); font-size: 12.5px; line-height: 1.4; }
+.ct-desk-tile figcaption [data-ct-desk-caption] { color: var(--text-secondary, #a8b3c7); }
+.ct-desk-place { font-weight: 650; color: var(--text-muted, #7386a3); }
+.ct-desk-tile figcaption a { color: var(--accent, #f97316); text-decoration: none; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 100%; }
+.ct-desk-ask { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 8px; padding: 10px 12px; border-radius: 12px; border: 1px solid color-mix(in srgb, var(--accent, #f97316) 45%, var(--line)); background: color-mix(in srgb, var(--accent, #f97316) 8%, var(--bg-card, #131c2e)); }
+.ct-desk-ask p { margin: 0; flex: 1 1 220px; font-size: 13.5px; color: var(--text-primary, #e6edf7); }
+.ct-desk-ask button { min-height: 44px; padding: 0 14px; border: 0; border-radius: 12px; background: var(--accent, #f97316); color: var(--on-accent, #fff); font: inherit; font-weight: 700; }
+
+/* The viewer: a phone gets the desk filling the screen, a computer gets it beside the steps. */
+.ct-viewer { position: fixed; inset: 0; z-index: 60; display: grid; grid-template-rows: minmax(0, 1fr) auto; background: #05070d; color: #e6edf7; }
+.ct-viewer[hidden] { display: none; }
+.ct-viewer-stage { position: relative; min-height: 0; overflow: hidden; background: #05070d; }
+.ct-viewer-stage img, .ct-viewer-stage iframe { position: absolute; inset: 0; width: 100%; height: 100%; border: 0; object-fit: contain; background: #05070d; }
+.ct-viewer-input { position: absolute; inset: 0; z-index: 3; display: none; touch-action: none; cursor: crosshair; outline: 0; }
+.ct-viewer[data-control="1"] .ct-viewer-input { display: block; }
+.ct-viewer-ring { position: absolute; z-index: 2; width: 24px; height: 24px; margin: -12px 0 0 -12px; border-radius: 50%; border: 2px solid #ef4444; box-shadow: 0 0 0 4px rgba(239, 68, 68, .35); pointer-events: none; }
+.ct-viewer-badge { position: absolute; z-index: 4; top: 10px; left: 10px; display: inline-flex; align-items: center; gap: 6px; padding: 3px 9px; border-radius: 999px; background: rgba(0, 0, 0, .65); color: #fff; font-size: 11px; font-weight: 750; letter-spacing: .04em; text-transform: uppercase; }
+.ct-viewer[data-state="control"] .ct-viewer-badge { background: color-mix(in srgb, var(--accent, #f97316) 75%, #05070d); }
+.ct-viewer-caption { position: absolute; z-index: 4; left: 10px; right: 10px; bottom: 10px; display: flex; flex-wrap: wrap; align-items: center; gap: 4px 10px; padding: 8px 10px; border-radius: 12px; background: rgba(2, 6, 14, .72); color: #e6edf7; font-size: 12.5px; }
+.ct-viewer-caption a { color: #fdba74; text-decoration: none; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; max-width: 100%; }
+.ct-viewer-side { display: none; }
+.ct-viewer-bar { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; padding: 8px 10px calc(72px + env(safe-area-inset-bottom, 0px)); background: linear-gradient(180deg, transparent, #05070d 40%); }
+.ct-viewer-bar button, .ct-viewer-note button { min-height: 44px; padding: 0 14px; border-radius: 12px; border: 1px solid var(--line); background: #0f1729; color: #e6edf7; font: inherit; font-weight: 700; cursor: pointer; }
+.ct-viewer-bar button.is-primary, .ct-viewer-note button { border: 0; background: var(--accent, #f97316); color: var(--on-accent, #fff); }
+.ct-viewer-bar button:disabled { opacity: .5; cursor: default; }
+.ct-viewer-hint { flex: 1 1 200px; margin: 0; color: #a8b3c7; font-size: 12.5px; line-height: 1.35; }
+.ct-viewer-note { display: flex; flex-wrap: wrap; gap: 8px; width: 100%; }
+.ct-viewer-note input { flex: 1 1 180px; min-height: 44px; padding: 0 12px; border-radius: 12px; border: 1px solid var(--line); background: #0b1220; color: inherit; font: inherit; }
+.ct-viewer[data-control="0"] .ct-viewer-note { display: none; }
+@media (min-width: 900px) {
+ .ct-viewer { grid-template-columns: minmax(0, 1.6fr) minmax(280px, .8fr); grid-template-rows: minmax(0, 1fr) auto; }
+ .ct-viewer-stage { grid-column: 1; grid-row: 1 / span 2; }
+ .ct-viewer-side { grid-column: 2; grid-row: 1; display: block; overflow: auto; padding: 16px; border-left: 1px solid var(--line); }
+ .ct-viewer-bar { grid-column: 2; grid-row: 2; display: grid; grid-template-columns: 1fr 1fr; padding: 10px 16px 16px; background: #05070d; }
+ .ct-viewer-hint, .ct-viewer-note { grid-column: 1 / -1; }
+ .ct-viewer-side h2 { margin: 0 0 8px; font-size: 15px; }
+ .ct-viewer-side .ovt-w-list { max-height: none; }
+}
+@media (max-width: 899px) {
+ .ct-viewer-bar { position: fixed; left: 0; right: 0; bottom: 0; z-index: 61; }
+ .ct-viewer { padding-bottom: 0; }
+}
+.ct-replay { margin-top: 10px; border: 1px solid var(--line); border-radius: 14px; padding: 0 12px; background: var(--bg-card, #131c2e); }
+.ct-replay > summary { display: flex; align-items: center; min-height: 44px; cursor: pointer; font-size: 13.5px; font-weight: 650; color: var(--text-secondary, #a8b3c7); list-style: none; }
+.ct-replay > summary::-webkit-details-marker { display: none; }
+.ct-replay-body { display: grid; gap: 8px; padding: 0 0 12px; }
+.ct-replay-body input[type=range] { width: 100%; min-height: 44px; accent-color: var(--accent, #f97316); }
+.ct-replay-shot { margin: 0; display: grid; gap: 6px; }
+.ct-replay-shot img { width: 100%; aspect-ratio: 16 / 9; object-fit: contain; border-radius: 12px; background: #05070d; }
+.ct-replay-none { display: grid; place-items: center; min-height: 120px; border-radius: 12px; background: #05070d; color: var(--text-muted, #7386a3); font-size: 13px; }
+.ct-replay-shot figcaption { display: flex; flex-wrap: wrap; justify-content: space-between; gap: 4px 10px; font-size: 12.5px; color: var(--text-muted, #7386a3); }
+.ct-replay-shot figcaption a { color: var(--accent, #f97316); }
+.ct-replay-steps { list-style: none; margin: 0; padding: 0; display: grid; gap: 4px; max-height: 220px; overflow: auto; }
+.ct-replay-steps button { display: flex; align-items: center; gap: 8px; width: 100%; min-height: 44px; padding: 0 8px; border: 0; border-radius: 9px; background: transparent; color: var(--text-secondary, #a8b3c7); font: inherit; font-size: 13px; text-align: left; cursor: pointer; }
+.ct-replay-steps button:hover, .ct-replay-steps button:focus-visible, .ct-replay-steps button.is-on { background: color-mix(in srgb, var(--accent, #f97316) 12%, transparent); color: var(--text-primary, #e6edf7); }
+.ct-replay-steps button span { flex: none; display: grid; place-items: center; width: 22px; height: 22px; border-radius: 50%; background: color-mix(in srgb, var(--accent, #f97316) 16%, transparent); color: var(--accent, #f97316); font-size: 11px; font-weight: 750; }
+@media (max-width: 640px) {
+ .ct-desk-open span { min-height: 44px; }
+ .ct-viewer-bar button, .ct-viewer-note button { min-height: 44px; }
+}
@@ -500,6 +500,7 @@
@@ -500,6 +500,7 @@
liveLine(turn);
liveLine(turn);
draft(turn);
draft(turn);
syncWait(turn, next);
syncWait(turn, next);
+ if (window.OVDesks) window.OVDesks.sync(turn, next);
} else {
} else {
stopScreen(turn);
stopScreen(turn);
// An answer the person already watched being written is not typed in again: it only takes its
// An answer the person already watched being written is not typed in again: it only takes its
@@ -507,6 +508,7 @@
@@ -507,6 +508,7 @@
if (turn.querySelector('[data-ovt-draft]')) turn.removeAttribute('data-open');
if (turn.querySelector('[data-ovt-draft]')) turn.removeAttribute('data-open');
turn._draft = null;
turn._draft = null;
if (window.OVThread) window.OVThread.replace(id, h); else turn.replaceWith(next);
if (window.OVThread) window.OVThread.replace(id, h); else turn.replaceWith(next);
+ if (window.OVDesks) window.OVDesks.scan();
var dot = chat.querySelector('.ovt-list a[aria-current] .ovt-dot'); if (dot) dot.remove();
var dot = chat.querySelector('.ovt-list a[aria-current] .ovt-dot'); if (dot) dot.remove();
if (note) note.textContent = '';
if (note) note.textContent = '';
syncSend();
syncSend();
new file mode 100644
new file mode 100644
@@ -0,0 +1,429 @@
@@ -0,0 +1,429 @@
+/* See it work (W1): the desk tile in a running task's card, the viewer with Take over / Pause / Stop, and a finished
+ * run's replay. The calls are the SDK's names (desks, takeOver, handBack, replay) over Actor's own routes until
+ * openvibe-sdk/agents carries them; a page can do what a script can.
+ */
+(function () {
+ 'use strict';
+ var api = {
+ async json(url, opts) {
+ var r = await fetch(url, Object.assign({ credentials: 'same-origin', headers: { Accept: 'application/json' } }, opts || {}));
+ var data = await r.json().catch(function () { return {}; });
+ if (!r.ok) throw new Error((data && (data.detail || data.error)) || 'That did not work.');
+ return data;
+ },
+ desks(run) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks'); },
+ takeOver(run, desk) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks/' + encodeURIComponent(desk) + '/take-over', { method: 'POST' }); },
+ handBack(run, desk, note) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks/' + encodeURIComponent(desk) + '/hand-back', { method: 'POST', headers: { Accept: 'application/json', 'Content-Type': 'application/json' }, body: JSON.stringify({ note: note || '' }) }); },
+ input(run, desk, value) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/desks/' + encodeURIComponent(desk) + '/input', { method: 'POST', headers: { Accept: 'application/json', 'Content-Type': 'application/json' }, body: JSON.stringify(value) }); },
+ replay(run) { return this.json('/api/v1/tasks/' + encodeURIComponent(run) + '/replay'); },
+ };
+ window.OVAgents = api;
+
+ var seen = new WeakSet();
+ var streams = new WeakMap(); // tile → { live, frame, timer, visible, paused, blob }
+ var viewer = null;
+
+ function taskOf(el) { return el.getAttribute('data-task'); }
+ function deskIdOf(el) { return el.getAttribute('data-desk'); }
+
+ function stopTimer(st) { if (st && st.timer) { clearInterval(st.timer); st.timer = 0; } }
+ function stopBlob(st) { if (st && st.blob) { URL.revokeObjectURL(st.blob); st.blob = null; } }
+
+ /** The stream is loaded only while its tile is on screen; Live's encoder stops when nobody watches. */
+ function visible(el) {
+ var r = el.getBoundingClientRect();
+ return r.bottom > 0 && r.top < (window.innerHeight || 0) && r.right > 0 && r.left < (window.innerWidth || 0);
+ }
+
+ function show(img, src) {
+ var st = streams.get(img);
+ if (!st) return;
+ fetch(src, { credentials: 'same-origin', cache: 'no-store' })
+ .then(function (r) { return r.ok ? r.blob() : null; })
+ .then(function (blob) {
+ if (!blob || !st.visible || st.live || st.paused) return;
+ stopBlob(st);
+ st.blob = URL.createObjectURL(blob);
+ img.src = st.blob;
+ img.hidden = false;
+ })
+ .catch(function () {});
+ }
+
+ function startStill(tile, st) {
+ stopTimer(st);
+ var img = tile.querySelector('[data-ct-desk-still]');
+ var src = tile.getAttribute('data-still');
+ if (!img || !src) return;
+ if (!st.visible || st.live || st.paused) return;
+ show(img, src);
+ st.timer = setInterval(function () { show(img, tile.getAttribute('data-still')); }, 1000);
+ }
+
+ /** Load the Live player while the tile is visible, and remove it when the tile leaves the screen. */
+ function startStream(tile, st) {
+ if (st.frame || !st.live || !st.live.embed_url || !st.visible || st.paused) return;
+ var box = tile.querySelector('[data-ct-desk-frame]');
+ if (!box) return;
+ var frame = document.createElement('iframe');
+ frame.src = st.live.embed_url;
+ frame.title = 'The desk Actor works on, live';
+ frame.setAttribute('allow', 'autoplay; fullscreen; clipboard-write');
+ frame.setAttribute('referrerpolicy', 'no-referrer');
+ frame.setAttribute('sandbox', 'allow-scripts allow-same-origin allow-forms allow-pointer-lock');
+ box.appendChild(frame);
+ st.frame = frame;
+ var img = tile.querySelector('[data-ct-desk-still]');
+ if (img) img.hidden = true;
+ var badge = tile.querySelector('[data-ct-desk-live]');
+ if (badge) badge.hidden = false;
+ stopTimer(st);
+ }
+ function stopStream(st) {
+ if (st.frame) { st.frame.remove(); st.frame = null; }
+ var tile = st.tile;
+ if (tile) {
+ var badge = tile.querySelector('[data-ct-desk-live]');
+ if (badge) badge.hidden = true;
+ var img = tile.querySelector('[data-ct-desk-still]');
+ if (img) img.hidden = false;
+ }
+ }
+
+ function syncMeta(tile, fresh) {
+ if (!tile || !fresh) return;
+ for (var key of ['data-desk', 'data-where', 'data-kind', 'data-label', 'data-still']) {
+ if (fresh.hasAttribute(key)) tile.setAttribute(key, fresh.getAttribute(key));
+ }
+ var cap = tile.querySelector('[data-ct-desk-caption]'), freshCap = fresh.querySelector('[data-ct-desk-caption]');
+ if (cap && freshCap) cap.textContent = freshCap.textContent;
+ var place = tile.querySelector('[data-ct-desk-place]'), freshPlace = fresh.querySelector('[data-ct-desk-place]');
+ if (place && freshPlace) place.textContent = freshPlace.textContent;
+ var ring = tile.querySelector('[data-ct-desk-ring]'), freshRing = fresh.querySelector('[data-ct-desk-ring]');
+ if (ring && freshRing) ring.outerHTML = freshRing.outerHTML;
+ else if (ring) ring.remove();
+ else if (freshRing) tile.querySelector('[data-ct-desk-frame]').appendChild(freshRing);
+ var url = tile.querySelector('[data-ct-desk-url]'), freshUrl = fresh.querySelector('[data-ct-desk-url]');
+ if (url && freshUrl) { url.href = freshUrl.href; url.textContent = freshUrl.textContent; url.hidden = false; }
+ else if (url) url.hidden = true;
+ else if (freshUrl) tile.querySelector('figcaption').appendChild(freshUrl);
+ var ask = tile.querySelector('[data-ct-desk-ask]'), freshAsk = fresh.querySelector('[data-ct-desk-ask]');
+ if (ask && freshAsk) ask.outerHTML = freshAsk.outerHTML;
+ else if (ask) ask.remove();
+ else if (freshAsk) tile.appendChild(freshAsk);
+ }
+
+ function scan() {
+ document.querySelectorAll('[data-ct-desk]').forEach(function (tile) {
+ if (seen.has(tile)) return;
+ seen.add(tile);
+ var st = { tile: tile, live: null, frame: null, timer: 0, visible: false, paused: false, blob: null };
+ streams.set(tile, st);
+ var task = taskOf(tile);
+ if (task) {
+ api.desks(task).then(function (j) {
+ var found = (j.desks || []).find(function (d) { return d.id === deskIdOf(tile); }) || (j.desks || [])[0];
+ if (!found) return;
+ st.live = found.live || null;
+ st.width = found.width || 1280; st.height = found.height || 800;
+ tile.setAttribute('data-width', String(st.width)); tile.setAttribute('data-height', String(st.height));
+ if (found.label) tile.setAttribute('data-label', found.label);
+ if (found.browser_url) {
+ var a = tile.querySelector('[data-ct-desk-url]');
+ if (a) { a.href = found.browser_url; a.textContent = found.browser_url.replace(/^https?:\/\//, ''); a.hidden = false; }
+ }
+ paint(tile, st);
+ }).catch(function () {});
+ }
+ paint(tile, st);
+ var io = typeof IntersectionObserver === 'function' ? new IntersectionObserver(function (entries) {
+ entries.forEach(function (entry) {
+ st.visible = entry.isIntersecting;
+ if (st.visible) { startStream(tile, st); startStill(tile, st); }
+ else { stopStream(st); stopTimer(st); }
+ });
+ }, { rootMargin: '80px' }) : null;
+ if (io) io.observe(tile); else { st.visible = true; paint(tile, st); }
+ });
+ scanReplays();
+ }
+
+ function paint(tile, st) {
+ st.visible = st.visible || visible(tile);
+ if (st.live && st.live.embed_url) startStream(tile, st);
+ else startStill(tile, st);
+ }
+
+ function openTile(tile, control) {
+ var task = taskOf(tile);
+ var desk = deskIdOf(tile);
+ if (!task || !desk) return;
+ var v = ensureViewer();
+ v.hidden = false;
+ v.dataset.task = task; v.dataset.desk = desk;
+ v.dataset.control = '0'; v.dataset.state = 'watch';
+ v._size = { width: Number(tile.getAttribute('data-width')) || 1280, height: Number(tile.getAttribute('data-height')) || 800 };
+ v.querySelector('[data-ct-viewer-badge]').textContent = 'Watching';
+ var badge = v.querySelector('[data-ct-viewer-badge]');
+ badge.textContent = tile.getAttribute('data-label') || 'The desk';
+ v.querySelector('[data-ct-viewer-caption-text]').textContent = (tile.querySelector('[data-ct-desk-caption]') || {}).textContent || '';
+ var page = tile.querySelector('[data-ct-desk-url]');
+ var capLink = v.querySelector('[data-ct-viewer-page]');
+ if (page && page.href) { capLink.href = page.href; capLink.textContent = page.textContent; capLink.hidden = false; } else capLink.hidden = true;
+ var stage = v.querySelector('[data-ct-viewer-stage]');
+ stage.querySelectorAll('iframe,img').forEach(function (n) { n.remove(); });
+ stage.querySelector('[data-ct-viewer-input]').hidden = false;
+ var img = document.createElement('img');
+ img.alt = 'The desk Actor works on';
+ img.hidden = false;
+ stage.insertBefore(img, stage.firstChild);
+ var same = streams.get(tile);
+ if (same) { stopStream(same); stopTimer(same); }
+ v._live = same && same.live && same.live.embed_url ? same.live : null;
+ v._img = img;
+ if (v._live) {
+ var frame = document.createElement('iframe');
+ frame.src = v._live.embed_url;
+ frame.title = 'The desk Actor works on, live';
+ frame.setAttribute('allow', 'autoplay; fullscreen; clipboard-write');
+ frame.setAttribute('referrerpolicy', 'no-referrer');
+ frame.setAttribute('sandbox', 'allow-scripts allow-same-origin allow-forms allow-pointer-lock');
+ stage.insertBefore(frame, stage.firstChild);
+ img.hidden = true;
+ }
+ // The picture polls at 1/s only when there is no stream (or the stream is paused).
+ v._still = function () {
+ if (v.hidden || v.dataset.paused === '1') return;
+ var src = tile.getAttribute('data-still');
+ if (!src || (same && same.live && !v.dataset.paused)) return;
+ fetch(src, { credentials: 'same-origin', cache: 'no-store' }).then(function (r) { return r.ok ? r.blob() : null; }).then(function (b) {
+ if (b && !v.hidden && v.dataset.paused !== '1') img.src = URL.createObjectURL(b);
+ }).catch(function () {});
+ };
+ v._still();
+ clearInterval(v._timer);
+ v._timer = setInterval(v._still, 1000);
+ var side = v.querySelector('[data-ct-viewer-steps]');
+ side.replaceChildren();
+ var turn = document.querySelector('.ovt-turn[data-task="' + task + '"]');
+ var list = turn && turn.querySelector('[data-ovt-work]');
+ if (list) side.appendChild(list.cloneNode(true));
+ var hint = v.querySelector('[data-ct-viewer-hint]');
+ hint.textContent = 'Watch Actor work, or take over to use the mouse and keyboard yourself.';
+ v.querySelector('[data-ct-viewer-take]').hidden = false;
+ v.querySelector('[data-ct-viewer-hand]').hidden = true;
+ v.querySelector('[data-ct-viewer-note]').hidden = true;
+ v.querySelector('[data-ct-viewer-pause]').textContent = 'Pause';
+ v.dataset.paused = '0';
+ v.querySelector('[data-ct-viewer-input]').focus();
+ if (control) takeOver(v);
+ else v.querySelector('[data-ct-viewer-take]').focus();
+ }
+
+ function ensureViewer() {
+ if (viewer) return viewer;
+ viewer = document.createElement('div');
+ viewer.className = 'ct-viewer';
+ viewer.setAttribute('role', 'dialog');
+ viewer.setAttribute('aria-modal', 'true');
+ viewer.setAttribute('aria-label', 'The desk Actor works on');
+ viewer.hidden = true;
+ viewer.innerHTML = '<div class="ct-viewer-stage" data-ct-viewer-stage><span class="ct-viewer-badge" data-ct-viewer-badge>Watching</span><div class="ct-viewer-input" data-ct-viewer-input tabindex="0" aria-label="Use the desk"></div><div class="ct-viewer-caption"><span data-ct-viewer-caption-text></span><a data-ct-viewer-page hidden></a></div></div>'
+ + '<aside class="ct-viewer-side"><h2>What Actor is doing</h2><div data-ct-viewer-steps></div></aside>'
+ + '<div class="ct-viewer-bar"><p class="ct-viewer-hint" data-ct-viewer-hint></p><button type="button" class="is-primary" data-ct-viewer-take>Take over</button><button type="button" data-ct-viewer-pause>Pause</button><button type="button" data-ct-viewer-hand hidden>Hand back</button><button type="button" data-ct-viewer-stop>Stop</button><button type="button" data-ct-viewer-close>Close</button><div class="ct-viewer-note" data-ct-viewer-note hidden><input type="text" maxlength="1000" placeholder="A note for Actor, optional" data-ct-viewer-note-input aria-label="A note for Actor"><button type="button" data-ct-viewer-note-send>Hand back with note</button></div></div>';
+ document.body.appendChild(viewer);
+ viewer.querySelector('[data-ct-viewer-close]').addEventListener('click', closeViewer);
+ viewer.querySelector('[data-ct-viewer-take]').addEventListener('click', function () { takeOver(viewer); });
+ viewer.querySelector('[data-ct-viewer-pause]').addEventListener('click', function () { pauseViewer(viewer); });
+ viewer.querySelector('[data-ct-viewer-stop]').addEventListener('click', function () { stopViewer(viewer); });
+ viewer.querySelector('[data-ct-viewer-hand]').addEventListener('click', function () { handBack(viewer); });
+ viewer.querySelector('[data-ct-viewer-note-send]').addEventListener('click', function () { handBack(viewer); });
+ viewer.querySelector('[data-ct-viewer-input]').addEventListener('pointerdown', onPointer);
+ viewer.querySelector('[data-ct-viewer-input]').addEventListener('wheel', onWheel, { passive: false });
+ viewer.querySelector('[data-ct-viewer-input]').addEventListener('keydown', onKey);
+ document.addEventListener('keydown', function (e) { if (e.key === 'Escape' && viewer && !viewer.hidden) closeViewer(); });
+ return viewer;
+ }
+
+ function closeViewer() {
+ if (!viewer) return;
+ viewer.hidden = true;
+ clearInterval(viewer._timer);
+ viewer.querySelectorAll('iframe').forEach(function (f) { f.remove(); });
+ document.querySelectorAll('[data-ct-desk]').forEach(function (tile) { var st = streams.get(tile); if (st) paint(tile, st); });
+ }
+
+ function stageSize(v) {
+ var stage = v.querySelector('[data-ct-viewer-stage]');
+ var img = stage.querySelector('img');
+ var size = v._size || {};
+ return { width: (img && img.naturalWidth) || size.width || 1280, height: (img && img.naturalHeight) || size.height || 800, rect: stage.getBoundingClientRect() };
+ }
+
+ function sendInput(v, value) {
+ var task = v.dataset.task, desk = v.dataset.desk;
+ if (!task || !desk) return;
+ api.input(task, desk, value).catch(function (err) {
+ var hint = v.querySelector('[data-ct-viewer-hint]');
+ if (hint) hint.textContent = err.message || 'The desk did not take that.';
+ });
+ }
+
+ function point(v, e) {
+ var s = stageSize(v);
+ var x = Math.round((e.clientX - s.rect.left) / Math.max(1, s.rect.width) * s.width);
+ var y = Math.round((e.clientY - s.rect.top) / Math.max(1, s.rect.height) * s.height);
+ var ring = v.querySelector('.ct-viewer-ring');
+ if (!ring) { ring = document.createElement('span'); ring.className = 'ct-viewer-ring'; v.querySelector('[data-ct-viewer-stage]').appendChild(ring); }
+ ring.style.left = ((x / Math.max(1, s.width)) * 100).toFixed(2) + '%';
+ ring.style.top = ((y / Math.max(1, s.height)) * 100).toFixed(2) + '%';
+ return { x: x, y: y, width: s.width, height: s.height };
+ }
+ function onPointer(e) {
+ if (!viewer || viewer.dataset.control !== '1') return;
+ e.preventDefault();
+ var p = point(viewer, e);
+ sendInput(viewer, { action: e.button === 2 ? 'right_click' : 'click', x: p.x, y: p.y, width: p.width, height: p.height });
+ viewer.querySelector('[data-ct-viewer-input]').focus();
+ }
+ function onWheel(e) {
+ if (!viewer || viewer.dataset.control !== '1') return;
+ e.preventDefault();
+ var p = point(viewer, e);
+ sendInput(viewer, { action: 'scroll', x: p.x, y: p.y, width: p.width, height: p.height, dy: e.deltaY > 0 ? 3 : -3 });
+ }
+ function onKey(e) {
+ if (!viewer || viewer.dataset.control !== '1') return;
+ var keys = { Enter: 'Return', Backspace: 'BackSpace', Tab: 'Tab', Escape: 'Escape', ArrowUp: 'Up', ArrowDown: 'Down', ArrowLeft: 'Left', ArrowRight: 'Right', Delete: 'Delete', ' ': 'space' };
+ if (keys[e.key]) { e.preventDefault(); sendInput(viewer, { action: 'key', keys: keys[e.key] }); return; }
+ if (e.key.length === 1 && !e.ctrlKey && !e.metaKey && !e.altKey) { e.preventDefault(); sendInput(viewer, { action: 'type', text: e.key }); }
+ }
+
+ async function takeOver(v) {
+ var hint = v.querySelector('[data-ct-viewer-hint]');
+ try {
+ await api.takeOver(v.dataset.task, v.dataset.desk);
+ v.dataset.control = '1';
+ v.dataset.state = 'control';
+ v.querySelector('[data-ct-viewer-badge]').textContent = 'You are driving';
+ v.querySelector('[data-ct-viewer-take]').hidden = true;
+ v.querySelector('[data-ct-viewer-hand]').hidden = false;
+ v.querySelector('[data-ct-viewer-note]').hidden = false;
+ hint.textContent = 'You are driving. Actor waits; click and type on the picture.';
+ v.querySelector('[data-ct-viewer-input]').focus();
+ } catch (err) {
+ hint.textContent = err.message || 'Actor could not hand over the desk.';
+ }
+ }
+ async function handBack(v) {
+ var note = (v.querySelector('[data-ct-viewer-note-input]') || {}).value || '';
+ try {
+ await api.handBack(v.dataset.task, v.dataset.desk, note);
+ v.dataset.control = '0';
+ v.dataset.state = 'watch';
+ v.querySelector('[data-ct-viewer-badge]').textContent = 'Actor is working';
+ v.querySelector('[data-ct-viewer-take]').hidden = false;
+ v.querySelector('[data-ct-viewer-hand]').hidden = true;
+ v.querySelector('[data-ct-viewer-note]').hidden = true;
+ v.querySelector('[data-ct-viewer-note-input]').value = '';
+ v.querySelector('[data-ct-viewer-hint]').textContent = note.trim() ? 'Handed back with your note. Actor goes on.' : 'Handed back. Actor goes on.';
+ } catch (err) {
+ v.querySelector('[data-ct-viewer-hint]').textContent = err.message || 'Actor could not take the desk back.';
+ }
+ }
+ function pauseViewer(v) {
+ var paused = v.dataset.paused === '1';
+ v.dataset.paused = paused ? '0' : '1';
+ v.querySelector('[data-ct-viewer-pause]').textContent = paused ? 'Pause' : 'Resume';
+ var stage = v.querySelector('[data-ct-viewer-stage]');
+ if (!paused) {
+ // Pause the picture: the Live player leaves, so the desk's encoder stops when nobody else watches.
+ stage.querySelectorAll('iframe').forEach(function (f) { f.remove(); });
+ if (v._img) v._img.hidden = false;
+ v._still();
+ } else if (v._live) {
+ var frame = document.createElement('iframe');
+ frame.src = v._live.embed_url;
+ frame.title = 'The desk Actor works on, live';
+ frame.setAttribute('allow', 'autoplay; fullscreen; clipboard-write');
+ frame.setAttribute('referrerpolicy', 'no-referrer');
+ frame.setAttribute('sandbox', 'allow-scripts allow-same-origin allow-forms allow-pointer-lock');
+ stage.insertBefore(frame, stage.firstChild);
+ if (v._img) v._img.hidden = true;
+ }
+ }
+ function stopViewer(v) {
+ fetch('/tasks/' + encodeURIComponent(v.dataset.task) + '/cancel', { method: 'POST', credentials: 'same-origin', redirect: 'manual' })
+ .catch(function () {}).then(function () { closeViewer(); });
+ }
+
+ // ── Replay ──────────────────────────────────────────────────────────────────────────────────────────────────────
+ function scanReplays() {
+ document.querySelectorAll('[data-ct-replay]').forEach(function (box) {
+ if (seen.has(box)) return;
+ seen.add(box);
+ var data = null;
+ try { data = JSON.parse((box.querySelector('[data-ct-replay-data]') || {}).textContent || '[]'); } catch { data = []; }
+ var range = box.querySelector('[data-ct-replay-range]');
+ var img = box.querySelector('[data-ct-replay-img]');
+ var caption = box.querySelector('[data-ct-replay-caption]');
+ var link = box.querySelector('[data-ct-replay-link]');
+ function show(i) {
+ var st = data[i] || null;
+ if (!st) return;
+ range.value = String(i);
+ caption.textContent = captionOf(st);
+ if (st.still && img) { img.src = st.still.url; img.hidden = false; if (link) { link.href = st.still.url; link.hidden = false; } }
+ else if (img) { img.hidden = true; if (link) link.hidden = true; }
+ box.querySelectorAll('[data-ct-replay-step]').forEach(function (b) { b.classList.toggle('is-on', Number(b.getAttribute('data-ct-replay-step')) === i); });
+ }
+ function captionOf(st) {
+ if (!st) return '';
+ if (st.tool === 'computer_input') {
+ if (st.action === 'click' || st.action === 'double_click' || st.action === 'right_click') return 'Clicked at ' + Math.round(st.x || 0) + ', ' + Math.round(st.y || 0);
+ if (st.action === 'scroll') return 'Scrolled';
+ if (st.action === 'key') return 'Pressed a key';
+ if (st.action === 'move') return 'Moved the mouse';
+ return 'Typed';
+ }
+ if (st.tool === 'computer_browser') {
+ if (st.action === 'open') return 'Opened a page' + (st.url ? ': ' + st.url : '');
+ if (st.action === 'read') return 'Read the page';
+ if (st.action === 'click') return 'Clicked on the page';
+ if (st.action === 'fill') return 'Filled a field';
+ if (st.action === 'press') return 'Pressed a key';
+ if (st.action === 'back') return 'Went back';
+ return 'Used the browser';
+ }
+ if (st.tool === 'computer_screen') return 'Looked at the screen';
+ return st.action === 'start' ? 'Started the desk' : st.action === 'stop' ? 'Stopped the desk' : 'Checked the desk';
+ }
+ if (range) range.addEventListener('input', function () { show(Number(range.value) || 0); });
+ box.querySelectorAll('[data-ct-replay-step]').forEach(function (b) {
+ b.addEventListener('click', function () { show(Number(b.getAttribute('data-ct-replay-step')) || 0); });
+ });
+ if (data.length) show(0);
+ });
+ }
+
+ function syncDesk(turn, next) {
+ var now = turn.querySelector('[data-ct-desk]'), fresh = next.querySelector('[data-ct-desk]');
+ if (now && fresh) syncMeta(now, fresh);
+ else if (now && !fresh) now.remove();
+ else if (!now && fresh) {
+ var reply = turn.querySelector('.ovt-reply');
+ if (reply) reply.insertBefore(fresh, reply.firstChild);
+ scan();
+ }
+ }
+
+ document.addEventListener('click', function (e) {
+ var take = e.target.closest && e.target.closest('[data-ct-takeover]');
+ if (take) { var t = take.closest('[data-ct-desk]'); if (t) openTile(t, true); return; }
+ var open = e.target.closest && e.target.closest('[data-ct-desk-open]');
+ if (open) { var tile = open.closest('[data-ct-desk]'); if (tile) openTile(tile, false); }
+ });
+ window.OVDesks = { scan: scan, sync: syncDesk, open: openTile };
+ if (document.readyState === 'loading') document.addEventListener('DOMContentLoaded', scan); else scan();
+ document.addEventListener('ov:boost:load', scan);
+})();
@@ -316,9 +316,8 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensO
@@ -316,9 +316,8 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensO
* Mouse, scroll, typing or one key on the machine (contracts 0.171.0 `input`): an acting command, so at level "ask"
* Mouse, scroll, typing or one key on the machine (contracts 0.171.0 `input`): an acting command, so at level "ask"
* it waits for the owner's approval there. x, y are measured in a screen picture of width × height (computer_screen's).
* it waits for the owner's approval there. x, y are measured in a screen picture of width × height (computer_screen's).
*/
*/
- async function input(subject, machine, args, { signal } = {}) {
- const bad = guard(subject, machine);
- if (bad) return bad;
+ /** A validated `input` value for a person's own mouse and keyboard, or { error }. */
+ function inputValue(args) {
const a = args && typeof args === 'object' ? args : {};
const a = args && typeof args === 'object' ? args : {};
if (!INPUT_ACTIONS.has(a.action)) return { error: `action is one of ${[...INPUT_ACTIONS].join(', ')}` };
if (!INPUT_ACTIONS.has(a.action)) return { error: `action is one of ${[...INPUT_ACTIONS].join(', ')}` };
const value = { action: a.action, ...onDesk(a) };
const value = { action: a.action, ...onDesk(a) };
@@ -343,9 +342,69 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensO
@@ -343,9 +342,69 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensO
if (typeof a.keys !== 'string' || !/^[A-Za-z0-9_+-]{1,64}$/.test(a.keys) || a.keys.startsWith('-')) return { error: 'key needs keys: one key or chord, for example Return, Escape, ctrl+s' };
if (typeof a.keys !== 'string' || !/^[A-Za-z0-9_+-]{1,64}$/.test(a.keys) || a.keys.startsWith('-')) return { error: 'key needs keys: one key or chord, for example Return, Escape, ctrl+s' };
value.keys = a.keys;
value.keys = a.keys;
}
}
- const r = await command(subject, machine, 'input', value, { signal, windowMs: ACT_WINDOW_MS });
+ return { value };
+ }
+ async function input(subject, machine, args, { signal } = {}) {
+ const bad = guard(subject, machine);
+ if (bad) return bad;
+ const made = inputValue(args);
+ if (made.error) return made;
+ const r = await command(subject, machine, 'input', made.value, { signal, windowMs: ACT_WINDOW_MS });
+ if (r.error) return r;
+ return { done: !!r.data.done, action: r.data.action || made.value.action, ...(r.data.x != null ? { x: r.data.x, y: r.data.y } : {}) };
+ }
+
+ /**
+ * The person's own mouse and keyboard on their Link desk (See it work, W1: "their taps and keys on the picture map
+ * to desk coordinates"). Actor passes the person's own Network token to Bot, never its service token: it is their
+ * input, at their own level, so Bot's "ask" does not ask them to approve themselves. A Run desk uses the session's
+ * person-commands route instead.
+ */
+ async function personInput(subject, token, machine, args, { signal } = {}) {
+ const bad = guard(subject, machine);
+ if (bad) return bad;
+ if (SESSION_MACHINE.test(String(machine || ''))) return { error: 'a Run desk takes the person\'s input through its own session' };
+ if (!token) return { error: 'sign in again: your own session is needed to use the desk' };
+ const made = inputValue(args);
+ if (made.error) return made;
+ const sig = signal ? AbortSignal.any([signal, AbortSignal.timeout(ACT_WINDOW_MS + 10_000)]) : AbortSignal.timeout(ACT_WINDOW_MS + 10_000);
+ let res;
+ try {
+ res = await fetchImpl(`${base}/api/v1/machines/${encodeURIComponent(machine)}/commands`, {
+ method: 'POST',
+ headers: { Accept: 'application/json', 'Content-Type': 'application/json', Authorization: `Bearer ${token}` },
+ body: JSON.stringify({ kind: 'input', value: made.value, ms: ACT_WINDOW_MS }), signal: sig,
+ });
+ } catch { return { error: 'Bot did not answer' }; }
+ const data = await res.json().catch(() => null);
+ if (!res.ok) return problem({ status: res.status, data }, 'your input');
+ if (!data || data.result !== 'ack') return { error: (data && (data.reason || (data.result === 'expired' ? 'the machine did not answer in time' : null))) || 'the machine refused it' };
+ return { done: true, action: (data.data && data.data.action) || made.value.action };
+ }
+
+ /** A private agent stream for the task's desk on Link (See it work, W1; Link docs/stream.md). */
+ const DESK_STREAM_ACTIONS = new Set(['start', 'stop', 'status']);
+ async function deskStream(subject, machine, args, { signal } = {}) {
+ const bad = guard(subject, machine);
+ if (bad) return bad;
+ if (SESSION_MACHINE.test(String(machine || ''))) return { error: 'a Run desk has no Link stream: use its still' };
+ const a = args && typeof args === 'object' ? args : {};
+ if (!DESK_STREAM_ACTIONS.has(a.action)) return { error: 'action is start, stop or status' };
+ const value = { action: a.action };
+ if (a.desk != null) value.desk = String(a.desk).slice(0, 40);
+ if (a.action === 'start') {
+ if (a.title != null) value.title = String(a.title).replace(/\s+/g, ' ').trim().slice(0, 120);
+ if (a.task && typeof a.task === 'object') value.task = { source: ['codes', 'actor', 'desk', 'node'].includes(a.task.source) ? a.task.source : 'actor', ref: String(a.task.ref || '').slice(0, 100), title: String(a.task.title || '').slice(0, 120), ...(typeof a.task.url === 'string' && /^https:\/\//.test(a.task.url) ? { url: a.task.url.slice(0, 300) } : {}) };
+ value.visibility = ['private', 'unlisted', 'public'].includes(a.visibility) ? a.visibility : 'private';
+ if (Array.isArray(a.allow_sites)) value.allow_sites = a.allow_sites.filter((u) => typeof u === 'string' && /^https:\/\//.test(u)).slice(0, 20);
+ }
+ const r = await command(subject, machine, 'desk.stream', value, { signal, windowMs: a.action === 'status' ? READ_WINDOW_MS : 120_000 });
if (r.error) return r;
if (r.error) return r;
- return { done: !!r.data.done, action: r.data.action || a.action, ...(r.data.x != null ? { x: r.data.x, y: r.data.y } : {}) };
+ const d = r.data || {};
+ if (a.action === 'stop') return { stopped: !!d.stopped, ...(d.id ? { id: d.id } : {}) };
+ if (a.action === 'status') return { streaming: !!d.streaming, ...(d.id ? { id: d.id } : {}), ...(d.page_url ? { page_url: d.page_url } : {}) };
+ if (!d.id || !/^ags_[0-9A-HJKMNP-TV-Z]{26}$/.test(String(d.id))) return { error: 'the machine did not start a stream' };
+ return { streaming: !!d.streaming, id: d.id, page_url: d.page_url || null, visibility: d.visibility || 'private', expires_at: d.expires_at || null, capture: d.capture || null };
}
}
/**
/**
@@ -476,7 +535,7 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensO
@@ -476,7 +535,7 @@ function createComputers({ config, fetchImpl = globalThis.fetch, tokens: tokensO
return { url: data.url, expires_at: data.expires_at || null, control: !!data.control, desk: data.desk || desk };
return { url: data.url, expires_at: data.expires_at || null, control: !!data.control, desk: data.desk || desk };
}
}
- return { enabled, mine, list, read, run, write, screen, input, browser, desk, deskTurn, watchUrl, watchLink, sessionList, profileSites, personSession, openSession, endSession, fillLogin, still, sessionsOn, isSession: (m) => SESSION_MACHINE.test(String(m || '')) };
+ return { enabled, mine, list, read, run, write, screen, input, personInput, browser, desk, deskStream, deskTurn, watchUrl, watchLink, sessionList, profileSites, personSession, openSession, endSession, fillLogin, still, sessionsOn, isSession: (m) => SESSION_MACHINE.test(String(m || '')) };
}
}
module.exports = { createComputers };
module.exports = { createComputers };
new file mode 100644
new file mode 100644
@@ -0,0 +1,49 @@
@@ -0,0 +1,49 @@
+'use strict';
+
+/**
+ * OpenVibe.Live, the small part Actor needs for "See it work" (Agent Streams S1): a view ticket for an agent stream,
+ * and the embed address that carries it. The desk publishes the stream (Link's desk.stream creates it with the machine's
+ * own node token); Actor only mints the person's ten-minute view ticket with the person's own token, the way Live's docs
+ * say a first-party page does, and puts it in `/embed/agent/<ags_id>?vt=…`.
+ *
+ * Actor never keeps the ticket: it is returned to the page request that asked, lives ten minutes, and renews there.
+ * Without a person token there is no ticket, and the page falls back to the still.
+ */
+const STREAM_RE = /^ags_[0-9A-HJKMNP-TV-Z]{26}$/;
+
+function createLive({ config, fetchImpl = globalThis.fetch } = {}) {
+ const base = config && config.services && config.services.liveUrl;
+ const publicBase = ((config && config.services && config.services.livePublicUrl) || base || '').replace(/\/+$/, '');
+ const enabled = !!base;
+
+ async function viewTicket(token, streamId, { signal } = {}) {
+ if (!enabled) return { error: 'OpenVibe.Live is not connected here' };
+ if (!token) return { error: 'a view ticket needs the person\'s own session' };
+ if (!STREAM_RE.test(String(streamId || ''))) return { error: 'not an agent stream' };
+ const sig = signal ? AbortSignal.any([signal, AbortSignal.timeout(8000)]) : AbortSignal.timeout(8000);
+ let res;
+ try {
+ res = await fetchImpl(`${base}/api/streams/agent/${encodeURIComponent(streamId)}/view-ticket`, {
+ method: 'POST',
+ headers: { Accept: 'application/json', 'Content-Type': 'application/json', Authorization: `Bearer ${token}` },
+ body: '{}', signal: sig,
+ });
+ } catch {
+ return { error: 'OpenVibe.Live did not answer' };
+ }
+ const data = await res.json().catch(() => null);
+ if (!res.ok || !data || !data.view_ticket) {
+ return { error: (data && (data.detail || data.error)) || `OpenVibe.Live answered ${res.status}`, status: res.status || 502 };
+ }
+ return {
+ stream_id: streamId,
+ view_ticket: data.view_ticket,
+ expires_at: data.expires_at || null,
+ page_url: `${publicBase}/embed/agent/${encodeURIComponent(streamId)}?vt=${encodeURIComponent(data.view_ticket)}`,
+ };
+ }
+
+ return { enabled, viewTicket, base, publicBase };
+}
+
+module.exports = { createLive, STREAM_RE };
@@ -23,6 +23,7 @@ const { createKeyStore } = require('./auth/keys');
@@ -23,6 +23,7 @@ const { createKeyStore } = require('./auth/keys');
const { createSso } = require('./auth/sso');
const { createSso } = require('./auth/sso');
const catalog = require('./agents/catalog');
const catalog = require('./agents/catalog');
const { createAdapters } = require('./agents/adapters');
const { createAdapters } = require('./agents/adapters');
+const { createLive } = require('./agents/live');
const { createModels } = require('./runtime/models');
const { createModels } = require('./runtime/models');
const createAiCatalogConfig = require('./ai-catalog-config').create;
const createAiCatalogConfig = require('./ai-catalog-config').create;
const { createRegistry } = require('./runtime/tools');
const { createRegistry } = require('./runtime/tools');
@@ -163,9 +164,11 @@ async function createApp(opts = {}) {
@@ -163,9 +164,11 @@ async function createApp(opts = {}) {
const computers = (opts.bot && opts.bot.computers) || require('./agents/computers').createComputers({ config, fetchImpl: providerFetch, log });
const computers = (opts.bot && opts.bot.computers) || require('./agents/computers').createComputers({ config, fetchImpl: providerFetch, log });
// An agent's money (plan HO7): the client that pays a tip through OpenVibe.Tips with the person's own token.
// An agent's money (plan HO7): the client that pays a tip through OpenVibe.Tips with the person's own token.
const tips = require('./agents/spend').createTips({ config, fetchImpl });
const tips = require('./agents/spend').createTips({ config, fetchImpl });
+ // OpenVibe.Live: a view ticket for an agent desk's private stream, minted with the person's own token.
+ const live = opts.live || createLive({ config, fetchImpl });
// An agent answers the mail sent to its own address (track HO6): the consumer below takes the event, the engine
// An agent answers the mail sent to its own address (track HO6): the consumer below takes the event, the engine
// reports each run's end back here, and the approvals decide() asks it to send an approved draft.
// reports each run's end back here, and the approvals decide() asks it to send an approved draft.
- const ctx = { config, s, keys, aiCatalogConfig, sso, stream, adapters, engine, webhooks, principal, projects, log, chatAsks, vibeDelivery, models, registry, loop, notifier, stats, checkpoints, connectors, waits, sharing, computers, triggers, skillRun, media, tips };
+ const ctx = { config, s, keys, aiCatalogConfig, sso, stream, adapters, engine, webhooks, principal, projects, log, chatAsks, vibeDelivery, models, registry, loop, notifier, stats, checkpoints, connectors, waits, sharing, computers, live, triggers, skillRun, media, tips };
const fabricMail = require('./mail/client').createMailClient({ config, fetchImpl: opts.mailFetch || fetchImpl, tokens: opts.mailTokens || null });
const fabricMail = require('./mail/client').createMailClient({ config, fetchImpl: opts.mailFetch || fetchImpl, tokens: opts.mailTokens || null });
ctx.fabric = require('./fabric/service').createFabric({ s, enabled: !!config.fabric?.enabled, readMail: async (args) => {
ctx.fabric = require('./fabric/service').createFabric({ s, enabled: !!config.fabric?.enabled, readMail: async (args) => {
const out = await fabricMail.getMessage(args);
const out = await fabricMail.getMessage(args);
@@ -247,7 +250,7 @@ async function createApp(opts = {}) {
@@ -247,7 +250,7 @@ async function createApp(opts = {}) {
// The navbar's Messages count asks Chat (openvibe-shared 4.130.0).
// The navbar's Messages count asks Chat (openvibe-shared 4.130.0).
connectSrc: ["'self'", 'https://openvibe.chat', 'https://openvibe.network', 'https://cloudflareinsights.com', 'https://openvibe.events'],
connectSrc: ["'self'", 'https://openvibe.chat', 'https://openvibe.network', 'https://cloudflareinsights.com', 'https://openvibe.events'],
// Bot's watch links: a chat turn frames the desk the agent works on, live (plan L55 C1b).
// Bot's watch links: a chat turn frames the desk the agent works on, live (plan L55 C1b).
- frameSrc: ["'self'", 'https://openvibe.network', ...(() => { try { return [new URL(config.services.botPublicUrl).origin]; } catch { return []; } })()],
+ frameSrc: ["'self'", 'https://openvibe.network', ...(() => { try { return [new URL(config.services.botPublicUrl).origin]; } catch { return []; } })(), ...(() => { try { return [new URL(config.services.livePublicUrl).origin]; } catch { return []; } })()],
frameAncestors: ["'none'"],
frameAncestors: ["'none'"],
objectSrc: ["'none'"],
objectSrc: ["'none'"],
baseUri: ["'self'"],
baseUri: ["'self'"],
@@ -177,6 +177,10 @@ function load(env = process.env) {
@@ -177,6 +177,10 @@ function load(env = process.env) {
// OpenVibe.Run: a browser session on a desk host when none of the person's own machines has a desk for the
// OpenVibe.Run: a browser session on a desk host when none of the person's own machines has a desk for the
// task (Desks plan D6). It is also where a skill's code runs (ADR-060). Empty turns it off.
// task (Desks plan D6). It is also where a skill's code runs (ADR-060). Empty turns it off.
runUrl: trim(env.ACTOR_RUN_URL != null ? env.ACTOR_RUN_URL : (selfHosted ? '' : 'http://127.0.0.1:4920')),
runUrl: trim(env.ACTOR_RUN_URL != null ? env.ACTOR_RUN_URL : (selfHosted ? '' : 'http://127.0.0.1:4920')),
+ // OpenVibe.Live: an agent desk's private stream and its embed (Agent Streams S1). Empty turns the live
+ // tile off (the still fallback stays). The person's own token mints the view ticket; Actor never stores it.
+ liveUrl: trim(env.ACTOR_LIVE_URL != null ? env.ACTOR_LIVE_URL : 'https://openvibe.live'),
+ livePublicUrl: trim(env.ACTOR_LIVE_PUBLIC_URL != null ? env.ACTOR_LIVE_PUBLIC_URL : 'https://openvibe.live'),
// OpenVibe.Media: where a sandbox skill's code bytes are stored, private to Actor's tenant (ADR-060).
// OpenVibe.Media: where a sandbox skill's code bytes are stored, private to Actor's tenant (ADR-060).
// Empty means a sandbox skill cannot be published (a prompt skill is unaffected).
// Empty means a sandbox skill cannot be published (a prompt skill is unaffected).
mediaUrl: trim(env.ACTOR_MEDIA_URL != null ? env.ACTOR_MEDIA_URL : (selfHosted ? '' : 'http://127.0.0.1:4100')),
mediaUrl: trim(env.ACTOR_MEDIA_URL != null ? env.ACTOR_MEDIA_URL : (selfHosted ? '' : 'http://127.0.0.1:4100')),
@@ -46,6 +46,8 @@ const timeline = require('../agents/timeline');
@@ -46,6 +46,8 @@ const timeline = require('../agents/timeline');
const assignments = require('../agents/assignments');
const assignments = require('../agents/assignments');
const modelPrefs = require('../runtime/model-prefs');
const modelPrefs = require('../runtime/model-prefs');
const driving = require('../agents/driving');
const driving = require('../agents/driving');
+const stills = require('../tasks/stills');
+const replay = require('../tasks/replay');
const canvases = require('../canvases');
const canvases = require('../canvases');
const skills = require('../skills');
const skills = require('../skills');
@@ -68,6 +70,13 @@ function createApi(ctx) {
@@ -68,6 +70,13 @@ function createApi(ctx) {
};
};
const triggerFail = (req, res, out) => problem(req, res, (out.error && out.error.status) || 502, (out.error && out.error.code) || 'actor.trigger.failed', (out.error && out.error.detail) || 'The trigger could not be saved.');
const triggerFail = (req, res, out) => problem(req, res, (out.error && out.error.status) || 502, (out.error && out.error.code) || 'actor.trigger.failed', (out.error && out.error.detail) || 'The trigger could not be saved.');
const triggerOwner = (req) => String(req.principal.requester || '').replace(/^user:/, '');
const triggerOwner = (req) => String(req.principal.requester || '').replace(/^user:/, '');
+ // The person's own token, for the calls only they may make: Live's view ticket and Bot's own input route.
+ const personToken = (req) => {
+ if (!req.principal || req.principal.kind !== 'user') return null;
+ const h = String(req.headers.authorization || '');
+ if (h.startsWith('Bearer ')) return h.slice(7).trim();
+ return req.token || (req.viewer && req.viewer.token) || null;
+ };
r.use(express.json({ limit: '64kb' }));
r.use(express.json({ limit: '64kb' }));
r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });
r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });
@@ -720,10 +729,141 @@ function createApi(ctx) {
@@ -720,10 +729,141 @@ function createApi(ctx) {
} catch { ctx.log.warn('[Actor] task view issue failed', req.params.id); res.sendStatus(503); }
} catch { ctx.log.warn('[Actor] task view issue failed', req.params.id); res.sendStatus(503); }
});
});
- // Take over / Give it back (plan 2026-11-07), built on the hand_to_person machinery: the person drives the desk
- // they were handed, the agent's input is refused while they do, and the task's timeline gets one hand-off step.
- // Only the task's own people (load()), and only while it runs on a Computer.
- async function handoff(req, res, { take }) {
+ // A still is its task's person's, and nobody else's: a signed-out visitor and an app token get the same 404 as a
+ // task that does not exist (never a 401 that would confirm one).
+ const stillPerson = (req, res, next) => ((req.principal && req.principal.kind === 'user') ? next() : problem(req, res, 404, 'actor.task.not_found', 'No such task.'));
+
+ // ── See it work (W1): the task's desk, its private Live stream, the person's control, and the replay ─────────
+ // One description of where a task's desk is, from its own steps: a Link desk on the person's machine, or a Run
+ // session on a desk host. Nothing here trusts the model; the desk and the Computer are read from tool events.
+ async function deskTarget(row) {
+ const computer = await store.taskComputer(s, row);
+ const desk = await store.taskDesk(s, row);
+ if (computer && computer.session_id) {
+ return { id: `run:${computer.session_id}`, where: 'run', session_id: computer.session_id,
+ machine: null, desk: null, kind: 'browser', computer, label: computer.where === 'yours' ? `${computer.machine || 'Your computer'} · a desk host` : 'OpenVibe computer' };
+ }
+ const place = desk && store.deskPlace(desk.url);
+ if (!place) return null;
+ return { id: `link:${place.machine}:${place.desk}`, where: 'link', session_id: null, machine: place.machine,
+ desk: place.desk, kind: desk.kind || 'desktop', computer, label: `${(computer && computer.machine) || 'Your PC'} · ${place.desk === 'main' ? 'your screen' : 'its own desk'}` };
+ }
+ async function browserUrlOf(row) {
+ const events = await store.eventsAfter(s, row.id, 0);
+ for (let i = events.length - 1; i >= 0; i--) {
+ const e = events[i];
+ if (e && e.kind === 'output' && e.step === 'tool_result' && e.tool === 'computer_browser' && !e.is_error) {
+ try { const d = JSON.parse(e.text || '{}'); if (d && typeof d.url === 'string' && /^https?:\/\//.test(d.url)) return d.url.slice(0, 500); } catch { /* not JSON */ }
+ }
+ }
+ return null;
+ }
+ async function deskView(row, req, target) {
+ const token = personToken(req);
+ const desk = await store.taskDesk(s, row);
+ let live = null;
+ if (target.where === 'link' && desk && desk.stream && desk.stream.id && token && ctx.live && ctx.live.enabled) {
+ const view = await ctx.live.viewTicket(token, desk.stream.id).catch(() => null);
+ // The embed must be Live's own, never an address another service convinced it to return.
+ let embed = null;
+ try {
+ const u = new URL(String(view && view.page_url || ''));
+ const expected = new URL(String(ctx.live.publicBase || 'https://openvibe.live'));
+ if (/^https?:$/.test(u.protocol) && u.origin === expected.origin) embed = u.href;
+ } catch { embed = null; }
+ if (view && !view.error && embed) {
+ live = { stream_id: desk.stream.id, embed_url: embed, page_url: embed, visibility: desk.stream.visibility || 'private', expires_at: view.expires_at || null };
+ }
+ }
+ const pictures = await stills.list(s, row.id).catch(() => []);
+ const last = pictures.length ? pictures[pictures.length - 1] : null;
+ return {
+ id: target.id, where: target.where, machine: target.machine, desk: target.desk, kind: target.kind,
+ label: target.label, browser_url: target.where === 'link' ? await browserUrlOf(row) : null,
+ width: (last && last.width) || 1280, height: (last && last.height) || 800,
+ live,
+ still: target.where === 'run' ? `${config.baseUrl}/api/v1/tasks/${row.id}/still` : `${config.baseUrl}/api/v1/tasks/${row.id}/desk-still?desk=${encodeURIComponent(target.desk || 'private')}`,
+ driving: driving.is(row.id, s.now()),
+ };
+ }
+ r.get('/tasks/:id/desks', principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {
+ const row = await load(req, res);
+ if (!row) return;
+ const target = await deskTarget(row);
+ res.json({ desks: target ? [await deskView(row, req, target)] : [] });
+ });
+
+ // The still fallback: Link's screen through Bot (at most once a second, served from a one-frame cache), or Run's
+ // session still. Only the task's own person; a picture must never confirm a task or a desk exists.
+ const stillFrame = new Map(); // task id -> { at, type, bytes }
+ r.get('/tasks/:id/desk-still', stillPerson, principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {
+ const row = await load(req, res);
+ if (!row) return;
+ const target = await deskTarget(row);
+ if (!target) return problem(req, res, 404, 'actor.task.no_still', 'This task has no desk picture.');
+ if (target.where === 'run') return problem(req, res, 404, 'actor.task.no_still', 'Use the session still.');
+ const key = `${row.id}:${target.desk}`;
+ const kept = stillFrame.get(key);
+ const at = Date.now();
+ if (kept && at - kept.at < 1000) {
+ res.set('Cache-Control', 'private, no-store'); res.set('X-Content-Type-Options', 'nosniff'); res.type(kept.type).send(kept.bytes); return;
+ }
+ const shot = await ctx.computers.screen(row.requester.slice(5), target.machine, { desk: target.desk, maxWidth: 640 });
+ if (!shot || shot.error) return problem(req, res, 502, 'actor.task.no_still', (shot && shot.error) || 'The desk sent no picture.');
+ const type = shot.format === 'png' ? 'image/png' : 'image/jpeg';
+ const bytes = Buffer.from(String(shot.content_b64 || ''), 'base64');
+ if (!bytes.length) return problem(req, res, 502, 'actor.task.no_still', 'The desk sent no picture.');
+ stillFrame.set(key, { at, type, bytes });
+ while (stillFrame.size > 500) stillFrame.delete(stillFrame.keys().next().value);
+ res.set('Cache-Control', 'private, no-store'); res.set('X-Content-Type-Options', 'nosniff'); res.type(type).send(bytes);
+ });
+
+ // The person's own input on the picture (Link: their token at Bot; Run: the session's person-commands route).
+ r.post('/tasks/:id/desks/:desk/input', principal.requireCapability('actor.task.create'), async (req, res) => {
+ const row = await load(req, res);
+ if (!row) return;
+ if (!store.OPEN.includes(row.state)) return problem(req, res, 409, 'actor.task.ended', 'This task has ended: its desk is gone.');
+ if (!driving.is(row.id, s.now())) return problem(req, res, 409, 'actor.task.not_driving', 'Take over the desk before using it.');
+ const target = await deskTarget(row);
+ if (!target || target.id !== String(req.params.desk || '')) return problem(req, res, 404, 'actor.task.no_computer', 'This task has no such desk.');
+ const subject = row.requester.slice(5);
+ const body = req.body && typeof req.body === 'object' ? req.body : {};
+ let out;
+ if (target.where === 'run') out = await ctx.computers.personSession(subject, target.session_id, 'input', { kind: 'input', value: body }).catch((err) => ({ error: (err && err.message) || 'the session refused it' }));
+ else {
+ const token = personToken(req);
+ if (!token) return problem(req, res, 401, 'actor.task.sign_in', 'Sign in again: your own session is needed to use the desk.');
+ out = await ctx.computers.personInput(subject, token, target.machine, { ...body, desk: target.desk });
+ }
+ if (!out || out.error) return problem(req, res, 502, 'actor.task.input_failed', (out && out.error) || 'The desk did not take that.');
+ return res.json({ done: true, action: out.action || body.action || null });
+ });
+
+ r.get('/tasks/:id/replay', principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {
+ const row = await load(req, res);
+ if (!row) return;
+ res.json({ task_id: row.id, steps: await replay.steps(s, row, { baseUrl: config.baseUrl }) });
+ });
+ r.get('/tasks/:id/replay/:seq/still', stillPerson, principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {
+ const row = await load(req, res);
+ if (!row) return;
+ const seq = Number.parseInt(req.params.seq, 10);
+ if (!Number.isInteger(seq) || seq < 1) return problem(req, res, 404, 'actor.task.no_still', 'No such picture.');
+ const shot = await stills.get(s, row.id, seq);
+ if (!shot || !shot.bytes) return problem(req, res, 404, 'actor.task.no_still', 'No such picture.');
+ const bytes = Buffer.isBuffer(shot.bytes) ? shot.bytes : Buffer.from(shot.bytes);
+ res.set('Cache-Control', 'private, no-store');
+ res.set('X-Content-Type-Options', 'nosniff');
+ res.set('Content-Type', shot.mime);
+ res.set('Content-Length', String(bytes.length));
+ return res.status(200).end(bytes);
+ });
+
+ // Take over / Give it back (plan 2026-11-07; See it work, W1): the person drives the desk they were handed, the
+ // agent's input is refused while they do, and the task's timeline gets one hand-off step. A Run desk's lease is
+ // really taken and given back; a Link desk's input is the person's own (their token at Bot). A note handed back is
+ // the person's own message to the run.
+ async function handoff(req, res, { take, deskId = null, note = null }) {
const row = await load(req, res);
const row = await load(req, res);
if (!row) return;
if (!row) return;
if (!store.OPEN.includes(row.state)) return problem(req, res, 409, 'actor.task.ended', 'This task has ended: its Computer is gone.');
if (!store.OPEN.includes(row.state)) return problem(req, res, 409, 'actor.task.ended', 'This task has ended: its Computer is gone.');
@@ -731,16 +871,26 @@ function createApi(ctx) {
@@ -731,16 +871,26 @@ function createApi(ctx) {
const desk = await store.taskDesk(s, row);
const desk = await store.taskDesk(s, row);
if (!computer && !desk) return problem(req, res, 409, 'actor.task.no_computer', 'This task has no Computer to hand over.');
if (!computer && !desk) return problem(req, res, 409, 'actor.task.no_computer', 'This task has no Computer to hand over.');
if (!String(row.requester).startsWith('user:')) return problem(req, res, 409, 'actor.task.no_person', 'A Computer is a person\'s: sign in as the task\'s own person to take it.');
if (!String(row.requester).startsWith('user:')) return problem(req, res, 409, 'actor.task.no_person', 'A Computer is a person\'s: sign in as the task\'s own person to take it.');
+ const target = await deskTarget(row);
+ if (deskId && (!target || target.id !== deskId)) return problem(req, res, 404, 'actor.task.no_computer', 'This task has no such desk.');
+ const subject = row.requester.slice(5);
+ if (target && target.where === 'run' && ctx.computers.personSession) {
+ const lease = take
+ ? await ctx.computers.personSession(subject, target.session_id, 'take').catch((err) => ({ error: err.message }))
+ : await ctx.computers.personSession(subject, target.session_id, 'back').catch((err) => ({ error: err.message }));
+ if (lease && lease.error) return problem(req, res, 502, 'actor.session.control_failed', lease.error);
+ }
const at = s.iso();
const at = s.iso();
const now = take ? (driving.take(row.id, s.now()), true) : (driving.give(row.id), false);
const now = take ? (driving.take(row.id, s.now()), true) : (driving.give(row.id), false);
- // The step in the task's own timeline (the same events /events streams), carrying the Computer.
+ if (!take && typeof note === 'string' && note.trim()) {
+ await engine.steer(row.id, note.trim().slice(0, 1000)).catch(() => null);
+ }
const event = await store.appendEvent(s, row.id, {
const event = await store.appendEvent(s, row.id, {
at, kind: 'output', step: 'handoff',
at, kind: 'output', step: 'handoff',
text: now ? 'You took over the desk. The agent waits until you give it back.' : 'You gave the desk back. The agent goes on.',
text: now ? 'You took over the desk. The agent waits until you give it back.' : 'You gave the desk back. The agent goes on.',
...(computer ? { computer } : {}),
...(computer ? { computer } : {}),
});
});
stream.publish(row.id, event);
stream.publish(row.id, event);
- // And one step in the one timeline behind the Dock (agent.timeline-entry@1), the same Computer.
try {
try {
await timeline.upsert(s, row.requester, {
await timeline.upsert(s, row.requester, {
step_id: `${row.id}:takeover`, mode: 'browse', product: 'actor', ref: row.id,
step_id: `${row.id}:takeover`, mode: 'browse', product: 'actor', ref: row.id,
@@ -756,14 +906,15 @@ function createApi(ctx) {
@@ -756,14 +906,15 @@ function createApi(ctx) {
res.json(out);
res.json(out);
}
}
r.post('/tasks/:id/takeover', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: true }));
r.post('/tasks/:id/takeover', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: true }));
- r.post('/tasks/:id/giveback', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: false }));
+ r.post('/tasks/:id/giveback', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: false, note: req.body && typeof req.body.note === 'string' ? req.body.note : null }));
+ r.post('/tasks/:id/desks/:desk/take-over', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: true, deskId: String(req.params.desk || '') }));
+ r.post('/tasks/:id/desks/:desk/hand-back', principal.requireCapability('actor.task.create'), async (req, res) => handoff(req, res, { take: false, deskId: String(req.params.desk || ''), note: req.body && typeof req.body.note === 'string' ? req.body.note : null }));
// The Computer's newest frame, fetched from the session's own service (Run) as Actor, acting for the task's
// The Computer's newest frame, fetched from the session's own service (Run) as Actor, acting for the task's
// person: the person's own token would have to cross an origin, and the picture is theirs. Owner/project only
// person: the person's own token would have to cross an origin, and the picture is theirs. Owner/project only
// (load()); a task with no Run session has no picture, and neither has anyone asking for another person's task.
// (load()); a task with no Run session has no picture, and neither has anyone asking for another person's task.
// A still is its task's person's, and nobody else's: a signed-out visitor and an app token get the same 404 as a
// A still is its task's person's, and nobody else's: a signed-out visitor and an app token get the same 404 as a
// task that does not exist (never a 401 that would confirm one).
// task that does not exist (never a 401 that would confirm one).
- const stillPerson = (req, res, next) => ((req.principal && req.principal.kind === 'user') ? next() : problem(req, res, 404, 'actor.task.not_found', 'No such task.'));
r.get('/tasks/:id/still', stillPerson, principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {
r.get('/tasks/:id/still', stillPerson, principal.requireCapability('actor.task.read'), limits.reads('actor.task.read'), async (req, res) => {
const row = await load(req, res);
const row = await load(req, res);
if (!row) return;
if (!row) return;
@@ -34,6 +34,7 @@ const inputs = require('../tasks/inputs');
@@ -34,6 +34,7 @@ const inputs = require('../tasks/inputs');
const modelPrefs = require('../runtime/model-prefs');
const modelPrefs = require('../runtime/model-prefs');
const aiCatalog = require('../runtime/ai-catalog');
const aiCatalog = require('../runtime/ai-catalog');
const canvases = require('../canvases');
const canvases = require('../canvases');
+const replay = require('../tasks/replay');
// The model a person's tasks run on (plan L55 C3; the owner, 11-01: "use your own API keys, or OpenVibe's"). Auto is
// The model a person's tasks run on (plan L55 C3; the owner, 11-01: "use your own API keys, or OpenVibe's"). Auto is
// OpenVibe's own; "own:<provider>" is the person's key, sealed in OpenVibe.AI, which Actor never sees.
// OpenVibe's own; "own:<provider>" is the person's key, sealed in OpenVibe.AI, which Actor never sees.
@@ -240,6 +241,131 @@ ${from ? html`<input type="hidden" name="from" value="${from}">` : ''}<label for
@@ -240,6 +241,131 @@ ${from ? html`<input type="hidden" name="from" value="${from}">` : ''}<label for
return raw(thread.work({ open, live: LIVE_TEXT[row.state] || 'Working', took: tookOf(row) || null, rows: workRows(lines) }));
return raw(thread.work({ open, live: LIVE_TEXT[row.state] || 'Working', took: tookOf(row) || null, rows: workRows(lines) }));
}
}
+ /**
+ * Where a running task's desk is (See it work, W1): Link's desk on the person's machine or a Run session on a desk
+ * host, read from the task's own tool results. → { where, id, machine, desk, kind, label, still } or null.
+ */
+ function deskContext(events) {
+ const desk = store.deskFrom(events);
+ const computer = store.computerFrom(events);
+ if (computer && computer.session_id) {
+ return { where: 'run', id: `run:${computer.session_id}`, session_id: computer.session_id, machine: null, desk: null,
+ kind: 'browser', label: computer.where === 'yours' ? `${computer.machine || 'Your computer'} · a desk host` : 'OpenVibe computer',
+ still: `/api/v1/tasks/__TASK__/still` };
+ }
+ const place = desk && store.deskPlace(desk.url);
+ // The person's own screen is never framed: an agent's desk is private or desk-<name>.
+ if (!place || place.desk === 'main') return null;
+ return { where: 'link', id: `link:${place.machine}:${place.desk}`, machine: place.machine, desk: place.desk,
+ kind: desk.kind || 'desktop', label: `${(computer && computer.machine) || 'Your PC'} · ${place.desk === 'main' ? 'your screen' : 'its own desk'}`,
+ still: `/api/v1/tasks/__TASK__/desk-still?desk=${encodeURIComponent(place.desk)}` };
+ }
+
+ /** The latest browser address the desk's browser reported (a page address, never model text). */
+ function browserUrlOf(events) {
+ for (let i = events.length - 1; i >= 0; i--) {
+ const e = events[i];
+ if (!e || e.kind !== 'output' || e.step !== 'tool_result' || e.tool !== 'computer_browser' || e.is_error) continue;
+ try { const d = JSON.parse(e.text || '{}'); if (d && typeof d.url === 'string' && /^https?:\/\//.test(d.url)) return d.url.slice(0, 500); } catch { /* not JSON */ }
+ }
+ return null;
+ }
+
+ /** The last click or move, for the ring: x,y in the screen picture's pixels, and that picture's size. */
+ function lastClick(events) {
+ for (let i = events.length - 1; i >= 0; i--) {
+ const e = events[i];
+ if (!e || e.kind !== 'output' || e.step !== 'tool_call' || e.tool !== 'computer_input') continue;
+ const a = e.args && typeof e.args === 'object' ? e.args : {};
+ if (!['click', 'double_click', 'right_click', 'move'].includes(String(a.action))) continue;
+ if (!Number.isFinite(a.x) || !Number.isFinite(a.y)) continue;
+ return { x: Number(a.x), y: Number(a.y), width: Number(a.width) || 1280, height: Number(a.height) || 800 };
+ }
+ return null;
+ }
+
+ /** The latest computer step in plain words, for the tile's one caption. Never the typed text. */
+ function deskCaption(events) {
+ for (let i = events.length - 1; i >= 0; i--) {
+ const e = events[i];
+ if (!e || e.kind !== 'output' || e.step !== 'tool_call') continue;
+ const a = e.args && typeof e.args === 'object' ? e.args : {};
+ if (e.tool === 'computer_input') {
+ if (a.action === 'type') return 'Typing';
+ if (a.action === 'click' || a.action === 'double_click' || a.action === 'right_click') return `Clicking at ${Math.round(Number(a.x) || 0)}, ${Math.round(Number(a.y) || 0)}`;
+ if (a.action === 'scroll') return 'Scrolling';
+ if (a.action === 'key') return 'Pressing a key';
+ if (a.action === 'move') return 'Moving the mouse';
+ return 'Using the computer';
+ }
+ if (e.tool === 'computer_browser') {
+ if (a.action === 'open') return 'Opening a page';
+ if (a.action === 'read') return 'Reading the page';
+ if (a.action === 'click') return 'Clicking on the page';
+ if (a.action === 'fill') return 'Typing in a field';
+ if (a.action === 'press') return 'Pressing a key';
+ if (a.action === 'back') return 'Going back';
+ return 'Using the browser';
+ }
+ if (e.tool === 'computer_screen') return 'Looking at the screen';
+ if (e.tool === 'computer_desk') return a.action === 'start' ? 'Starting its desk' : a.action === 'stop' ? 'Stopping its desk' : 'Checking its desk';
+ }
+ return 'Getting ready';
+ }
+
+ /** A step waiting for the person (hand_to_person) becomes an Ask whose button opens the viewer in control. */
+ function pendingAsk(events) {
+ // The public body keeps the tool but not call_id; a hand_to_person stays open until its result arrives.
+ let open = null;
+ for (const e of events) {
+ if (!e || e.kind !== 'output') continue;
+ if (e.step === 'tool_call' && e.tool === 'hand_to_person') open = String((e.args && e.args.what) || 'Actor needs you on its desk').slice(0, 300);
+ if (e.step === 'tool_result' && e.tool === 'hand_to_person') open = null;
+ }
+ return open;
+ }
+
+ function deskTile(row, events, readOnly = false) {
+ const d = deskContext(events);
+ if (!d || readOnly) return '';
+ const still = d.still.replace('__TASK__', row.id);
+ const click = lastClick(events);
+ const ring = click ? (() => {
+ const left = Math.max(0, Math.min(100, (click.x / Math.max(1, click.width)) * 100));
+ const top = Math.max(0, Math.min(100, (click.y / Math.max(1, click.height)) * 100));
+ return `<span class="ct-desk-ring" data-ct-desk-ring aria-hidden="true" style="left:${left.toFixed(2)}%;top:${top.toFixed(2)}%"></span>`;
+ })() : '';
+ const page = browserUrlOf(events);
+ const ask = pendingAsk(events);
+ return html`<figure class="ct-desk-tile" data-ct-desk data-task="${row.id}" data-desk="${d.id}" data-where="${d.where}" data-kind="${d.kind}" data-label="${d.label}" data-still="${still}">
+<div class="ct-desk-frame" data-ct-desk-frame>
+<img data-ct-desk-still src="${still}" alt="Actor's desk" loading="lazy" decoding="async">
+<span class="ct-desk-live" data-ct-desk-live hidden>Live</span>
+<button type="button" class="ct-desk-open" data-ct-desk-open aria-label="Open the desk viewer" title="Open the desk viewer"><span>Watch</span></button>
+${raw(ring)}
+</div>
+<figcaption><span data-ct-desk-caption>${deskCaption(events)}</span><span class="ct-desk-place">${d.label}</span>${page ? html`<a data-ct-desk-url href="${page}" target="_blank" rel="noopener nofollow" hidden>${String(page).replace(/^https?:\/\//, '').slice(0, 80)}</a>` : ''}</figcaption>
+${ask ? html`<div class="ct-desk-ask" data-ct-desk-ask><p><b>Actor needs you:</b> ${ask}</p><button type="button" class="ct-desk-take" data-ct-takeover>Take over</button></div>` : ''}
+</figure>`;
+ }
+
+ /** A finished task's replay: the steps with stills, and the first picture until the person scrubs. */
+ async function replayBlock(row) {
+ const steps = await replay.steps(s, row);
+ if (!steps.length) return '';
+ const first = steps.find((x) => x.still) || null;
+ const json = raw(JSON.stringify(steps).replace(/</g, '\\u003c').replace(/&/g, '\\u0026'));
+ return html`<details class="ct-replay" data-ct-replay data-task="${row.id}">
+<summary>Replay · ${String(steps.length)} step${steps.length === 1 ? '' : 's'}</summary>
+<div class="ct-replay-body">
+<input type="range" min="0" max="${String(Math.max(0, steps.length - 1))}" value="0" step="1" data-ct-replay-range aria-label="Scrub the replay">
+<figure class="ct-replay-shot">${first ? html`<img data-ct-replay-img src="${first.still.url}" alt="What the step saw" loading="lazy">` : html`<span class="ct-replay-none">No pictures were kept for this run.</span>`}
+<figcaption><span data-ct-replay-caption>${replay.caption(steps[0])}</span>${first ? html`<a data-ct-replay-link href="${first.still.url}" target="_blank" rel="noopener">Open the picture</a>` : ''}</figcaption></figure>
+<ol class="ct-replay-steps">${steps.map((st, i) => html`<li><button type="button" data-ct-replay-step="${String(i)}"><span>${String(i + 1)}</span>${replay.caption(st)}${st.still ? '' : ' · no picture'}</button></li>`)}</ol>
+<script type="application/json" data-ct-replay-data>${json}</script>
+</div></details>`;
+ }
+
/**
/**
* A turn that is running is a card in the conversation (one-agent system, stage 1: the owner, 11-12, "a more
* A turn that is running is a card in the conversation (one-agent system, stage 1: the owner, 11-12, "a more
* modular system… reused in a shared modular way"), not a separate console panel: its title, its state, the last
* modular system… reused in a shared modular way"), not a separate console panel: its title, its state, the last
@@ -262,6 +388,7 @@ ${from ? html`<input type="hidden" name="from" value="${from}">` : ''}<label for
@@ -262,6 +388,7 @@ ${from ? html`<input type="hidden" name="from" value="${from}">` : ''}<label for
return `<li class="ovt-w-step"><span>${esc(clip(r.text, 200))}</span></li>`;
return `<li class="ovt-w-step"><span>${esc(clip(r.text, 200))}</span></li>`;
};
};
return html`<article class="ovt-card ct-run" data-ct-run="${row.id}">
return html`<article class="ovt-card ct-run" data-ct-run="${row.id}">
+${deskTile(row, events, readOnly)}
<p class="ovt-card-h"><span class="ct-run-title">${clip(row.task, 160)}</span><span class="ct-run-state" data-ct-run-state>${LIVE_TEXT[row.state] || 'Working'}…</span></p>
<p class="ovt-card-h"><span class="ct-run-title">${clip(row.task, 160)}</span><span class="ct-run-state" data-ct-run-state>${LIVE_TEXT[row.state] || 'Working'}…</span></p>
<p class="ct-run-live" data-ovt-live>${live}</p>
<p class="ct-run-live" data-ovt-live>${live}</p>
<ol class="ct-run-steps" data-ovt-work>${raw(rows.map(item).join(''))}</ol>
<ol class="ct-run-steps" data-ovt-work>${raw(rows.map(item).join(''))}</ol>
@@ -523,7 +650,7 @@ ${shown.length ? html`<div class="ct-trig-runs">${shown}</div>` : ''}`;
@@ -523,7 +650,7 @@ ${shown.length ? html`<div class="ct-trig-runs">${shown}</div>` : ''}`;
async function turnHtml(row, { events = null, files = null, readOnly = false } = {}) {
async function turnHtml(row, { events = null, files = null, readOnly = false } = {}) {
const attached = files || (await inputs.list(s, [row.id])).get(row.id) || [];
const attached = files || (await inputs.list(s, [row.id])).get(row.id) || [];
const t = store.toWire(row, { baseUrl: config.baseUrl });
const t = store.toWire(row, { baseUrl: config.baseUrl });
- const evs = events || await store.eventsAfter(s, row.id, 0);
+ const evs = events || await store.eventsDetailed(s, row.id, 0);
const open = OPEN.includes(row.state);
const open = OPEN.includes(row.state);
let reply;
let reply;
if (row.state === 'succeeded') reply = html`${workBlock(row, evs, false)}${screenBlock(row, evs, false)}${answerBlock(row, t)}`;
if (row.state === 'succeeded') reply = html`${workBlock(row, evs, false)}${screenBlock(row, evs, false)}${answerBlock(row, t)}`;
@@ -534,7 +661,14 @@ ${shown.length ? html`<div class="ct-trig-runs">${shown}</div>` : ''}`;
@@ -534,7 +661,14 @@ ${shown.length ? html`<div class="ct-trig-runs">${shown}</div>` : ''}`;
// A running turn is a card in the conversation (one-agent system, stage 1). A turn parked on a wait still
// A running turn is a card in the conversation (one-agent system, stage 1). A turn parked on a wait still
// says what it waits for (plan A2).
// says what it waits for (plan A2).
const parked = ctx.waits && typeof ctx.waits.listForTask === 'function' ? (await ctx.waits.listForTask(row.id).catch(() => [])).find((w) => w.status === 'waiting') : null;
const parked = ctx.waits && typeof ctx.waits.listForTask === 'function' ? (await ctx.waits.listForTask(row.id).catch(() => [])).find((w) => w.status === 'waiting') : null;
- reply = html`${runCard(row, evs, readOnly)}${screenBlock(row, evs, true)}${waitBlock(row, parked, readOnly)}`;
+ // The desk's own tile is the live picture (See it work, W1): the old separate screen panel stays only for a
+ // run that has no desk to stream (a single look), so there is never a second, empty Live box under the card.
+ reply = html`${runCard(row, evs, readOnly)}${deskContext(evs) ? '' : screenBlock(row, evs, true)}${waitBlock(row, parked, readOnly)}`;
+ }
+ // See it work (W1): a finished run keeps a replay of the pictures its steps saw.
+ if (!readOnly && ['succeeded', 'failed', 'cancelled'].includes(row.state)) {
+ const shown = await replayBlock(row).catch(() => '');
+ if (shown) reply = html`${reply}${shown}`;
}
}
// A routine the turn made is its card, after whatever the turn said (plan §12 A-4).
// A routine the turn made is its card, after whatever the turn said (plan §12 A-4).
const routines = await triggerBlocks(row, evs, readOnly);
const routines = await triggerBlocks(row, evs, readOnly);
@@ -674,7 +808,7 @@ ${parent ? html`<input type="hidden" name="parent" value="${parent}">` : ''}<tex
@@ -674,7 +808,7 @@ ${parent ? html`<input type="hidden" name="parent" value="${parent}">` : ''}<tex
viewer: req.viewer, config, path: req.originalUrl, title, app: true, noReferrer: true,
viewer: req.viewer, config, path: req.originalUrl, title, app: true, noReferrer: true,
// The network's sidebar (kit.css, kit-web.js) and its conversation UI (thread.css, ov-thread.js); composer
// The network's sidebar (kit.css, kit-web.js) and its conversation UI (thread.css, ov-thread.js); composer
// v2 draws the box (ov-composer.js, 4.215.0).
// v2 draws the box (ov-composer.js, 4.215.0).
- css: ['css/chat.css', 'css/skill-draft.css'], scripts: ['js/chat.js'], styles: ['kit.css', 'thread.css', 'ov-share.css', 'ov-upload.css'],
+ css: ['css/chat.css', 'css/skill-draft.css'], scripts: ['js/desks.js', 'js/chat.js'], styles: ['kit.css', 'thread.css', 'ov-share.css', 'ov-upload.css'],
sharedScripts: ['kit-web.js', 'ov-thread.js', 'ov-composer.js', 'ov-share.js', 'ov-canvas.js', 'ov-upload.js'],
sharedScripts: ['kit-web.js', 'ov-thread.js', 'ov-composer.js', 'ov-share.js', 'ov-canvas.js', 'ov-upload.js'],
body: html`${busy && !shared ? raw('<noscript><meta http-equiv="refresh" content="5"></noscript>') : ''}
body: html`${busy && !shared ? raw('<noscript><meta http-equiv="refresh" content="5"></noscript>') : ''}
<div class="chat" data-chat="${first ? first.id : ''}">${raw(thread.page({
<div class="chat" data-chat="${first ? first.id : ''}">${raw(thread.page({
@@ -20,7 +20,7 @@ const PERSON_QUIET_MS = 20_000;
@@ -20,7 +20,7 @@ const PERSON_QUIET_MS = 20_000;
const AGENT_PROFILE = 'agent';
const AGENT_PROFILE = 'agent';
function deskRun(taskId = null) {
function deskRun(taskId = null) {
- return { name: `desk-${crypto.randomBytes(5).toString('hex')}`, taskId, named: null, kinds: new Map(), keeps: new Set(), used: new Set(), placed: null, names: new Map() };
+ return { name: `desk-${crypto.randomBytes(5).toString('hex')}`, taskId, named: null, kinds: new Map(), keeps: new Set(), used: new Set(), placed: null, names: new Map(), streams: new Map() };
}
}
/** The person's own word for their machine (their name for it), remembered from list_my_computers. */
/** The person's own word for their machine (their name for it), remembered from list_my_computers. */
@@ -72,11 +72,46 @@ async function closeDesks(run, requester, core) {
@@ -72,11 +72,46 @@ async function closeDesks(run, requester, core) {
await core.pcs.endSession(person, run.placed).catch(() => null); // the task's desk on a desk host ends with it
await core.pcs.endSession(person, run.placed).catch(() => null); // the task's desk on a desk host ends with it
run.placed = null;
run.placed = null;
}
}
- if (!run || !person || !run.used.size) return;
+ if (!run || !person) return;
+ // The Live stream goes before the desk it shows (See it work, W1): the desk's WHIP session ends with it, and the
+ // private stream is ended, not left publishing a screen nobody owns.
+ if (run.streams && run.streams.size && typeof core.pcs.deskStream === 'function') {
+ await Promise.all([...run.streams.entries()].map(([machine, s]) => core.pcs.deskStream(person, machine, { action: 'stop', desk: s.desk }).catch(() => null)));
+ run.streams.clear();
+ }
+ if (!run.used.size) return;
await Promise.all([...run.used].map((machine) => core.pcs.desk(person, machine, { action: 'stop', desk: run.name }).catch(() => null)));
await Promise.all([...run.used].map((machine) => core.pcs.desk(person, machine, { action: 'stop', desk: run.name }).catch(() => null)));
run.used.clear();
run.used.clear();
}
}
+/**
+ * Start the task's private Live stream for the desk it just started (See it work, W1). The stream is the same desk the
+ * agent works on, private to the person, and its id/page travel in the computer_desk result; closeDesks ends it when the
+ * task does. A refusal leaves the desk working and the tile falls back to the still.
+ */
+async function startDeskStream(run, person, core, ctx, machine, desk, out) {
+ if (!run || !out || out.error || !person || !ctx || !ctx.taskId) return null;
+ if (core.pcs.isSession && core.pcs.isSession(machine)) return null; // a Run desk has its still, no Link stream
+ if (typeof core.pcs.deskStream !== 'function') return null;
+ const title = String(ctx.personText || 'Actor task').replace(/\s+/g, ' ').trim().slice(0, 120) || 'Actor task';
+ const stream = await core.pcs.deskStream(person, machine, {
+ action: 'start', desk, title, visibility: 'private',
+ task: { source: 'actor', ref: ctx.taskId, title, url: `${(ctx.config && ctx.config.baseUrl) || 'https://openvibe.actor'}/chat/${ctx.taskId}` },
+ signal: ctx.signal,
+ }).catch((err) => ({ error: (err && err.message) || 'the stream did not start' }));
+ if (!stream || stream.error) return null;
+ run.streams.set(machine, { desk, id: stream.id });
+ return stream;
+}
+
+async function stopDeskStream(run, person, core, machine, desk) {
+ if (!run || !person || !core.pcs || typeof core.pcs.deskStream !== 'function') return null;
+ const kept = run.streams && run.streams.get(machine);
+ if (!kept || (desk && kept.desk !== desk)) return null;
+ run.streams.delete(machine);
+ return core.pcs.deskStream(person, machine, { action: 'stop', desk: kept.desk }).catch(() => null);
+}
+
/** A look at a computer's screen: the picture to the vision tier with the question, its answer back. */
/** A look at a computer's screen: the picture to the vision tier with the question, its answer back. */
async function lookAtScreen(person, a, ctx, core) {
async function lookAtScreen(person, a, ctx, core) {
if (!ctx.models) return { error: 'no vision model is configured on this server, so the screen cannot be read' };
if (!ctx.models) return { error: 'no vision model is configured on this server, so the screen cannot be read' };
@@ -158,6 +193,6 @@ async function prepareDesk(tool, a, person, core, ctx) {
@@ -158,6 +193,6 @@ async function prepareDesk(tool, a, person, core, ctx) {
}
}
module.exports = {
module.exports = {
- personOf, deskRun, deskFor, deskKind, placementOrder, closeDesks, lookAtScreen, handToPerson, prepareDesk, SCREEN_SYSTEM,
+ personOf, deskRun, deskFor, deskKind, placementOrder, closeDesks, startDeskStream, stopDeskStream, lookAtScreen, handToPerson, prepareDesk, SCREEN_SYSTEM,
PERSON_WAIT_MS, PERSON_POLL_MS, PERSON_QUIET_MS, AGENT_PROFILE, DESK_TOOLS, rememberMachines, machineWord,
PERSON_WAIT_MS, PERSON_POLL_MS, PERSON_QUIET_MS, AGENT_PROFILE, DESK_TOOLS, rememberMachines, machineWord,
};
};
@@ -1,7 +1,7 @@
@@ -1,7 +1,7 @@
'use strict';
'use strict';
const { serviceFor, json, personOf } = require('./_openvibe');
const { serviceFor, json, personOf } = require('./_openvibe');
-const { prepareDesk, deskKind, placementOrder, AGENT_PROFILE, machineWord } = require('./_person');
+const { prepareDesk, deskKind, placementOrder, AGENT_PROFILE, machineWord, startDeskStream, stopDeskStream } = require('./_person');
/**
/**
* The Computer this task opened, as platform.task@1 carries it (plan 2026-11-07): the one id every surface reads.
* The Computer this task opened, as platform.task@1 carries it (plan 2026-11-07): the one id every surface reads.
@@ -39,7 +39,8 @@ async function startDesk(a, person, core, ctx) {
@@ -39,7 +39,8 @@ async function startDesk(a, person, core, ctx) {
// kept for their agent on that machine, and saves it back when it stops. Never the model's choice.
// kept for their agent on that machine, and saves it back when it stops. Never the model's choice.
delete body.profile;
delete body.profile;
if (body.desk !== 'private' && desks && desks.keeps && desks.keeps.has(a.machine)) body.profile = AGENT_PROFILE;
if (body.desk !== 'private' && desks && desks.keeps && desks.keeps.has(a.machine)) body.profile = AGENT_PROFILE;
- return core.pcs.desk(person, body.machine, body, { signal: ctx.signal });
+ const out = await core.pcs.desk(person, body.machine, body, { signal: ctx.signal });
+ return out && !out.error ? { ...out, desk: body.desk || 'private' } : out;
}
}
module.exports = {
module.exports = {
@@ -78,8 +79,10 @@ module.exports = {
@@ -78,8 +79,10 @@ module.exports = {
out = await startDesk({ ...a, machine: id }, person, core, ctx);
out = await startDesk({ ...a, machine: id }, person, core, ctx);
if (!(out && out.error && /limit|max_desks|already runs/i.test(out.error))) {
if (!(out && out.error && /limit|max_desks|already runs/i.test(out.error))) {
if (out && !out.error && desks && !desks.placed) desks.placed = id;
if (out && !out.error && desks && !desks.placed) desks.placed = id;
+ const stream = out && !out.error ? await startDeskStream(desks, person, core, ctx, id, out.desk || a.desk, out) : null;
const computer = out && !out.error ? computerOf({ run: desks, machine: id }) : undefined;
const computer = out && !out.error ? computerOf({ run: desks, machine: id }) : undefined;
- return { text: json(out && !out.error ? { machine: id, ...out } : out), data: out, ...(computer ? { computer } : {}) };
+ const watch = out && !out.error ? core.pcs.watchUrl(id, out.desk || a.desk) : null;
+ return { text: json(out && !out.error ? { watch, machine: id, ...out, ...(stream ? { stream } : {}) } : out), data: out, ...(computer ? { computer } : {}) };
}
}
}
}
// None of the person's own machines has room (or none runs desks): a browser desk on a desk host,
// None of the person's own machines has room (or none runs desks): a browser desk on a desk host,
@@ -103,9 +106,11 @@ module.exports = {
@@ -103,9 +106,11 @@ module.exports = {
if (a.action === 'start') {
if (a.action === 'start') {
const out = await startDesk(a, person, core, ctx);
const out = await startDesk(a, person, core, ctx);
if (out && !out.error && desks && !desks.placed) desks.placed = a.machine;
if (out && !out.error && desks && !desks.placed) desks.placed = a.machine;
+ const stream = out && !out.error ? await startDeskStream(desks, person, core, ctx, a.machine, out.desk || a.desk, out) : null;
const computer = out && !out.error ? computerOf({ run: desks, machine: a.machine }) : undefined;
const computer = out && !out.error ? computerOf({ run: desks, machine: a.machine }) : undefined;
- return { text: json(out && out.running && !(core.pcs.isSession && core.pcs.isSession(a.machine)) ? { watch: core.pcs.watchUrl(a.machine, a.desk), ...out } : out), data: out, ...(computer ? { computer } : {}) };
+ return { text: json(out && out.running && !(core.pcs.isSession && core.pcs.isSession(a.machine)) ? { watch: core.pcs.watchUrl(a.machine, a.desk), ...out, ...(stream ? { stream } : {}) } : out), data: out, ...(computer ? { computer } : {}) };
}
}
+ if (a.action === 'stop' && desks) await stopDeskStream(desks, person, core, a.machine, a.desk);
const out = await core.pcs.desk(person, a.machine, a, { signal: ctx.signal });
const out = await core.pcs.desk(person, a.machine, a, { signal: ctx.signal });
if (desks && out && !out.error) {
if (desks && out && !out.error) {
if (a.action === 'stop' && desks.placed === a.machine) desks.placed = null;
if (a.action === 'stop' && desks.placed === a.machine) desks.placed = null;
@@ -1,7 +1,8 @@
@@ -1,7 +1,8 @@
'use strict';
'use strict';
const { serviceFor, json, personOf } = require('./_openvibe');
const { serviceFor, json, personOf } = require('./_openvibe');
-const { handToPerson } = require('./_person');
+const { handToPerson, PERSON_WAIT_MS, PERSON_POLL_MS } = require('./_person');
+const driving = require('../../agents/driving');
const machine = { type: 'string', description: 'the machine id (dev_…) from list_my_computers' };
const machine = { type: 'string', description: 'the machine id (dev_…) from list_my_computers' };
@@ -24,7 +25,21 @@ module.exports = {
@@ -24,7 +25,21 @@ module.exports = {
if (ctx.day && core.pcs.isSession && core.pcs.isSession(target)) {
if (ctx.day && core.pcs.isSession && core.pcs.isSession(target)) {
return ctx.day.ask(ctx.taskId, { kind: 'sign_in', text: input.what, options: ['Sign in on its screen', 'Done'], session_id: target.slice(4) });
return ctx.day.ask(ctx.taskId, { kind: 'sign_in', text: input.what, options: ['Sign in on its screen', 'Done'], session_id: target.slice(4) });
}
}
- if (core.pcs.isSession && core.pcs.isSession(target)) return { text: json({ done: false, note: 'Your desk is on a desk host, which the person cannot take over from here: tell them in your answer what is left for them to do, and where.' }), data: null };
+ if (core.pcs.isSession && core.pcs.isSession(target)) {
+ // A Run desk's Ask: the viewer takes the session's lease and says so in Actor's driving marker. Wait for the
+ // person to take over and hand back, then let the agent look again. No note is passed in a tool result.
+ const what = String(input.what || '').replace(/\s+/g, ' ').trim().slice(0, 300);
+ if (ctx.emit) await ctx.emit({ step: 'text', text: `Needs you: ${what}` });
+ const end = Date.now() + PERSON_WAIT_MS;
+ let took = false;
+ const pause = (ms) => new Promise((resolve) => { const timer = setTimeout(resolve, ms); if (ctx.signal) ctx.signal.addEventListener('abort', () => { clearTimeout(timer); resolve(); }, { once: true }); });
+ while (Date.now() < end && !(ctx.signal && ctx.signal.aborted)) {
+ if (driving.is(ctx.taskId)) took = true;
+ else if (took) return { text: json({ done: true, note: 'The person used the desk and handed it back. Look at it again before you go on.', session_id: target.slice(4) }), data: { done: true } };
+ await pause(PERSON_POLL_MS);
+ }
+ return { text: json({ done: false, note: took ? 'The person was still using the desk when the wait ended: tell them what is left.' : 'The person did not take the desk in time: tell them in your answer what is left for them to do.' }), data: { done: false } };
+ }
const data = await handToPerson(person, input, ctx, core);
const data = await handToPerson(person, input, ctx, core);
return { text: json(data), data };
return { text: json(data), data };
},
},
@@ -19,6 +19,8 @@ const PERSON_WAIT_MAX_MS = 15 * 60_000;
@@ -19,6 +19,8 @@ const PERSON_WAIT_MAX_MS = 15 * 60_000;
* At most config.maxRunning tasks run at once; the rest wait queued in order.
* At most config.maxRunning tasks run at once; the rest wait queued in order.
*/
*/
const store = require('./store');
const store = require('./store');
+const stills = require('./stills');
+const looks = require('../agents/looks');
const modelPrefs = require('../runtime/model-prefs');
const modelPrefs = require('../runtime/model-prefs');
// The person's own model choice rides with the models layer (plan L55 C3). A stand-in models object (a test's)
// The person's own model choice rides with the models layer (plan L55 C3). A stand-in models object (a test's)
// does not carry it, and a run then keeps today's behaviour.
// does not carry it, and a run then keeps today's behaviour.
@@ -188,6 +190,12 @@ function createEngine({ config, s, adapters, stream, webhooks = null, now = () =
@@ -188,6 +190,12 @@ function createEngine({ config, s, adapters, stream, webhooks = null, now = () =
const saved = await store.appendEvent(s, id, event);
const saved = await store.appendEvent(s, id, event);
if (day) await day.event(id, event);
if (day) await day.event(id, event);
stream.publish(id, saved);
stream.publish(id, saved);
+ // See it work (W1): every computer step keeps the picture it saw, for the finished run's replay. The desk's
+ // own curtain has already blanked a secret screen before the shot reached Actor; typed text is not a picture.
+ if (event.kind === 'output' && event.step === 'tool_result' && /^computer_/.test(String(event.tool || '')) && saved.seq) {
+ const shot = looks.get(id);
+ if (shot) await stills.save(s, id, saved.seq, { ...shot, tool: event.tool }).catch(() => null);
+ }
return saved;
return saved;
}
}
/**
/**
new file mode 100644
new file mode 100644
@@ -0,0 +1,79 @@
@@ -0,0 +1,79 @@
+'use strict';
+
+/**
+ * A finished run's replay (See it work, W1): every computer step, the action it took and the still it saw. Built from
+ * the task's own events and the pictures kept by tasks/stills; the model never supplies an action or a picture.
+ *
+ * Typed text is never returned: a step may say that something was typed or filled, never what. A click keeps its x,y in
+ * the picture's pixels so a page can put a ring where it happened.
+ */
+const store = require('./store');
+const stills = require('./stills');
+
+const COMPUTER = /^computer_(input|browser|screen|desk)$/;
+
+async function steps(s, row, { baseUrl = '' } = {}) {
+ const events = await store.eventsDetailed(s, row.id, 0);
+ const pictures = await stills.list(s, row.id);
+ const bySeq = new Map(pictures.map((p) => [p.seq, p]));
+ const results = new Map();
+ for (const e of events) if (e && e.kind === 'output' && e.step === 'tool_result' && e.call_id) results.set(e.call_id, e);
+ const out = [];
+ for (const e of events) {
+ if (!e || e.kind !== 'output' || e.step !== 'tool_call' || !COMPUTER.test(String(e.tool || ''))) continue;
+ const result = results.get(e.call_id) || null;
+ const args = e.args && typeof e.args === 'object' ? e.args : {};
+ let action = args.action || null, x = null, y = null, width = null, height = null, url = null, secret = false;
+ if (e.tool === 'computer_input') {
+ action = String(args.action || 'input');
+ for (const [k, v] of [['x', args.x], ['y', args.y], ['width', args.width], ['height', args.height]]) {
+ if (Number.isFinite(v)) { if (k === 'x') x = Number(v); else if (k === 'y') y = Number(v); else if (k === 'width') width = Number(v); else height = Number(v); }
+ }
+ if (action === 'type') secret = true;
+ } else if (e.tool === 'computer_browser') {
+ action = String(args.action || 'browser');
+ if (action === 'open' && typeof args.url === 'string') url = args.url.slice(0, 400);
+ if (action === 'fill') secret = true;
+ } else if (e.tool === 'computer_screen') action = 'look';
+ else action = String(args.action || 'desk');
+ if (result) {
+ try { const d = JSON.parse(result.text || '{}'); if (d && typeof d.url === 'string' && /^https?:\/\//.test(d.url)) url = d.url.slice(0, 400); } catch { /* not JSON */ }
+ }
+ const seq = result ? result.seq : e.seq;
+ let picture = bySeq.get(seq);
+ for (let p = seq; p >= 1 && !picture; p--) picture = bySeq.get(p);
+ out.push({
+ seq, at: result && result.at ? result.at : e.at || null, tool: e.tool, action: secret ? 'secret' : action,
+ x, y, width, height, url, secret,
+ text: result ? String(result.text || '').slice(0, 400) : null,
+ error: !!(result && result.is_error),
+ still: picture ? { seq: picture.seq, url: `${baseUrl}/api/v1/tasks/${row.id}/replay/${picture.seq}/still`, width: picture.width, height: picture.height, mime: picture.mime } : null,
+ });
+ }
+ return out.slice(-200);
+}
+
+/** One line for the replay's caption, in plain words. */
+function caption(step) {
+ if (!step) return '';
+ if (step.tool === 'computer_input') {
+ if (step.action === 'click' || step.action === 'double_click' || step.action === 'right_click') return `Clicked at ${Math.round(step.x || 0)}, ${Math.round(step.y || 0)}`;
+ if (step.action === 'move') return `Moved to ${Math.round(step.x || 0)}, ${Math.round(step.y || 0)}`;
+ if (step.action === 'scroll') return 'Scrolled';
+ if (step.action === 'key') return 'Pressed a key';
+ return 'Typed';
+ }
+ if (step.tool === 'computer_browser') {
+ if (step.action === 'open') return 'Opened a page';
+ if (step.action === 'read') return 'Read the page';
+ if (step.action === 'click') return 'Clicked on the page';
+ if (step.action === 'fill') return 'Filled a field';
+ if (step.action === 'press') return 'Pressed a key';
+ if (step.action === 'back') return 'Went back';
+ return 'Used the browser';
+ }
+ if (step.tool === 'computer_screen') return 'Looked at the screen';
+ return step.action === 'start' ? 'Started the desk' : step.action === 'stop' ? 'Stopped the desk' : 'Checked the desk';
+}
+
+module.exports = { steps, caption, COMPUTER };
new file mode 100644
new file mode 100644
@@ -0,0 +1,63 @@
@@ -0,0 +1,63 @@
+'use strict';
+
+/**
+ * The pictures a task's steps saw (See it work, W1): one still per computer step, kept with its event seq so a finished
+ * run's replay can scrub through them. The desk's own curtain blanks a secret screen before this ever sees it; typed
+ * text is never part of a still.
+ *
+ * In Actor's own database, gone with the task (store.prune deletes the orphans; tasks are kept 30 days). At most
+ * MAX_PER_TASK per task, 640 px wide, WebP where sharp is available, otherwise the JPEG/PNG the machine sent.
+ */
+const MAX_PER_TASK = 120;
+const MAX_BYTES = 400 * 1024;
+const MAX_KEEP = 640;
+
+const sharp = (() => { try { return require('sharp'); } catch { return null; } })();
+
+/** Shrink to a replay picture and encode it. → { mime, bytes, width, height } */
+async function shrink(shot) {
+ const bytes = Buffer.isBuffer(shot.data) ? shot.data : Buffer.from(String(shot.data || ''), 'base64');
+ if (!bytes.length) return null;
+ const format = shot.format === 'png' ? 'png' : 'jpeg';
+ if (sharp) {
+ try {
+ const out = await sharp(bytes).rotate().resize({ width: MAX_KEEP, withoutEnlargement: true }).webp({ quality: 72 }).toBuffer({ resolveWithObject: true });
+ if (out.data.length && out.data.length <= MAX_BYTES) {
+ return { mime: 'image/webp', bytes: out.data, width: out.info.width || null, height: out.info.height || null };
+ }
+ } catch { /* fall through to the original */ }
+ }
+ if (bytes.length > MAX_BYTES) return null;
+ return { mime: `image/${format}`, bytes, width: Number(shot.width) || null, height: Number(shot.height) || null };
+}
+
+/** Remember the still a computer step saw at its result event's seq. Best effort: a picture never fails the run. */
+async function save(s, taskId, seq, shot) {
+ if (!s || !taskId || !Number.isInteger(seq) || seq < 1 || !shot) return false;
+ const made = await shrink(shot);
+ if (!made) return false;
+ const count = await s.db.value('SELECT count(*)::int FROM actor_task_stills WHERE task_id = $1', [taskId]).catch(() => 0);
+ if (Number(count) >= MAX_PER_TASK) return false;
+ await s.db.query(
+ `INSERT INTO actor_task_stills (task_id, seq, tool, mime, width, height, bytes, at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
+ ON CONFLICT (task_id, seq) DO NOTHING`,
+ [taskId, seq, shot.tool || null, made.mime, made.width, made.height, made.bytes, s.iso()],
+ );
+ return true;
+}
+
+async function list(s, taskId) {
+ return s.db.many('SELECT seq, tool, mime, width, height, at FROM actor_task_stills WHERE task_id = $1 ORDER BY seq', [taskId]);
+}
+
+async function get(s, taskId, seq) {
+ return s.db.maybe('SELECT seq, tool, mime, width, height, bytes, at FROM actor_task_stills WHERE task_id = $1 AND seq = $2', [taskId, seq]);
+}
+
+/** Drop the pictures of tasks that are gone (called by store.prune). */
+async function prune(s) {
+ await s.db.query('DELETE FROM actor_task_stills WHERE task_id NOT IN (SELECT id FROM tasks)');
+}
+
+module.exports = { save, list, get, prune, MAX_PER_TASK };
@@ -134,8 +134,13 @@ function deskFrom(events) {
@@ -134,8 +134,13 @@ function deskFrom(events) {
if (!e || e.step !== 'tool_result' || e.tool !== 'computer_desk' || e.is_error) continue;
if (!e || e.step !== 'tool_result' || e.tool !== 'computer_desk' || e.is_error) continue;
const t = String(e.text || '');
const t = String(e.text || '');
const m = DESK_WATCH.exec(t);
const m = DESK_WATCH.exec(t);
- if (m && /"running":true/.test(t)) desk = { url: m[1], kind: (/"kind":"(browser|desktop)"/.exec(t) || [])[1] || null };
- else if (/"running":false/.test(t)) desk = null;
+ if (m && /"running":true/.test(t)) {
+ let parsed = null;
+ try { parsed = JSON.parse(t); } catch { parsed = null; }
+ desk = { url: m[1], kind: (/"kind":"(browser|desktop)"/.exec(t) || [])[1] || null };
+ // See it work: the private Live stream the desk started for this task (Link's desk.stream), if any.
+ if (parsed && parsed.stream && typeof parsed.stream === 'object') desk.stream = parsed.stream;
+ } else if (/"running":false/.test(t)) desk = null;
}
}
return desk;
return desk;
}
}
@@ -181,6 +186,20 @@ async function eventsAfter(s, taskId, after = 0) {
@@ -181,6 +186,20 @@ async function eventsAfter(s, taskId, after = 0) {
return rows.map((r) => parse(r.body));
return rows.map((r) => parse(r.body));
}
}
+/**
+ * The events with the detail a replay needs (See it work, W1): the tool args that appendEvent keeps out of the public
+ * body (a click's x,y; a browser action). They were stripped of credential-shaped fields before they were stored, and
+ * the replay blanks typed text again. Only the task's own replay reads this.
+ */
+async function eventsDetailed(s, taskId, after = 0) {
+ const rows = await s.db.many('SELECT body, ui_detail FROM task_events WHERE task_id = $1 AND seq > $2 ORDER BY seq', [taskId, after]);
+ return rows.map((r) => {
+ const body = parse(r.body) || {};
+ const detail = parse(r.ui_detail) || {};
+ return { ...body, ...(detail.args !== undefined ? { args: detail.args } : {}), ...(detail.call_id ? { call_id: detail.call_id } : {}) };
+ });
+}
+
/** Add to a requester's and everyone's spend today (tasks counts only the requester's). */
/** Add to a requester's and everyone's spend today (tasks counts only the requester's). */
async function addSpend(s, requester, usd, { tasks = 0 } = {}) {
async function addSpend(s, requester, usd, { tasks = 0 } = {}) {
const day = dayOf(s.now());
const day = dayOf(s.now());
@@ -256,6 +275,8 @@ async function prune(s) {
@@ -256,6 +275,8 @@ async function prune(s) {
const cutoff = new Date(s.now() - RETENTION_DAYS * 86_400_000).toISOString();
const cutoff = new Date(s.now() - RETENTION_DAYS * 86_400_000).toISOString();
const n = await s.db.exec('DELETE FROM tasks WHERE created_at < $1', [cutoff]);
const n = await s.db.exec('DELETE FROM tasks WHERE created_at < $1', [cutoff]);
await s.db.query('DELETE FROM spend_daily WHERE day < $1', [cutoff.slice(0, 10)]);
await s.db.query('DELETE FROM spend_daily WHERE day < $1', [cutoff.slice(0, 10)]);
+ // See it work: a replay's pictures go with the task that saw them.
+ await require('./stills').prune(s).catch(() => null);
return n;
return n;
}
}
@@ -290,4 +311,4 @@ function toWire(row, { baseUrl }) {
@@ -290,4 +311,4 @@ function toWire(row, { baseUrl }) {
return t;
return t;
}
}
-module.exports = { insertTask, getTask, byIdempotency, listTasks, updateTask, appendEvent, eventsAfter, lastEvent, needsYou, deskFrom, deskPlace, taskDesk, computerFrom, taskComputer, addSpend, spendToday, spendDays, lockSpendToday, failInterrupted, openTasks, prune, toWire, subjectRef, dayOf, parseJson: parse, OPEN, RETENTION_DAYS, followContext, followUps, forks, recentSucceeded };
+module.exports = { insertTask, getTask, byIdempotency, listTasks, updateTask, appendEvent, eventsAfter, eventsDetailed, lastEvent, needsYou, deskFrom, deskPlace, taskDesk, computerFrom, taskComputer, addSpend, spendToday, spendDays, lockSpendToday, failInterrupted, openTasks, prune, toWire, subjectRef, dayOf, parseJson: parse, OPEN, RETENTION_DAYS, followContext, followUps, forks, recentSucceeded };
@@ -37,10 +37,11 @@ const MACHINE = 'dev_01JZ0000000000000000000000';
@@ -37,10 +37,11 @@ const MACHINE = 'dev_01JZ0000000000000000000000';
await s.db.query("UPDATE tasks SET state = 'running' WHERE id = $1", [id]);
await s.db.query("UPDATE tasks SET state = 'running' WHERE id = $1", [id]);
await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`);
await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`);
let page = await t.get(`/chat/${id}`, { as: kim });
let page = await t.get(`/chat/${id}`, { as: kim });
- assert.match(page.text, new RegExp(`data-watch="/chat/${id}/watch"`), 'the figure says where to ask');
+ assert.match(page.text, new RegExp(`<figure class="ct-desk-tile" data-ct-desk data-task="${id}" data-desk="link:${MACHINE}:desk-trains"`), 'the desk tile says where the desk is and opens the viewer');
+ assert.match(page.text, /data-ct-desk-open/, 'the tile opens the viewer');
await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen`);
await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen`);
page = await t.get(`/chat/${id}`, { as: kim });
page = await t.get(`/chat/${id}`, { as: kim });
- assert.ok(!page.text.includes('data-watch='), 'the owner\'s own screen is never framed');
+ assert.ok(!page.text.includes('data-ct-desk'), 'the owner\'s own screen is never framed');
await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`);
await deskEvent(`https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`);
});
});
@@ -30,6 +30,15 @@ const ROUTE_EXTENSIONS = new Map([
@@ -30,6 +30,15 @@ const ROUTE_EXTENSIONS = new Map([
['GET /api/v1/agent/timeline/:id', 'actor.timeline.read'],
['GET /api/v1/agent/timeline/:id', 'actor.timeline.read'],
['POST /api/v1/agent/approvals/:product/:id/decide', 'actor.approval.decide'],
['POST /api/v1/agent/approvals/:product/:id/decide', 'actor.approval.decide'],
['POST /api/v1/agent/kill-switch', 'actor.resource.control'],
['POST /api/v1/agent/kill-switch', 'actor.resource.control'],
+ // See it work (W1): the desk, its still, the person's control and the replay. They are read/create on the task
+ // the person already holds; Contracts lists them after this stage (the same pattern as the trigger routes).
+ ['GET /api/v1/tasks/:id/desks', 'actor.task.read'],
+ ['GET /api/v1/tasks/:id/desk-still', 'actor.task.read'],
+ ['POST /api/v1/tasks/:id/desks/:desk/input', 'actor.task.create'],
+ ['POST /api/v1/tasks/:id/desks/:desk/take-over', 'actor.task.create'],
+ ['POST /api/v1/tasks/:id/desks/:desk/hand-back', 'actor.task.create'],
+ ['GET /api/v1/tasks/:id/replay', 'actor.task.read'],
+ ['GET /api/v1/tasks/:id/replay/:seq/still', 'actor.task.read'],
// Triggers made in a conversation (plan §12 A-4). The routes exist before Contracts lists actor.trigger.*; a
// Triggers made in a conversation (plan §12 A-4). The routes exist before Contracts lists actor.trigger.*; a
// trigger is the person's own work record, so the task capabilities the person already holds guard it.
// trigger is the person's own work record, so the task capabilities the person already holds guard it.
['GET /api/v1/triggers', 'actor.task.read'],
['GET /api/v1/triggers', 'actor.task.read'],
@@ -31,6 +31,9 @@ async function boot(opts = {}) {
@@ -31,6 +31,9 @@ async function boot(opts = {}) {
// A test boot makes many runs in one "day"; the free allowance is measured in free-allowance.test.js, which
// A test boot makes many runs in one "day"; the free allowance is measured in free-allowance.test.js, which
// sets its own. This keeps the other suites about what they are about.
// sets its own. This keeps the other suites about what they are about.
ACTOR_FREE_PER_DAY_USD: '1000',
ACTOR_FREE_PER_DAY_USD: '1000',
+ // Tests stand Live in when they need a view ticket; no test calls openvibe.live.
+ ACTOR_LIVE_URL: '',
+ ACTOR_LIVE_PUBLIC_URL: 'https://openvibe.live',
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
OV_OAUTH_CLIENT_ID: 'actor', OV_OAUTH_CLIENT_SECRET: 'actor-secret', COOKIE_SECURE: 'false',
OV_OAUTH_CLIENT_ID: 'actor', OV_OAUTH_CLIENT_SECRET: 'actor-secret', COOKIE_SECURE: 'false',
...providers.env,
...providers.env,
@@ -45,7 +48,7 @@ async function boot(opts = {}) {
@@ -45,7 +48,7 @@ async function boot(opts = {}) {
const { createStore } = require('../../server/db');
const { createStore } = require('../../server/db');
const testdb = await require('./db').testDb();
const testdb = await require('./db').testDb();
const store = createStore(testdb.db, { now: opts.now });
const store = createStore(testdb.db, { now: opts.now });
- const built = await createApp({ config, env, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, adapters: opts.adapters, accountSend: opts.accountSend, webhookPost: opts.webhookPost, chatClient: opts.chatClient, secretSigner: opts.secretSigner, bot: opts.bot, projects: opts.projects, mailFetch: opts.mailFetch, mailTokens: opts.mailTokens, notifier: opts.notifier, models: opts.models, registry: opts.registry, loop: opts.loop, connectors: opts.connectors, guard: opts.guard });
+ const built = await createApp({ config, env, store, log, telemetrySink: opts.telemetrySink, limitsNow: opts.limitsNow, callerLimits: opts.callerLimits === true, adapters: opts.adapters, accountSend: opts.accountSend, webhookPost: opts.webhookPost, chatClient: opts.chatClient, secretSigner: opts.secretSigner, bot: opts.bot, live: opts.live, projects: opts.projects, mailFetch: opts.mailFetch, mailTokens: opts.mailTokens, notifier: opts.notifier, models: opts.models, registry: opts.registry, loop: opts.loop, connectors: opts.connectors, guard: opts.guard });
await built.ctx.keys.ensure();
await built.ctx.keys.ensure();
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const server = await new Promise((resolve) => { const s = http.createServer(built.app); s.listen(0, '127.0.0.1', () => resolve(s)); });
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
@@ -109,6 +109,7 @@ const SAME = { 'sec-fetch-site': 'same-origin' };
@@ -109,6 +109,7 @@ const SAME = { 'sec-fetch-site': 'same-origin' };
'server/agents/providers.js': 'the configured provider base URLs (DeepSeek, OpenAI, the open model) with the operator\'s keys',
'server/agents/providers.js': 'the configured provider base URLs (DeepSeek, OpenAI, the open model) with the operator\'s keys',
'server/agents/codes.js': 'the configured OpenVibe.Codes base (ACTOR_CODES_URL) with Actor\'s own service token; the subject must be usr_<ULID> and every task id tsk_<ULID> before it goes in a path; the repository named in the task is data for the person\'s own machine, never fetched here (test/codes.test.js)',
'server/agents/codes.js': 'the configured OpenVibe.Codes base (ACTOR_CODES_URL) with Actor\'s own service token; the subject must be usr_<ULID> and every task id tsk_<ULID> before it goes in a path; the repository named in the task is data for the person\'s own machine, never fetched here (test/codes.test.js)',
'server/agents/computers.js': 'the configured OpenVibe.Bot base (ACTOR_BOT_URL) with Actor\'s own service token (bot.computer.control); the machine id must be dev_<ULID> and the subject usr_<ULID> before anything is sent; a path or program is data for the person\'s own machine, never fetched here (test/computers.test.js)',
'server/agents/computers.js': 'the configured OpenVibe.Bot base (ACTOR_BOT_URL) with Actor\'s own service token (bot.computer.control); the machine id must be dev_<ULID> and the subject usr_<ULID> before anything is sent; a path or program is data for the person\'s own machine, never fetched here (test/computers.test.js)',
+ 'server/agents/live.js': "the configured OpenVibe.Live base (ACTOR_LIVE_URL) for one POST /api/streams/agent/<ags_ULID>/view-ticket, with the person's own bearer token; the stream id must be ags_<ULID> and is URL-encoded, and the returned embed is checked to Live's configured origin before the page gets it (test/see-it-work.test.js)",
'server/agents/timeline.js': 'the approval proxy answers the mode that owns an approval: the product base is the configured ACTOR_<PRODUCT>_URL (or the config map) and the approval id is URL-encoded; the person\'s own bearer token is forwarded, and a stop uses the same base with the step\'s own ref (codes cancel, run cancel, live end); no URL from a step\'s content (test/timeline.test.js)',
'server/agents/timeline.js': 'the approval proxy answers the mode that owns an approval: the product base is the configured ACTOR_<PRODUCT>_URL (or the config map) and the approval id is URL-encoded; the person\'s own bearer token is forwarded, and a stop uses the same base with the step\'s own ref (codes cancel, run cancel, live end); no URL from a step\'s content (test/timeline.test.js)',
'server/chat/client.js': 'the configured OpenVibe.Chat base (ACTOR_CHAT_URL, loopback) with Actor\'s own service token; the room and the message it answers come from a signed chat.message.created, never from the task (test/chat-asks.test.js)',
'server/chat/client.js': 'the configured OpenVibe.Chat base (ACTOR_CHAT_URL, loopback) with Actor\'s own service token; the room and the message it answers come from a signed chat.message.created, never from the task (test/chat-asks.test.js)',
'server/mail/client.js': 'the configured OpenVibe.Mail base (ACTOR_MAIL_URL, loopback) with Actor\'s own service token (mail.agent.read, mail.agent.reply, mail.agent.answering); the person is the task\'s owner in X-OV-Subject, the agent name is slugified and URL-encoded, and the message id is Mail\'s own and URL-encoded; no address or URL from the letter is ever fetched here (test/agent-mail.test.js)',
'server/mail/client.js': 'the configured OpenVibe.Mail base (ACTOR_MAIL_URL, loopback) with Actor\'s own service token (mail.agent.read, mail.agent.reply, mail.agent.answering); the person is the task\'s owner in X-OV-Subject, the agent name is slugified and URL-encoded, and the message id is Mail\'s own and URL-encoded; no address or URL from the letter is ever fetched here (test/agent-mail.test.js)',
new file mode 100644
new file mode 100644
@@ -0,0 +1,111 @@
@@ -0,0 +1,111 @@
+'use strict';
+// Screenshots for See it work (W1), not run by test/run.js: a task with a live desk tile, the viewer in control, and a
+// finished run's replay, at 390 and 1280. The Live embed is a local stand-in (the test page never calls openvibe.live).
+// Usage: node test/see-it-work-shots.js <outdir>
+const fs = require('fs');
+const http = require('http');
+const path = require('path');
+const h = require('openvibe-shared/browser-harness');
+const { boot } = require('./helpers/boot');
+const store = require('../server/tasks/store');
+const stills = require('../server/tasks/stills');
+
+const MACHINE = 'dev_01JZ0000000000000000000000';
+const STREAM = 'ags_01JZ0000000000000000000000';
+const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==';
+
+(async () => {
+ const out = process.argv[2] || path.join(__dirname, '..', '..', 'research-2026-11-12', 'actor-shots', 'see-it-work');
+ fs.mkdirSync(out, { recursive: true });
+ // A stand-in Live embed: what a viewer sees on openvibe.live, without leaving this machine.
+ const live = http.createServer((req, res) => {
+ res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
+ res.end(`<!doctype html><meta charset="utf-8"><style>html,body{margin:0;height:100%;background:#05070d;color:#e6edf7;font:14px system-ui;overflow:hidden}
+.window{position:absolute;inset:10px;border-radius:10px;background:#0f1729;border:1px solid #334155;overflow:hidden}
+.bar{height:34px;display:flex;align-items:center;gap:6px;padding:0 10px;background:#1e293b;border-bottom:1px solid #334155}
+.dot{width:7px;height:7px;border-radius:50%;background:#64748b}.url{flex:1;height:20px;border-radius:6px;background:#0b1220;border:1px solid #334155;color:#94a3b8;font-size:11px;line-height:20px;padding:0 8px}
+.page{padding:22px}.page h1{margin:0 0 8px;font-size:22px}.page p{color:#94a3b8;margin:0 0 18px}.card{height:86px;border-radius:10px;background:#162033;border:1px solid #334155;margin:8px 0}
+.tag{position:absolute;left:18px;bottom:16px;display:inline-flex;align-items:center;gap:7px;padding:3px 10px;border-radius:999px;background:#0b1220cc;border:1px solid #334155;font-size:11px;font-weight:700;letter-spacing:.06em;text-transform:uppercase}
+.tag::before{content:"";width:7px;height:7px;border-radius:50%;background:#ef4444;box-shadow:0 0 0 3px #ef444455}</style>
+<div class="window"><div class="bar"><span class="dot"></span><span class="dot"></span><span class="dot"></span><span class="url">example.com/trains</span></div>
+<div class="page"><h1>Trains to Porto</h1><p>Tuesday, 14:10 · from Lisbon</p><div class="card"></div><div class="card"></div><div class="card"></div></div></div><span class="tag">AI agent · live</span>`);
+ });
+ await new Promise((r) => live.listen(0, '127.0.0.1', r));
+ const liveBase = `http://127.0.0.1:${live.address().port}`;
+ const t = await boot({ env: { ACTOR_LIVE_URL: '', ACTOR_LIVE_PUBLIC_URL: liveBase } });
+ // The page's own view ticket call: the same shape Live answers, pointing at the stand-in embed.
+ t.ctx.live.enabled = true;
+ t.ctx.live.viewTicket = async (token, id) => ({ stream_id: id, view_ticket: 'wvt_test', expires_at: new Date(Date.now() + 600e3).toISOString(), page_url: `${liveBase}/embed/agent/${id}?vt=wvt_test` });
+ const kim = t.network.addUser('kim');
+ const s = t.ctx.s;
+ const owner = `user:${kim.subject}`;
+ const event = (id, body) => store.appendEvent(s, id, { kind: 'output', at: s.iso(), ...body });
+ const make = async (state) => {
+ const id = s.newId('tsk');
+ await store.insertTask(s, { id, requester: owner, task: 'Book the cheapest train to Porto', mode: 'balanced', budget_task: 0.05, budget_day: 0.5, created_at: s.iso() });
+ await store.updateTask(s, id, { state });
+ return id;
+ };
+ const makeLive = async () => {
+ const id = await make('running');
+ await event(id, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ watch: `https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`, running: true, kind: 'browser', stream: { id: STREAM, page_url: `${liveBase}/embed/agent/${STREAM}`, visibility: 'private' } }) });
+ await event(id, { step: 'tool_call', tool: 'computer_input', call_id: 'c1', args: { action: 'click', x: 470, y: 250, width: 1280, height: 800 }, input: 'click' });
+ await event(id, { step: 'tool_result', tool: 'computer_input', call_id: 'c1', text: '{"done":true}' });
+ await event(id, { step: 'tool_call', tool: 'computer_browser', call_id: 'c2', args: { action: 'open', url: 'https://example.com/trains' }, input: 'open' });
+ await event(id, { step: 'tool_result', tool: 'computer_browser', call_id: 'c2', text: '{"url":"https://example.com/trains","title":"Trains"}' });
+ return id;
+ };
+
+ const replayId = await make('succeeded');
+ await store.updateTask(s, replayId, { finished_at: s.iso(), result: { answer: 'The 14:10 from Lisbon is the cheapest: €29.50.' } });
+ const type = await event(replayId, { step: 'tool_call', tool: 'computer_input', call_id: 'r1', args: { action: 'type', text: 'Porto', desk: 'private' }, input: 'type' });
+ await stills.save(s, replayId, type.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 1280, height: 800 });
+ await event(replayId, { step: 'tool_result', tool: 'computer_input', call_id: 'r1', text: '{"done":true}' });
+ const click = await event(replayId, { step: 'tool_call', tool: 'computer_input', call_id: 'r2', args: { action: 'click', x: 320, y: 180, width: 1280, height: 800 }, input: 'click' });
+ await stills.save(s, replayId, click.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 1280, height: 800 });
+ await event(replayId, { step: 'tool_result', tool: 'computer_input', call_id: 'r2', text: '{"done":true}' });
+
+ const browser = await h.launch({});
+ try {
+ for (const width of [390, 1280]) {
+ const liveId = await makeLive();
+ const cases = [
+ ['tile', false, async () => {}, liveId],
+ ['viewer', true, async (page) => {
+ await page.evaluate(`document.querySelector('[data-ct-desk-open]').click()`);
+ await new Promise((r) => setTimeout(r, 800));
+ await page.evaluate(`document.querySelector('[data-ct-viewer-take]').click()`);
+ await new Promise((r) => setTimeout(r, 1000));
+ }, liveId],
+ ['replay', false, async (page) => {
+ await page.evaluate(`(() => { const d = document.querySelector('[data-ct-replay]'); if (d) { d.open = true; const b = d.querySelector('[data-ct-replay-step="1"]'); if (b) b.click(); } })()`);
+ await new Promise((r) => setTimeout(r, 700));
+ }, replayId],
+ ];
+ for (const [name, viewer, act, id] of cases) {
+ const page = await h.openPage(browser, await browser.newContext(), { width, height: 1500 });
+ await page.send('Network.setCookie', { name: 'actor_at', value: t.signIn(kim).replace(/^actor_at=/, ''), url: t.base, path: '/' });
+ await page.send('Network.setExtraHTTPHeaders', { headers: { Authorization: `Bearer ${t.network.userToken(kim)}` } });
+ await page.goto(`${t.base}/chat/${id}`, { maxMs: 10000 });
+ await new Promise((r) => setTimeout(r, 900));
+ await act(page);
+ const shot = path.join(out, `${name}-${width}.png`);
+ const png = await page.send('Page.captureScreenshot', { format: 'png' });
+ fs.writeFileSync(shot, Buffer.from(png.data, 'base64'));
+ const read = await page.evaluate(`(() => {
+ const wide = [...document.querySelectorAll('main *')].filter((e) => { const r = e.getBoundingClientRect(); return r.width > 0 && (r.right > innerWidth + 1 || r.left < -1); }).slice(0, 5).map((e) => e.tagName.toLowerCase());
+ const small = [...document.querySelectorAll('.ct-desk-tile button, .ct-viewer button, .ct-viewer input, .ct-replay button, .ct-replay input')].filter((e) => { const r = e.getBoundingClientRect(); return !e.hidden && r.width > 0 && r.height > 0 && (r.width < 44 || r.height < 44); }).map((e) => e.tagName.toLowerCase());
+ return { sw: document.documentElement.scrollWidth, iw: innerWidth, wide, small, live: !!document.querySelector('[data-ct-desk-frame] iframe'), viewer: !!document.querySelector('.ct-viewer:not([hidden])'), replay: !!document.querySelector('[data-ct-replay][open]') };
+ })()`);
+ process.stdout.write(`${shot} scrollWidth=${read.sw}/${read.iw} live=${read.live} viewer=${read.viewer} replay=${read.replay}${read.wide.length ? ` WIDE ${read.wide.join(',')}` : ''}${read.small.length ? ` SMALL ${read.small.join(',')}` : ''}\n`);
+ await page.close();
+ }
+ await store.updateTask(s, liveId, { state: 'cancelled', finished_at: s.iso() }).catch(() => null);
+ }
+ } finally {
+ await browser.close();
+ live.close();
+ await t.close();
+ }
+ process.exit(0);
+})().catch((e) => { process.stderr.write(String(e && e.stack)); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,217 @@
@@ -0,0 +1,217 @@
+'use strict';
+// See it work (W1): a running task's desk tile, the private Live stream's view ticket, the still fallback, the person's
+// take-over and hand-back (Run lease / Link input as the person), the Ask that opens the viewer in control, and a
+// finished run's replay with the still each step saw. Typed text is never in a replay.
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const store = require('../server/tasks/store');
+const stills = require('../server/tasks/stills');
+
+const SAME = { 'sec-fetch-site': 'same-origin' };
+const MACHINE = 'dev_01JZ0000000000000000000000';
+const SESSION = 'ses_01JZ0000000000000000000000';
+const STREAM = 'ags_01JZ0000000000000000000000';
+const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==';
+
+(async () => {
+ const calls = [];
+ const computers = {
+ async still(subject, id) { calls.push(['still', subject, id]); return { bytes: Buffer.from(PNG, 'base64'), type: 'image/png' }; },
+ async screen(subject, machine, opts) { calls.push(['screen', subject, machine, opts]); return { format: 'png', width: 640, height: 400, content_b64: PNG }; },
+ async personSession(subject, id, action, body) { calls.push(['session', subject, id, action, body]); return { lease: { holder: action === 'take' ? 'person' : 'ai' }, done: true }; },
+ async personInput(subject, token, machine, args) { calls.push(['input', subject, token, machine, args]); return { done: true, action: args.action }; },
+ async deskStream() { return { error: 'not used in this test' }; },
+ };
+ const live = {
+ enabled: true,
+ async viewTicket(token, id) { calls.push(['ticket', token, id]); return { view_ticket: 'wvt_private', expires_at: new Date(Date.now() + 600e3).toISOString(), page_url: `https://openvibe.live/embed/agent/${id}?vt=wvt_private` }; },
+ };
+ const t = await boot({ bot: { computers }, live });
+ const kim = t.network.addUser('kim');
+ const bo = t.network.addUser('bo');
+ const s = t.ctx.s;
+ const owner = `user:${kim.subject}`;
+ const token = t.network.userToken(kim);
+ const event = (id, body) => store.appendEvent(s, id, { kind: 'output', at: s.iso(), ...body });
+ const make = async (state = 'running') => {
+ const id = s.newId('tsk');
+ await store.insertTask(s, { id, requester: owner, task: 'Book the cheapest train to Porto', mode: 'balanced', budget_task: 0.05, budget_day: 0.5, created_at: s.iso() });
+ if (state !== 'queued') await store.updateTask(s, id, { state });
+ return id;
+ };
+ try {
+ const id = await make();
+ await event(id, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ watch: `https://openvibe.link/machines/${MACHINE}/screen?desk=desk-trains`, running: true, kind: 'browser', stream: { id: STREAM, page_url: `https://openvibe.live/embed/agent/${STREAM}`, visibility: 'private', expires_at: new Date(Date.now() + 4 * 3600e3).toISOString() } }) });
+ await event(id, { step: 'tool_call', tool: 'computer_input', call_id: 'c1', args: { action: 'click', x: 320, y: 200, width: 1280, height: 800 }, input: 'click' });
+ await event(id, { step: 'tool_result', tool: 'computer_input', call_id: 'c1', text: '{"done":true}' });
+ await event(id, { step: 'tool_call', tool: 'computer_browser', call_id: 'c2', args: { action: 'open', url: 'https://example.com/trains' }, input: 'open' });
+ await event(id, { step: 'tool_result', tool: 'computer_browser', call_id: 'c2', text: '{"url":"https://example.com/trains","title":"Trains"}' });
+
+ await check('the running card has the desk tile: label, caption, ring, page address and the watch button', async () => {
+ const page = await t.get(`/chat/${id}`, { as: kim });
+ assert.strictEqual(page.status, 200);
+ assert.match(page.text, new RegExp(`<figure class="ct-desk-tile" data-ct-desk data-task="${id}" data-desk="link:${MACHINE}:desk-trains" data-where="link" data-kind="browser"`));
+ assert.match(page.text, /Opening a page/);
+ assert.match(page.text, /style="left:25\.00%;top:25\.00%"/, 'the ring is where it clicked');
+ assert.match(page.text, /example\.com\/trains/, 'the browser address');
+ assert.match(page.text, /Your PC · its own desk/);
+ assert.match(page.text, /data-ct-desk-open/);
+ assert.match(page.text, new RegExp(`src="/api/v1/tasks/${id}/desk-still\\?desk=desk-trains"`), 'the still fallback');
+ });
+
+ await check('the desks call names the private Live stream with a view ticket; the still is Bot\'s picture at 640', async () => {
+ const r = await t.get(`/api/v1/tasks/${id}/desks`, { as: kim, bearer: token, headers: SAME });
+ assert.strictEqual(r.status, 200, r.text);
+ const d = r.json().desks[0];
+ assert.strictEqual(d.id, `link:${MACHINE}:desk-trains`);
+ assert.strictEqual(d.label, 'Your PC · its own desk');
+ assert.strictEqual(d.live.stream_id, STREAM);
+ assert.match(d.live.embed_url, new RegExp(`/embed/agent/${STREAM}\\?vt=wvt_private`));
+ assert.ok(calls.some((c) => c[0] === 'ticket' && c[1] === token && c[2] === STREAM), 'Live got the person\'s own token');
+ const still = await t.get(`/api/v1/tasks/${id}/desk-still?desk=desk-trains`, { as: kim });
+ assert.strictEqual(still.status, 200);
+ assert.match(still.headers.get('content-type'), /^image\/png/);
+ assert.deepStrictEqual(still.buffer, Buffer.from(PNG, 'base64'));
+ assert.strictEqual((await t.get(`/api/v1/tasks/${id}/desks`, { as: bo })).status, 404, 'another person: not there');
+ });
+
+ await check('take over, person input, hand back with a note (Link: the person\'s own token to Bot)', async () => {
+ const take = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/take-over`, { as: kim, method: 'POST', headers: SAME });
+ assert.strictEqual(take.status, 200, take.text);
+ assert.ok(take.json().desk, 'the desk is in the answer');
+ assert.strictEqual((await t.get(`/api/v1/tasks/${id}/desks`, { as: kim, bearer: token })).json().desks[0].driving, true);
+ const input = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/input`, { as: kim, bearer: token, json: { action: 'click', x: 100, y: 120, width: 640, height: 400 }, headers: SAME });
+ assert.strictEqual(input.status, 200, input.text);
+ const got = calls.find((c) => c[0] === 'input');
+ assert.deepStrictEqual([got[1], got[2], got[3], got[4].action, got[4].x, got[4].y, got[4].width, got[4].height], [kim.subject, token, MACHINE, 'click', 100, 120, 640, 400]);
+ let steered = null;
+ const before = t.ctx.engine.steer;
+ t.ctx.engine.steer = async (taskId, note) => { steered = [taskId, note]; return { ok: true }; };
+ try {
+ const back = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/hand-back`, { as: kim, json: { note: 'signed in, carry on' }, headers: SAME });
+ assert.strictEqual(back.status, 200, back.text);
+ assert.strictEqual((await t.get(`/api/v1/tasks/${id}/desks`, { as: kim, bearer: token })).json().desks[0].driving, false);
+ assert.deepStrictEqual(steered, [id, 'signed in, carry on'], 'the note goes to the task as the person\'s message');
+ } finally { t.ctx.engine.steer = before; }
+ const again = await t.get(`/api/v1/tasks/${id}/desks/${encodeURIComponent(`link:${MACHINE}:desk-trains`)}/input`, { as: kim, bearer: token, json: { action: 'click', x: 1, y: 1 }, headers: SAME });
+ assert.strictEqual(again.status, 409, 'not driving: no input');
+ });
+
+ await check('a step that meets a sign-in becomes an Ask whose button opens the viewer in control', async () => {
+ const other = await make();
+ await event(other, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ watch: `https://openvibe.link/machines/${MACHINE}/screen?desk=desk-signin`, running: true, stream: { id: STREAM, visibility: 'private' } }) });
+ await event(other, { step: 'tool_call', tool: 'hand_to_person', call_id: 'h1', args: { machine: MACHINE, what: 'Sign in to your train account.' } });
+ const page = await t.get(`/chat/${other}`, { as: kim });
+ assert.match(page.text, /data-ct-desk-ask/);
+ assert.match(page.text, /Actor needs you:/);
+ assert.match(page.text, /Sign in to your train account\./);
+ assert.match(page.text, /data-ct-takeover>Take over</);
+ });
+
+ await check('a Run desk uses its session still and the Run lease; take over, input and hand back go through the session', async () => {
+ const runId = await make();
+ await event(runId, { step: 'tool_result', tool: 'computer_desk', text: JSON.stringify({ machine: `run:${SESSION}`, running: true, kind: 'browser', where: 'an OpenVibe computer', computer: { session_id: SESSION, machine: 'an OpenVibe desk host', where: 'network' } }), computer: { session_id: SESSION, machine: 'an OpenVibe desk host', where: 'network' } });
+ const listed = (await t.get(`/api/v1/tasks/${runId}/desks`, { as: kim, bearer: token })).json();
+ assert.strictEqual(listed.desks[0].id, `run:${SESSION}`);
+ assert.strictEqual(listed.desks[0].live, null);
+ assert.match(listed.desks[0].still, new RegExp(`/api/v1/tasks/${runId}/still$`));
+ calls.length = 0;
+ const take = await t.get(`/api/v1/tasks/${runId}/desks/${encodeURIComponent(`run:${SESSION}`)}/take-over`, { as: kim, method: 'POST', headers: SAME });
+ assert.strictEqual(take.status, 200, take.text);
+ assert.deepStrictEqual(calls.find((c) => c[0] === 'session').slice(0, 4), ['session', kim.subject, SESSION, 'take']);
+ const input = await t.get(`/api/v1/tasks/${runId}/desks/${encodeURIComponent(`run:${SESSION}`)}/input`, { as: kim, bearer: token, json: { action: 'key', keys: 'Return' }, headers: SAME });
+ assert.strictEqual(input.status, 200, input.text);
+ const sent = calls.filter((c) => c[0] === 'session').at(-1);
+ assert.deepStrictEqual(sent[4], { kind: 'input', value: { action: 'key', keys: 'Return' } });
+ const back = await t.get(`/api/v1/tasks/${runId}/desks/${encodeURIComponent(`run:${SESSION}`)}/hand-back`, { as: kim, json: {}, headers: SAME });
+ assert.strictEqual(back.status, 200, back.text);
+ assert.deepStrictEqual(calls.filter((c) => c[0] === 'session').at(-1).slice(0, 4), ['session', kim.subject, SESSION, 'back']);
+ });
+
+ await check('a finished run has a replay: each step keeps its still, and typed text is never returned', async () => {
+ const doneId = await make('succeeded');
+ await store.updateTask(s, doneId, { finished_at: s.iso(), result: { answer: 'Booked.' } });
+ const typeCall = await event(doneId, { step: 'tool_call', tool: 'computer_input', call_id: 't1', args: { action: 'type', text: 'hunter2', desk: 'private' }, input: 'type' });
+ await stills.save(s, doneId, typeCall.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 1280, height: 800, tool: 'computer_input' });
+ await event(doneId, { step: 'tool_result', tool: 'computer_input', call_id: 't1', text: '{"done":true}' });
+ const clickCall = await event(doneId, { step: 'tool_call', tool: 'computer_input', call_id: 't2', args: { action: 'click', x: 64, y: 32, width: 640, height: 400 }, input: 'click' });
+ await stills.save(s, doneId, clickCall.seq + 1, { format: 'png', data: Buffer.from(PNG, 'base64'), width: 640, height: 400, tool: 'computer_input' });
+ await event(doneId, { step: 'tool_result', tool: 'computer_input', call_id: 't2', text: '{"done":true}' });
+ const r = await t.get(`/api/v1/tasks/${doneId}/replay`, { as: kim });
+ assert.strictEqual(r.status, 200, r.text);
+ const steps = r.json().steps;
+ assert.strictEqual(steps.length, 2);
+ assert.strictEqual(steps[0].secret, true, 'typed text is marked secret');
+ assert.strictEqual(steps[0].action, 'secret');
+ assert.ok(!JSON.stringify(steps).includes('hunter2'), 'the typed text is never returned');
+ assert.strictEqual(steps[1].action, 'click');
+ assert.deepStrictEqual([steps[1].x, steps[1].y, steps[1].width, steps[1].height], [64, 32, 640, 400]);
+ assert.ok(steps[0].still && steps[1].still, 'both steps keep a picture');
+ const shot = await t.get(new URL(steps[0].still.url).pathname, { as: kim });
+ assert.strictEqual(shot.status, 200);
+ assert.match(shot.headers.get('content-type'), /^image\/webp/);
+ assert.strictEqual((await t.get(new URL(steps[0].still.url).pathname, { as: bo })).status, 404, 'another person: no picture');
+ const page = await t.get(`/chat/${doneId}`, { as: kim });
+ assert.match(page.text, /data-ct-replay/);
+ assert.match(page.text, /Replay · 2 steps/);
+ assert.match(page.text, /Clicked at 64, 32/);
+ assert.ok(!page.text.includes('hunter2'), 'the page does not carry the typed text either');
+ });
+ await check('the Live client only mints a person\'s view ticket for an ags_ stream', async () => {
+ const { createLive } = require('../server/agents/live');
+ const asked = [];
+ const live = createLive({ config: { services: { liveUrl: 'https://live.test', livePublicUrl: 'https://openvibe.live' } }, fetchImpl: async (url, init) => {
+ asked.push([url, init.headers.Authorization]);
+ return new Response(JSON.stringify({ view_ticket: 'wvt_x', expires_at: '2030-01-01T00:00:00Z' }), { status: 200, headers: { 'content-type': 'application/json' } });
+ } });
+ const view = await live.viewTicket('person-token', STREAM);
+ assert.match(view.page_url, new RegExp(`https://openvibe\\.live/embed/agent/${STREAM}\\?vt=wvt_x`));
+ assert.deepStrictEqual(asked, [[`https://live.test/api/streams/agent/${STREAM}/view-ticket`, 'Bearer person-token']]);
+ assert.match((await live.viewTicket('person-token', 'not-a-stream')).error, /not an agent stream/);
+ assert.strictEqual(asked.length, 1, 'a malformed id never reaches Live');
+ assert.match((await live.viewTicket(null, STREAM)).error, /own session/);
+ const replayLib = require('../server/tasks/replay');
+ assert.strictEqual(typeof replayLib.steps, 'function');
+ assert.strictEqual(replayLib.caption({ tool: 'computer_screen' }), 'Looked at the screen');
+ assert.strictEqual(require('../server/runtime/tools/computer_desk').name, 'computer_desk');
+ assert.ok(require('fs').existsSync(require('path').join(__dirname, '..', 'public', 'js', 'desks.js')));
+ });
+
+ await check('the page uses the SDK names, pauses a stream off screen, and polls a still at 1/s', async () => {
+ const js = require('fs').readFileSync(require('path').join(__dirname, '..', 'public', 'js', 'desks.js'), 'utf8');
+ assert.match(js, /desks\(run\)/);
+ assert.match(js, /takeOver\(run, desk\)/);
+ assert.match(js, /handBack\(run, desk, note\)/);
+ assert.match(js, /replay\(run\)/);
+ assert.match(js, /IntersectionObserver/);
+ assert.match(js, /stopStream\(st\)/);
+ assert.match(js, /setInterval\(function \(\) \{ show\(img, tile\.getAttribute\('data-still'\)\); \}, 1000\)/);
+ assert.match(js, /data-ct-replay-step/);
+ const css = require('fs').readFileSync(require('path').join(__dirname, '..', 'public', 'css', 'chat.css'), 'utf8');
+ assert.match(css, /\.ct-desk-frame \{[^}]*aspect-ratio: 16 \/ 9/);
+ assert.match(css, /@media \(min-width: 900px\)[\s\S]*\.ct-viewer-side \{/);
+ assert.match(css, /\.ct-viewer-bar button[^}]*min-height: 44px/);
+ });
+
+ await check('the desk stream starts with the desk and stops with the task (Link desk.stream)', async () => {
+ const { startDeskStream, closeDesks } = require('../server/runtime/tools/_person');
+ const core = { pcs: {
+ isSession: () => false,
+ async deskStream(subject, machine, args) { calls.push(['stream', subject, machine, args]); return args.action === 'start' ? { id: STREAM, page_url: 'https://openvibe.live/embed/agent/' + STREAM } : { stopped: true }; },
+ async desk(subject, machine, args) { calls.push(['desk', subject, machine, args]); return { running: false }; },
+ } };
+ const run = { streams: new Map(), used: new Set([MACHINE]), placed: null, name: 'desk-stream-test' };
+ await startDeskStream(run, kim.subject, core, { taskId: id, personText: 'Book the cheapest train to Porto', config: { baseUrl: 'https://openvibe.actor' } }, MACHINE, 'desk-trains', { running: true });
+ const start = calls.find((c) => c[0] === 'stream' && c[3].action === 'start');
+ assert.ok(start, 'the stream started');
+ assert.deepStrictEqual([start[3].desk, start[3].visibility, start[3].task.ref], ['desk-trains', 'private', id]);
+ assert.ok(run.streams.has(MACHINE), 'the run remembers the stream for its end');
+ await closeDesks(run, owner, core);
+ const stop = calls.find((c) => c[0] === 'stream' && c[3].action === 'stop');
+ assert.ok(stop, 'the stream stopped with the task');
+ assert.strictEqual(run.streams.size, 0);
+ });
+
+ } finally { await t.close(); }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });