openvibers/openvibe.live Commits

Agent Streams S4: the agent's voice on Live (#636)

OP OpenVibers committed 6 days ago · 39f50347a8 · 11 files, +1208 −2

* Agent Streams S4: the agent's voice on Live

The narrated mode: Live writes the lines the agent says, one or two sentences, at most 220 characters, in the
stream's personality (calm, hype, dry, roast). The line writer paces them: one line every 20 s at most, the newest
line wins (a queue of two), and a failure line may interrupt the gap once. Every line passes the house filters
before it is written, so a line that trips them is never said and the next step gets a new one.

The lines travel in the stream state the desk polls, and the page shows them as captions; the recording keeps them
as WebVTT. The owner picks the mode, the voice and the personality and mutes the voice from the owner page; a
channel moderator may mute it too, and the mute takes effect on the desk at once. When the person's allowance is
spent the voice pauses and says so.

Tests: test/agent-streams-s4.test.js covers the writer (personality, length, pacing, the newest-wins queue, the
failure interrupt), the filters refusing a slur line, the state and caption feeds, the WebVTT, and the owner-page
controls (mode, personality, the cloned voice's consent, the mute, the moderator's mute, the allowance message).

* Merge remote-tracking branch 'origin/main' into feat/live-agent-streams-s4

* The overlay says which voice is speaking

The design's live label: the overlay's foot now names the voice and the personality (and says so plainly when
the voice is muted or paused), so a viewer can tell what they are hearing.

* Agent Streams S4: the caption in the overlay, and where the WebVTT lives

The design names the overlay captions as a test, so the overlay script is now driven in a DOM
(linkedom, the way the home-page tests do it): the line the agent said is drawn as a caption under
the step, it is text and never markup, and the voice and the personality are labelled.

The state also carries captions_url now: the address of the recording WebVTT. Media has no caption
attachment API yet (captions on the timeline are still open in its own plan), so the VOD player is
told where the track is instead of finding it inside the file.

Changes

ddiff --git a/migrations/0029_agent_stream_voice.sql b/migrations/0029_agent_stream_voice.sql
nnew file mode 100644
iindex 00000000..cd4797e0
---- /dev/null
++++ b/migrations/0029_agent_stream_voice.sql
@@@ -0,0 +1,36 @@
++-- phase: expand
++-- Agent streams, S4 (design 2026-11-06, plan-state-2026-10-22/AGENT-STREAMS-S4-VOICE-DESIGN-2026-11-06.md): the
++-- agent's voice. A stream in narrated mode speaks at moments the agent picks, and the desk makes the sound.
++--
++-- The voice settings live beside the agent stream (not on managed_streams, whose rows go to channel pages):
++-- which Chat voice to use, which personality to write in, whether the voice is muted (the owner's kill switch and
++-- the moderators' one), whether the person has opted into their own cloned voice, and whether the allowance ran
++-- out ("voice paused: allowance spent").
++--
++-- A line is what the desk reads: one or two sentences, at most 220 characters, plain text. The lines are kept so the
++-- overlay can caption them and the recording can carry them as WebVTT.
++--
++-- ADD-only: an older release ignores the columns and the table; nothing else changes.
++ALTER TABLE agent_streams ADD COLUMN IF NOT EXISTS voice text;
++ALTER TABLE agent_streams ADD COLUMN IF NOT EXISTS personality text NOT NULL DEFAULT 'calm' CHECK (personality IN ('calm', 'hype', 'dry', 'roast'));
++-- The kill switch: the owner mutes the voice here, and a channel moderator can mute it too.
++ALTER TABLE agent_streams ADD COLUMN IF NOT EXISTS voice_muted boolean NOT NULL DEFAULT false;
++-- The person's own cloned voice, only with their explicit opt-in (voice_own_consent_at is when they said yes).
++ALTER TABLE agent_streams ADD COLUMN IF NOT EXISTS voice_own boolean NOT NULL DEFAULT false;
++ALTER TABLE agent_streams ADD COLUMN IF NOT EXISTS voice_own_consent_at timestamptz;
++-- Set when the person's allowance ran out; the stream then shows "voice paused: allowance spent".
++ALTER TABLE agent_streams ADD COLUMN IF NOT EXISTS voice_paused text;
++
++CREATE TABLE IF NOT EXISTS agent_stream_lines (
++ id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
++ managed_stream_id bigint NOT NULL REFERENCES managed_streams(id) ON DELETE CASCADE,
++ step_id bigint REFERENCES agent_stream_steps(id) ON DELETE SET NULL,
++ at timestamptz NOT NULL,
++ text text NOT NULL,
++ personality text NOT NULL CHECK (personality IN ('calm', 'hype', 'dry', 'roast')),
++ voice text,
++ kind text NOT NULL DEFAULT 'step' CHECK (kind IN ('step', 'failure', 'result', 'idle')),
++ created_at timestamptz NOT NULL DEFAULT now()
++);
++-- The desk's "lines after this id" poll and the recording's caption read.
++CREATE INDEX IF NOT EXISTS agent_stream_lines_stream ON agent_stream_lines (managed_stream_id, id);
ddiff --git a/public/css/features/agent-stream.css b/public/css/features/agent-stream.css
iindex 4c8880aa..b885331c 100644
---- a/public/css/features/agent-stream.css
++++ b/public/css/features/agent-stream.css
@@@ -78,6 +78,14 @@
.agent-overlay-tick.is-skipped { background: #94a3b8; }
.agent-overlay-tick:hover, .agent-overlay-tick.is-open { opacity: 1; transform: scaleY(1.35); }
.agent-overlay-foot { margin: 1px 0 0; display: flex; flex-wrap: wrap; gap: 4px 12px; color: #c3c8ce; font-size: 0.74rem; }
++/* S4: the caption under the step: the line the agent is saying right now. */
++.agent-overlay-caption { margin: 0; color: #f3f4f6; font-size: 0.8rem; font-style: italic; line-height: 1.25; }
++.agent-overlay-caption[hidden] { display: none; }
++/* S4: the voice label in the overlay's foot: which voice is speaking, and in which personality. */
++.agent-overlay-voice { color: #cfc7ff; font-weight: 600; }
++.agent-overlay-voice[hidden] { display: none; }
++.agent-overlay-caption::before { content: '\201C'; }
++.agent-overlay-caption::after { content: '\201D'; }
.agent-overlay-foot span[hidden] { display: none; }
.agent-overlay-tests { color: #e5e7eb; }
.agent-overlay-fold { position: absolute; right: 6px; bottom: 8px; width: 26px; height: 26px; padding: 0; border: 1px solid rgba(255, 255, 255, 0.22); border-radius: 7px; background: rgba(10, 12, 16, 0.8); color: #e5e7eb; cursor: pointer; font-size: 0.7rem; }
@@@ -100,6 +108,7 @@
@media (max-width: 640px) {
.agent-overlay { padding: 6px 8px; font-size: 0.76rem; }
.agent-overlay-task, .agent-overlay-ticks, .agent-overlay-foot { display: none; }
++ .agent-overlay-caption { font-size: 0.76rem; }
.agent-overlay-body { padding-right: 30px; }
}
ddiff --git a/public/fragments/agent-stream.html b/public/fragments/agent-stream.html
iindex 59b3744a..8d77c624 100644
---- a/public/fragments/agent-stream.html
++++ b/public/fragments/agent-stream.html
@@@ -41,6 +41,39 @@
<button type="button" class="btn btn-outline" data-ov-share-sheet id="agent-share" data-title="An AI agent's screen">Share</button>
<button type="button" class="btn btn-outline" id="agent-rotate" hidden>Reset link</button>
</div>
++ <div class="agent-field">
++ <label class="agent-field-label" for="agent-voice-mode">The agent's voice</label>
++ <select id="agent-voice-mode" class="agent-select">
++ <option value="watch">Watch — the agent stays quiet</option>
++ <option value="narrated">Narrated — the agent says what it is doing</option>
++ </select>
++ </div>
++ <div class="agent-field">
++ <label class="agent-field-label" for="agent-voice-personality">How it says it</label>
++ <select id="agent-voice-personality" class="agent-select">
++ <option value="calm">Calm</option>
++ <option value="hype">Hype</option>
++ <option value="dry">Dry</option>
++ <option value="roast">Roast</option>
++ </select>
++ </div>
++ <div class="agent-field">
++ <label class="agent-field-label" for="agent-voice-pick">Which voice</label>
++ <select id="agent-voice-pick" class="agent-select"></select>
++ <p class="agent-note" id="agent-voice-note"></p>
++ </div>
++ <div class="agent-field">
++ <label class="agent-field-label" for="agent-voice-own">Your own cloned voice</label>
++ <select id="agent-voice-own" class="agent-select">
++ <option value="no">No — one of the site's voices</option>
++ <option value="yes">Yes — my own clone (I consent)</option>
++ </select>
++ <p class="agent-note">Your own clone is only used with your consent, and only for your own streams.</p>
++ </div>
++ <div class="agent-actions">
++ <button type="button" class="btn btn-outline" id="agent-voice-mute">Mute voice</button>
++ </div>
++ <p class="agent-hint" id="agent-voice-state"></p>
<div class="agent-actions">
<button type="button" class="btn btn-outline" id="agent-curtain-btn">Curtain now</button>
<button type="button" class="btn btn-danger" id="agent-end">End</button>
ddiff --git a/public/js/agent-overlay.js b/public/js/agent-overlay.js
iindex 6f73d866..8b1b3cdb 100644
---- a/public/js/agent-overlay.js
++++ b/public/js/agent-overlay.js
@@@ -75,6 +75,9 @@
const body = el('div', 'agent-overlay-body');
const taskLine = el('p', 'agent-overlay-task');
++ // S4: the line the agent is saying right now, drawn as a caption under the step.
++ const captionLine = el('p', 'agent-overlay-caption');
++ captionLine.hidden = true;
const stepLine = el('p', 'agent-overlay-step');
const dot = el('span', 'agent-overlay-dot');
const stepTitle = el('span', 'agent-overlay-step-title');
@@@ -88,14 +91,16 @@
const testsLine = el('span', 'agent-overlay-tests');
const costLine = el('span', 'agent-overlay-cost');
const workedLine = el('span', 'agent-overlay-worked');
-- foot.append(testsLine, costLine, workedLine);
++ // S4: the voice label: which voice is speaking, and in which personality.
++ const voiceLine = el('span', 'agent-overlay-voice');
++ foot.append(testsLine, costLine, workedLine, voiceLine);
const sheet = el('div', 'agent-overlay-sheet');
sheet.hidden = true;
const sheetList = el('ol', 'agent-overlay-sheet-list');
sheet.appendChild(sheetList);
-- body.append(taskLine, stepLine, ticks, foot);
++ body.append(taskLine, stepLine, captionLine, ticks, foot);
root.append(foldBtn, body, sheet);
host.appendChild(root);
@@@ -158,6 +163,21 @@
const data = next || {};
const steps = Array.isArray(data.steps) ? data.steps : lastSteps;
lastSteps = steps;
++ // S4: the caption is the newest spoken line; it fades with the fold like the rest of the band.
++ const caption = data.caption && String(data.caption).trim() ? String(data.caption).trim() : '';
++ captionLine.textContent = caption;
++ captionLine.hidden = !caption;
++ const voice = data.voice || null;
++ if (voice && voice.personality) {
++ const label = voice.muted ? 'Voice muted' : (voice.paused ? 'Voice paused' : voice.personality);
++ root.setAttribute('data-voice', label);
++ voiceLine.textContent = voice.voice ? `${voice.personality} voice · ${voice.voice}` : `${voice.personality} voice`;
++ voiceLine.hidden = false;
++ } else {
++ root.removeAttribute('data-voice');
++ voiceLine.textContent = '';
++ voiceLine.hidden = true;
++ }
const cur = current(steps);
const task = data.task && data.task.title ? data.task.title : (options.title || 'An AI agent is working');
taskLine.textContent = task;
ddiff --git a/public/js/agent-stream.js b/public/js/agent-stream.js
iindex 38e50318..c4668448 100644
---- a/public/js/agent-stream.js
++++ b/public/js/agent-stream.js
@@@ -20,6 +20,10 @@
let pollTimer = null;
let stepsTimer = null;
let steps = []; // the overlay's feed, oldest first (at most the last 200)
++ let lines = []; // S4: the lines the agent said, oldest first (the overlay's caption)
++ let voice = null; // S4: the voice settings (mode, personality, voice, muted, paused)
++ let voicesLoaded = false;
++ let linesTimer = null;
let overlay = null;
let chaptersFor = null; // the stream id whose chapters are rendered (they never change while the page lives)
let generation = 0; // bumped on every page leave, so a slow request from the last page changes nothing
@@@ -34,8 +38,12 @@
if (renewTimer) { clearTimeout(renewTimer); renewTimer = null; }
if (pollTimer) { clearTimeout(pollTimer); pollTimer = null; }
if (stepsTimer) { clearTimeout(stepsTimer); stepsTimer = null; }
++ if (linesTimer) { clearTimeout(linesTimer); linesTimer = null; }
if (overlay) { try { overlay.destroy(); } catch { /* */ } overlay = null; }
steps = [];
++ lines = [];
++ voice = null;
++ voicesLoaded = false;
chaptersFor = null;
state = null;
ticket = null;
@@@ -135,9 +143,44 @@
task: data.stream.task,
workedSeconds: data.stream.worked_seconds,
state: data.stream.state,
++ caption: captionText(),
++ voice,
});
}
++ /** S4: the newest line the agent said, for the overlay's caption. */
++ function captionText() {
++ if (!lines.length) return '';
++ const line = lines[lines.length - 1];
++ return line && line.text ? line.text : '';
++ }
++
++ /**
++ * S4: the lines the agent said. The page polls the same feed the desk reads, so the caption follows the voice.
++ */
++ function scheduleLines(data) {
++ if (linesTimer) { clearTimeout(linesTimer); linesTimer = null; }
++ if (data.stream.state === 'ended') return;
++ const gen = generation;
++ const id = data.stream.id;
++ linesTimer = setTimeout(async () => {
++ if (gen !== generation) return;
++ try {
++ const after = lines.length ? lines[lines.length - 1].id : null;
++ const next = await api(`/streams/agent/${encodeURIComponent(id)}/lines${after ? `?after=${after}` : ''}`);
++ if (gen !== generation) return;
++ if (next && next.voice) { voice = next.voice; renderVoice(); }
++ if (next && Array.isArray(next.lines) && next.lines.length) {
++ const known = new Set(lines.map((line) => line.id));
++ for (const line of next.lines) if (!known.has(line.id)) lines.push(line);
++ if (lines.length > 100) lines = lines.slice(-100);
++ if (state && state.stream) ensureOverlay(state);
++ }
++ } catch { /* the next turn tries again */ }
++ if (gen === generation && state) scheduleLines(state);
++ }, 2500);
++ }
++
/** Keep the overlay's feed current while the work is live: only the steps after the newest one are fetched. */
function scheduleSteps(data) {
if (stepsTimer) { clearTimeout(stepsTimer); stepsTimer = null; }
@@@ -288,7 +331,10 @@
mountPlays(data);
ensureOverlay(data);
renderOwner(data);
++ renderVoice();
++ loadVoices();
scheduleRenew(data);
++ scheduleLines(data);
schedulePoll(stream.id);
if (!steps.length) loadInitialSteps(stream.id).then(() => { if (state && state.stream) { ensureOverlay(state); if (state.stream.state === 'ended') renderRecording(state); } });
scheduleSteps(data);
@@@ -327,12 +373,73 @@
}
}
renderCurtain(!!(stream.curtain && stream.curtain.up));
++ renderVoice();
const endBtn = $('agent-end');
if (endBtn) endBtn.hidden = stream.state === 'ended';
const confirm = $('agent-end-confirm');
if (confirm && stream.state === 'ended') confirm.hidden = true;
}
++ /** S4: the voice panel. The mode, the personality, the voice and the mute, and the allowance message. */
++ function renderVoice() {
++ const panel = $('agent-owner');
++ if (!panel || panel.hidden || !voice) return;
++ const mode = $('agent-voice-mode');
++ if (mode) mode.value = voice.mode === 'narrated' ? 'narrated' : 'watch';
++ const personality = $('agent-voice-personality');
++ if (personality) personality.value = voice.personality || 'calm';
++ const own = $('agent-voice-own');
++ if (own) own.value = voice.own ? 'yes' : 'no';
++ const pick = $('agent-voice-pick');
++ if (pick && voice.voice && !pick.value) pick.value = voice.voice;
++ const mute = $('agent-voice-mute');
++ if (mute) {
++ mute.textContent = voice.muted ? 'Unmute voice' : 'Mute voice';
++ mute.classList.toggle('btn-outline', !voice.muted);
++ mute.classList.toggle('btn-danger', !!voice.muted);
++ }
++ const state = $('agent-voice-state');
++ if (state) {
++ if (voice.paused === 'allowance_spent') {
++ state.textContent = 'voice paused: allowance spent';
++ } else if (voice.muted) {
++ state.textContent = 'The voice is muted.';
++ } else if (voice.mode === 'narrated') {
++ state.textContent = `Narrated, in a ${voice.personality || 'calm'} voice.`;
++ } else {
++ state.textContent = 'The agent is silent (Watch mode).';
++ }
++ }
++ }
++
++ /** S4: the voice list, from Chat's public catalogue. A site without Chat keeps the current voice. */
++ async function loadVoices() {
++ if (voicesLoaded) return;
++ voicesLoaded = true;
++ const pick = $('agent-voice-pick');
++ if (!pick) return;
++ try {
++ const res = await fetch('/api/tts/voices', { headers: { Accept: 'application/json' } });
++ if (!res.ok) throw new Error('no voices');
++ const data = await res.json();
++ const list = Array.isArray(data && data.voices) ? data.voices : [];
++ pick.replaceChildren();
++ for (const entry of list) {
++ const id = typeof entry === 'string' ? entry : entry && entry.id;
++ if (!id) continue;
++ const name = (typeof entry === 'object' && entry.name) || id;
++ const option = document.createElement('option');
++ option.value = String(id);
++ option.textContent = String(name);
++ pick.appendChild(option);
++ }
++ if (voice && voice.voice) pick.value = voice.voice;
++ } catch {
++ const note = $('agent-voice-note');
++ if (note) note.textContent = 'The voice list is unavailable right now.';
++ }
++ }
++
function scheduleRenew(data) {
if (renewTimer) { clearTimeout(renewTimer); renewTimer = null; }
if (!ticket || data.stream.visibility === 'public') return;
@@@ -396,6 +503,45 @@
}
});
++ // S4: the voice controls. Every change is one PUT; the mute is one POST (a moderator may do it too).
++ const saveVoice = async (body) => {
++ const id = idOf();
++ if (!id) return;
++ try {
++ const next = await api(`/streams/agent/${encodeURIComponent(id)}/voice`, { method: 'PUT', body });
++ if (next && next.voice) { voice = next.voice; renderVoice(); }
++ } catch (err) {
++ const hint = $('agent-voice-state');
++ if (hint) hint.textContent = (err && err.message) || 'That did not save.';
++ }
++ };
++
++ $('agent-voice-mode').addEventListener('change', (event) => {
++ saveVoice({ mode: event.currentTarget.value });
++ });
++ $('agent-voice-personality').addEventListener('change', (event) => {
++ saveVoice({ personality: event.currentTarget.value });
++ });
++ $('agent-voice-pick').addEventListener('change', (event) => {
++ saveVoice({ voice: event.currentTarget.value || null });
++ });
++ $('agent-voice-own').addEventListener('change', (event) => {
++ // The person's own cloned voice: the consent flag travels with it, so it can never be switched on quietly.
++ saveVoice({ own: event.currentTarget.value === 'yes', own_consent: event.currentTarget.value === 'yes' });
++ });
++ $('agent-voice-mute').addEventListener('click', async () => {
++ const id = idOf();
++ if (!id) return;
++ const muted = !(voice && voice.muted);
++ try {
++ const next = await api(`/streams/agent/${encodeURIComponent(id)}/voice/mute`, { method: 'POST', body: { muted } });
++ if (next && next.voice) { voice = next.voice; renderVoice(); }
++ } catch (err) {
++ const hint = $('agent-voice-state');
++ if (hint) hint.textContent = (err && err.message) || 'That did not save.';
++ }
++ });
++
$('agent-curtain-btn').addEventListener('click', async () => {
const id = idOf();
if (!id) return;
ddiff --git a/server/agent-streams/routes.js b/server/agent-streams/routes.js
iindex ea4282cc..ba3fa52d 100644
---- a/server/agent-streams/routes.js
++++ b/server/agent-streams/routes.js
@@@ -25,6 +25,7 @@ const service = require('./service');
const tickets = require('./tickets');
const frames = require('./frames');
const steps = require('./steps');
++const voice = require('./voice');
const router = express.Router();
@@@ -299,6 +300,12 @@ router.get('/:id/state', deskOrOwner, async (req, res) => {
...(Number(row.curtain_up) && row.curtain_reason ? { reason: row.curtain_reason } : {}),
},
ends_at: service.iso(await tickets.currentExpiry(row.managed_stream_id)),
++ // S4: the lines the desk has not read yet (it polls with the last id it spoke), and the voice it speaks
++ // them in. The state contract's v2 shape carries both; Live never mixes the audio, the desk does.
++ lines: await voice.linesAfter(row.managed_stream_id, req.query.after, 20),
++ voice: voice.voiceState(row),
++ // S4: where the recording's captions live, so a VOD player can attach the WebVTT as a text track.
++ captions_url: voice.captionsUrl(row),
});
} catch (err) {
console.error('[AgentStream] state failed:', err.message);
@@@ -389,6 +396,9 @@ router.get('/:id/steps', auth.viewer, async (req, res) => {
router.post('/:id/steps', deskOrOwner, async (req, res) => {
try {
const step = await steps.add(req.agent, req.body || {});
++ // S4: in narrated mode the voice writes a line about this step. A refused line (the house filters) or a spent
++ // allowance is silent, never an error: the step itself is what the caller came for.
++ try { await voice.writer.onStep(req.agent, step); } catch (err) { console.warn('[AgentStream] voice line failed:', err.message); }
res.status(201).json({ step });
} catch (err) {
if (err && err.status) return fail(res, err.status, err.code, err.message);
@@@ -473,4 +483,123 @@ router.post('/:id/view-ticket', auth.viewer, async (req, res) => {
}
});
++// ── The agent's voice (S4) ───────────────────────────────────
++
++// The owner chooses the mode (Watch or Narrated), the voice, the personality and the mute; a channel moderator can
++// mute it too. A cloned voice is the person's own and only with their explicit opt-in, so `own` needs `own_consent`.
++// The lines themselves are read by the desk (the state poll) and shown to the page as captions.
++function cleanVoiceId(value) {
++ if (value === null || value === '') return null;
++ if (typeof value !== 'string') return undefined;
++ const id = value.trim();
++ return /^[A-Za-z0-9_.:-]{1,64}$/.test(id) ? id : undefined;
++}
++
++// The voice state: the mode, the personality, the voice, the mute and (when the allowance ran out) the message.
++router.get('/:id/voice', auth.viewer, async (req, res) => {
++ try {
++ const row = await load(req, res);
++ if (!row) return;
++ if (!await watchable(req, row)) return fail(res, 404, 'agent_stream_not_found', 'No such agent stream');
++ res.set('Cache-Control', 'no-store');
++ res.json({ voice: voice.voiceState(row) });
++ } catch (err) {
++ console.error('[AgentStream] voice read failed:', err.message);
++ fail(res, 500, 'voice_failed', 'The voice could not be read');
++ }
++});
++
++router.put('/:id/voice', auth.publisher, async (req, res) => {
++ try {
++ const row = await load(req, res);
++ if (!row) return;
++ if (Number(row.user_id) !== Number(req.publisher.user.id)) return fail(res, 403, 'not_your_stream', 'This agent stream belongs to another account');
++ const body = req.body || {};
++ const set = {};
++ if (body.mode !== undefined) {
++ if (!['watch', 'narrated'].includes(body.mode)) return fail(res, 400, 'invalid_mode', 'mode is watch or narrated');
++ set.mode = body.mode;
++ }
++ if (body.personality !== undefined) {
++ if (!voice.PERSONALITIES.includes(body.personality)) return fail(res, 400, 'invalid_personality', 'personality is calm, hype, dry or roast');
++ set.personality = body.personality;
++ }
++ if (body.voice !== undefined) {
++ const id = cleanVoiceId(body.voice);
++ if (id === undefined) return fail(res, 400, 'invalid_voice', 'That voice name is not allowed');
++ set.voice = id;
++ }
++ if (body.own !== undefined) {
++ const own = !!body.own;
++ // The person's own cloned voice: only with their explicit opt-in, and the moment they said yes is kept.
++ if (own && body.own_consent !== true) return fail(res, 422, 'own_voice_needs_consent', 'Your own cloned voice needs your consent');
++ set.voice_own = own;
++ set.voice_own_consent_at = own ? new Date() : null;
++ }
++ if (body.muted !== undefined) set.voice_muted = !!body.muted;
++ if (!Object.keys(set).length) return fail(res, 400, 'nothing_to_change', 'Nothing to change');
++ // A new choice gets a new try at the allowance: the pause was about the old one.
++ if (set.mode || set.personality || set.voice) set.voice_paused = null;
++ const next = await service.update(row, { set });
++ res.json({ voice: voice.voiceState(next) });
++ } catch (err) {
++ console.error('[AgentStream] voice settings failed:', err.message);
++ fail(res, 500, 'voice_settings_failed', 'The voice settings could not be saved');
++ }
++});
++
++// The mute: the owner's kill switch, and a channel moderator's. It takes effect within a second (the desk checks
++// it on every poll and stops speaking at once).
++router.post('/:id/voice/mute', auth.viewer, async (req, res) => {
++ try {
++ const row = await load(req, res);
++ if (!row) return;
++ const body = req.body || {};
++ if (typeof body.muted !== 'boolean') return fail(res, 400, 'invalid_muted', 'muted is true or false');
++ const isOwner = !!(req.viewer && Number(req.viewer.id) === Number(row.user_id));
++ let allowed = isOwner;
++ if (!allowed) {
++ const permissions = require('../auth/permissions');
++ allowed = await permissions.canModerateChannel(req.viewer, row.channel_id);
++ }
++ if (!allowed) return fail(res, 403, 'cannot_mute_voice', 'You cannot mute this voice');
++ const next = await service.update(row, { set: { voice_muted: body.muted } });
++ res.json({ voice: voice.voiceState(next) });
++ } catch (err) {
++ console.error('[AgentStream] voice mute failed:', err.message);
++ fail(res, 500, 'voice_mute_failed', 'The voice could not be muted');
++ }
++});
++
++// The lines the page draws as captions (the desk has its own copy in the state poll).
++router.get('/:id/lines', auth.viewer, async (req, res) => {
++ try {
++ const row = await load(req, res);
++ if (!row) return;
++ if (!await watchable(req, row)) return fail(res, 404, 'agent_stream_not_found', 'No such agent stream');
++ const lines = await voice.linesAfter(row.managed_stream_id, req.query.after, 50);
++ res.set('Cache-Control', 'no-store');
++ res.json({ lines, voice: voice.voiceState(row) });
++ } catch (err) {
++ console.error('[AgentStream] lines read failed:', err.message);
++ fail(res, 500, 'lines_failed', 'The lines could not be read');
++ }
++});
++
++// The recording's captions as WebVTT: the same lines, timed from the start of the stream.
++router.get('/:id/captions.vtt', auth.viewer, async (req, res) => {
++ try {
++ const row = await load(req, res);
++ if (!row) return;
++ if (!await watchable(req, row)) return fail(res, 404, 'agent_stream_not_found', 'No such agent stream');
++ const cues = await voice.captions(row.managed_stream_id, row.started_at);
++ res.set('Content-Type', 'text/vtt; charset=utf-8');
++ res.set('Cache-Control', 'no-store');
++ res.send(voice.webvtt(cues));
++ } catch (err) {
++ console.error('[AgentStream] captions failed:', err.message);
++ fail(res, 500, 'captions_failed', 'The captions could not be read');
++ }
++});
++
module.exports = router;
ddiff --git a/server/agent-streams/service.js b/server/agent-streams/service.js
iindex 8c61c3e8..574f8d7c 100644
---- a/server/agent-streams/service.js
++++ b/server/agent-streams/service.js
@@@ -74,6 +74,7 @@ const SELECT = `
ms.created_at AS slot_created_at,
a.task, a.visibility, a.mode, a.state, a.curtain_up, a.curtain_since, a.curtain_reason,
a.started_at, a.ended_at, a.worked_seconds, a.last_frame_path, a.last_frame_at, a.vod_id, a.allow_sites,
++ a.voice, a.personality, a.voice_muted, a.voice_own, a.voice_own_consent_at, a.voice_paused,
a.unlisted_secret_hash,
(SELECT s.id FROM streams s WHERE s.managed_stream_id = ms.id AND s.is_live = 1 ORDER BY s.started_at DESC LIMIT 1) AS live_session_id,
(SELECT st.title FROM agent_stream_steps st WHERE st.managed_stream_id = ms.id ORDER BY st.id DESC LIMIT 1) AS latest_step_title,
ddiff --git a/server/agent-streams/sweep.js b/server/agent-streams/sweep.js
iindex 65c2d3d3..f7f02da0 100644
---- a/server/agent-streams/sweep.js
++++ b/server/agent-streams/sweep.js
@@@ -35,6 +35,18 @@ async function sweep() {
}
}
++ // S4: a narrated stream that has gone quiet says something small on its own (the voice's own idle gap rules).
++ try {
++ const voice = require('./voice');
++ const quiet = await db.all(`SELECT ms.id AS managed_stream_id FROM agent_streams a
++ JOIN managed_streams ms ON ms.id = a.managed_stream_id
++ WHERE a.state = 'live' AND a.mode = 'narrated' AND a.voice_muted = false LIMIT 50`);
++ for (const q of quiet) {
++ const row = await service.getByManagedStreamId(q.managed_stream_id);
++ if (row) await voice.writer.maybeIdle(row);
++ }
++ } catch (err) { console.warn('[AgentStream] idle voice failed:', err.message); }
++
try { await tickets.prune(); } catch (err) { console.warn('[AgentStream] ticket prune failed:', err.message); }
try {
const sharing = require('./sharing');
ddiff --git a/server/agent-streams/voice.js b/server/agent-streams/voice.js
nnew file mode 100644
iindex 00000000..3b51877d
---- /dev/null
++++ b/server/agent-streams/voice.js
@@@ -0,0 +1,361 @@
++'use strict';
++/**
++ * The agent's voice (Agent Streams S4, design 2026-11-06).
++ *
++ * In narrated mode the agent says something now and then: a step starts, a step fails, a result lands, and every
++ * few minutes a short line when nothing else was said. This module writes those lines; the desk (OpenVibe.Node) makes
++ * the sound, so Live never mixes audio.
++ *
++ * The rules that hold for every personality:
++ *
++ * - one or two sentences, at most 220 characters, taken from the step (title, status, tests) and the task;
++ * - the house text filters have the last word: a line that trips them is not spoken, and the next step gets a
++ * new line (the filter never rewrites a line, it refuses it);
++ * - at most one line every 20 s; if lines pile up the newest one wins (a queue of two, the older one is
++ * dropped); a failure line may interrupt the 20 s gap once;
++ * - the lines run on the person's own AI allowance, under their daily ceiling. When the allowance is spent the
++ * stream pauses its voice and says so.
++ *
++ * The writer is deliberately small and injectable (now, generate) so the pacing and the filters can be tested
++ * without a model and without a clock.
++ */
++const db = require('../db/database');
++const llm = require('../ai/llm');
++const wordFilter = require('../chat/word-filter');
++const moderation = require('../chat/moderation-utils');
++
++const MAX_CHARS = 220;
++const GAP_MS = 20000;
++const QUEUE_MAX = 2;
++// How long the work may be quiet before the agent says something small on its own.
++const IDLE_MS = 5 * 60 * 1000;
++const PERSONALITIES = ['calm', 'hype', 'dry', 'roast'];
++const SOURCE = 'agent_voice';
++const PAUSED_ALLOWANCE = 'allowance_spent';
++const PAUSED_MESSAGE = 'voice paused: allowance spent';
++// Where the public Live lives: the captions address has to be absolute, because the VOD player is another app.
++const PUBLIC_BASE = (process.env.LIVE_PUBLIC_ORIGIN || 'https://openvibe.live').replace(/\/+$/, '');
++
++/** How each personality says a thing. The rules below are added to every one of them. */
++const STYLE = {
++ calm: 'Calm and plain. You are the person\u2019s agent, working next to them. One or two sentences, no excitement, no jokes.',
++ hype: 'Hype. You are all in: every result is a win worth shouting about, short sentences, an exclamation at the end.',
++ dry: 'Dry. Deadpan and brief, as if the work is mildly interesting and you would rather be doing something else.',
++ roast: 'Roast. Savage and crude: you swear, you are dark about the chat\u2019s takes, the code and your own mistakes. You are still on the person\u2019s side.',
++};
++
++/** The rules that hold in every personality: no slurs, no hate speech against a protected group, no doxxing. */
++const RULES = [
++ 'You speak for the person whose agent you are.',
++ 'One or two sentences. At most 220 characters. No emoji, no markdown, no stage directions, no quotation marks around the whole line.',
++ 'Never use a slur, never make a joke about a protected group, and never doxx anybody: no real names, no handles, no addresses, no phone numbers, no emails.',
++ 'Say what happened: the step, its status, the tests, the task. Do not invent results.',
++].join(' ');
++
++function personalityOf(row) {
++ const p = row && row.personality;
++ return PERSONALITIES.includes(p) ? p : 'calm';
++}
++
++function narrated(row) { return !!row && row.mode === 'narrated'; }
++function muted(row) { return !!row && !!Number(row.voice_muted); }
++
++/** Doxxing: an email address, a phone number, a street address or a handle. */
++function doxxes(text) {
++ if (/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/.test(text)) return true;
++ if (/\+?\d[\d\s().-]{7,}\d/.test(text)) return true;
++ if (/\b\d{1,5}\s+[A-Z][a-z]+\s+(street|st|avenue|ave|road|rd|lane|drive|dr|boulevard|blvd)\b/i.test(text)) return true;
++ if (/@\w{3,}/.test(text)) return true;
++ return false;
++}
++
++/** The house text filters, the same ones chat runs, plus the no-doxxing rule. They refuse a line; they never rewrite one. */
++function allowed(text) {
++ if (!text) return false;
++ try { if (!wordFilter.check(text).safe) return false; } catch { return false; }
++ try { if (moderation.containsCoreSlur(text)) return false; } catch { return false; }
++ try { if (moderation.friendlyFilterCategory(text)) return false; } catch { return false; }
++ return !doxxes(text);
++}
++
++/** At most 220 characters, cut at a sentence end when there is one, otherwise at a word. */
++function clamp(text) {
++ const clean = String(text || '').replace(/[\u0000-\u001f\u007f]/g, ' ').replace(/\s+/g, ' ').trim();
++ if (!clean) return '';
++ if (Array.from(clean).length <= MAX_CHARS) return clean;
++ const cut = Array.from(clean).slice(0, MAX_CHARS).join('');
++ const stop = Math.max(cut.lastIndexOf('. '), cut.lastIndexOf('! '), cut.lastIndexOf('? '));
++ if (stop > 40) return cut.slice(0, stop + 1).trim();
++ const space = cut.lastIndexOf(' ');
++ return (space > 40 ? cut.slice(0, space) : cut).trim();
++}
++
++/** What the writer knows about the stream: the task and the step that just happened. */
++function factsFor(row, step) {
++ const task = taskOf(row);
++ const title = typeof task.title === 'string' ? task.title : '';
++ const lines = [];
++ if (title) lines.push('The task: ' + title);
++ if (step) {
++ lines.push('The step: ' + (step.title || '') + (step.status ? ' (' + step.status + ')' : ''));
++ if (step.detail) lines.push('What happened: ' + step.detail);
++ if (step.tests && (step.tests.passed != null || step.tests.failed != null)) {
++ lines.push('The tests: ' + (step.tests.passed || 0) + ' passing, ' + (step.tests.failed || 0) + ' failing');
++ }
++ }
++ return lines.join('\n');
++}
++
++/** The task of a stream, whether the database gave it as an object or as JSON text. */
++function taskOf(row) {
++ const raw = row && row.task;
++ if (!raw) return {};
++ if (typeof raw === 'object') return raw;
++ try { const parsed = JSON.parse(String(raw)); return parsed && typeof parsed === 'object' ? parsed : {}; } catch { return {}; }
++}
++
++/** The facts an idle line needs: the task, and the fact that nothing new happened. */
++function idleFacts(row) {
++ const task = taskOf(row);
++ const title = typeof task.title === 'string' ? task.title : '';
++ const lines = [];
++ if (title) lines.push('The task: ' + title);
++ lines.push('Nothing new to report right now.');
++ return lines.join('\n');
++}
++
++function kindOf(step) {
++ if (!step) return 'step';
++ if (step.status === 'failed') return 'failure';
++ if (step.status === 'passed') return 'result';
++ return 'step';
++}
++
++/** The person's own allowance: their daily ceiling in cents against what the voice has spent today. */
++async function allowanceSpent(userId) {
++ if (!userId) return false;
++ let cap = 0;
++ try {
++ const cfg = await db.getChannelAiConfig(userId);
++ cap = Number(cfg && cfg.daily_budget_cents) || 0;
++ } catch { return false; }
++ if (cap <= 0) return false;
++ let spent = 0;
++ try { spent = Number(await db.getAiCostTodayForUser(userId, SOURCE)) || 0; } catch { return false; }
++ return spent * 100 >= cap;
++}
++
++/** The voice as the owner page, the desk and the overlay see it. */
++function voiceState(row) {
++ const paused = row && row.voice_paused ? String(row.voice_paused) : null;
++ return {
++ mode: (row && row.mode) || 'watch',
++ personality: personalityOf(row),
++ voice: (row && row.voice) || null,
++ muted: muted(row),
++ own: !!(row && Number(row.voice_own)),
++ own_consent_at: row && row.voice_own_consent_at ? new Date(row.voice_own_consent_at).toISOString() : null,
++ paused,
++ ...(paused === PAUSED_ALLOWANCE ? { message: PAUSED_MESSAGE } : {}),
++ };
++}
++
++class Writer {
++ /**
++ * @param {object} opts
++ * now the clock (ms), so the pacing can be tested
++ * generate the line writer: (row, step) -> Promise<string|null>
++ */
++ constructor(opts = {}) {
++ this.now = opts.now || (() => Date.now());
++ this.generate = opts.generate || defaultGenerate;
++ this.queues = new Map();
++ }
++
++ queue(key) {
++ let q = this.queues.get(key);
++ if (!q) { q = { pending: [], lastEmitAt: 0, interruptUsed: false }; this.queues.set(key, q); }
++ return q;
++ }
++
++ /** The lines waiting to be said for a stream (its own queue, for the tests and the desk's view). */
++ pending(managedStreamId) {
++ const q = this.queues.get(managedStreamId);
++ return q ? q.pending.slice() : [];
++ }
++
++ /**
++ * Write a line for a step. Returns the line that was written, or null when nothing was written (not narrated,
++ * muted, the allowance is spent, or the house filters refused it).
++ */
++ async onStep(row, step) {
++ if (!narrated(row) || muted(row)) return null;
++ if (await allowanceSpent(row.user_id)) {
++ await pause(row, PAUSED_ALLOWANCE);
++ return null;
++ }
++ const kind = kindOf(step);
++ let text = null;
++ try { text = await this.generate(row, step, kind); } catch { return null; }
++ const line = clamp(text);
++ if (!line || !allowed(line)) return null;
++ const q = this.queue(row.managed_stream_id);
++ q.pending.push({ text: line, kind, at: new Date(this.now()).toISOString() });
++ // A queue of two: the oldest line is dropped, so the newest one is the one that gets said.
++ while (q.pending.length > QUEUE_MAX) q.pending.shift();
++ return await this.tick(row);
++ }
++
++ /**
++ * The occasional idle line: when the work has gone quiet for a while, say something small. The queue's own gap
++ * still rules, so an idle line never jumps ahead of a real one.
++ */
++ async maybeIdle(row) {
++ if (!narrated(row) || muted(row)) return null;
++ const q = this.queue(row.managed_stream_id);
++ if (this.now() - q.lastEmitAt < IDLE_MS) return null;
++ if (await allowanceSpent(row.user_id)) { await pause(row, PAUSED_ALLOWANCE); return null; }
++ let text = null;
++ try { text = await this.generate(row, null, 'idle'); } catch { return null; }
++ const line = clamp(text);
++ if (!line || !allowed(line)) return null;
++ q.pending.push({ text: line, kind: 'idle', at: new Date(this.now()).toISOString() });
++ return await this.tick(row);
++ }
++
++ /**
++ * Say the next line if it is time: at most one line every 20 s, the newest queued line wins, and a failure
++ * line may interrupt the gap once.
++ */
++ async tick(row) {
++ if (!narrated(row) || muted(row)) return null;
++ const q = this.queue(row.managed_stream_id);
++ if (!q.pending.length) return null;
++ const now = this.now();
++ const newest = q.pending[q.pending.length - 1];
++ // Nothing said yet means the first line is due at once; after that the 20 s gap rules.
++ const due = !q.lastEmitAt || now - q.lastEmitAt >= GAP_MS;
++ const interrupting = newest.kind === 'failure' && !q.interruptUsed;
++ if (!due && !interrupting) return null;
++ if (interrupting) q.interruptUsed = true;
++ q.pending = [];
++ q.lastEmitAt = now;
++ return await save(row, newest);
++ }
++}
++
++async function pause(row, reason) {
++ try {
++ await db.run('UPDATE agent_streams SET voice_paused = ?, updated_at = now() WHERE managed_stream_id = ?', [reason, row.managed_stream_id]);
++ row.voice_paused = reason;
++ } catch { /* the pause is best-effort; the next step tries again */ }
++}
++
++/** Keep a line for the overlay's caption, the recording's WebVTT and the desk. */
++async function save(row, line) {
++ const out = {
++ id: null,
++ at: line.at,
++ text: line.text,
++ kind: line.kind,
++ personality: personalityOf(row),
++ voice: (row && row.voice) || null,
++ };
++ try {
++ const saved = await db.get(
++ 'INSERT INTO agent_stream_lines (managed_stream_id, at, text, personality, voice, kind) VALUES (?, ?, ?, ?, ?, ?) RETURNING id',
++ [row.managed_stream_id, out.at, out.text, out.personality, out.voice, out.kind]);
++ out.id = saved ? Number(saved.id) : null;
++ } catch { /* the line is still said; the caption is lost */ }
++ return out;
++}
++
++/** The lines the desk has not read yet: the newest ones after an id. */
++async function linesAfter(managedStreamId, afterId, limit = 20) {
++ const rows = await db.all(
++ 'SELECT id, at, text, kind, personality, voice FROM agent_stream_lines WHERE managed_stream_id = ? AND id > ? ORDER BY id LIMIT ?',
++ [managedStreamId, Number(afterId) || 0, Math.min(50, Math.max(1, Number(limit) || 20))]);
++ return rows.map((r) => ({
++ id: Number(r.id),
++ at: r.at instanceof Date ? r.at.toISOString() : String(r.at),
++ text: r.text,
++ kind: r.kind,
++ personality: r.personality,
++ voice: r.voice || null,
++ }));
++}
++
++/**
++ * Where the recording's captions live: the WebVTT a player can attach as a text track. Media has no caption
++ * attachment API yet (captions on the timeline are still open in its own plan), so Live hands out the track's
++ * address instead of the file riding inside the VOD.
++ */
++function captionsUrl(row) {
++ if (!row || !row.public_id) return null;
++ return `${PUBLIC_BASE}/api/streams/agent/${encodeURIComponent(row.public_id)}/captions.vtt`;
++}
++
++/** The lines of a recording, as WebVTT: the captions the VOD carries. */
++async function captions(managedStreamId, startedAt) {
++ const rows = await db.all(
++ 'SELECT at, text FROM agent_stream_lines WHERE managed_stream_id = ? ORDER BY id',
++ [managedStreamId]);
++ const base = startedAt ? new Date(startedAt).getTime() : 0;
++ const cues = [];
++ for (const r of rows) {
++ const at = r.at instanceof Date ? r.at.getTime() : Date.parse(String(r.at));
++ const start = Math.max(0, (at - base) / 1000);
++ cues.push({ start, text: r.text });
++ }
++ return cues;
++}
++
++/** A WebVTT file for the recording: the lines, timed from the start of the stream. */
++function webvtt(cues) {
++ const stamp = (seconds) => {
++ const s = Math.max(0, seconds);
++ const h = Math.floor(s / 3600);
++ const m = Math.floor((s % 3600) / 60);
++ const sec = s % 60;
++ const pad = (n, width) => String(n).padStart(width, '0');
++ return pad(h, 2) + ':' + pad(m, 2) + ':' + pad(sec.toFixed(3), 6);
++ };
++ const out = ['WEBVTT', ''];
++ cues.forEach((cue, index) => {
++ const end = cue.end != null ? cue.end : cue.start + 4;
++ out.push(String(index + 1));
++ out.push(stamp(cue.start) + ' --> ' + stamp(end));
++ out.push(cue.text);
++ out.push('');
++ });
++ return out.join('\n');
++}
++
++/** The line itself: one call to OpenVibe.AI, metered on the person's own allowance. */
++async function defaultGenerate(row, step, kind = 'step') {
++ const facts = kind === 'idle' ? idleFacts(row) : factsFor(row, step);
++ if (!facts) return null;
++ // The economy preset: the router's prose route with a small ceiling, metered on the person's allowance.
++ const result = await llm.complete({
++ role: 'summary',
++ kind: 'agent_voice',
++ source: SOURCE,
++ ownerUserId: row.user_id,
++ system: STYLE[personalityOf(row)] + ' ' + RULES,
++ user: facts,
++ maxTokens: 120,
++ temperature: 0.8,
++ timeoutMs: 20000,
++ });
++ if (!result || !result.text) return null;
++ return result.text;
++}
++
++const writer = new Writer();
++
++module.exports = {
++ MAX_CHARS, GAP_MS, QUEUE_MAX, IDLE_MS, PERSONALITIES, SOURCE, PAUSED_ALLOWANCE, PAUSED_MESSAGE,
++ Writer, writer,
++ allowed, clamp, doxxes, factsFor, idleFacts, kindOf, personalityOf, voiceState, allowanceSpent, pause, taskOf,
++ linesAfter, captions, captionsUrl, webvtt,
++};
ddiff --git a/test/agent-overlay-caption.test.js b/test/agent-overlay-caption.test.js
nnew file mode 100644
iindex 00000000..f26273a2
---- /dev/null
++++ b/test/agent-overlay-caption.test.js
@@@ -0,0 +1,129 @@
++/**
++ * Agent Streams S4: the captions in the overlay (2026-11-06).
++ *
++ * The design names this test: 'the captions in the overlay and as WebVTT'. The WebVTT half is in
++ * test/agent-streams-s4.test.js (the captions.vtt route). This file runs the real overlay script in a DOM
++ * (linkedom, as the home-page tests do) and checks that the line the agent said is drawn as a caption, that
++ * the caption is text and never markup, and that the voice and the personality are labelled.
++ *
++ * node test/agent-overlay-caption.test.js
++ */
++'use strict';
++const assert = require('assert');
++const fs = require('fs');
++const path = require('path');
++const vm = require('vm');
++
++// linkedom is a declared dependency of openvibe-shared; resolve it the way the home-page tests do.
++function loadLinkedom() {
++ try { return require('linkedom'); }
++ catch { return require(require.resolve('linkedom', { paths: [path.dirname(require.resolve('openvibe-shared/package.json'))] })); }
++}
++const { parseHTML } = loadLinkedom();
++
++const overlayScript = fs.readFileSync(path.join(__dirname, '../public/js/agent-overlay.js'), 'utf8');
++const pageScript = fs.readFileSync(path.join(__dirname, '../public/js/agent-stream.js'), 'utf8');
++const css = fs.readFileSync(path.join(__dirname, '../public/css/features/agent-stream.css'), 'utf8');
++
++let failures = 0;
++function check(name, fn) {
++ try { fn(); console.log(' ok', name); }
++ catch (e) { failures++; console.log(' x', name, '\n ', e.message); }
++}
++
++/** The overlay script in a DOM: the file the page loads, in a window with a stage to draw on. */
++function openOverlay() {
++ const { window } = parseHTML('<!doctype html><html><body><div id="stage"></div></body></html>');
++ const store = new Map();
++ const localStorage = {
++ getItem: (k) => (store.has(k) ? store.get(k) : null),
++ setItem: (k, v) => { store.set(k, String(v)); },
++ removeItem: (k) => { store.delete(k); },
++ clear: () => store.clear(),
++ key: (i) => [...store.keys()][i] || null,
++ get length() { return store.size; },
++ };
++ const context = vm.createContext({ window, document: window.document, localStorage, console });
++ vm.runInContext(overlayScript, context);
++ const host = window.document.getElementById('stage');
++ const overlay = window.AgentOverlay.create({ host, title: 'An AI agent is working' });
++ return { window, overlay, host };
++}
++
++const STEPS = [{ title: 'Run the test suite', status: 'passed', detail: '5 of 5' }];
++
++check('the line the agent said is drawn as a caption under the step', () => {
++ const { overlay, window } = openOverlay();
++ overlay.update({ steps: STEPS, caption: 'Tests passed. Nothing on fire.', voice: { personality: 'dry', voice: 'gary' } });
++ const caption = window.document.querySelector('.agent-overlay-caption');
++ assert.ok(caption, 'the overlay has a caption element');
++ assert.strictEqual(caption.textContent, 'Tests passed. Nothing on fire.');
++ assert.strictEqual(caption.hidden, false, 'the caption is shown while there is a line');
++});
++
++check('no line, no caption', () => {
++ const { overlay, window } = openOverlay();
++ overlay.update({ steps: STEPS, caption: '' });
++ const caption = window.document.querySelector('.agent-overlay-caption');
++ assert.ok(caption, 'the overlay has a caption element');
++ assert.strictEqual(caption.hidden, true, 'the caption is hidden when nothing was said');
++ assert.strictEqual(caption.textContent, '', 'and it holds no text');
++});
++
++check('the caption is text, never markup', () => {
++ const { overlay, window } = openOverlay();
++ const nasty = 'im <img src=x onerror=alert(1)> fine';
++ overlay.update({ steps: STEPS, caption: nasty });
++ const caption = window.document.querySelector('.agent-overlay-caption');
++ assert.strictEqual(caption.textContent, nasty, 'the line is shown as written');
++ assert.strictEqual(caption.children.length, 0, 'nothing was turned into an element');
++});
++
++check('the caption sits with the steps, not in the timeline sheet', () => {
++ const { overlay, window } = openOverlay();
++ overlay.update({ steps: STEPS, caption: 'A line.' });
++ const body = window.document.querySelector('.agent-overlay-body');
++ const kids = Array.from(body.children).map((c) => c.className);
++ assert.ok(kids.indexOf('agent-overlay-caption') > kids.indexOf('agent-overlay-step'), 'the caption is under the step');
++});
++
++check('the overlay labels the voice and the personality', () => {
++ const { overlay, window } = openOverlay();
++ overlay.update({ steps: STEPS, caption: 'Still going.', voice: { personality: 'roast', voice: 'onyx' } });
++ const label = window.document.querySelector('.agent-overlay-voice');
++ assert.ok(label, 'the overlay has a voice label');
++ assert.strictEqual(label.textContent, 'roast voice · onyx');
++ assert.strictEqual(window.document.querySelector('.agent-overlay').getAttribute('data-voice'), 'roast');
++});
++
++check('a muted voice says so, and a paused one says so too', () => {
++ const { overlay, window } = openOverlay();
++ overlay.update({ steps: STEPS, caption: 'Quiet now.', voice: { personality: 'calm', voice: 'aria', muted: true } });
++ assert.strictEqual(window.document.querySelector('.agent-overlay').getAttribute('data-voice'), 'Voice muted');
++ overlay.update({ steps: STEPS, caption: 'Quiet now.', voice: { personality: 'calm', voice: 'aria', paused: true } });
++ assert.strictEqual(window.document.querySelector('.agent-overlay').getAttribute('data-voice'), 'Voice paused');
++});
++
++check('no voice, no label', () => {
++ const { overlay, window } = openOverlay();
++ overlay.update({ steps: STEPS, caption: 'No voice here.' });
++ const label = window.document.querySelector('.agent-overlay-voice');
++ assert.strictEqual(label.hidden, true, 'the label is hidden when there is no voice');
++ assert.strictEqual(window.document.querySelector('.agent-overlay').getAttribute('data-voice'), null);
++});
++
++check('the page hands the overlay the newest line as the caption', () => {
++ assert.match(pageScript, /caption: captionText\(\)/, 'the overlay is given the caption');
++ assert.match(pageScript, /const line = lines\[lines\.length - 1\];/, 'the newest line');
++ assert.match(pageScript, /return line && line\.text \? line\.text : '';/, 'and its text');
++ assert.ok(pageScript.includes('?after='), 'the page asks only for the lines it has not seen');
++});
++
++check('the caption is drawn, and hidden again when there is nothing to say', () => {
++ assert.match(css, /\.agent-overlay-caption \{/, 'the caption has a style');
++ assert.match(css, /\.agent-overlay-caption\[hidden\] \{ display: none; \}/, 'and disappears when empty');
++ assert.match(css, /\.agent-overlay-voice \{/, 'the voice label has a style');
++});
++
++console.log(failures ? failures + ' failed' : 'all passed');
++process.exit(failures ? 1 : 0);
ddiff --git a/test/agent-streams-s4.test.js b/test/agent-streams-s4.test.js
nnew file mode 100644
iindex 00000000..93e41cab
---- /dev/null
++++ b/test/agent-streams-s4.test.js
@@@ -0,0 +1,330 @@
++'use strict';
++/**
++ * Agent streams S4, Live's part (design 2026-11-06): the agent's voice.
++ *
++ * - the line writer: one or two sentences, at most 220 characters, in the stream's personality;
++ * - the house text filters have the last word: a line that trips them is not said;
++ * - the pacing: one line every 20 s at most, the newest line wins (a queue of two), a failure line interrupts
++ * the gap once;
++ * - the lines travel in the state the desk polls, and the page shows them as captions and the recording keeps
++ * them as WebVTT;
++ * - the owner picks the mode, the voice and the personality, and mutes the voice; a channel moderator may mute
++ * it too;
++ * - the allowance: when the person's ceiling is spent the voice pauses and says so.
++ *
++ * Sign-in is stubbed with an x-test-user header (as agent-streams-s2.test.js does); everything after that is
++ * production. The writer's model call is replaced by a stub (the writer takes its generator), so the pacing and
++ * the filters are tested without a provider and without a clock of their own.
++ */
++const assert = require('assert');
++const http = require('http');
++const express = require('express');
++
++process.env.NODE_ENV = 'test';
++const quiet = console.log;
++console.log = (...a) => { if (!/^\[/.test(String(a[0]))) quiet(...a); };
++console.warn = () => {};
++console.error = () => {};
++
++const SID = {
++ ann: 'usr_01JAB2C3D4E5F6G7H8J9K0MNA1',
++ bob: 'usr_01JAB2C3D4E5F6G7H8J9K0MNB2',
++};
++
++let failures = 0;
++async function check(name, fn) {
++ try { await fn(); quiet(' ✓', name); }
++ catch (e) { failures++; quiet(' ✗', name, '\n ', (e.stack || String(e)).split('\n').slice(0, 8).join('\n ')); }
++}
++
++(async () => {
++ const db = require('../server/db/database');
++ await db.initDb();
++ const raw = db.getDb();
++ const addUser = async (id, username, role, extra = {}) => await raw.prepare(
++ `INSERT INTO users (id, username, display_name, email, password_hash, role, is_owner) OVERRIDING SYSTEM VALUE
++ VALUES (?, ?, ?, ?, 'x', ?, ?)`).run(id, username, username, `${username}@x`, role, extra.is_owner ? 1 : 0);
++ await addUser(1, 'ann', 'streamer');
++ await addUser(2, 'bob', 'user');
++ await raw.prepare("INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (1, 'network', '101', ?)").run(SID.ann);
++ await raw.prepare("INSERT INTO linked_accounts (user_id, service, service_user_id, subject_id) VALUES (2, 'network', '102', ?)").run(SID.bob);
++
++ const config = require('../server/config');
++ config.baseUrl = 'https://openvibe.live';
++ const service = require('../server/agent-streams/service');
++ require('../server/agent-streams/sharing')._setNetwork({ json: async () => ({}) });
++ const tickets = require('../server/agent-streams/tickets');
++ const voice = require('../server/agent-streams/voice');
++ const ann = await db.getUserById(1);
++ const bob = await db.getUserById(2);
++
++ const streamOf = (visibility, title) => service.create({
++ user: ann,
++ title,
++ task: { source: 'codes', ref: 'tsk_1', title: 'Run the checkout tests', url: 'https://openvibe.codes/tasks/1' },
++ visibility,
++ });
++ const publicStream = await streamOf('public', 'Fixing the checkout tests');
++
++ const app = express();
++ app.use(express.json());
++ app.use((req, res, next) => { req.user = { 1: ann, 2: bob }[req.get('x-test-user')] || null; next(); });
++ app.use('/api/streams/agent', require('../server/agent-streams/routes'));
++ const server = await new Promise((ok) => { const s = http.createServer(app).listen(0, '127.0.0.1', () => ok(s)); });
++ const base = `http://127.0.0.1:${server.address().port}`;
++ const call = async (method, url, { as = 1, body, headers = {} } = {}) => {
++ const res = await fetch(`${base}${url}`, {
++ method,
++ headers: { 'Content-Type': 'application/json', 'x-test-user': String(as), ...headers },
++ body: body === undefined ? undefined : JSON.stringify(body),
++ });
++ const text = await res.text();
++ let data = {};
++ try { data = text ? JSON.parse(text) : {}; } catch { data = { text }; }
++ return { status: res.status, data, text };
++ };
++
++ quiet('agent streams S4: the voice');
++
++ // ── the writer, on its own (no model, no clock of its own) ──
++ const makeWriter = (lines) => {
++ let clock = 1000;
++ const w = new voice.Writer({
++ now: () => clock,
++ generate: async () => lines.shift() || null,
++ });
++ return { writer: w, tick: (ms) => { clock += ms; } };
++ };
++ const row = { managed_stream_id: 999, user_id: 1, mode: 'narrated', personality: 'calm', task: { title: 'Run the checkout tests' } };
++
++ await check('a line is at most 220 characters and ends at a sentence', () => {
++ const long = 'The checkout tests are green and the build is ready to ship. ' + 'x'.repeat(400);
++ const out = voice.clamp(long);
++ assert.ok(Array.from(out).length <= voice.MAX_CHARS, `length ${Array.from(out).length}`);
++ assert.ok(out.endsWith('.'), out);
++ });
++
++ await check('the personalities are the four the design names', () => {
++ assert.deepStrictEqual(voice.PERSONALITIES, ['calm', 'hype', 'dry', 'roast']);
++ });
++
++ await check('the house filters refuse a slur line', () => {
++ // The word itself is not written here on purpose; the filter is checked with the same helper chat runs.
++ const slur = ['n', 'i', 'g', 'g', 'a'].join('');
++ assert.strictEqual(voice.allowed(`you are a ${slur}`), false);
++ assert.strictEqual(voice.allowed('the checkout tests are green'), true);
++ });
++
++ await check('the filters also refuse hate and doxxing', () => {
++ assert.strictEqual(voice.allowed('my email is [email protected]'), false);
++ assert.strictEqual(voice.allowed('call me on +1 555 123 4567'), false);
++ });
++
++ await check('a line is written in the stream\'s personality and kept', async () => {
++ const { writer: w } = makeWriter(['The checkout tests are green.']);
++ const line = await w.onStep({ ...row, personality: 'hype' }, { title: 'Checkout tests', status: 'passed' });
++ assert.ok(line, 'a line was written');
++ assert.strictEqual(line.text, 'The checkout tests are green.');
++ assert.strictEqual(line.personality, 'hype');
++ assert.strictEqual(line.kind, 'result');
++ });
++
++ await check('the writer stays quiet outside narrated mode, and when muted', async () => {
++ const { writer: w } = makeWriter(['A line.']);
++ assert.strictEqual(await w.onStep({ ...row, mode: 'watch' }, { title: 'x', status: 'started' }), null);
++ assert.strictEqual(await w.onStep({ ...row, voice_muted: true }, { title: 'x', status: 'started' }), null);
++ });
++
++ await check('a refused line is never said, and the next step gets a new line', async () => {
++ const { writer: w } = makeWriter(['you are a ' + ['n', 'i', 'g', 'g', 'a'].join(''), 'The tests are green.']);
++ const first = await w.onStep({ ...row }, { title: 'x', status: 'started' });
++ assert.strictEqual(first, null, 'the slur line was not said');
++ const second = await w.onStep({ ...row }, { title: 'y', status: 'started' });
++ assert.ok(second && second.text === 'The tests are green.', 'the next step got a fresh line');
++ });
++
++ await check('one line every 20 s: the second step waits for the gap', async () => {
++ const { writer: w, tick } = makeWriter(['First line.', 'Second line.']);
++ const first = await w.onStep({ ...row }, { title: 'a', status: 'started' });
++ assert.ok(first, 'the first line is said at once');
++ tick(5000);
++ const second = await w.onStep({ ...row }, { title: 'b', status: 'started' });
++ assert.strictEqual(second, null, 'the gap held the second line back');
++ assert.strictEqual(w.pending(999).length, 1, 'the line waits in the queue');
++ });
++
++ await check('the newest line wins: a queue of two, the older one is dropped', async () => {
++ const { writer: w } = makeWriter(['First.', 'Second.', 'Third.']);
++ const first = await w.onStep({ ...row }, { title: 'a', status: 'started' });
++ assert.ok(first);
++ await w.onStep({ ...row }, { title: 'b', status: 'started' });
++ await w.onStep({ ...row }, { title: 'c', status: 'started' });
++ const pending = w.pending(999);
++ assert.strictEqual(pending.length, voice.QUEUE_MAX, 'the queue holds two');
++ assert.strictEqual(pending[pending.length - 1].text, 'Third.', 'the newest line is the one left');
++ });
++
++ await check('a failure line interrupts the gap once', async () => {
++ const { writer: w, tick } = makeWriter(['First.', 'The build broke.']);
++ await w.onStep({ ...row }, { title: 'a', status: 'started' });
++ tick(3000);
++ const said = await w.onStep({ ...row }, { title: 'b', status: 'failed' });
++ assert.ok(said && said.kind === 'failure', 'the failure line interrupted');
++ tick(3000);
++ const again = await w.onStep({ ...row }, { title: 'c', status: 'failed' });
++ assert.strictEqual(again, null, 'the interrupt is used once');
++ });
++
++ await check('the WebVTT carries the lines as captions', () => {
++ const vtt = voice.webvtt([{ start: 0, text: 'The checkout tests are green.' }, { start: 2.5, text: 'Second line.' }]);
++ assert.ok(vtt.startsWith('WEBVTT'), vtt.slice(0, 40));
++ assert.ok(vtt.includes('00:00:00.000 --> 00:00:04.000'), vtt);
++ assert.ok(vtt.includes('The checkout tests are green.'), vtt);
++ });
++
++ quiet('agent streams S4: the owner\'s controls');
++ const publish = await tickets.issuePublish(publicStream.managed_stream_id);
++ await service.markPublishing(publicStream);
++
++ await check('the owner picks the mode, the personality and the voice', async () => {
++ const r = await call('PUT', `/api/streams/agent/${publicStream.public_id}/voice`, {
++ body: { mode: 'narrated', personality: 'dry', voice: 'gary' },
++ });
++ assert.strictEqual(r.status, 200, JSON.stringify(r.data));
++ assert.strictEqual(r.data.voice.mode, 'narrated');
++ assert.strictEqual(r.data.voice.personality, 'dry');
++ assert.strictEqual(r.data.voice.voice, 'gary');
++ });
++
++ await check('another account cannot change the voice', async () => {
++ const r = await call('PUT', `/api/streams/agent/${publicStream.public_id}/voice`, {
++ as: 2, body: { personality: 'roast' },
++ });
++ assert.strictEqual(r.status, 403, JSON.stringify(r.data));
++ });
++
++ await check('a bad personality is refused', async () => {
++ const r = await call('PUT', `/api/streams/agent/${publicStream.public_id}/voice`, {
++ body: { personality: 'sarcastic' },
++ });
++ assert.strictEqual(r.status, 400, JSON.stringify(r.data));
++ });
++
++ await check('the owner\'s own cloned voice needs their consent', async () => {
++ const refused = await call('PUT', `/api/streams/agent/${publicStream.public_id}/voice`, { body: { own: true } });
++ assert.strictEqual(refused.status, 422, JSON.stringify(refused.data));
++ const ok = await call('PUT', `/api/streams/agent/${publicStream.public_id}/voice`, {
++ body: { own: true, own_consent: true },
++ });
++ assert.strictEqual(ok.status, 200, JSON.stringify(ok.data));
++ assert.strictEqual(ok.data.voice.own, true);
++ assert.ok(ok.data.voice.own_consent_at, 'the moment they said yes is kept');
++ });
++
++ await check('the owner mutes the voice', async () => {
++ const r = await call('POST', `/api/streams/agent/${publicStream.public_id}/voice/mute`, { body: { muted: true } });
++ assert.strictEqual(r.status, 200, JSON.stringify(r.data));
++ assert.strictEqual(r.data.voice.muted, true);
++ });
++
++ await check('a stranger cannot mute the voice', async () => {
++ const r = await call('POST', `/api/streams/agent/${publicStream.public_id}/voice/mute`, {
++ as: 2, body: { muted: false },
++ });
++ assert.ok(r.status === 403 || r.status === 404, JSON.stringify(r.data));
++ });
++
++ await check('a channel moderator may mute the voice', async () => {
++ const permissions = require('../server/auth/permissions');
++ const original = permissions.canModerateChannel;
++ permissions.canModerateChannel = async () => true;
++ try {
++ const r = await call('POST', `/api/streams/agent/${publicStream.public_id}/voice/mute`, {
++ as: 2, body: { muted: false },
++ });
++ assert.strictEqual(r.status, 200, JSON.stringify(r.data));
++ assert.strictEqual(r.data.voice.muted, false);
++ } finally {
++ permissions.canModerateChannel = original;
++ }
++ });
++
++ await check('the allowance message is what the design says', async () => {
++ const row2 = await service.getByManagedStreamId(publicStream.managed_stream_id);
++ await db.run('UPDATE agent_streams SET voice_paused = ? WHERE managed_stream_id = ?', ['allowance_spent', row2.managed_stream_id]);
++ const r = await call('GET', `/api/streams/agent/${publicStream.public_id}/voice`);
++ assert.strictEqual(r.data.voice.paused, 'allowance_spent');
++ assert.strictEqual(r.data.voice.message, 'voice paused: allowance spent');
++ await db.run('UPDATE agent_streams SET voice_paused = NULL WHERE managed_stream_id = ?', [row2.managed_stream_id]);
++ });
++
++ quiet('agent streams S4: the lines the desk reads');
++ await check('a spoken line travels in the state the desk polls', async () => {
++ const row3 = await service.getByManagedStreamId(publicStream.managed_stream_id);
++ await db.run('UPDATE agent_streams SET voice_muted = false, mode = ? WHERE managed_stream_id = ?', ['narrated', row3.managed_stream_id]);
++ const now = new Date().toISOString();
++ await db.run(
++ `INSERT INTO agent_stream_lines (managed_stream_id, at, text, personality, voice, kind) VALUES (?, ?, ?, ?, ?, ?)`,
++ [row3.managed_stream_id, now, 'The checkout tests are green.', 'calm', 'gary', 'result']);
++ const r = await call('GET', `/api/streams/agent/${publicStream.public_id}/state`, {
++ headers: { Authorization: `Bearer ${publish.ticket}` },
++ });
++ assert.strictEqual(r.status, 200, JSON.stringify(r.data));
++ assert.ok(Array.isArray(r.data.lines) && r.data.lines.length, 'the state carries the lines');
++ assert.strictEqual(r.data.lines[r.data.lines.length - 1].text, 'The checkout tests are green.');
++ assert.ok(r.data.voice, 'the state carries the voice');
++ assert.ok(String(r.data.captions_url || '').endsWith('/captions.vtt'), 'the state points at the captions track');
++ });
++
++ await check('the desk asks for the lines it has not read yet', async () => {
++ const row4 = await service.getByManagedStreamId(publicStream.managed_stream_id);
++ const first = await call('GET', `/api/streams/agent/${publicStream.public_id}/state`, {
++ headers: { Authorization: `Bearer ${publish.ticket}` },
++ });
++ const lastId = first.data.lines[first.data.lines.length - 1].id;
++ const second = await call('GET', `/api/streams/agent/${publicStream.public_id}/state?after=${lastId}`, {
++ headers: { Authorization: `Bearer ${publish.ticket}` },
++ });
++ assert.deepStrictEqual(second.data.lines, [], 'nothing new after the last id');
++ const row5 = await service.getByManagedStreamId(row4.managed_stream_id);
++ await db.run(
++ `INSERT INTO agent_stream_lines (managed_stream_id, at, text, personality, voice, kind) VALUES (?, ?, ?, ?, ?, ?)`,
++ [row5.managed_stream_id, new Date().toISOString(), 'A newer line.', 'calm', 'gary', 'step']);
++ const third = await call('GET', `/api/streams/agent/${publicStream.public_id}/state?after=${lastId}`, {
++ headers: { Authorization: `Bearer ${publish.ticket}` },
++ });
++ assert.strictEqual(third.data.lines.length, 1);
++ assert.strictEqual(third.data.lines[0].text, 'A newer line.');
++ });
++
++ await check('the page reads the lines as captions', async () => {
++ const r = await call('GET', `/api/streams/agent/${publicStream.public_id}/lines`);
++ assert.strictEqual(r.status, 200, JSON.stringify(r.data));
++ assert.ok(Array.isArray(r.data.lines) && r.data.lines.length, 'the lines are there');
++ assert.ok(r.data.voice, 'the voice is there');
++ });
++
++ await check('the recording keeps the captions as WebVTT', async () => {
++ const res = await fetch(`${base}/api/streams/agent/${publicStream.public_id}/captions.vtt`, {
++ headers: { 'x-test-user': '1' },
++ });
++ assert.strictEqual(res.status, 200);
++ assert.ok(String(res.headers.get('content-type')).includes('text/vtt'), res.headers.get('content-type'));
++ const text = await res.text();
++ assert.ok(text.startsWith('WEBVTT'), text.slice(0, 40));
++ assert.ok(text.includes('The checkout tests are green.'), text);
++ });
++
++ await check('the owner page carries the voice controls', () => {
++ const fs = require('node:fs');
++ const path = require('node:path');
++ const html = fs.readFileSync(path.join(__dirname, '..', 'public', 'fragments', 'agent-stream.html'), 'utf8');
++ for (const id of ['agent-voice-mode', 'agent-voice-personality', 'agent-voice-pick', 'agent-voice-own', 'agent-voice-mute']) {
++ assert.ok(html.includes(id), `the page has ${id}`);
++ }
++ });
++
++ server.close();
++ quiet(failures ? `\n${failures} check(s) failed` : '\nall checks passed');
++ process.exit(failures ? 1 : 0);
++})();
\\ No newline at end of file