openvibers/openvibe.chat Commits

Persisted TTS and sound queue: every TTS utterance, channel !sound and 101soundboards clip is a row of audio_requests (server/chat/audio-queue.js) played one at a time per room, queued -> playing -> played, or skipped / failed. The audio is made when its turn comes and paced by the clip's length (WAV header, MP3 frame bitrate, or the sound's stored duration), so skip and clear take effect for everything not yet delivered. The broadcaster and moderators control it with /skiptts [id] and /cleartts (logged as tts_skip / tts_clear) and GET /api/tts/queue, POST /api/tts/queue/skip, /clear and /:id/report (the playing client's played / failed). Synthesis errors, missing clips and repeated crashes are recorded as failed and the queue moves on. A keyed request (the TTS of message m<id>) is queued once per room, replacing the 30 s in-memory dedupe, and the per-user limit (tts_max_queue_per_user) now holds. A restart keeps the queue: rows that were playing are finished, rows older than 5 minutes expire, and the rest play once the room has listeners again (a second after a socket rejoins, 8 s at most). Clients get the same tts-audio / soundboard-audio frames plus request_id, and two new frames they may ignore: audio-skip and audio-clear. test/audio-queue.test.js covers the state machine, pacing, permissions, failures, dedupe and a real SIGTERM restart

OP OpenVibers committed 23 Sept 2026 · bf88760b24 · 7 files, +933 −93


Changes

ddiff --git a/README.md b/README.md
iindex a679d7f..37aeef9 100644
---- a/README.md
++++ b/README.md
@@@ -28,6 +28,16 @@ before changing behaviour.** Browser JavaScript does not change; nginx routes th
DMs (`dm.js`, `/api/dm/*`), chat history (`history-store.js`, `/api/chat/*`), TTS (`tts-engine.js`,
`/api/tts/*`), 101soundboards and channel sounds (`/api/sounds*`), moderation utils and the word
filter, the deploy-notice card. Changes are confined to where they touched Live's data.
++- **The TTS and sound queue** (`audio-queue.js`, table `audio_requests`). TTS, channel `!sounds` and
++ 101soundboards clips are persisted requests played one at a time per room — `queued → playing →
++ played`, or `skipped` / `failed` — so the queue survives a Chat restart (what was playing is
++ finished, what waited plays once the room has listeners again, requests older than 5 minutes
++ expire) and a keyed request (the TTS of message `m<id>`) is read once. The broadcaster and
++ moderators control it: `/skiptts [id]`, `/cleartts`, and `GET /api/tts/queue`,
++ `POST /api/tts/queue/skip|clear`, `POST /api/tts/queue/:id/report` (the playing client's
++ `played`/`failed`). Browsers get the same `tts-audio` / `soundboard-audio` frames as before, now
++ with `request_id` and paced by the clip's length, plus two new frames old clients ignore:
++ `audio-skip { request_id }` and `audio-clear { request_ids }`.
- **One adapter to Live — `server/live-context.js`.** Accounts and roles, streams, slots and channels,
channel moderation settings and moderators, bans and IP rules, follows, cosmetics and tags, site
settings and anon numbers are read through it; coins, AI viewers, arena, media queue, hardware,
@@@ -130,8 +140,9 @@ docs/ cutover.md, live-patch.diff, capabilities-proposal/
- TTS/audio/soundboard/media-request queues with skip/clear/failure lifecycle
- call signalling metadata and the `pending/ringing/active/ended/missed/declined/failed` lifecycle
--(Wave 6 moved messages, DMs, TTS, sounds and the chat side of moderation; bans, channel moderation
--settings, the media-request queue and calls are still Live's and reached through `live-context`.)
++(Wave 6 moved messages, DMs, TTS, sounds and the chat side of moderation, and gave TTS and sounds a
++persisted queue with skip/clear/failed states; bans, channel moderation settings, the media-request
++queue and calls are still Live's and reached through `live-context`.)
## Does not own
ddiff --git a/docs/cutover.md b/docs/cutover.md
iindex c3707b6..ee11f05 100644
---- a/docs/cutover.md
++++ b/docs/cutover.md
@@@ -24,7 +24,7 @@ nginx locations (`deploy/nginx/openvibe.live-chat.locations.conf`, included in L
| `^~ /ws/chat` | the chat WebSocket (`?stream=`, `?token=`; protocol unchanged) |
| `^~ /api/chat/` | `GET gif/providers`, `GET gif/trending`, `GET gif/search`, `POST send`, `GET search`, `GET user/:userId/history`, `GET user/:username/profile`, `GET relay-user/:platform/:username`, `GET relay-user/:platform/:username/logs`, `GET anon/:anonId`, `GET anon/:anonId/logs`, `GET filters/friendly`, `GET global/history`, `GET :streamId/replay`, `GET :streamId/history`, `GET channel/:userId/history`, `GET :streamId/users`, `POST admin/purge/preview`, `DELETE admin/purge`, `GET admin/logs`, `GET admin/logs/export` |
| `^~ /api/dm/` | `GET/POST conversations`, `GET/PATCH conversations/:id`, `GET/POST conversations/:id/messages`, `DELETE conversations/:id/messages/:msgId`, `POST conversations/:id/read`, `POST conversations/:id/participants`, `DELETE conversations/:id/participants/:userId`, `GET unread`, `GET users/search`, `GET blocks`, `POST/DELETE blocks/:userId`, `GET blocks/check/:userId` |
--| `^~ /api/tts/` | `GET voices`, `GET settings`, `GET/PUT admin/settings`, `POST admin/test`, `GET audio/:file` (Chat's clips; Live's arena/mod-preview clips are fetched from Live under the same URL) |
++| `^~ /api/tts/` | `GET voices`, `GET settings`, `GET/PUT admin/settings`, `POST admin/test`, `GET audio/:file` (Chat's clips; Live's arena/mod-preview clips are fetched from Live under the same URL), `GET queue`, `POST queue/skip`, `POST queue/clear`, `POST queue/:id/report` (the TTS and sound queue) |
| `= /api/sounds` and `^~ /api/sounds/` | `POST /api/sounds` (upload), `GET channel/:userId`, `GET all/:streamId`, `DELETE :id`, `PATCH command`, `GET file/:filename`, `GET alert/mine`, `POST/DELETE alert/:kind` |
Stay on Live (do not route): `/api/chat-ai/*` (the trailing slash in `/api/chat/` keeps it out),
ddiff --git a/server/chat/audio-queue.js b/server/chat/audio-queue.js
nnew file mode 100644
iindex 0000000..13bf57e
---- /dev/null
++++ b/server/chat/audio-queue.js
@@@ -0,0 +1,325 @@
++/**
++ * OpenVibe.Chat — the persisted TTS and sound queue (roadmap Wave 6 D4).
++ *
++ * Every TTS utterance, channel !sound and 101soundboards clip a room asks for is a row of
++ * `audio_requests`, played one at a time per room:
++ *
++ * queued ──▶ playing ──▶ played
++ * │ │
++ * ├──────────┴──▶ skipped (a moderator or the broadcaster: /skiptts, /cleartts, /api/tts/queue)
++ * └──────────┴──▶ failed (synthesis or the clip failed, it expired, or the playing client said so)
++ *
++ * queued accepted, not delivered yet. The audio is made (synthesized / read / fetched) when its
++ * turn comes, so a skipped or cleared request costs nothing.
++ * playing delivered to the room (the `tts-audio` / `soundboard-audio` frame, now with request_id)
++ * and inside its play window (the clip's length, estimated from the audio). When the
++ * window ends it is played and the next one goes out.
++ * played, skipped, failed final.
++ *
++ * A Chat restart keeps the queue: `recover()` at boot finishes rows that were playing (they reached
++ * the room before the restart), fails rows queued too long ago to still matter, and plays the rest
++ * once the room has listeners again: a second after the first socket rejoins it (clients come back
++ * 1–4 s after a restart notice), or after RESUME_GRACE_MS whatever happens.
++ * A keyed request (a TTS of chat message m<id>) is queued once per room, so a retried bridge call
++ * or a double delivery never reads it twice, even across a restart.
++ *
++ * Browsers keep their own playback queue and the frames they already know; the server pacing means
++ * that queue holds at most the clip playing now, so skip and clear on the server take effect at
++ * once for everything not yet delivered. Two new frames, ignored by clients that don't know them,
++ * let a client stop the clip in hand: `{ type: 'audio-skip', request_id }` and
++ * `{ type: 'audio-clear', request_ids }`.
++ */
++'use strict';
++
++const db = require('../db/database');
++
++const STATES = ['queued', 'playing', 'played', 'skipped', 'failed'];
++const TRANSITIONS = {
++ queued: ['playing', 'skipped', 'failed'],
++ playing: ['played', 'skipped', 'failed'],
++ played: [],
++ skipped: [],
++ failed: [],
++};
++const KINDS = ['tts', 'channel-sound', 'soundboard'];
++
++const GAP_MS = 250; // between two clips
++const MIN_PLAY_MS = 500;
++const MAX_PLAY_MS = 60 * 1000;
++const STALE_MS = 5 * 60 * 1000; // a request still queued this long after it was made is dropped
++const MAX_ATTEMPTS = 3; // a request whose making crashed Chat this often fails
++const KEEP_MS = 7 * 24 * 3600 * 1000;
++const RESUME_GRACE_MS = 8000; // after a restart, a room with waiting requests resumes by then…
++const RESUME_SETTLE_MS = 1000; // …or this long after a socket rejoins it
++
++function canTransition(from, to) {
++ return !!(TRANSITIONS[from] && TRANSITIONS[from].includes(to));
++}
++
++function roomKey({ streamId, channelUserId }) {
++ if (streamId) return `stream:${Number(streamId)}`;
++ if (channelUserId) return `channel:${Number(channelUserId)}`;
++ return null;
++}
++
++// ── Clip length ─────────────────────────────────────────────────────────────────────────────────
++const MP3_BITRATES = {
++ // [version 1, version 2/2.5] × layer III kbps
++ 1: [0, 32, 40, 48, 56, 64, 80, 96, 112, 128, 160, 192, 224, 256, 320],
++ 2: [0, 8, 16, 24, 32, 40, 48, 56, 64, 80, 96, 112, 128, 144, 160],
++};
++/** Bitrate (bits/s) of the first MPEG layer III frame, skipping an ID3v2 tag; null if none. */
++function mp3Bitrate(buf) {
++ let i = 0;
++ if (buf.length > 10 && buf.toString('latin1', 0, 3) === 'ID3') {
++ i = 10 + (((buf[6] & 0x7f) << 21) | ((buf[7] & 0x7f) << 14) | ((buf[8] & 0x7f) << 7) | (buf[9] & 0x7f));
++ }
++ for (const end = Math.min(buf.length - 4, i + 64 * 1024); i < end; i++) {
++ if (buf[i] !== 0xff || (buf[i + 1] & 0xe0) !== 0xe0) continue;
++ const version = (buf[i + 1] >> 3) & 0x03; // 3 = MPEG1, 2 = MPEG2, 0 = MPEG2.5
++ const layer = (buf[i + 1] >> 1) & 0x03; // 1 = layer III
++ const idx = (buf[i + 2] >> 4) & 0x0f;
++ if (version === 1 || layer !== 1 || idx === 0 || idx === 15) continue;
++ return MP3_BITRATES[version === 3 ? 1 : 2][idx] * 1000;
++ }
++ return null;
++}
++/** How long a clip plays, in ms: a known length, a WAV header, an MP3 frame's bitrate, or a guess. */
++function estimatePlayMs({ audio, mimeType, seconds, speed } = {}) {
++ const rate = Number(speed) > 0 ? Number(speed) : 1;
++ let ms = null;
++ if (Number(seconds) > 0) ms = Number(seconds) * 1000;
++ else if (audio) {
++ let buf = null;
++ try { buf = Buffer.from(String(audio), 'base64'); } catch { buf = null; }
++ if (buf && buf.length) {
++ if (buf.length > 44 && buf.toString('latin1', 0, 4) === 'RIFF' && buf.toString('latin1', 8, 12) === 'WAVE') {
++ const byteRate = buf.readUInt32LE(28);
++ if (byteRate > 0) ms = ((buf.length - 44) / byteRate) * 1000;
++ } else {
++ const bps = /mpeg|mp3/.test(String(mimeType || '')) ? mp3Bitrate(buf) : null;
++ ms = (buf.length * 8 / (bps || 128000)) * 1000;
++ }
++ }
++ }
++ if (ms == null) ms = 3000;
++ return Math.round(Math.min(MAX_PLAY_MS, Math.max(MIN_PLAY_MS, ms / rate)));
++}
++
++// ── Store ───────────────────────────────────────────────────────────────────────────────────────
++function getRequest(id) { return db.get('SELECT * FROM audio_requests WHERE id = ?', [id]) || null; }
++
++/**
++ * Move one request along the state machine. Returns true when it moved (the row was in a state
++ * that allows `to`); false when something else moved it first.
++ */
++function transition(id, to, { error = null, actor = null, durationMs = null, now = Date.now() } = {}) {
++ const from = Object.keys(TRANSITIONS).filter((s) => TRANSITIONS[s].includes(to));
++ if (!from.length) return false;
++ const final = to !== 'playing';
++ const r = db.run(
++ `UPDATE audio_requests SET state = ?, error = COALESCE(?, error), actor = COALESCE(?, actor),
++ duration_ms = COALESCE(?, duration_ms),
++ started_at = CASE WHEN ? = 'playing' THEN ? ELSE started_at END,
++ finished_at = CASE WHEN ? THEN ? ELSE finished_at END
++ WHERE id = ? AND state IN (${from.map(() => '?').join(',')})`,
++ [to, error, actor, durationMs, to, now, final ? 1 : 0, now, id, ...from],
++ );
++ return r.changes > 0;
++}
++
++// ── Runtime ─────────────────────────────────────────────────────────────────────────────────────
++const performers = {}; // kind → async (row, payload) → { frame, durationMs } | null
++let deliver = null; // (row, frame) → void
++const rooms = new Map(); // room → { busy, timer, playingId }
++let stopped = false;
++
++function roomState(room) {
++ if (!rooms.has(room)) rooms.set(room, { busy: false, timer: null, playingId: null, holdUntil: 0, holdTimer: null });
++ return rooms.get(room);
++}
++
++/** Hold a room's queue until `until` (restart recovery), then play. A shorter hold replaces a longer one. */
++function holdRoom(room, until) {
++ const st = roomState(room);
++ if (st.holdUntil && st.holdUntil <= until) return;
++ st.holdUntil = until;
++ if (st.holdTimer) clearTimeout(st.holdTimer);
++ st.holdTimer = setTimeout(() => { st.holdUntil = 0; st.holdTimer = null; pump(room); }, Math.max(0, until - Date.now()));
++ if (st.holdTimer.unref) st.holdTimer.unref();
++}
++
++/** A socket joined a room: a queue held since a restart plays shortly. */
++function roomJoined(room) {
++ const st = room && rooms.get(room);
++ if (st && st.holdUntil > Date.now()) holdRoom(room, Date.now() + RESUME_SETTLE_MS);
++}
++
++/**
++ * Wire the queue to the chat server: how each kind is made, and how a frame reaches a room.
++ * `performers[kind](row, payload)` resolves { frame, durationMs } or null (nothing to play).
++ */
++function init({ performers: p, deliver: d }) {
++ Object.assign(performers, p || {});
++ deliver = d;
++ stopped = false;
++}
++
++/** Stop pacing timers (shutdown). States stay as they are; recover() picks them up at boot. */
++function stop() {
++ stopped = true;
++ for (const st of rooms.values()) {
++ if (st.timer) clearTimeout(st.timer);
++ if (st.holdTimer) clearTimeout(st.holdTimer);
++ st.timer = null; st.holdTimer = null;
++ }
++ rooms.clear();
++}
++
++/**
++ * Accept a request. Returns { id, queued: true } or { queued: false, reason } — `duplicate` (this
++ * room already has this key), `full` (the room or this requester is at its limit).
++ */
++function enqueue({ kind, streamId = null, channelUserId = null, requestedBy = null, identityKey = null, label = null, payload = {}, dedupeKey = null, maxRoom = null, maxPerRequester = null, now = Date.now() }) {
++ if (!KINDS.includes(kind)) throw new Error(`audio-queue: unknown kind ${kind}`);
++ const room = roomKey({ streamId, channelUserId });
++ if (!room) return { queued: false, reason: 'no_room' };
++ const result = db.transaction(() => {
++ if (dedupeKey && db.get('SELECT 1 FROM audio_requests WHERE room = ? AND dedupe_key = ?', [room, dedupeKey])) return { queued: false, reason: 'duplicate' };
++ if (maxRoom) {
++ const n = db.get("SELECT COUNT(*) AS n FROM audio_requests WHERE room = ? AND state IN ('queued', 'playing')", [room]).n;
++ if (n >= maxRoom) return { queued: false, reason: 'full' };
++ }
++ if (maxPerRequester && identityKey) {
++ const n = db.get("SELECT COUNT(*) AS n FROM audio_requests WHERE room = ? AND identity_key = ? AND state IN ('queued', 'playing')", [room, identityKey]).n;
++ if (n >= maxPerRequester) return { queued: false, reason: 'full' };
++ }
++ const r = db.run(
++ `INSERT INTO audio_requests (room, stream_id, channel_user_id, kind, state, requested_by, identity_key, label, payload, dedupe_key, created_at)
++ VALUES (?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?)`,
++ [room, streamId || null, channelUserId || null, kind, requestedBy, identityKey, label ? String(label).slice(0, 300) : null, JSON.stringify(payload || {}), dedupeKey, now],
++ );
++ return { queued: true, id: Number(r.lastInsertRowid), room };
++ });
++ if (result.queued) setImmediate(() => pump(result.room));
++ return result;
++}
++
++/** Play the next request of a room if none is playing. */
++async function pump(room) {
++ if (stopped || !room) return;
++ const st = roomState(room);
++ if (st.busy || st.playingId || st.holdUntil > Date.now()) return;
++ st.busy = true;
++ try {
++ for (;;) {
++ if (stopped) return;
++ let row;
++ try { row = db.get("SELECT * FROM audio_requests WHERE room = ? AND state = 'queued' ORDER BY id LIMIT 1", [room]); } catch { return; }
++ if (!row) return;
++ if (row.attempts >= MAX_ATTEMPTS) { transition(row.id, 'failed', { error: 'gave up after repeated attempts', actor: 'system' }); continue; }
++ db.run('UPDATE audio_requests SET attempts = attempts + 1 WHERE id = ?', [row.id]);
++ const perform = performers[row.kind];
++ let made = null, error = null;
++ try {
++ made = perform ? await perform(row, JSON.parse(row.payload || '{}')) : null;
++ if (!perform) error = `no performer for ${row.kind}`;
++ else if (!made || !made.frame) error = 'no audio';
++ } catch (err) { error = (err && err.message) || 'failed'; }
++ if (stopped) return;
++ if (error) { transition(row.id, 'failed', { error: String(error).slice(0, 300), actor: 'system' }); continue; }
++ const durationMs = Math.round(Math.min(MAX_PLAY_MS, Math.max(MIN_PLAY_MS, Number(made.durationMs) || estimatePlayMs(made.frame))));
++ // Skipped or cleared while it was being made: nothing goes out.
++ if (!transition(row.id, 'playing', { durationMs })) continue;
++ try { if (deliver) deliver(row, { ...made.frame, request_id: row.id }); } catch (err) { console.warn('[AudioQueue] deliver:', err.message); }
++ st.playingId = row.id;
++ st.timer = setTimeout(() => finish(room, row.id, 'played'), durationMs + GAP_MS);
++ if (st.timer.unref) st.timer.unref();
++ return;
++ }
++ } finally { st.busy = false; }
++}
++
++/** End the playing request of a room (its window ended, it was skipped, or the client reported). */
++function finish(room, id, to, opts = {}) {
++ const st = roomState(room);
++ let moved = false;
++ try { moved = transition(id, to, opts); } catch { moved = false; }
++ if (st.playingId === id) {
++ if (st.timer) clearTimeout(st.timer);
++ st.timer = null;
++ st.playingId = null;
++ setImmediate(() => pump(room));
++ }
++ return moved;
++}
++
++/**
++ * Skip one request of a room: `id`, or else the one playing, or else the next queued.
++ * Returns the skipped row or null.
++ */
++function skip(room, { id = null, actor = null } = {}) {
++ let row = null;
++ if (id) row = db.get("SELECT * FROM audio_requests WHERE id = ? AND room = ? AND state IN ('queued', 'playing')", [id, room]);
++ else row = db.get("SELECT * FROM audio_requests WHERE room = ? AND state = 'playing' ORDER BY id LIMIT 1", [room])
++ || db.get("SELECT * FROM audio_requests WHERE room = ? AND state = 'queued' ORDER BY id LIMIT 1", [room]);
++ if (!row) return null;
++ const moved = row.state === 'playing' ? finish(room, row.id, 'skipped', { actor }) : transition(row.id, 'skipped', { actor });
++ if (!moved) return null;
++ if (row.state === 'playing' && deliver) { try { deliver(row, { type: 'audio-skip', request_id: row.id, kind: row.kind }); } catch { /* */ } }
++ return { ...row, state: 'skipped' };
++}
++
++/** Skip everything queued or playing in a room. Returns the ids skipped. */
++function clear(room, { actor = null } = {}) {
++ const rows = db.all("SELECT * FROM audio_requests WHERE room = ? AND state IN ('queued', 'playing') ORDER BY id", [room]);
++ const ids = [];
++ let sample = null;
++ for (const row of rows) {
++ const moved = row.state === 'playing' ? finish(room, row.id, 'skipped', { actor }) : transition(row.id, 'skipped', { actor });
++ if (moved) { ids.push(row.id); sample = sample || row; }
++ }
++ if (ids.length && deliver) { try { deliver(sample, { type: 'audio-clear', request_ids: ids }); } catch { /* */ } }
++ return ids;
++}
++
++/** The playing client's report: its clip ended ('played') or could not play ('failed'). */
++function report(room, id, state, { error = null, actor = null } = {}) {
++ if (state !== 'played' && state !== 'failed') return false;
++ const row = db.get("SELECT * FROM audio_requests WHERE id = ? AND room = ? AND state = 'playing'", [id, room]);
++ if (!row) return false;
++ return finish(room, row.id, state, { error: state === 'failed' ? String(error || 'playback failed').slice(0, 300) : null, actor });
++}
++
++/** A room's queue: what plays now, what waits, and the latest finished requests. */
++function list(room, { recent = 20 } = {}) {
++ const shape = (r) => ({
++ id: r.id, kind: r.kind, state: r.state, requested_by: r.requested_by, label: r.label,
++ error: r.error, actor: r.actor, duration_ms: r.duration_ms,
++ created_at: r.created_at, started_at: r.started_at, finished_at: r.finished_at,
++ });
++ const playing = db.get("SELECT * FROM audio_requests WHERE room = ? AND state = 'playing' ORDER BY id LIMIT 1", [room]);
++ const queued = db.all("SELECT * FROM audio_requests WHERE room = ? AND state = 'queued' ORDER BY id", [room]);
++ const done = db.all("SELECT * FROM audio_requests WHERE room = ? AND state IN ('played', 'skipped', 'failed') ORDER BY COALESCE(finished_at, created_at) DESC, id DESC LIMIT ?", [room, Math.min(Math.max(parseInt(recent, 10) || 0, 0), 100)]);
++ return { room, playing: playing ? shape(playing) : null, queued: queued.map(shape), recent: done.map(shape) };
++}
++
++/**
++ * Boot: rows playing when Chat stopped were delivered, so they are played; rows queued too long
++ * ago fail as expired; every room with queued rows starts playing again.
++ */
++function recover({ now = Date.now(), graceMs = RESUME_GRACE_MS } = {}) {
++ const played = db.run("UPDATE audio_requests SET state = 'played', finished_at = ?, error = 'delivered before a restart' WHERE state = 'playing'", [now]).changes;
++ const expired = db.run("UPDATE audio_requests SET state = 'failed', finished_at = ?, error = 'expired', actor = 'system' WHERE state = 'queued' AND created_at < ?", [now, now - STALE_MS]).changes;
++ db.run("DELETE FROM audio_requests WHERE state IN ('played', 'skipped', 'failed') AND COALESCE(finished_at, created_at) < ?", [now - KEEP_MS]);
++ const pending = db.all("SELECT DISTINCT room FROM audio_requests WHERE state = 'queued'").map((r) => r.room);
++ for (const room of pending) holdRoom(room, now + graceMs);
++ if (played || expired || pending.length) console.log(`[AudioQueue] recovered: ${played} finished, ${expired} expired, ${pending.length} room(s) resuming`);
++ return { played, expired, rooms: pending };
++}
++
++module.exports = {
++ STATES, TRANSITIONS, KINDS, STALE_MS, GAP_MS, RESUME_GRACE_MS, RESUME_SETTLE_MS,
++ canTransition, roomKey, estimatePlayMs, mp3Bitrate,
++ init, stop, enqueue, pump, skip, clear, report, list, recover, roomJoined, getRequest, transition,
++};
ddiff --git a/server/chat/chat-server.js b/server/chat/chat-server.js
iindex 33e9868..8f835ce 100644
---- a/server/chat/chat-server.js
++++ b/server/chat/chat-server.js
@@@ -28,6 +28,7 @@ const permissions = require('../auth/permissions');
const wordFilter = require('./word-filter');
const ttsEngine = require('./tts-engine');
const soundboard = require('./soundboard-service');
++const audioQueue = require('./audio-queue');
const dm = require('./dm');
// Cosmetics and tags are Live's (monetization/cosmetics, game/tags); read through live-context.
const cosmetics = { getCosmeticProfile: (userId) => ctx.getCosmeticProfile(userId) };
@@@ -79,10 +80,7 @@ class ChatServer {
/** @type {Map<number, number>} streamId → slow mode ms (0 = off, default rate limit applies) */
this.slowModeByStream = new Map();
this.heartbeatInterval = null;
-- /** @type {Map<number, number>} streamId → current TTS queue size */
-- this.ttsQueueSize = new Map();
-- /** @type {Map<string, number>} `${streamId}:${userId}` → user's TTS queue count */
-- this.ttsUserCounts = new Map();
++ // TTS and sound requests queue in Chat's database (./audio-queue.js), one room at a time.
/** @type {Map<string, number>} `${streamId}:${userKey}` → last soundboard trigger */
this.soundboardRateLimits = new Map();
/** @type {Map<number, {count: number, windowStart: number}>} streamId → stream-level soundboard rate window */
@@@ -212,6 +210,18 @@ class ChatServer {
this.handleConnection(ws, req);
});
++ // The TTS and sound queue: how each kind of request is made when its turn comes, and
++ // where its frame goes. recover() resumes what a restart interrupted.
++ audioQueue.init({
++ performers: {
++ tts: (row, p) => this._makeTtsAudio(p),
++ 'channel-sound': (row, p) => this._makeChannelSoundAudio(p),
++ soundboard: (row, p) => this._makeSoundboardAudio(p),
++ },
++ deliver: (row, frame) => this._broadcastTtsPayload(row.stream_id, row.channel_user_id, frame),
++ });
++ try { audioQueue.recover(); } catch (err) { console.warn('[AudioQueue] recover failed:', err.message); }
++
console.log('[Chat] WebSocket chat server initialized');
return this.wss;
}
@@@ -424,6 +434,8 @@ class ChatServer {
}
client.channelUserId = channelUserId;
client._modStream = null;
++ // A TTS/sound queue held since a restart plays once its room has listeners again.
++ audioQueue.roomJoined(audioQueue.roomKey({ streamId: client.streamId, channelUserId: client.streamId ? null : channelUserId }));
// Update viewer counts for old and new streams
if (oldStream !== client.streamId) {
if (oldStream) this.broadcastUserCount(oldStream);
@@@ -1311,12 +1323,8 @@ class ChatServer {
return;
}
-- // Read + encode the clip
++ // The clip is read when its turn in the room's audio queue comes.
if (!sound.url || !fs.existsSync(sound.url)) return;
-- let audioB64;
-- try {
-- audioB64 = fs.readFileSync(sound.url).toString('base64');
-- } catch { return; }
this.soundboardRateLimits.set(rateKey, now);
streamWindow.count += 1;
@@@ -1377,17 +1385,14 @@ class ChatServer {
this.broadcastToStream(client.streamId, soundMsg);
// Also surface it on the global chat feed / global overlay (with stream_channel).
this.forwardToGlobal(client.streamId, soundMsg);
-- this.broadcastToStream(client.streamId, {
-- type: 'soundboard-audio',
-- username,
-- title: `!${cmd}`,
-- audio: audioB64,
-- mimeType: sound.mime || 'audio/mpeg',
-- pitch: mods.pitch,
-- speed: mods.speed,
-- pitchShift: mods.pitchShift,
-- source: 'channel-sound',
-- timestamp: new Date().toISOString(),
++ audioQueue.enqueue({
++ kind: 'channel-sound',
++ streamId: client.streamId,
++ requestedBy: username,
++ identityKey: relay ? `relay:${relay.username}` : (client.user ? `user:${client.user.username}` : `anon:${client.anonId}`),
++ label: `!${cmd}`,
++ payload: { username, title: `!${cmd}`, file: sound.url, mimeType: sound.mime || 'audio/mpeg', seconds: Number(sound.duration_seconds) || 0, pitch: mods.pitch, speed: mods.speed, pitchShift: mods.pitchShift },
++ dedupeKey: soundMsg.id ? `m${soundMsg.id}` : null,
});
} catch (err) {
console.error('[ChannelSound] trigger error:', err.message);
@@@ -1419,7 +1424,7 @@ class ChatServer {
message: `Commands: /help, /tts <message>, /color <#hex>, /viewers, /uptime, /me <action>, /paste <content>` +
`\nMedia: !sr/!yt/!youtube/!req/!request <url>, !queue, !nowplaying` +
(this.canModerate(client)
-- ? `\nMod: /ban <user>, /unban <user>, /timeout <user> [seconds], /clear, /slow <seconds>`
++ ? `\nMod: /ban <user>, /unban <user>, /timeout <user> [seconds], /clear, /slow <seconds>, /skiptts [id], /cleartts`
: ''),
});
break;
@@@ -1548,6 +1553,28 @@ class ChatServer {
this.handleModAction(ws, client, 'timeout', args);
break;
++ case 'skiptts':
++ case 'cleartts': {
++ // The room's TTS and sound queue (./audio-queue.js): moderators and the broadcaster.
++ // /skiptts [request id] skips the clip playing now (or that request); /cleartts
++ // skips everything playing or waiting.
++ if (!this.canModerate(client)) { this.sendTo(ws, { type: 'system', message: 'You do not have permission.' }); return; }
++ const room = audioQueue.roomKey({ streamId: client.streamId, channelUserId: client.streamId ? null : client.channelUserId });
++ if (!room) { this.sendTo(ws, { type: 'system', message: 'No TTS or sounds here.' }); return; }
++ const actor = client.user ? `user:${client.user.username}` : null;
++ if (cmd === 'skiptts') {
++ const id = parseInt(argParts[0], 10);
++ const skipped = audioQueue.skip(room, { id: Number.isFinite(id) && id > 0 ? id : null, actor });
++ this.sendTo(ws, { type: 'system', message: skipped ? `Skipped ${skipped.kind === 'tts' ? 'TTS' : 'sound'} #${skipped.id}${skipped.requested_by ? ` from ${skipped.requested_by}` : ''}.` : 'Nothing to skip.' });
++ if (skipped) this.logChatModeration(client, 'tts_skip', { request_id: skipped.id, kind: skipped.kind });
++ } else {
++ const ids = audioQueue.clear(room, { actor });
++ this.sendTo(ws, { type: 'system', message: ids.length ? `Cleared ${ids.length} TTS/sound request${ids.length === 1 ? '' : 's'}.` : 'The TTS and sound queue is empty.' });
++ if (ids.length) this.logChatModeration(client, 'tts_clear', { request_ids: ids });
++ }
++ return;
++ }
++
case 'clear':
if (this.canModerate(client)) {
this._broadcastToRoom(client, { type: 'clear' });
@@@ -1740,24 +1767,30 @@ class ChatServer {
// streamId may be null for OFFLINE channel chat — pass channelUserId instead and
// the audio is delivered to the channel room rather than a stream room.
-- async synthesizeAndBroadcastTTS(streamId, username, text, voiceFX, sourcePlatform = null, identityKey = null, channelUserId = null, ttsKey = null) {
++ //
++ // Queues the utterance (./audio-queue.js); it is synthesized and sent when its turn comes.
++ // Returns the queue's answer ({ queued, id } or { queued: false, reason }) or undefined when
++ // the message is not read at all.
++ synthesizeAndBroadcastTTS(streamId, username, text, voiceFX, sourcePlatform = null, identityKey = null, channelUserId = null, ttsKey = null) {
// Queue accounting key: per-stream when live, per-channel when offline.
const queueKey = streamId || (channelUserId ? `ch:${channelUserId}` : null);
// Duplicate suppression by MESSAGE IDENTITY, not content: the same chat
// message synthesized twice (duplicated bridge, double-fired relay) is a dupe;
// a user legitimately typing the same text again is NOT and must be read
-- // again. Callers pass ttsKey (the persisted message id). The content fallback
-- // for id-less paths uses a 2s window — wide enough for a racing double
-- // delivery, far too narrow to eat a real repeat.
-- if (!this._recentTtsDedupe) this._recentTtsDedupe = new Map();
-- const dedupeKey = ttsKey ? `k:${queueKey}|${ttsKey}` : `${queueKey}|${identityKey || username}|${String(text).slice(0, 200)}`;
-- const windowMs = ttsKey ? 30000 : 2000;
-- const nowMs = Date.now();
-- const lastMs = this._recentTtsDedupe.get(dedupeKey);
-- if (lastMs && nowMs - lastMs < windowMs) { console.log(`[TTS] deduped duplicate synth for ${username} in ${queueKey} (${ttsKey ? 'same message id' : 'same content <2s'})`); return; }
-- this._recentTtsDedupe.set(dedupeKey, nowMs);
-- if (this._recentTtsDedupe.size > 500) {
-- for (const [k, t] of this._recentTtsDedupe) if (nowMs - t > 60000) this._recentTtsDedupe.delete(k);
++ // again. Callers pass ttsKey (the persisted message id) — the queue keeps it, so a keyed
++ // message is queued once per room even across a restart. The content fallback for
++ // id-less paths uses a 2s window — wide enough for a racing double delivery, far too
++ // narrow to eat a real repeat.
++ if (!ttsKey) {
++ if (!this._recentTtsDedupe) this._recentTtsDedupe = new Map();
++ const dedupeKey = `${queueKey}|${identityKey || username}|${String(text).slice(0, 200)}`;
++ const nowMs = Date.now();
++ const lastMs = this._recentTtsDedupe.get(dedupeKey);
++ if (lastMs && nowMs - lastMs < 2000) { console.log(`[TTS] deduped duplicate synth for ${username} in ${queueKey} (same content <2s)`); return; }
++ this._recentTtsDedupe.set(dedupeKey, nowMs);
++ if (this._recentTtsDedupe.size > 500) {
++ for (const [k, t] of this._recentTtsDedupe) if (nowMs - t > 60000) this._recentTtsDedupe.delete(k);
++ }
}
try {
// "." prefix = user opted this message out of TTS — never synthesize or broadcast it.
@@@ -1766,68 +1799,119 @@ class ChatServer {
const settings = ttsEngine.getTTSSettings();
if (!settings.enabled) return;
-- // Queue limit checks
const limits = ttsEngine.getQueueLimits();
-- const globalCount = this.ttsQueueSize.get(queueKey) || 0;
-- if (globalCount >= limits.maxGlobal) return;
--
-- // Determine voice ID from equipped cosmetic
-- let voiceId = null;
-- if (voiceFX?.itemId && ttsEngine.VOICE_CATALOG[voiceFX.itemId]) {
-- voiceId = voiceFX.itemId;
-- }
--
-- // Increment queue counter
-- this.ttsQueueSize.set(queueKey, globalCount + 1);
++ const r = audioQueue.enqueue({
++ kind: 'tts',
++ streamId: streamId || null,
++ channelUserId: streamId ? null : channelUserId,
++ requestedBy: username || null,
++ identityKey: identityKey || null,
++ label: String(text || '').slice(0, 300),
++ payload: { streamId: streamId || null, channelUserId: channelUserId || null, username, text, voiceFX: voiceFX || null, sourcePlatform: sourcePlatform || null, identityKey: identityKey || null, ttsKey: ttsKey || null },
++ dedupeKey: ttsKey ? `tts:${ttsKey}` : null,
++ maxRoom: limits.maxGlobal,
++ maxPerRequester: limits.maxPerUser,
++ });
++ if (!r.queued && r.reason === 'duplicate') console.log(`[TTS] deduped duplicate synth for ${username} in ${queueKey} (same message id)`);
++ return r;
++ } catch (err) {
++ console.error('[TTS] queue error:', err.message);
++ }
++ }
-- // Honor the channel's configured TTS length (streamers can raise it up to 1200);
-- // falls back to the site default when unset. Without this the server synth always
-- // truncated at the global 200 even when the channel allowed more.
-- let ttsMaxOverride;
-- try { ttsMaxOverride = Number(this._getChannelChatSettings(streamId).tts_max_length) || undefined; } catch { /* use engine default */ }
++ /** The queue's TTS performer: synthesize now, return the tts-audio frame (./audio-queue.js). */
++ async _makeTtsAudio(p) {
++ const settings = ttsEngine.getTTSSettings();
++ if (!settings.enabled) throw new Error('TTS is disabled');
-- let result;
-- if (!voiceId && settings.perUserVoices) {
-- // No equipped cosmetic voice → give this chatter a stable per-username voice.
-- const idKey = identityKey || username || 'anon';
-- result = await ttsEngine.synthesizeUserVoice(text, idKey, username, ttsMaxOverride);
-- } else {
-- result = await ttsEngine.synthesize(text, voiceId || settings.defaultVoice, username, ttsMaxOverride);
-- }
++ // Determine voice ID from equipped cosmetic
++ let voiceId = null;
++ if (p.voiceFX?.itemId && ttsEngine.VOICE_CATALOG[p.voiceFX.itemId]) {
++ voiceId = p.voiceFX.itemId;
++ }
-- // Decrement queue counter
-- const current = this.ttsQueueSize.get(queueKey) || 1;
-- this.ttsQueueSize.set(queueKey, Math.max(0, current - 1));
++ // Honor the channel's configured TTS length (streamers can raise it up to 1200);
++ // falls back to the site default when unset. Without this the server synth always
++ // truncated at the global 200 even when the channel allowed more.
++ let ttsMaxOverride;
++ try { ttsMaxOverride = Number(this._getChannelChatSettings(p.streamId).tts_max_length) || undefined; } catch { /* use engine default */ }
-- if (!result) return;
++ let result;
++ if (!voiceId && settings.perUserVoices) {
++ // No equipped cosmetic voice → give this chatter a stable per-username voice.
++ const idKey = p.identityKey || p.username || 'anon';
++ result = await ttsEngine.synthesizeUserVoice(p.text, idKey, p.username, ttsMaxOverride);
++ } else {
++ result = await ttsEngine.synthesize(p.text, voiceId || settings.defaultVoice, p.username, ttsMaxOverride);
++ }
++ if (!result) return null;
-- // Broadcast TTS audio to all clients in the stream
-- this._broadcastTtsPayload(streamId, channelUserId, {
++ return {
++ durationMs: audioQueue.estimatePlayMs({ audio: result.audio, mimeType: result.mimeType }),
++ frame: {
type: 'tts-audio',
-- username,
++ username: p.username,
// Stable sender identity ("user:<login>" / "anon:<id>") so clients can
// skip their OWN message's TTS locally — senders were hearing their
// message twice (their tab + the stream audio).
-- sender_key: identityKey || undefined,
++ sender_key: p.identityKey || undefined,
// Unique per utterance — clients dedupe playback on this, so identical
// TEXT from separate messages still reads every time.
-- ttsKey: ttsKey || undefined,
-- message: text,
++ ttsKey: p.ttsKey || undefined,
++ message: p.text,
audio: result.audio,
mimeType: result.mimeType,
engine: result.engine,
voiceName: result.voiceName,
voiceId: result.voiceId,
fallback: result.fallback || false,
-- source_platform: sourcePlatform || undefined,
++ source_platform: p.sourcePlatform || undefined,
timestamp: new Date().toISOString(),
-- });
-- } catch (err) {
-- // Ensure queue counter is decremented on error
-- const current = this.ttsQueueSize.get(queueKey) || 1;
-- this.ttsQueueSize.set(queueKey, Math.max(0, current - 1));
-- console.error('[TTS] Synthesis broadcast error:', err.message);
-- }
++ },
++ };
++ }
++
++ /** The queue's channel !sound performer: read the clip now, return the soundboard-audio frame. */
++ async _makeChannelSoundAudio(p) {
++ if (!p.file || !fs.existsSync(p.file)) throw new Error('sound file missing');
++ const audio = (await fs.promises.readFile(p.file)).toString('base64');
++ return {
++ durationMs: audioQueue.estimatePlayMs({ audio, mimeType: p.mimeType, seconds: p.seconds, speed: p.speed }),
++ frame: {
++ type: 'soundboard-audio',
++ username: p.username,
++ title: p.title,
++ audio,
++ mimeType: p.mimeType || 'audio/mpeg',
++ pitch: p.pitch,
++ speed: p.speed,
++ pitchShift: p.pitchShift,
++ source: 'channel-sound',
++ timestamp: new Date().toISOString(),
++ },
++ };
++ }
++
++ /** The queue's 101soundboards performer: the clip (cached since the request), as a soundboard-audio frame. */
++ async _makeSoundboardAudio(p) {
++ const result = await soundboard.getSoundboardAudio(p.soundId);
++ if (!result) return null;
++ return {
++ durationMs: audioQueue.estimatePlayMs({ audio: result.audio, mimeType: result.mimeType, speed: p.speed }),
++ frame: {
++ type: 'soundboard-audio',
++ username: p.username,
++ audio: result.audio,
++ mimeType: result.mimeType,
++ soundId: result.soundId,
++ title: result.title,
++ sourceUrl: result.sourceUrl,
++ pitch: p.pitch,
++ speed: p.speed,
++ pitchShift: p.pitchShift,
++ timestamp: new Date().toISOString(),
++ },
++ };
}
/**
@@@ -1947,18 +2031,14 @@ class ChatServer {
} catch { /* non-critical */ }
this.broadcastToStream(streamId, sbMsg);
-- this.broadcastToStream(streamId, {
-- type: 'soundboard-audio',
-- username,
-- audio: result.audio,
-- mimeType: result.mimeType,
-- soundId: result.soundId,
-- title: result.title,
-- sourceUrl: result.sourceUrl,
-- pitch: parsed.pitch,
-- speed: parsed.speed,
-- pitchShift: parsed.pitchShift,
-- timestamp: new Date().toISOString(),
++ audioQueue.enqueue({
++ kind: 'soundboard',
++ streamId,
++ requestedBy: username,
++ identityKey: client.user ? `user:${client.user.username}` : `anon:${client.anonId}`,
++ label: result.title,
++ payload: { username, soundId: result.soundId, pitch: parsed.pitch, speed: parsed.speed, pitchShift: parsed.pitchShift },
++ dedupeKey: sbMsg.id ? `m${sbMsg.id}` : null,
});
} catch (err) {
if (String(text || '').trim().startsWith('!sb')) {
@@@ -2434,6 +2514,7 @@ class ChatServer {
clearInterval(this._autoDeleteSweepInterval);
this._autoDeleteSweepInterval = null;
}
++ audioQueue.stop();
this.wss.clients.forEach(ws => ws.close());
this.wss.close();
}
ddiff --git a/server/chat/tts-routes.js b/server/chat/tts-routes.js
iindex fdcc35b..6dfc226 100644
---- a/server/chat/tts-routes.js
++++ b/server/chat/tts-routes.js
@@@ -12,6 +12,15 @@
* Admin routes:
* GET /api/tts/admin/settings — Full TTS config with API keys
* PUT /api/tts/admin/settings — Update TTS config
++ *
++ * The room's TTS and sound queue (./audio-queue.js) — the broadcaster and moderators of the room
++ * (channel mods, global mods, admins). The room is `stream_id`, or `channel_user_id` for offline
++ * channel chat (query string for GET, body for POST):
++ * GET /api/tts/queue — { room, playing, queued, recent }
++ * POST /api/tts/queue/skip — { id? } skip that request, else the one playing (else the next)
++ * POST /api/tts/queue/clear — skip everything playing or waiting
++ * POST /api/tts/queue/:id/report — { state: 'played' | 'failed', error? } the playing client's
++ * report that its clip ended or could not play
*/
const express = require('express');
const router = express.Router();
@@@ -19,8 +28,11 @@ const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const { requireAuth, requireAdmin } = require('../auth/auth');
--const { isOwner } = require('../auth/permissions');
++const permissions = require('../auth/permissions');
++const { isOwner } = permissions;
const ttsEngine = require('./tts-engine');
++const audioQueue = require('./audio-queue');
++const db = require('../db/database');
const ctx = require('../live-context');
const config = require('../config');
@@@ -161,6 +173,89 @@ router.post('/admin/test', requireAuth, requireAdmin, async (req, res) => {
}
});
++// ── The room's TTS and sound queue ────────────────────────────
++/**
++ * Resolve the room a queue request names and check the caller may control it: the broadcaster
++ * (stream or channel owner), the channel's moderators, global mods and admins. Sends the error and
++ * returns null when not.
++ */
++async function queueRoom(req, res) {
++ const src = req.method === 'GET' ? req.query : (req.body || {});
++ const streamId = parseInt(src.stream_id, 10) || null;
++ const channelUserId = streamId ? null : (parseInt(src.channel_user_id, 10) || null);
++ if (!streamId && !channelUserId) { res.status(400).json({ error: 'stream_id or channel_user_id is required' }); return null; }
++ let allowed = false;
++ let channel = null;
++ if (streamId) {
++ const stream = await ctx.ensureStream(streamId).catch(() => null);
++ if (!stream) { res.status(404).json({ error: 'No such stream' }); return null; }
++ channel = stream.channel_id ? ctx.getChannelById(stream.channel_id) : ctx.getChannelByUserId(stream.user_id);
++ if (channel) await ctx.ensurePolicy(channel.id).catch(() => {}); // channel moderators
++ allowed = permissions.canModerateStream(req.user, streamId);
++ } else {
++ channel = ctx.getChannelByUserId(channelUserId);
++ if (channel) await ctx.ensurePolicy(channel.id).catch(() => {});
++ allowed = req.user.id === channelUserId || permissions.isGlobalModOrAbove(req.user)
++ || (!!channel && permissions.isChannelMod(req.user, channel.id));
++ }
++ if (!allowed) { res.status(403).json({ error: 'Only the broadcaster and moderators can manage the TTS and sound queue' }); return null; }
++ return { room: audioQueue.roomKey({ streamId, channelUserId }), streamId, channelUserId, channel };
++}
++
++function logQueueAction(req, r, action, details) {
++ try {
++ db.logModerationAction({
++ scope_type: r.channel ? 'channel' : 'site',
++ scope_id: r.channel ? r.channel.id : undefined,
++ actor_user_id: req.user.id,
++ action_type: action,
++ details: { stream_id: r.streamId, channel_user_id: r.channelUserId, ...details },
++ });
++ } catch { /* non-critical */ }
++}
++
++router.get('/queue', requireAuth, async (req, res) => {
++ try {
++ const r = await queueRoom(req, res);
++ if (!r) return;
++ res.json(audioQueue.list(r.room, { recent: req.query.recent }));
++ } catch (e) { res.status(500).json({ error: e.message }); }
++});
++
++router.post('/queue/skip', requireAuth, async (req, res) => {
++ try {
++ const r = await queueRoom(req, res);
++ if (!r) return;
++ const id = parseInt(req.body && req.body.id, 10) || null;
++ const skipped = audioQueue.skip(r.room, { id, actor: `user:${req.user.username}` });
++ if (!skipped) return res.status(id ? 404 : 409).json({ error: id ? 'No such request waiting or playing' : 'Nothing to skip' });
++ logQueueAction(req, r, 'tts_skip', { request_id: skipped.id, kind: skipped.kind });
++ res.json({ skipped: { id: skipped.id, kind: skipped.kind, requested_by: skipped.requested_by }, queue: audioQueue.list(r.room) });
++ } catch (e) { res.status(500).json({ error: e.message }); }
++});
++
++router.post('/queue/clear', requireAuth, async (req, res) => {
++ try {
++ const r = await queueRoom(req, res);
++ if (!r) return;
++ const ids = audioQueue.clear(r.room, { actor: `user:${req.user.username}` });
++ if (ids.length) logQueueAction(req, r, 'tts_clear', { request_ids: ids });
++ res.json({ cleared: ids, queue: audioQueue.list(r.room) });
++ } catch (e) { res.status(500).json({ error: e.message }); }
++});
++
++router.post('/queue/:id/report', requireAuth, async (req, res) => {
++ try {
++ const r = await queueRoom(req, res);
++ if (!r) return;
++ const state = String((req.body && req.body.state) || '');
++ if (state !== 'played' && state !== 'failed') return res.status(400).json({ error: "state must be 'played' or 'failed'" });
++ const ok = audioQueue.report(r.room, parseInt(req.params.id, 10), state, { error: req.body && req.body.error, actor: `user:${req.user.username}` });
++ if (!ok) return res.status(409).json({ error: 'That request is not playing' });
++ res.json({ ok: true, queue: audioQueue.list(r.room) });
++ } catch (e) { res.status(500).json({ error: e.message }); }
++});
++
// Cached one-off clips (admin tests, previews) — streamed same-origin so no blob:/data: is needed.
router.get('/audio/:file', async (req, res) => {
try {
ddiff --git a/server/db/schema.sql b/server/db/schema.sql
iindex cb037b3..e2488ba 100644
---- a/server/db/schema.sql
++++ b/server/db/schema.sql
@@@ -401,6 +401,33 @@ CREATE TABLE IF NOT EXISTS bridge_applied (
applied_at INTEGER NOT NULL
);
++-- The TTS and sound queue (server/chat/audio-queue.js): one row per TTS utterance, channel !sound
++-- or 101soundboards clip, played one at a time per room ('stream:<id>' | 'channel:<userId>').
++-- queued → playing → played, or skipped (moderators, the broadcaster) / failed. payload holds what
++-- is needed to make the audio when its turn comes, so the queue survives a restart. Times are ms.
++CREATE TABLE IF NOT EXISTS audio_requests (
++ id INTEGER PRIMARY KEY AUTOINCREMENT,
++ room TEXT NOT NULL,
++ stream_id INTEGER,
++ channel_user_id INTEGER,
++ kind TEXT NOT NULL CHECK(kind IN ('tts', 'channel-sound', 'soundboard')),
++ state TEXT NOT NULL DEFAULT 'queued' CHECK(state IN ('queued', 'playing', 'played', 'skipped', 'failed')),
++ requested_by TEXT,
++ identity_key TEXT,
++ label TEXT,
++ payload TEXT NOT NULL DEFAULT '{}',
++ dedupe_key TEXT,
++ attempts INTEGER NOT NULL DEFAULT 0,
++ duration_ms INTEGER,
++ error TEXT,
++ actor TEXT,
++ created_at INTEGER NOT NULL,
++ started_at INTEGER,
++ finished_at INTEGER,
++ UNIQUE(room, dedupe_key)
++);
++CREATE INDEX IF NOT EXISTS idx_audio_requests_room_state ON audio_requests(room, state, id);
++
-- Live's placeholder ids (a large negative number per Live boot) → the real chat_messages id, so a
-- later op of the same Live boot that carries the placeholder is rewritten even after a Chat
-- restart (the batch that acknowledged the insert and the one carrying its broadcast can straddle
ddiff --git a/test/audio-queue.test.js b/test/audio-queue.test.js
nnew file mode 100644
iindex 0000000..b6b612d
---- /dev/null
++++ b/test/audio-queue.test.js
@@@ -0,0 +1,301 @@
++'use strict';
++/**
++ * The persisted TTS and sound queue (server/chat/audio-queue.js, roadmap Wave 6 D4): requests are
++ * rows that go queued → playing → played, or skipped / failed; one plays at a time per room; the
++ * broadcaster and moderators skip and clear (/skiptts, /cleartts, /api/tts/queue); failures are
++ * recorded and the queue moves on; a keyed request is queued once; and a Chat restart (a real
++ * process, SIGTERM) keeps the queue: what was playing is finished, what waited plays, in order,
++ * once. The frames clients already know (tts-audio, soundboard-audio) are unchanged apart from a
++ * new request_id.
++ */
++const assert = require('assert');
++const fs = require('fs');
++const path = require('path');
++const Database = require('better-sqlite3');
++const { boot, suite } = require('./helpers');
++
++const t = suite('audio-queue');
++let h, aq, tts, streamer, mod, viewer, listenerUser, chatters, channelId, streamId, listener;
++const sockets = [];
++const realSynth = {};
++
++/** 8 kHz 8-bit mono WAV of `seconds`, base64 (1 s = 8000 bytes of audio). */
++function wav(seconds) {
++ const n = Math.floor(8000 * seconds);
++ const b = Buffer.alloc(44 + n);
++ b.write('RIFF', 0); b.writeUInt32LE(36 + n, 4); b.write('WAVE', 8); b.write('fmt ', 12);
++ b.writeUInt32LE(16, 16); b.writeUInt16LE(1, 20); b.writeUInt16LE(1, 22); b.writeUInt32LE(8000, 24);
++ b.writeUInt32LE(8000, 28); b.writeUInt16LE(1, 32); b.writeUInt16LE(8, 34); b.write('data', 36); b.writeUInt32LE(n, 40);
++ b.fill(128, 44);
++ return b;
++}
++
++let synthPlan = []; // per call: seconds of audio, null (no audio) or an Error
++function stubSynth() {
++ const fake = async (text) => {
++ const step = synthPlan.length ? synthPlan.shift() : 0.6;
++ if (step instanceof Error) throw step;
++ if (step == null) return null;
++ return { audio: wav(step).toString('base64'), mimeType: 'audio/wav', engine: 'test', voiceName: 'test', voiceId: 'test', text };
++ };
++ tts.synthesize = fake;
++ tts.synthesizeUserVoice = fake;
++}
++
++async function joined(user, ip, stream = streamId) {
++ const ws = await h.ws({ ip, token: user && user.token, stream });
++ sockets.push(ws);
++ ws.sendJson({ type: 'join', streamId: stream, ...(user ? { token: user.token } : {}) });
++ await ws.next((m) => m.type === 'auth');
++ return ws;
++}
++const rows = () => h.db.all('SELECT * FROM audio_requests ORDER BY id');
++const byLabel = (label) => h.db.get('SELECT * FROM audio_requests WHERE label = ?', [label]);
++async function until(fn, ms = 4000) {
++ const end = Date.now() + ms;
++ for (;;) { const v = fn(); if (v) return v; if (Date.now() > end) throw new Error('condition not met in time'); await h.sleep(25); }
++}
++
++t('boot', async () => {
++ h = await boot();
++ aq = require('../server/chat/audio-queue');
++ tts = require('../server/chat/tts-engine');
++ realSynth.synthesize = tts.synthesize;
++ realSynth.synthesizeUserVoice = tts.synthesizeUserVoice;
++ streamer = h.addUser('streamer', { role: 'streamer' });
++ mod = h.addUser('moddy');
++ viewer = h.addUser('viewer');
++ listenerUser = h.addUser('listener');
++ chatters = ['ann', 'ben', 'cat', 'dan'].map((n) => h.addUser(n));
++ channelId = h.addChannel(streamer.id, { moderators: [mod.id] });
++ streamId = h.addStream(streamer.id, channelId);
++ await h.ctx.sync();
++ listener = await joined(listenerUser, '198.51.100.90');
++});
++
++t('the state machine and clip lengths', async () => {
++ assert.deepStrictEqual(aq.STATES, ['queued', 'playing', 'played', 'skipped', 'failed']);
++ const allowed = [];
++ for (const a of aq.STATES) for (const b of aq.STATES) if (aq.canTransition(a, b)) allowed.push(`${a}>${b}`);
++ assert.deepStrictEqual(allowed.sort(), ['playing>failed', 'playing>played', 'playing>skipped', 'queued>failed', 'queued>playing', 'queued>skipped'].sort());
++ assert.strictEqual(aq.estimatePlayMs({ audio: wav(1.5).toString('base64'), mimeType: 'audio/wav' }), 1500);
++ // MPEG-1 layer III, 128 kbit/s frames behind an ID3 tag: 16000 bytes = 1 s.
++ const mp3 = Buffer.alloc(16000);
++ mp3.write('ID3', 0); mp3[3] = 4; mp3[9] = 20; // tag of 20 bytes after the 10-byte header
++ mp3[30] = 0xff; mp3[31] = 0xfb; mp3[32] = 0x90;
++ assert.strictEqual(aq.mp3Bitrate(mp3), 128000);
++ assert.strictEqual(aq.estimatePlayMs({ audio: mp3.toString('base64'), mimeType: 'audio/mpeg' }), 1000);
++ assert.strictEqual(aq.estimatePlayMs({ seconds: 2, speed: 0.5 }), 4000, 'a slowed clip plays longer');
++ assert.strictEqual(aq.estimatePlayMs({ seconds: 600 }), 60000, 'capped');
++});
++
++let first, second, third;
++t('TTS of chat messages: queued rows, delivered one at a time, each frame carries request_id', async () => {
++ stubSynth();
++ synthPlan = [1.2, 1.2, 6];
++ const t0 = Date.now();
++ for (const [i, c] of chatters.slice(0, 3).entries()) {
++ const ws = await joined(c, `198.51.100.${100 + i}`);
++ ws.sendJson({ type: 'chat', message: `hello from ${c.username}` });
++ await ws.next((m) => m.type === 'chat' && m.message === `hello from ${c.username}`);
++ }
++ first = await listener.next((m) => m.type === 'tts-audio' && m.message === 'hello from ann');
++ assert.ok(first.request_id > 0);
++ assert.ok(first.audio && first.mimeType === 'audio/wav' && first.ttsKey, 'the frame clients already play');
++ assert.strictEqual(aq.getRequest(first.request_id).state, 'playing');
++ assert.strictEqual(byLabel('hello from ben').state, 'queued');
++ assert.strictEqual(byLabel('hello from cat').state, 'queued');
++ assert.ok(await listener.none((m) => m.type === 'tts-audio' && m.message === 'hello from ben', 400), 'the next waits for the one playing');
++ second = await listener.next((m) => m.type === 'tts-audio' && m.message === 'hello from ben', 3000);
++ assert.ok(Date.now() - t0 >= 1200, 'paced by the clip length');
++ const r1 = aq.getRequest(first.request_id);
++ assert.strictEqual(r1.state, 'played');
++ assert.strictEqual(r1.duration_ms, 1200);
++ assert.strictEqual(r1.kind, 'tts');
++ assert.strictEqual(r1.room, `stream:${streamId}`);
++});
++
++t('only the broadcaster and moderators skip: /skiptts skips the playing clip and the next goes out at once', async () => {
++ const v = await joined(viewer, '198.51.100.110');
++ v.sendJson({ type: 'chat', message: '/skiptts' });
++ await v.next((m) => m.type === 'system' && m.message === 'You do not have permission.');
++ assert.strictEqual(aq.getRequest(second.request_id).state, 'playing');
++
++ const s = await joined(streamer, '198.51.100.111');
++ const at = Date.now();
++ s.sendJson({ type: 'chat', message: '/skiptts' });
++ const reply = await s.next((m) => m.type === 'system' && /^Skipped TTS/.test(m.message));
++ assert.match(reply.message, new RegExp(`#${second.request_id} from Ben`));
++ const stop = await listener.next((m) => m.type === 'audio-skip');
++ assert.strictEqual(stop.request_id, second.request_id);
++ const row = aq.getRequest(second.request_id);
++ assert.strictEqual(row.state, 'skipped');
++ assert.strictEqual(row.actor, 'user:streamer');
++ third = await listener.next((m) => m.type === 'tts-audio' && m.message === 'hello from cat');
++ assert.ok(Date.now() - at < 900, 'no waiting out the skipped clip');
++ const log = h.db.get("SELECT * FROM moderation_actions WHERE action_type = 'tts_skip' ORDER BY id DESC LIMIT 1");
++ assert.strictEqual(log.actor_user_id, streamer.id);
++});
++
++t('the queue over REST: list, skip by id, clear — moderators yes, others 403', async () => {
++ // Fill the room: cat plays, three more wait (through Live's bridge, keyed like chat messages).
++ synthPlan = [5, 5, 5];
++ const BRIDGE = h.serviceToken(['chat.live_bridge.write', 'chat.message.send']);
++ const r = await h.http('POST', '/internal/live/calls', { token: BRIDGE, body: { boot: 'b', ops: [1, 2, 3].map((n) => ({ seq: n, op: 'synthesizeAndBroadcastTTS', args: [streamId, 'ChatBot', `bot line ${n}`, null, 'ai', 'ai:bot', null, `m9${n}`] })) } });
++ assert.ok(r.body.results.every((x) => x.ok), r.text);
++
++ assert.strictEqual((await h.http('GET', `/api/tts/queue?stream_id=${streamId}`, { token: viewer.token })).status, 403);
++ assert.strictEqual((await h.http('GET', `/api/tts/queue?stream_id=${streamId}`)).status, 401);
++ const list = await h.http('GET', `/api/tts/queue?stream_id=${streamId}`, { token: mod.token });
++ assert.strictEqual(list.status, 200, list.text);
++ assert.strictEqual(list.body.playing.id, third.request_id);
++ assert.deepStrictEqual(list.body.queued.map((x) => x.label), ['bot line 1', 'bot line 2', 'bot line 3']);
++ assert.ok(list.body.recent.some((x) => x.id === second.request_id && x.state === 'skipped'));
++
++ const two = list.body.queued[1].id;
++ assert.strictEqual((await h.http('POST', '/api/tts/queue/skip', { token: viewer.token, body: { stream_id: streamId, id: two } })).status, 403);
++ const sk = await h.http('POST', '/api/tts/queue/skip', { token: mod.token, body: { stream_id: streamId, id: two } });
++ assert.strictEqual(sk.status, 200, sk.text);
++ assert.strictEqual(aq.getRequest(two).state, 'skipped');
++ assert.strictEqual(aq.getRequest(third.request_id).state, 'playing', 'skipping a waiting request leaves the playing one');
++ assert.ok(await listener.none((m) => m.type === 'audio-skip' && m.request_id === two, 150), 'nothing to stop on clients for a request never delivered');
++
++ const cl = await h.http('POST', '/api/tts/queue/clear', { token: mod.token, body: { stream_id: streamId } });
++ assert.strictEqual(cl.status, 200, cl.text);
++ assert.strictEqual(cl.body.cleared.length, 3, 'the playing one and the two still waiting');
++ const frame = await listener.next((m) => m.type === 'audio-clear');
++ assert.deepStrictEqual(frame.request_ids.sort((a, b) => a - b), cl.body.cleared.sort((a, b) => a - b));
++ assert.strictEqual(cl.body.queue.playing, null);
++ assert.deepStrictEqual(cl.body.queue.queued, []);
++ assert.ok(await listener.none((m) => m.type === 'tts-audio' && /^bot line/.test(m.message), 300), 'nothing cleared is read');
++ assert.ok(h.db.get("SELECT 1 FROM moderation_actions WHERE action_type = 'tts_clear' AND actor_user_id = ?", [mod.id]));
++ assert.strictEqual((await h.http('POST', '/api/tts/queue/skip', { token: mod.token, body: { stream_id: streamId } })).status, 409, 'nothing to skip');
++});
++
++t('failures are recorded and the queue moves on', async () => {
++ synthPlan = [null, new Error('voice service down'), 6];
++ const BRIDGE = h.serviceToken(['chat.live_bridge.write', 'chat.message.send']);
++ await h.http('POST', '/internal/live/calls', { token: BRIDGE, body: { boot: 'b', ops: ['no audio', 'throws', 'fine'].map((l, i) => ({ seq: i + 1, op: 'synthesizeAndBroadcastTTS', args: [streamId, 'ChatBot', l, null, 'ai', `ai:bot${i}`, null, `mf${i}`] })) } });
++ await listener.next((m) => m.type === 'tts-audio' && m.message === 'fine');
++ assert.strictEqual(byLabel('no audio').state, 'failed');
++ assert.strictEqual(byLabel('no audio').error, 'no audio');
++ assert.strictEqual(byLabel('throws').state, 'failed');
++ assert.strictEqual(byLabel('throws').error, 'voice service down');
++ assert.strictEqual(byLabel('fine').state, 'playing');
++});
++
++t('the playing client reports: failed with its reason; final states stay final', async () => {
++ const fine = byLabel('fine');
++ const bad = await h.http('POST', `/api/tts/queue/${fine.id}/report`, { token: streamer.token, body: { stream_id: streamId, state: 'skipped' } });
++ assert.strictEqual(bad.status, 400);
++ const rep = await h.http('POST', `/api/tts/queue/${fine.id}/report`, { token: streamer.token, body: { stream_id: streamId, state: 'failed', error: 'NotAllowedError' } });
++ assert.strictEqual(rep.status, 200, rep.text);
++ assert.strictEqual(aq.getRequest(fine.id).state, 'failed');
++ assert.strictEqual(aq.getRequest(fine.id).error, 'NotAllowedError');
++ assert.strictEqual((await h.http('POST', `/api/tts/queue/${fine.id}/report`, { token: streamer.token, body: { stream_id: streamId, state: 'played' } })).status, 409, 'final states stay final');
++});
++
++t('a keyed request is queued once; a requester’s limit holds', async () => {
++ synthPlan = [3, 3, 3, 3, 3, 3];
++ const before = rows().length;
++ const a = h.chatServer.synthesizeAndBroadcastTTS(streamId, 'Ann', 'same message', null, null, 'user:ann', null, 'm777');
++ const b = h.chatServer.synthesizeAndBroadcastTTS(streamId, 'Ann', 'same message', null, null, 'user:ann', null, 'm777');
++ assert.strictEqual(a.queued, true);
++ assert.deepStrictEqual(b, { queued: false, reason: 'duplicate' });
++ // tts_max_queue_per_user defaults to 3 (queued or playing).
++ const more = [1, 2, 3].map((n) => h.chatServer.synthesizeAndBroadcastTTS(streamId, 'Ann', `more ${n}`, null, null, 'user:ann', null, `m78${n}`));
++ assert.deepStrictEqual(more.map((x) => x.queued), [true, true, false]);
++ assert.strictEqual(more[2].reason, 'full');
++ assert.strictEqual(rows().length, before + 3);
++ assert.strictEqual(h.chatServer.synthesizeAndBroadcastTTS(streamId, 'Ann', '.not read', null, null, 'user:ann', null, 'm790'), undefined, "'.' opts a message out");
++ const s = await joined(streamer, '198.51.100.112');
++ s.sendJson({ type: 'chat', message: '/cleartts' });
++ await s.next((m) => m.type === 'system' && /^Cleared 3 TTS\/sound requests/.test(m.message));
++});
++
++t('channel !sounds go through the same queue (announce at once, audio in turn)', async () => {
++ const file = path.join(process.env.SOUNDS_PATH, 'honk-test.wav');
++ fs.writeFileSync(file, wav(0.8));
++ h.db.createChannelSound({ channel_owner_id: streamer.id, command: 'honk', url: file, mime: 'audio/wav', duration_seconds: 0.8 });
++ const c = await joined(chatters[3], '198.51.100.113');
++ c.sendJson({ type: 'chat', message: '!honk' });
++ const ann = await listener.next((m) => m.type === 'chat' && m.message_type === 'channel-sound');
++ assert.strictEqual(ann.message, 'played !honk');
++ const audio = await listener.next((m) => m.type === 'soundboard-audio' && m.title === '!honk');
++ assert.strictEqual(audio.source, 'channel-sound');
++ assert.strictEqual(Buffer.from(audio.audio, 'base64').length, fs.statSync(file).size);
++ const row = aq.getRequest(audio.request_id);
++ assert.strictEqual(row.kind, 'channel-sound');
++ assert.strictEqual(row.duration_ms, 800);
++ assert.strictEqual(row.dedupe_key, `m${ann.id}`);
++ await until(() => aq.getRequest(audio.request_id).state === 'played', 3000);
++});
++
++// ── Restart: Chat as its own process ────────────────────────────────────────────────────────────
++let chat, port, dbFile;
++const raw = (fn) => { const d = new Database(dbFile); try { return fn(d); } finally { d.close(); } };
++t('restart: what was playing is finished, what waited plays in order once, stale requests expire', async () => {
++ tts.synthesize = realSynth.synthesize;
++ tts.synthesizeUserVoice = realSynth.synthesizeUserVoice;
++ dbFile = path.join(h.tmp, 'chat.db');
++ for (const w of sockets) { try { w.close(); } catch { /* */ } }
++ await h.detach();
++ // Long enough clips that the queue is still full when Chat stops.
++ const file = path.join(process.env.SOUNDS_PATH, 'long-test.wav');
++ fs.writeFileSync(file, wav(2));
++ raw((d) => {
++ d.prepare("INSERT INTO channel_sounds (channel_owner_id, command, url, mime, duration_seconds) VALUES (?, 'long', ?, 'audio/wav', 2)").run(streamer.id, file);
++ // Left over from an earlier life: one queued an hour ago, one that crashed Chat three times.
++ const ins = d.prepare("INSERT INTO audio_requests (room, stream_id, kind, state, label, payload, created_at, attempts) VALUES (?, ?, 'channel-sound', 'queued', ?, ?, ?, ?)");
++ ins.run(`stream:${streamId}`, streamId, 'stale', JSON.stringify({ file, title: '!stale' }), Date.now() - 3600e3, 0);
++ ins.run(`stream:${streamId}`, streamId, 'crashy', JSON.stringify({ file, title: '!crashy' }), Date.now(), 3);
++ });
++ port = await h.freePort();
++ chat = await h.spawnChat({ port });
++ assert.strictEqual(raw((d) => d.prepare("SELECT state, error FROM audio_requests WHERE label = 'stale'").get()).error, 'expired');
++ // The room's queue is held until it has listeners again (a second after the first rejoins).
++ await h.sleep(300);
++ assert.strictEqual(raw((d) => d.prepare("SELECT state FROM audio_requests WHERE label = 'crashy'").get().state), 'queued', 'held for listeners');
++ const ears = await joined(listenerUser, '198.51.100.120');
++ await until(() => raw((d) => d.prepare("SELECT state FROM audio_requests WHERE label = 'crashy'").get().state) === 'failed');
++ assert.strictEqual(raw((d) => d.prepare("SELECT error FROM audio_requests WHERE label = 'crashy'").get()).error, 'gave up after repeated attempts');
++
++ for (const [i, c] of chatters.slice(0, 3).entries()) {
++ const ws = await joined(c, `198.51.100.${121 + i}`);
++ ws.sendJson({ type: 'chat', message: '!long' });
++ await ws.next((m) => m.type === 'chat' && m.message_type === 'channel-sound');
++ ws.close();
++ }
++ const firstClip = await ears.next((m) => m.type === 'soundboard-audio' && m.title === '!long');
++ const queued = raw((d) => d.prepare("SELECT id, state FROM audio_requests WHERE label = '!long' ORDER BY id").all());
++ assert.deepStrictEqual(queued.map((x) => x.state), ['playing', 'queued', 'queued']);
++ assert.strictEqual(queued[0].id, firstClip.request_id);
++
++ const exit = await h.stopChat(chat);
++ assert.deepStrictEqual(exit, { code: 0, signal: null }, chat.log);
++ chat = await h.spawnChat({ port });
++ await h.sleep(300);
++ assert.strictEqual(raw((d) => d.prepare('SELECT state FROM audio_requests WHERE id = ?').get(queued[1].id).state), 'queued', 'nothing plays into an empty room');
++ const ears2 = await joined(listenerUser, '198.51.100.130');
++ const got = [];
++ got.push(await ears2.next((m) => m.type === 'soundboard-audio' && m.title === '!long', 4000));
++ got.push(await ears2.next((m) => m.type === 'soundboard-audio' && m.title === '!long', 5000));
++ assert.deepStrictEqual(got.map((m) => m.request_id), [queued[1].id, queued[2].id], 'the waiting clips, in order');
++ assert.ok(await ears2.none((m) => m.type === 'soundboard-audio' && m.request_id === queued[0].id, 100), 'the clip delivered before the restart is not played again');
++ const after = raw((d) => d.prepare("SELECT id, state, error FROM audio_requests WHERE label = '!long' ORDER BY id").all());
++ assert.strictEqual(after[0].state, 'played');
++ assert.strictEqual(after[0].error, 'delivered before a restart');
++ assert.strictEqual(after[1].state, 'played');
++ assert.strictEqual(after[2].state, 'playing');
++
++ // Moderation still works on the new process.
++ const s = await joined(mod, '198.51.100.131');
++ s.sendJson({ type: 'chat', message: '/cleartts' });
++ await s.next((m) => m.type === 'system' && /^Cleared 1 TTS\/sound request\./.test(m.message));
++ assert.strictEqual(raw((d) => d.prepare('SELECT state FROM audio_requests WHERE id = ?').get(queued[2].id).state), 'skipped');
++});
++
++t.run(async () => {
++ for (const w of sockets) { try { w.close(); } catch { /* */ } }
++ if (h) await h.close();
++});