openvibers/openvibe.live Commits

What was heard, as caption cues (item 151, C15) (#583)

OP OpenVibers committed 10 days ago · 795324c509 · 5 files, +177 −22

GET /api/vods/:id/sounds gives the VOD's sounds as caption cues in its own time: [laughter], [applause], [music],
[dog barking]. sound-captions.js maps a short list of classifier labels, each above its own bar (set from production:
the long tail like Vehicle and Animal is mostly wrong and never captioned; explosions and gunfire need 0.85 and must
be rare in the stream), merges a burst into one cue, and gives music one cue where it starts. Nothing is written by
a model. The memories route's shift into a part's own time is now inVodTime, shared by both.

Changes

ddiff --git a/docs/vods-and-clips.md b/docs/vods-and-clips.md
iindex af23bf7a..cc19bf59 100644
---- a/docs/vods-and-clips.md
++++ b/docs/vods-and-clips.md
@@@ -145,7 +145,13 @@ From the dashboard:
Every recording and clip has its page on OpenVibe.Video (`https://openvibe.video/v/<id>`, `/c/<id>`;
`server/web/video-url.js`, `VIDEO_PUBLIC_URL` overrides the origin): the player, AI chapters and captions, the chat
--replay, comments, clipping, and the owner's visibility and delete, for private items too (to their owner). Live's old
++replay, comments, clipping, and the owner's visibility and delete, for private items too (to their owner). The
++captions name what was heard as well as what was said: `GET /api/vods/:id/sounds` gives cues like "[laughter]",
++"[applause]" and "[music]" in the VOD's own time (item 151, C15; `server/ai/sound-captions.js`). Each tag comes from
++a classifier label above its own bar, set from production (applause and laughter from 0.7, a dog or a siren from 0.8);
++nothing is written by a model, so nothing is invented. A burst is one cue of at most 6 s; music gives one cue where it
++starts; explosions and gunfire need 0.85 and must be rare in the stream (in a shooter they are the game); the
++classifier's long tail ("Vehicle", "Animal", "Glass") is never captioned. Same visibility rule as `memories`. Live's old
`/vod/<id>` and `/clip/<id>` URLs answer a 301 there (`server/web/video-handover.js`, mounted before the SEO renderer
like the paste handover; a `?t=` start time goes along as seconds). Every watch link Live renders points there: the
SPA through `videoHref()` (the shell's `ov-video-base` meta names the origin), the server through `videoUrl()`. Live's
@@@ -185,7 +191,7 @@ answers 204 whether or not it counted.
## Who sees a private VOD or clip
--A private item (visibility `private`, or a legacy row with no visibility and `is_public = 0`) is visible only to its owners — the uploader or clipper, the clipped channel's streamer, the streamer whose stream it came from — and to staff (admin, global mod). Everyone else gets exactly the answer an unknown id gets: the same 404 and body on `/api/vods/:id` and its `live-info`/`memories`/`context`, `/api/clips/:id`, comments (including a comment's replies, edit and delete), thumbnail regeneration and the write routes (a 403 would confirm the id exists). It is also left out of `/api/chat-ai/vod-transcripts`, the server-rendered `/vod/:id` and `/clip/:id` pages, and `/api/streams/recent`. The rule lives in `server/media-proxy/access.js`; Media applies the same rule to `openvibe.media/v/:id` and `/c/:id`. Unlisted items stay reachable by direct link.
++A private item (visibility `private`, or a legacy row with no visibility and `is_public = 0`) is visible only to its owners — the uploader or clipper, the clipped channel's streamer, the streamer whose stream it came from — and to staff (admin, global mod). Everyone else gets exactly the answer an unknown id gets: the same 404 and body on `/api/vods/:id` and its `live-info`/`memories`/`sounds`/`context`, `/api/clips/:id`, comments (including a comment's replies, edit and delete), thumbnail regeneration and the write routes (a 403 would confirm the id exists). It is also left out of `/api/chat-ai/vod-transcripts`, the server-rendered `/vod/:id` and `/clip/:id` pages, and `/api/streams/recent`. The rule lives in `server/media-proxy/access.js`; Media applies the same rule to `openvibe.media/v/:id` and `/c/:id`. Unlisted items stay reachable by direct link.
## Comments are OpenVibe.Community threads
ddiff --git a/server/ai/sound-captions.js b/server/ai/sound-captions.js
nnew file mode 100644
iindex 00000000..e625906a
---- /dev/null
++++ b/server/ai/sound-captions.js
@@@ -0,0 +1,84 @@
++'use strict';
++
++/**
++ * sound-captions.js — what a stream's audio classifier heard, as caption cues: "[laughter]", "[applause]", "[music]"
++ * (item 151, C15: grounded audio descriptions). The VOD's captions on OpenVibe.Video carry them between the spoken
++ * lines, the way captions for deaf and hard-of-hearing viewers name the sounds that matter.
++ *
++ * Grounded by construction: every cue is one of a short list of tags, each from a classifier label (YAMNet's
++ * AudioSet names, server/ai/audio-events.js) above that tag's own bar. Nothing is written by a model, so nothing can be
++ * invented: no speaker, no reason, no "possibly". The bars come from production (2026-11-03, read-only): applause
++ * averaged 0.77 and is kept from 0.7, laughter is rare and noisy and is kept from 0.7, while explosions and gunfire
++ * fill a shooter's evening at 0.6: they are captioned only from 0.85 and only where they are rare in the stream (at
++ * most 12 an hour; in a shooter they are the game, not news). The long tail ("Vehicle", "Animal", "Glass", "Liquid")
++ * is never captioned: it is mostly wrong.
++ *
++ * cues(events, { spanSec }) → [{ start, end, text }] events: [{ start_sec, end_sec, label, confidence }]
++ *
++ * A burst becomes one cue (same tag within 4 s), 1.5 to 6 seconds long. Background music (audio-events.js spans,
++ * label "Music") gives one "[music]" cue where it starts, not one under every line.
++ */
++
++const clipScore = require('./clip-score');
++
++// [label pattern, tag, minimum confidence, only where rare in the stream]
++const TAGS = [
++ [/^(laughter|giggle|chuckle, chortle|belly laugh|snicker|baby laughter)$/i, 'laughter', 0.7, false],
++ [/^(applause|clapping)$/i, 'applause', 0.7, false],
++ [/^(cheering|crowd|children shouting)$/i, 'cheering', 0.7, false],
++ [/^whistling$/i, 'whistling', 0.75, false],
++ [/^(crying, sobbing|baby cry, infant cry|whimper)$/i, 'crying', 0.75, false],
++ [/^screaming$/i, 'screaming', 0.8, false],
++ [/^(dog|bark|yip|howl|bow-wow)$/i, 'dog barking', 0.8, false],
++ [/^(cat|meow|purr)$/i, 'cat meowing', 0.8, false],
++ [/^(doorbell|ding-dong)$/i, 'doorbell', 0.8, false],
++ [/^(telephone bell ringing|ringtone)$/i, 'phone ringing', 0.8, false],
++ [/^(siren|police car \(siren\)|ambulance \(siren\)|fire engine, fire truck \(siren\)|civil defense siren)$/i, 'siren', 0.8, false],
++ [/^explosion$/i, 'explosion', 0.85, true],
++ [/^(gunshot, gunfire|machine gun|fusillade)$/i, 'gunfire', 0.85, true],
++];
++const MERGE_GAP = 4;
++const MIN_LEN = 1.5;
++const MAX_LEN = 6;
++const MUSIC_LEN = 3;
++
++function tagOf(label) {
++ const l = String(label || '').trim();
++ for (const [re, tag, min, rareOnly] of TAGS) if (re.test(l)) return { tag, min, rareOnly };
++ return null;
++}
++
++function cues(events, { spanSec = 0 } = {}) {
++ const list = (Array.isArray(events) ? events : []).filter((e) => e && Number.isFinite(Number(e.start_sec)));
++ const rare = new Set(clipScore.rareSounds(list, spanSec).map((e) => String(e.label || '')));
++ const raw = [];
++ for (const e of list) {
++ const start = Math.max(0, Number(e.start_sec));
++ const end = Number.isFinite(Number(e.end_sec)) && Number(e.end_sec) > start ? Number(e.end_sec) : start + MIN_LEN;
++ if (String(e.label || '') === 'Music') {
++ if (Number(e.confidence || 0) >= 0.5) raw.push({ start, end: start + MUSIC_LEN, text: '[music]' });
++ continue;
++ }
++ const t = tagOf(e.label);
++ if (!t || Number(e.confidence || 0) < t.min) continue;
++ if (t.rareOnly && !rare.has(String(e.label || ''))) continue;
++ raw.push({ start, end, text: `[${t.tag}]` });
++ }
++ raw.sort((a, b) => a.start - b.start || a.text.localeCompare(b.text));
++ const out = [];
++ const last = new Map(); // tag → its latest cue, so a burst becomes one cue
++ for (const c of raw) {
++ const prev = last.get(c.text);
++ if (prev && c.start - prev.end <= MERGE_GAP && c.text !== '[music]') {
++ prev.end = Math.min(prev.start + MAX_LEN, Math.max(prev.end, c.end));
++ continue;
++ }
++ if (prev && c.text === '[music]' && c.start - prev.end <= MERGE_GAP) continue;
++ const cue = { start: c.start, end: Math.min(c.start + MAX_LEN, Math.max(c.start + MIN_LEN, c.end)), text: c.text };
++ out.push(cue);
++ last.set(c.text, cue);
++ }
++ return out.map((c) => ({ start: Math.round(c.start * 100) / 100, end: Math.round(c.end * 100) / 100, text: c.text }));
++}
++
++module.exports = { cues, tagOf, TAGS };
ddiff --git a/server/media-proxy/vods.js b/server/media-proxy/vods.js
iindex 091d047d..8e1bfc3a 100644
---- a/server/media-proxy/vods.js
++++ b/server/media-proxy/vods.js
@@@ -420,6 +420,28 @@ router.post('/bulk-delete-old', requireAuth, async (req, res) => {
}
});
++/**
++ * A stream's rows (stream time, in `key`) in this VOD's own time. The first part of a joined recording answers for the
++ * whole video: every row that was recorded, in the video's time (streaming/recordings.js; the gaps between parts were
++ * not). A VOD that is one part of its stream (the broadcaster reconnected) keeps the rows inside it, moved to its time:
++ * its chapters (OpenVibe.Video) pointed eleven minutes late on part 2 of stream 2795. `round` keeps whole seconds.
++ */
++async function inVodTime(vod, rows, key, { round = true } = {}) {
++ const fix = (v) => (round ? Math.round(v) : Math.round(v * 100) / 100);
++ if (vod.recording && Number(vod.recording.vod_id) === Number(vod.id) && vod.recording.hls_url && (vod.recording.parts || []).length > 1) {
++ const recordings = require('../streaming/recordings');
++ const parts = await recordings.presentParts(vod.recording);
++ return rows.map((r) => ({ r, at: recordings.toRecordingTime(parts, Number(r[key])) }))
++ .filter((x) => x.at != null).map(({ r, at }) => ({ ...r, [key]: fix(at), ...(r.end != null ? { end: fix(at + (Number(r.end) - Number(r[key]))) } : {}) }));
++ }
++ const part = await require('../streaming/stream-history').partForVod(vod.id);
++ if (!part) return rows;
++ const len = part.end != null ? part.end - part.at : Infinity;
++ return rows
++ .filter((r) => Number(r[key]) >= part.at && Number(r[key]) - part.at <= len)
++ .map((r) => ({ ...r, [key]: fix(Number(r[key]) - part.at), ...(r.end != null ? { end: fix(Number(r.end) - part.at) } : {}) }));
++}
++
// AI "memory" timeline for a VOD (stream_memories stays in live.db).
router.get('/:id/memories', optionalAuth, async (req, res) => {
try {
@@@ -427,35 +449,38 @@ router.get('/:id/memories', optionalAuth, async (req, res) => {
if (!vod) return;
const streamId = vod.stream_id || null;
if (!streamId) return res.json({ memories: [] });
-- let memories = (await db.getStreamMemories(streamId) || []).map(m => ({
++ const memories = (await db.getStreamMemories(streamId) || []).map(m => ({
offset_seconds: m.offset_seconds,
description: db.publicMemoryText(m.description),
tags: m.tags ? (() => { try { return JSON.parse(m.tags); } catch { return []; } })() : [],
}));
-- // The first part of a joined recording answers for the whole video: every moment that was recorded, in the
-- // video's time (streaming/recordings.js; the gaps between parts were not).
-- if (vod.recording && Number(vod.recording.vod_id) === Number(vod.id) && vod.recording.hls_url && (vod.recording.parts || []).length > 1) {
-- const recordings = require('../streaming/recordings');
-- const parts = await recordings.presentParts(vod.recording);
-- memories = memories.map((m) => ({ ...m, offset_seconds: recordings.toRecordingTime(parts, m.offset_seconds) }))
-- .filter((m) => m.offset_seconds != null).map((m) => ({ ...m, offset_seconds: Math.round(m.offset_seconds) }));
-- return res.json({ memories });
-- }
-- // A VOD that is one part of its stream (the broadcaster reconnected): the moments inside it, in its own time.
-- // Its chapters (OpenVibe.Video) pointed eleven minutes late on part 2 of stream 2795.
-- const part = await require('../streaming/stream-history').partForVod(vod.id);
-- if (part) {
-- const len = part.end != null ? part.end - part.at : Infinity;
-- memories = memories
-- .filter((m) => Number(m.offset_seconds) >= part.at && Number(m.offset_seconds) - part.at <= len)
-- .map((m) => ({ ...m, offset_seconds: Math.round(Number(m.offset_seconds) - part.at) }));
-- }
-- res.json({ memories });
++ res.json({ memories: await inVodTime(vod, memories, 'offset_seconds') });
} catch {
res.status(500).json({ error: 'Failed to load AI timeline' });
}
});
++/**
++ * GET /api/vods/:id/sounds — what was heard in the VOD, as caption cues in its own time: "[laughter]", "[applause]",
++ * "[music]" (item 151, C15; server/ai/sound-captions.js: a short list of tags, each above its own bar, nothing written
++ * by a model). OpenVibe.Video puts them in the VOD's captions between the spoken lines. Empty for a stream recorded
++ * before the timeline, or with nothing worth a tag.
++ */
++router.get('/:id/sounds', optionalAuth, async (req, res) => {
++ try {
++ const vod = await loadVisibleVod(req, res, req.params.id);
++ if (!vod) return;
++ const streamId = vod.stream_id || null;
++ if (!streamId) return res.json({ cues: [] });
++ const events = await db.getTimeline(streamId, { kind: 'sound', limit: 20000 }) || [];
++ const last = events.reduce((m, e) => Math.max(m, Number(e.end_sec) || Number(e.start_sec) || 0), 0);
++ const cues = require('../ai/sound-captions').cues(events, { spanSec: last });
++ res.json({ cues: await inVodTime(vod, cues, 'start', { round: false }) });
++ } catch {
++ res.status(500).json({ error: 'Failed to load the sounds' });
++ }
++});
++
/**
* GET /api/vods/:id/context — everything around a VOD that isn't the file: the stream it came
* from, its activity numbers (chat, viewers, mic moments, follows, clips), the clips cut from
ddiff --git a/test/sound-captions.test.js b/test/sound-captions.test.js
nnew file mode 100644
iindex 00000000..cd849a61
---- /dev/null
++++ b/test/sound-captions.test.js
@@@ -0,0 +1,25 @@
++'use strict';
++// What was heard, as caption cues (item 151, C15; server/ai/sound-captions.js): a short list of tags, each from a
++// classifier label above its own bar, nothing written by a model. A burst is one cue; background music gives one cue
++// where it starts; explosions and gunfire only where they are rare (in a shooter they are the game); the classifier's
++// long tail is never captioned.
++const assert = require('assert');
++const sc = require('../server/ai/sound-captions');
++const ev = (start_sec, label, confidence, end_sec) => ({ start_sec, end_sec, label, confidence });
++
++assert.deepStrictEqual(sc.cues([ev(10, 'Laughter', 0.9, 11), ev(12, 'Giggle', 0.8, 13.2), ev(30, 'Laughter', 0.75)]),
++ [{ start: 10, end: 13.2, text: '[laughter]' }, { start: 30, end: 31.5, text: '[laughter]' }], 'a burst is one cue; one 17 s later is another');
++assert.deepStrictEqual(sc.cues([ev(10, 'Laughter', 0.6), ev(20, 'Applause', 0.69)]), [], 'below the bars, nothing');
++assert.deepStrictEqual(sc.cues([ev(5, 'Vehicle', 0.95), ev(6, 'Animal', 0.95), ev(7, 'Glass', 0.95), ev(8, 'Speech', 0.99)]), [], 'the long tail is never captioned');
++assert.deepStrictEqual(sc.cues([ev(100, 'Applause', 0.8, 160)]), [{ start: 100, end: 106, text: '[applause]' }], 'a cue lasts at most 6 s');
++assert.deepStrictEqual(sc.cues([ev(0, 'Music', 0.7, 300), ev(302, 'Music', 0.7, 400), ev(500, 'Music', 0.4, 600)]),
++ [{ start: 0, end: 3, text: '[music]' }, { start: 302, end: 305, text: '[music]' }], 'one cue where music starts, not under every line');
++
++// Explosions: captioned from 0.85, and only where rare in the stream.
++const shooter = Array.from({ length: 40 }, (_, i) => ev(i * 90, 'Explosion', 0.9, i * 90 + 1));
++assert.deepStrictEqual(sc.cues(shooter, { spanSec: 3600 }), [], '40 explosions an hour are the game');
++assert.deepStrictEqual(sc.cues([ev(50, 'Explosion', 0.9, 51), ev(70, 'Gunshot, gunfire', 0.84)], { spanSec: 3600 }),
++ [{ start: 50, end: 51.5, text: '[explosion]' }], 'one rare explosion is news; gunfire below 0.85 is not captioned');
++assert.deepStrictEqual(sc.cues(null), []);
++assert.strictEqual(sc.tagOf('Dog').tag, 'dog barking');
++console.log('sound captions: all checks passed');
ddiff --git a/test/vod-downloads.test.js b/test/vod-downloads.test.js
iindex ac2b152f..0368766c 100644
---- a/test/vod-downloads.test.js
++++ b/test/vod-downloads.test.js
@@@ -58,6 +58,11 @@ async function check(name, fn) {
for (const [off, text] of [[100, 'birds in part one'], [900, 'robot in part two'], [1500, 'squirrels in part two']]) {
await raw.prepare('INSERT INTO stream_memories (stream_id, user_id, offset_seconds, description) VALUES (?, 80, ?, ?)').run(sid, off, text);
}
++ // What was heard (C15): a laugh and a rare explosion in part two, applause in part one, and a "Vehicle" the
++ // classifier hears everywhere and is never captioned.
++ for (const [at, end, label, conf] of [[150, 152, 'Applause', 0.8], [920, 921, 'Laughter', 0.9], [922, 923.5, 'Laughter', 0.8], [930, 931, 'Vehicle', 0.9], [1600, 1601, 'Explosion', 0.9]]) {
++ await raw.prepare(`INSERT INTO stream_timeline_events (stream_id, user_id, kind, start_sec, end_sec, label, confidence) VALUES (?, 80, 'sound', ?, ?, ?, ?)`).run(sid, at, end, label, conf);
++ }
const media = require('../server/media-client');
const VODS = {
@@@ -138,6 +143,16 @@ async function check(name, fn) {
assert.deepStrictEqual(m1.json.memories.map((m) => [m.offset_seconds, m.description]), [[72, 'birds in part one']]);
});
++ await check('what was heard, as caption cues in the part\'s own time (C15)', async () => {
++ const s2 = await call('/api/vods/6026/sounds');
++ assert.strictEqual(s2.status, 200, s2.text);
++ assert.deepStrictEqual(s2.json.cues, [{ start: 211, end: 214.5, text: '[laughter]' }, { start: 891, end: 892.5, text: '[explosion]' }],
++ 'one cue for the burst of laughter; no "[vehicle]"');
++ const s1 = await call('/api/vods/6025/sounds');
++ assert.deepStrictEqual(s1.json.cues.map((c) => [c.start, c.text]), [[122, '[applause]']]);
++ assert.strictEqual((await call('/api/vods/99999/sounds')).status, 404);
++ });
++
server.close();
if (failures) { quiet(`vod downloads: ${failures} failed`); process.exit(1); }
quiet('vod downloads: ok');