openvibers/openvibe.live Commits
What was heard, as caption cues (item 151, C15) (#583)
Changes
@@@ -145,7 +145,13 @@ From the dashboard:Every recording and clip has its page on OpenVibe.Video (`https://openvibe.video/v/<id>`, `/c/<id>`;`server/web/video-url.js`, `VIDEO_PUBLIC_URL` overrides the origin): the player, AI chapters and captions, the chat--replay, comments, clipping, and the owner's visibility and delete, for private items too (to their owner). Live's old++replay, comments, clipping, and the owner's visibility and delete, for private items too (to their owner). The++captions name what was heard as well as what was said: `GET /api/vods/:id/sounds` gives cues like "[laughter]",++"[applause]" and "[music]" in the VOD's own time (item 151, C15; `server/ai/sound-captions.js`). Each tag comes from++a classifier label above its own bar, set from production (applause and laughter from 0.7, a dog or a siren from 0.8);++nothing is written by a model, so nothing is invented. A burst is one cue of at most 6 s; music gives one cue where it++starts; explosions and gunfire need 0.85 and must be rare in the stream (in a shooter they are the game); the++classifier's long tail ("Vehicle", "Animal", "Glass") is never captioned. Same visibility rule as `memories`. Live's old`/vod/<id>` and `/clip/<id>` URLs answer a 301 there (`server/web/video-handover.js`, mounted before the SEO rendererlike the paste handover; a `?t=` start time goes along as seconds). Every watch link Live renders points there: theSPA through `videoHref()` (the shell's `ov-video-base` meta names the origin), the server through `videoUrl()`. Live's@@@ -185,7 +191,7 @@ answers 204 whether or not it counted.## Who sees a private VOD or clip--A private item (visibility `private`, or a legacy row with no visibility and `is_public = 0`) is visible only to its owners — the uploader or clipper, the clipped channel's streamer, the streamer whose stream it came from — and to staff (admin, global mod). Everyone else gets exactly the answer an unknown id gets: the same 404 and body on `/api/vods/:id` and its `live-info`/`memories`/`context`, `/api/clips/:id`, comments (including a comment's replies, edit and delete), thumbnail regeneration and the write routes (a 403 would confirm the id exists). It is also left out of `/api/chat-ai/vod-transcripts`, the server-rendered `/vod/:id` and `/clip/:id` pages, and `/api/streams/recent`. The rule lives in `server/media-proxy/access.js`; Media applies the same rule to `openvibe.media/v/:id` and `/c/:id`. Unlisted items stay reachable by direct link.++A private item (visibility `private`, or a legacy row with no visibility and `is_public = 0`) is visible only to its owners — the uploader or clipper, the clipped channel's streamer, the streamer whose stream it came from — and to staff (admin, global mod). Everyone else gets exactly the answer an unknown id gets: the same 404 and body on `/api/vods/:id` and its `live-info`/`memories`/`sounds`/`context`, `/api/clips/:id`, comments (including a comment's replies, edit and delete), thumbnail regeneration and the write routes (a 403 would confirm the id exists). It is also left out of `/api/chat-ai/vod-transcripts`, the server-rendered `/vod/:id` and `/clip/:id` pages, and `/api/streams/recent`. The rule lives in `server/media-proxy/access.js`; Media applies the same rule to `openvibe.media/v/:id` and `/c/:id`. Unlisted items stay reachable by direct link.## Comments are OpenVibe.Community threadsnnew file mode 100644@@@ -0,0 +1,84 @@++'use strict';++++/**++ * sound-captions.js — what a stream's audio classifier heard, as caption cues: "[laughter]", "[applause]", "[music]"++ * (item 151, C15: grounded audio descriptions). The VOD's captions on OpenVibe.Video carry them between the spoken++ * lines, the way captions for deaf and hard-of-hearing viewers name the sounds that matter.++ *++ * Grounded by construction: every cue is one of a short list of tags, each from a classifier label (YAMNet's++ * AudioSet names, server/ai/audio-events.js) above that tag's own bar. Nothing is written by a model, so nothing can be++ * invented: no speaker, no reason, no "possibly". The bars come from production (2026-11-03, read-only): applause++ * averaged 0.77 and is kept from 0.7, laughter is rare and noisy and is kept from 0.7, while explosions and gunfire++ * fill a shooter's evening at 0.6: they are captioned only from 0.85 and only where they are rare in the stream (at++ * most 12 an hour; in a shooter they are the game, not news). The long tail ("Vehicle", "Animal", "Glass", "Liquid")++ * is never captioned: it is mostly wrong.++ *++ * cues(events, { spanSec }) → [{ start, end, text }] events: [{ start_sec, end_sec, label, confidence }]++ *++ * A burst becomes one cue (same tag within 4 s), 1.5 to 6 seconds long. Background music (audio-events.js spans,++ * label "Music") gives one "[music]" cue where it starts, not one under every line.++ */++++const clipScore = require('./clip-score');++++// [label pattern, tag, minimum confidence, only where rare in the stream]++const TAGS = [++ [/^(laughter|giggle|chuckle, chortle|belly laugh|snicker|baby laughter)$/i, 'laughter', 0.7, false],++ [/^(applause|clapping)$/i, 'applause', 0.7, false],++ [/^(cheering|crowd|children shouting)$/i, 'cheering', 0.7, false],++ [/^whistling$/i, 'whistling', 0.75, false],++ [/^(crying, sobbing|baby cry, infant cry|whimper)$/i, 'crying', 0.75, false],++ [/^screaming$/i, 'screaming', 0.8, false],++ [/^(dog|bark|yip|howl|bow-wow)$/i, 'dog barking', 0.8, false],++ [/^(cat|meow|purr)$/i, 'cat meowing', 0.8, false],++ [/^(doorbell|ding-dong)$/i, 'doorbell', 0.8, false],++ [/^(telephone bell ringing|ringtone)$/i, 'phone ringing', 0.8, false],++ [/^(siren|police car \(siren\)|ambulance \(siren\)|fire engine, fire truck \(siren\)|civil defense siren)$/i, 'siren', 0.8, false],++ [/^explosion$/i, 'explosion', 0.85, true],++ [/^(gunshot, gunfire|machine gun|fusillade)$/i, 'gunfire', 0.85, true],++];++const MERGE_GAP = 4;++const MIN_LEN = 1.5;++const MAX_LEN = 6;++const MUSIC_LEN = 3;++++function tagOf(label) {++ const l = String(label || '').trim();++ for (const [re, tag, min, rareOnly] of TAGS) if (re.test(l)) return { tag, min, rareOnly };++ return null;++}++++function cues(events, { spanSec = 0 } = {}) {++ const list = (Array.isArray(events) ? events : []).filter((e) => e && Number.isFinite(Number(e.start_sec)));++ const rare = new Set(clipScore.rareSounds(list, spanSec).map((e) => String(e.label || '')));++ const raw = [];++ for (const e of list) {++ const start = Math.max(0, Number(e.start_sec));++ const end = Number.isFinite(Number(e.end_sec)) && Number(e.end_sec) > start ? Number(e.end_sec) : start + MIN_LEN;++ if (String(e.label || '') === 'Music') {++ if (Number(e.confidence || 0) >= 0.5) raw.push({ start, end: start + MUSIC_LEN, text: '[music]' });++ continue;++ }++ const t = tagOf(e.label);++ if (!t || Number(e.confidence || 0) < t.min) continue;++ if (t.rareOnly && !rare.has(String(e.label || ''))) continue;++ raw.push({ start, end, text: `[${t.tag}]` });++ }++ raw.sort((a, b) => a.start - b.start || a.text.localeCompare(b.text));++ const out = [];++ const last = new Map(); // tag → its latest cue, so a burst becomes one cue++ for (const c of raw) {++ const prev = last.get(c.text);++ if (prev && c.start - prev.end <= MERGE_GAP && c.text !== '[music]') {++ prev.end = Math.min(prev.start + MAX_LEN, Math.max(prev.end, c.end));++ continue;++ }++ if (prev && c.text === '[music]' && c.start - prev.end <= MERGE_GAP) continue;++ const cue = { start: c.start, end: Math.min(c.start + MAX_LEN, Math.max(c.start + MIN_LEN, c.end)), text: c.text };++ out.push(cue);++ last.set(c.text, cue);++ }++ return out.map((c) => ({ start: Math.round(c.start * 100) / 100, end: Math.round(c.end * 100) / 100, text: c.text }));++}++++module.exports = { cues, tagOf, TAGS };@@@ -420,6 +420,28 @@ router.post('/bulk-delete-old', requireAuth, async (req, res) => {}});++/**++ * A stream's rows (stream time, in `key`) in this VOD's own time. The first part of a joined recording answers for the++ * whole video: every row that was recorded, in the video's time (streaming/recordings.js; the gaps between parts were++ * not). A VOD that is one part of its stream (the broadcaster reconnected) keeps the rows inside it, moved to its time:++ * its chapters (OpenVibe.Video) pointed eleven minutes late on part 2 of stream 2795. `round` keeps whole seconds.++ */++async function inVodTime(vod, rows, key, { round = true } = {}) {++ const fix = (v) => (round ? Math.round(v) : Math.round(v * 100) / 100);++ if (vod.recording && Number(vod.recording.vod_id) === Number(vod.id) && vod.recording.hls_url && (vod.recording.parts || []).length > 1) {++ const recordings = require('../streaming/recordings');++ const parts = await recordings.presentParts(vod.recording);++ return rows.map((r) => ({ r, at: recordings.toRecordingTime(parts, Number(r[key])) }))++ .filter((x) => x.at != null).map(({ r, at }) => ({ ...r, [key]: fix(at), ...(r.end != null ? { end: fix(at + (Number(r.end) - Number(r[key]))) } : {}) }));++ }++ const part = await require('../streaming/stream-history').partForVod(vod.id);++ if (!part) return rows;++ const len = part.end != null ? part.end - part.at : Infinity;++ return rows++ .filter((r) => Number(r[key]) >= part.at && Number(r[key]) - part.at <= len)++ .map((r) => ({ ...r, [key]: fix(Number(r[key]) - part.at), ...(r.end != null ? { end: fix(Number(r.end) - part.at) } : {}) }));++}++// AI "memory" timeline for a VOD (stream_memories stays in live.db).router.get('/:id/memories', optionalAuth, async (req, res) => {try {@@@ -427,35 +449,38 @@ router.get('/:id/memories', optionalAuth, async (req, res) => {if (!vod) return;const streamId = vod.stream_id || null;if (!streamId) return res.json({ memories: [] });-- let memories = (await db.getStreamMemories(streamId) || []).map(m => ({++ const memories = (await db.getStreamMemories(streamId) || []).map(m => ({offset_seconds: m.offset_seconds,description: db.publicMemoryText(m.description),tags: m.tags ? (() => { try { return JSON.parse(m.tags); } catch { return []; } })() : [],}));-- // The first part of a joined recording answers for the whole video: every moment that was recorded, in the-- // video's time (streaming/recordings.js; the gaps between parts were not).-- if (vod.recording && Number(vod.recording.vod_id) === Number(vod.id) && vod.recording.hls_url && (vod.recording.parts || []).length > 1) {-- const recordings = require('../streaming/recordings');-- const parts = await recordings.presentParts(vod.recording);-- memories = memories.map((m) => ({ ...m, offset_seconds: recordings.toRecordingTime(parts, m.offset_seconds) }))-- .filter((m) => m.offset_seconds != null).map((m) => ({ ...m, offset_seconds: Math.round(m.offset_seconds) }));-- return res.json({ memories });-- }-- // A VOD that is one part of its stream (the broadcaster reconnected): the moments inside it, in its own time.-- // Its chapters (OpenVibe.Video) pointed eleven minutes late on part 2 of stream 2795.-- const part = await require('../streaming/stream-history').partForVod(vod.id);-- if (part) {-- const len = part.end != null ? part.end - part.at : Infinity;-- memories = memories-- .filter((m) => Number(m.offset_seconds) >= part.at && Number(m.offset_seconds) - part.at <= len)-- .map((m) => ({ ...m, offset_seconds: Math.round(Number(m.offset_seconds) - part.at) }));-- }-- res.json({ memories });++ res.json({ memories: await inVodTime(vod, memories, 'offset_seconds') });} catch {res.status(500).json({ error: 'Failed to load AI timeline' });}});++/**++ * GET /api/vods/:id/sounds — what was heard in the VOD, as caption cues in its own time: "[laughter]", "[applause]",++ * "[music]" (item 151, C15; server/ai/sound-captions.js: a short list of tags, each above its own bar, nothing written++ * by a model). OpenVibe.Video puts them in the VOD's captions between the spoken lines. Empty for a stream recorded++ * before the timeline, or with nothing worth a tag.++ */++router.get('/:id/sounds', optionalAuth, async (req, res) => {++ try {++ const vod = await loadVisibleVod(req, res, req.params.id);++ if (!vod) return;++ const streamId = vod.stream_id || null;++ if (!streamId) return res.json({ cues: [] });++ const events = await db.getTimeline(streamId, { kind: 'sound', limit: 20000 }) || [];++ const last = events.reduce((m, e) => Math.max(m, Number(e.end_sec) || Number(e.start_sec) || 0), 0);++ const cues = require('../ai/sound-captions').cues(events, { spanSec: last });++ res.json({ cues: await inVodTime(vod, cues, 'start', { round: false }) });++ } catch {++ res.status(500).json({ error: 'Failed to load the sounds' });++ }++});++/*** GET /api/vods/:id/context — everything around a VOD that isn't the file: the stream it came* from, its activity numbers (chat, viewers, mic moments, follows, clips), the clips cut fromnnew file mode 100644@@@ -0,0 +1,25 @@++'use strict';++// What was heard, as caption cues (item 151, C15; server/ai/sound-captions.js): a short list of tags, each from a++// classifier label above its own bar, nothing written by a model. A burst is one cue; background music gives one cue++// where it starts; explosions and gunfire only where they are rare (in a shooter they are the game); the classifier's++// long tail is never captioned.++const assert = require('assert');++const sc = require('../server/ai/sound-captions');++const ev = (start_sec, label, confidence, end_sec) => ({ start_sec, end_sec, label, confidence });++++assert.deepStrictEqual(sc.cues([ev(10, 'Laughter', 0.9, 11), ev(12, 'Giggle', 0.8, 13.2), ev(30, 'Laughter', 0.75)]),++ [{ start: 10, end: 13.2, text: '[laughter]' }, { start: 30, end: 31.5, text: '[laughter]' }], 'a burst is one cue; one 17 s later is another');++assert.deepStrictEqual(sc.cues([ev(10, 'Laughter', 0.6), ev(20, 'Applause', 0.69)]), [], 'below the bars, nothing');++assert.deepStrictEqual(sc.cues([ev(5, 'Vehicle', 0.95), ev(6, 'Animal', 0.95), ev(7, 'Glass', 0.95), ev(8, 'Speech', 0.99)]), [], 'the long tail is never captioned');++assert.deepStrictEqual(sc.cues([ev(100, 'Applause', 0.8, 160)]), [{ start: 100, end: 106, text: '[applause]' }], 'a cue lasts at most 6 s');++assert.deepStrictEqual(sc.cues([ev(0, 'Music', 0.7, 300), ev(302, 'Music', 0.7, 400), ev(500, 'Music', 0.4, 600)]),++ [{ start: 0, end: 3, text: '[music]' }, { start: 302, end: 305, text: '[music]' }], 'one cue where music starts, not under every line');++++// Explosions: captioned from 0.85, and only where rare in the stream.++const shooter = Array.from({ length: 40 }, (_, i) => ev(i * 90, 'Explosion', 0.9, i * 90 + 1));++assert.deepStrictEqual(sc.cues(shooter, { spanSec: 3600 }), [], '40 explosions an hour are the game');++assert.deepStrictEqual(sc.cues([ev(50, 'Explosion', 0.9, 51), ev(70, 'Gunshot, gunfire', 0.84)], { spanSec: 3600 }),++ [{ start: 50, end: 51.5, text: '[explosion]' }], 'one rare explosion is news; gunfire below 0.85 is not captioned');++assert.deepStrictEqual(sc.cues(null), []);++assert.strictEqual(sc.tagOf('Dog').tag, 'dog barking');++console.log('sound captions: all checks passed');@@@ -58,6 +58,11 @@ async function check(name, fn) {for (const [off, text] of [[100, 'birds in part one'], [900, 'robot in part two'], [1500, 'squirrels in part two']]) {await raw.prepare('INSERT INTO stream_memories (stream_id, user_id, offset_seconds, description) VALUES (?, 80, ?, ?)').run(sid, off, text);}++ // What was heard (C15): a laugh and a rare explosion in part two, applause in part one, and a "Vehicle" the++ // classifier hears everywhere and is never captioned.++ for (const [at, end, label, conf] of [[150, 152, 'Applause', 0.8], [920, 921, 'Laughter', 0.9], [922, 923.5, 'Laughter', 0.8], [930, 931, 'Vehicle', 0.9], [1600, 1601, 'Explosion', 0.9]]) {++ await raw.prepare(`INSERT INTO stream_timeline_events (stream_id, user_id, kind, start_sec, end_sec, label, confidence) VALUES (?, 80, 'sound', ?, ?, ?, ?)`).run(sid, at, end, label, conf);++ }const media = require('../server/media-client');const VODS = {@@@ -138,6 +143,16 @@ async function check(name, fn) {assert.deepStrictEqual(m1.json.memories.map((m) => [m.offset_seconds, m.description]), [[72, 'birds in part one']]);});++ await check('what was heard, as caption cues in the part\'s own time (C15)', async () => {++ const s2 = await call('/api/vods/6026/sounds');++ assert.strictEqual(s2.status, 200, s2.text);++ assert.deepStrictEqual(s2.json.cues, [{ start: 211, end: 214.5, text: '[laughter]' }, { start: 891, end: 892.5, text: '[explosion]' }],++ 'one cue for the burst of laughter; no "[vehicle]"');++ const s1 = await call('/api/vods/6025/sounds');++ assert.deepStrictEqual(s1.json.cues.map((c) => [c.start, c.text]), [[122, '[applause]']]);++ assert.strictEqual((await call('/api/vods/99999/sounds')).status, 404);++ });++server.close();if (failures) { quiet(`vod downloads: ${failures} failed`); process.exit(1); }quiet('vod downloads: ok');
@@ -145,7 +145,13 @@ From the dashboard:
@@ -145,7 +145,13 @@ From the dashboard:
Every recording and clip has its page on OpenVibe.Video (`https://openvibe.video/v/<id>`, `/c/<id>`;
Every recording and clip has its page on OpenVibe.Video (`https://openvibe.video/v/<id>`, `/c/<id>`;
`server/web/video-url.js`, `VIDEO_PUBLIC_URL` overrides the origin): the player, AI chapters and captions, the chat
`server/web/video-url.js`, `VIDEO_PUBLIC_URL` overrides the origin): the player, AI chapters and captions, the chat
-replay, comments, clipping, and the owner's visibility and delete, for private items too (to their owner). Live's old
+replay, comments, clipping, and the owner's visibility and delete, for private items too (to their owner). The
+captions name what was heard as well as what was said: `GET /api/vods/:id/sounds` gives cues like "[laughter]",
+"[applause]" and "[music]" in the VOD's own time (item 151, C15; `server/ai/sound-captions.js`). Each tag comes from
+a classifier label above its own bar, set from production (applause and laughter from 0.7, a dog or a siren from 0.8);
+nothing is written by a model, so nothing is invented. A burst is one cue of at most 6 s; music gives one cue where it
+starts; explosions and gunfire need 0.85 and must be rare in the stream (in a shooter they are the game); the
+classifier's long tail ("Vehicle", "Animal", "Glass") is never captioned. Same visibility rule as `memories`. Live's old
`/vod/<id>` and `/clip/<id>` URLs answer a 301 there (`server/web/video-handover.js`, mounted before the SEO renderer
`/vod/<id>` and `/clip/<id>` URLs answer a 301 there (`server/web/video-handover.js`, mounted before the SEO renderer
like the paste handover; a `?t=` start time goes along as seconds). Every watch link Live renders points there: the
like the paste handover; a `?t=` start time goes along as seconds). Every watch link Live renders points there: the
SPA through `videoHref()` (the shell's `ov-video-base` meta names the origin), the server through `videoUrl()`. Live's
SPA through `videoHref()` (the shell's `ov-video-base` meta names the origin), the server through `videoUrl()`. Live's
@@ -185,7 +191,7 @@ answers 204 whether or not it counted.
@@ -185,7 +191,7 @@ answers 204 whether or not it counted.
## Who sees a private VOD or clip
## Who sees a private VOD or clip
-A private item (visibility `private`, or a legacy row with no visibility and `is_public = 0`) is visible only to its owners — the uploader or clipper, the clipped channel's streamer, the streamer whose stream it came from — and to staff (admin, global mod). Everyone else gets exactly the answer an unknown id gets: the same 404 and body on `/api/vods/:id` and its `live-info`/`memories`/`context`, `/api/clips/:id`, comments (including a comment's replies, edit and delete), thumbnail regeneration and the write routes (a 403 would confirm the id exists). It is also left out of `/api/chat-ai/vod-transcripts`, the server-rendered `/vod/:id` and `/clip/:id` pages, and `/api/streams/recent`. The rule lives in `server/media-proxy/access.js`; Media applies the same rule to `openvibe.media/v/:id` and `/c/:id`. Unlisted items stay reachable by direct link.
+A private item (visibility `private`, or a legacy row with no visibility and `is_public = 0`) is visible only to its owners — the uploader or clipper, the clipped channel's streamer, the streamer whose stream it came from — and to staff (admin, global mod). Everyone else gets exactly the answer an unknown id gets: the same 404 and body on `/api/vods/:id` and its `live-info`/`memories`/`sounds`/`context`, `/api/clips/:id`, comments (including a comment's replies, edit and delete), thumbnail regeneration and the write routes (a 403 would confirm the id exists). It is also left out of `/api/chat-ai/vod-transcripts`, the server-rendered `/vod/:id` and `/clip/:id` pages, and `/api/streams/recent`. The rule lives in `server/media-proxy/access.js`; Media applies the same rule to `openvibe.media/v/:id` and `/c/:id`. Unlisted items stay reachable by direct link.
## Comments are OpenVibe.Community threads
## Comments are OpenVibe.Community threads
new file mode 100644
new file mode 100644
@@ -0,0 +1,84 @@
@@ -0,0 +1,84 @@
+'use strict';
+
+/**
+ * sound-captions.js — what a stream's audio classifier heard, as caption cues: "[laughter]", "[applause]", "[music]"
+ * (item 151, C15: grounded audio descriptions). The VOD's captions on OpenVibe.Video carry them between the spoken
+ * lines, the way captions for deaf and hard-of-hearing viewers name the sounds that matter.
+ *
+ * Grounded by construction: every cue is one of a short list of tags, each from a classifier label (YAMNet's
+ * AudioSet names, server/ai/audio-events.js) above that tag's own bar. Nothing is written by a model, so nothing can be
+ * invented: no speaker, no reason, no "possibly". The bars come from production (2026-11-03, read-only): applause
+ * averaged 0.77 and is kept from 0.7, laughter is rare and noisy and is kept from 0.7, while explosions and gunfire
+ * fill a shooter's evening at 0.6: they are captioned only from 0.85 and only where they are rare in the stream (at
+ * most 12 an hour; in a shooter they are the game, not news). The long tail ("Vehicle", "Animal", "Glass", "Liquid")
+ * is never captioned: it is mostly wrong.
+ *
+ * cues(events, { spanSec }) → [{ start, end, text }] events: [{ start_sec, end_sec, label, confidence }]
+ *
+ * A burst becomes one cue (same tag within 4 s), 1.5 to 6 seconds long. Background music (audio-events.js spans,
+ * label "Music") gives one "[music]" cue where it starts, not one under every line.
+ */
+
+const clipScore = require('./clip-score');
+
+// [label pattern, tag, minimum confidence, only where rare in the stream]
+const TAGS = [
+ [/^(laughter|giggle|chuckle, chortle|belly laugh|snicker|baby laughter)$/i, 'laughter', 0.7, false],
+ [/^(applause|clapping)$/i, 'applause', 0.7, false],
+ [/^(cheering|crowd|children shouting)$/i, 'cheering', 0.7, false],
+ [/^whistling$/i, 'whistling', 0.75, false],
+ [/^(crying, sobbing|baby cry, infant cry|whimper)$/i, 'crying', 0.75, false],
+ [/^screaming$/i, 'screaming', 0.8, false],
+ [/^(dog|bark|yip|howl|bow-wow)$/i, 'dog barking', 0.8, false],
+ [/^(cat|meow|purr)$/i, 'cat meowing', 0.8, false],
+ [/^(doorbell|ding-dong)$/i, 'doorbell', 0.8, false],
+ [/^(telephone bell ringing|ringtone)$/i, 'phone ringing', 0.8, false],
+ [/^(siren|police car \(siren\)|ambulance \(siren\)|fire engine, fire truck \(siren\)|civil defense siren)$/i, 'siren', 0.8, false],
+ [/^explosion$/i, 'explosion', 0.85, true],
+ [/^(gunshot, gunfire|machine gun|fusillade)$/i, 'gunfire', 0.85, true],
+];
+const MERGE_GAP = 4;
+const MIN_LEN = 1.5;
+const MAX_LEN = 6;
+const MUSIC_LEN = 3;
+
+function tagOf(label) {
+ const l = String(label || '').trim();
+ for (const [re, tag, min, rareOnly] of TAGS) if (re.test(l)) return { tag, min, rareOnly };
+ return null;
+}
+
+function cues(events, { spanSec = 0 } = {}) {
+ const list = (Array.isArray(events) ? events : []).filter((e) => e && Number.isFinite(Number(e.start_sec)));
+ const rare = new Set(clipScore.rareSounds(list, spanSec).map((e) => String(e.label || '')));
+ const raw = [];
+ for (const e of list) {
+ const start = Math.max(0, Number(e.start_sec));
+ const end = Number.isFinite(Number(e.end_sec)) && Number(e.end_sec) > start ? Number(e.end_sec) : start + MIN_LEN;
+ if (String(e.label || '') === 'Music') {
+ if (Number(e.confidence || 0) >= 0.5) raw.push({ start, end: start + MUSIC_LEN, text: '[music]' });
+ continue;
+ }
+ const t = tagOf(e.label);
+ if (!t || Number(e.confidence || 0) < t.min) continue;
+ if (t.rareOnly && !rare.has(String(e.label || ''))) continue;
+ raw.push({ start, end, text: `[${t.tag}]` });
+ }
+ raw.sort((a, b) => a.start - b.start || a.text.localeCompare(b.text));
+ const out = [];
+ const last = new Map(); // tag → its latest cue, so a burst becomes one cue
+ for (const c of raw) {
+ const prev = last.get(c.text);
+ if (prev && c.start - prev.end <= MERGE_GAP && c.text !== '[music]') {
+ prev.end = Math.min(prev.start + MAX_LEN, Math.max(prev.end, c.end));
+ continue;
+ }
+ if (prev && c.text === '[music]' && c.start - prev.end <= MERGE_GAP) continue;
+ const cue = { start: c.start, end: Math.min(c.start + MAX_LEN, Math.max(c.start + MIN_LEN, c.end)), text: c.text };
+ out.push(cue);
+ last.set(c.text, cue);
+ }
+ return out.map((c) => ({ start: Math.round(c.start * 100) / 100, end: Math.round(c.end * 100) / 100, text: c.text }));
+}
+
+module.exports = { cues, tagOf, TAGS };
@@ -420,6 +420,28 @@ router.post('/bulk-delete-old', requireAuth, async (req, res) => {
@@ -420,6 +420,28 @@ router.post('/bulk-delete-old', requireAuth, async (req, res) => {
}
}
});
});
+/**
+ * A stream's rows (stream time, in `key`) in this VOD's own time. The first part of a joined recording answers for the
+ * whole video: every row that was recorded, in the video's time (streaming/recordings.js; the gaps between parts were
+ * not). A VOD that is one part of its stream (the broadcaster reconnected) keeps the rows inside it, moved to its time:
+ * its chapters (OpenVibe.Video) pointed eleven minutes late on part 2 of stream 2795. `round` keeps whole seconds.
+ */
+async function inVodTime(vod, rows, key, { round = true } = {}) {
+ const fix = (v) => (round ? Math.round(v) : Math.round(v * 100) / 100);
+ if (vod.recording && Number(vod.recording.vod_id) === Number(vod.id) && vod.recording.hls_url && (vod.recording.parts || []).length > 1) {
+ const recordings = require('../streaming/recordings');
+ const parts = await recordings.presentParts(vod.recording);
+ return rows.map((r) => ({ r, at: recordings.toRecordingTime(parts, Number(r[key])) }))
+ .filter((x) => x.at != null).map(({ r, at }) => ({ ...r, [key]: fix(at), ...(r.end != null ? { end: fix(at + (Number(r.end) - Number(r[key]))) } : {}) }));
+ }
+ const part = await require('../streaming/stream-history').partForVod(vod.id);
+ if (!part) return rows;
+ const len = part.end != null ? part.end - part.at : Infinity;
+ return rows
+ .filter((r) => Number(r[key]) >= part.at && Number(r[key]) - part.at <= len)
+ .map((r) => ({ ...r, [key]: fix(Number(r[key]) - part.at), ...(r.end != null ? { end: fix(Number(r.end) - part.at) } : {}) }));
+}
+
// AI "memory" timeline for a VOD (stream_memories stays in live.db).
// AI "memory" timeline for a VOD (stream_memories stays in live.db).
router.get('/:id/memories', optionalAuth, async (req, res) => {
router.get('/:id/memories', optionalAuth, async (req, res) => {
try {
try {
@@ -427,35 +449,38 @@ router.get('/:id/memories', optionalAuth, async (req, res) => {
@@ -427,35 +449,38 @@ router.get('/:id/memories', optionalAuth, async (req, res) => {
if (!vod) return;
if (!vod) return;
const streamId = vod.stream_id || null;
const streamId = vod.stream_id || null;
if (!streamId) return res.json({ memories: [] });
if (!streamId) return res.json({ memories: [] });
- let memories = (await db.getStreamMemories(streamId) || []).map(m => ({
+ const memories = (await db.getStreamMemories(streamId) || []).map(m => ({
offset_seconds: m.offset_seconds,
offset_seconds: m.offset_seconds,
description: db.publicMemoryText(m.description),
description: db.publicMemoryText(m.description),
tags: m.tags ? (() => { try { return JSON.parse(m.tags); } catch { return []; } })() : [],
tags: m.tags ? (() => { try { return JSON.parse(m.tags); } catch { return []; } })() : [],
}));
}));
- // The first part of a joined recording answers for the whole video: every moment that was recorded, in the
- // video's time (streaming/recordings.js; the gaps between parts were not).
- if (vod.recording && Number(vod.recording.vod_id) === Number(vod.id) && vod.recording.hls_url && (vod.recording.parts || []).length > 1) {
- const recordings = require('../streaming/recordings');
- const parts = await recordings.presentParts(vod.recording);
- memories = memories.map((m) => ({ ...m, offset_seconds: recordings.toRecordingTime(parts, m.offset_seconds) }))
- .filter((m) => m.offset_seconds != null).map((m) => ({ ...m, offset_seconds: Math.round(m.offset_seconds) }));
- return res.json({ memories });
- }
- // A VOD that is one part of its stream (the broadcaster reconnected): the moments inside it, in its own time.
- // Its chapters (OpenVibe.Video) pointed eleven minutes late on part 2 of stream 2795.
- const part = await require('../streaming/stream-history').partForVod(vod.id);
- if (part) {
- const len = part.end != null ? part.end - part.at : Infinity;
- memories = memories
- .filter((m) => Number(m.offset_seconds) >= part.at && Number(m.offset_seconds) - part.at <= len)
- .map((m) => ({ ...m, offset_seconds: Math.round(Number(m.offset_seconds) - part.at) }));
- }
- res.json({ memories });
+ res.json({ memories: await inVodTime(vod, memories, 'offset_seconds') });
} catch {
} catch {
res.status(500).json({ error: 'Failed to load AI timeline' });
res.status(500).json({ error: 'Failed to load AI timeline' });
}
}
});
});
+/**
+ * GET /api/vods/:id/sounds — what was heard in the VOD, as caption cues in its own time: "[laughter]", "[applause]",
+ * "[music]" (item 151, C15; server/ai/sound-captions.js: a short list of tags, each above its own bar, nothing written
+ * by a model). OpenVibe.Video puts them in the VOD's captions between the spoken lines. Empty for a stream recorded
+ * before the timeline, or with nothing worth a tag.
+ */
+router.get('/:id/sounds', optionalAuth, async (req, res) => {
+ try {
+ const vod = await loadVisibleVod(req, res, req.params.id);
+ if (!vod) return;
+ const streamId = vod.stream_id || null;
+ if (!streamId) return res.json({ cues: [] });
+ const events = await db.getTimeline(streamId, { kind: 'sound', limit: 20000 }) || [];
+ const last = events.reduce((m, e) => Math.max(m, Number(e.end_sec) || Number(e.start_sec) || 0), 0);
+ const cues = require('../ai/sound-captions').cues(events, { spanSec: last });
+ res.json({ cues: await inVodTime(vod, cues, 'start', { round: false }) });
+ } catch {
+ res.status(500).json({ error: 'Failed to load the sounds' });
+ }
+});
+
/**
/**
* GET /api/vods/:id/context — everything around a VOD that isn't the file: the stream it came
* GET /api/vods/:id/context — everything around a VOD that isn't the file: the stream it came
* from, its activity numbers (chat, viewers, mic moments, follows, clips), the clips cut from
* from, its activity numbers (chat, viewers, mic moments, follows, clips), the clips cut from
new file mode 100644
new file mode 100644
@@ -0,0 +1,25 @@
@@ -0,0 +1,25 @@
+'use strict';
+// What was heard, as caption cues (item 151, C15; server/ai/sound-captions.js): a short list of tags, each from a
+// classifier label above its own bar, nothing written by a model. A burst is one cue; background music gives one cue
+// where it starts; explosions and gunfire only where they are rare (in a shooter they are the game); the classifier's
+// long tail is never captioned.
+const assert = require('assert');
+const sc = require('../server/ai/sound-captions');
+const ev = (start_sec, label, confidence, end_sec) => ({ start_sec, end_sec, label, confidence });
+
+assert.deepStrictEqual(sc.cues([ev(10, 'Laughter', 0.9, 11), ev(12, 'Giggle', 0.8, 13.2), ev(30, 'Laughter', 0.75)]),
+ [{ start: 10, end: 13.2, text: '[laughter]' }, { start: 30, end: 31.5, text: '[laughter]' }], 'a burst is one cue; one 17 s later is another');
+assert.deepStrictEqual(sc.cues([ev(10, 'Laughter', 0.6), ev(20, 'Applause', 0.69)]), [], 'below the bars, nothing');
+assert.deepStrictEqual(sc.cues([ev(5, 'Vehicle', 0.95), ev(6, 'Animal', 0.95), ev(7, 'Glass', 0.95), ev(8, 'Speech', 0.99)]), [], 'the long tail is never captioned');
+assert.deepStrictEqual(sc.cues([ev(100, 'Applause', 0.8, 160)]), [{ start: 100, end: 106, text: '[applause]' }], 'a cue lasts at most 6 s');
+assert.deepStrictEqual(sc.cues([ev(0, 'Music', 0.7, 300), ev(302, 'Music', 0.7, 400), ev(500, 'Music', 0.4, 600)]),
+ [{ start: 0, end: 3, text: '[music]' }, { start: 302, end: 305, text: '[music]' }], 'one cue where music starts, not under every line');
+
+// Explosions: captioned from 0.85, and only where rare in the stream.
+const shooter = Array.from({ length: 40 }, (_, i) => ev(i * 90, 'Explosion', 0.9, i * 90 + 1));
+assert.deepStrictEqual(sc.cues(shooter, { spanSec: 3600 }), [], '40 explosions an hour are the game');
+assert.deepStrictEqual(sc.cues([ev(50, 'Explosion', 0.9, 51), ev(70, 'Gunshot, gunfire', 0.84)], { spanSec: 3600 }),
+ [{ start: 50, end: 51.5, text: '[explosion]' }], 'one rare explosion is news; gunfire below 0.85 is not captioned');
+assert.deepStrictEqual(sc.cues(null), []);
+assert.strictEqual(sc.tagOf('Dog').tag, 'dog barking');
+console.log('sound captions: all checks passed');
@@ -58,6 +58,11 @@ async function check(name, fn) {
@@ -58,6 +58,11 @@ async function check(name, fn) {
for (const [off, text] of [[100, 'birds in part one'], [900, 'robot in part two'], [1500, 'squirrels in part two']]) {
for (const [off, text] of [[100, 'birds in part one'], [900, 'robot in part two'], [1500, 'squirrels in part two']]) {
await raw.prepare('INSERT INTO stream_memories (stream_id, user_id, offset_seconds, description) VALUES (?, 80, ?, ?)').run(sid, off, text);
await raw.prepare('INSERT INTO stream_memories (stream_id, user_id, offset_seconds, description) VALUES (?, 80, ?, ?)').run(sid, off, text);
}
}
+ // What was heard (C15): a laugh and a rare explosion in part two, applause in part one, and a "Vehicle" the
+ // classifier hears everywhere and is never captioned.
+ for (const [at, end, label, conf] of [[150, 152, 'Applause', 0.8], [920, 921, 'Laughter', 0.9], [922, 923.5, 'Laughter', 0.8], [930, 931, 'Vehicle', 0.9], [1600, 1601, 'Explosion', 0.9]]) {
+ await raw.prepare(`INSERT INTO stream_timeline_events (stream_id, user_id, kind, start_sec, end_sec, label, confidence) VALUES (?, 80, 'sound', ?, ?, ?, ?)`).run(sid, at, end, label, conf);
+ }
const media = require('../server/media-client');
const media = require('../server/media-client');
const VODS = {
const VODS = {
@@ -138,6 +143,16 @@ async function check(name, fn) {
@@ -138,6 +143,16 @@ async function check(name, fn) {
assert.deepStrictEqual(m1.json.memories.map((m) => [m.offset_seconds, m.description]), [[72, 'birds in part one']]);
assert.deepStrictEqual(m1.json.memories.map((m) => [m.offset_seconds, m.description]), [[72, 'birds in part one']]);
});
});
+ await check('what was heard, as caption cues in the part\'s own time (C15)', async () => {
+ const s2 = await call('/api/vods/6026/sounds');
+ assert.strictEqual(s2.status, 200, s2.text);
+ assert.deepStrictEqual(s2.json.cues, [{ start: 211, end: 214.5, text: '[laughter]' }, { start: 891, end: 892.5, text: '[explosion]' }],
+ 'one cue for the burst of laughter; no "[vehicle]"');
+ const s1 = await call('/api/vods/6025/sounds');
+ assert.deepStrictEqual(s1.json.cues.map((c) => [c.start, c.text]), [[122, '[applause]']]);
+ assert.strictEqual((await call('/api/vods/99999/sounds')).status, 404);
+ });
+
server.close();
server.close();
if (failures) { quiet(`vod downloads: ${failures} failed`); process.exit(1); }
if (failures) { quiet(`vod downloads: ${failures} failed`); process.exit(1); }
quiet('vod downloads: ok');
quiet('vod downloads: ok');