openvibers/openvibe.live Commits

A sound the stream hears all evening is not a moment (item 151, C08 follow-up) (#582)

OP OpenVibers committed 10 days ago · 7e15069939 · 5 files, +84 −10

The audio classifier hears a shooter's explosions and gunfire constantly: in production 'Explosion' and 'Gunshot,
gunfire' at 0.6 or more ran into the thousands across 200+ streams while laughter was rare, so every explosion was a
'strong sound' for the clip score and a place for the VOD picker to cut. clip-score.rareSounds keeps the labels heard
at most 12 times an hour in the stream; the VOD picker's evidence and fallback use only those, and the live score
uses those of the last hour. The AI still hears every sound.

Changes

ddiff --git a/docs/vods-and-clips.md b/docs/vods-and-clips.md
iindex e7b7a50a..af23bf7a 100644
---- a/docs/vods-and-clips.md
++++ b/docs/vods-and-clips.md
@@@ -214,7 +214,7 @@ Two jobs turn stream moments into content: `server/ai/ai-moments-job.js` (image
- The paste job no longer clips the paste's own moment; it asks for a **second, different beat** from the same VOD for the clip. The clip backfill avoids every offset a paste used.
- **Live pastes** — the stream-memory vision call now also answers "is this frame screenshot-worthy?" with a caption (no extra call). When it says yes, `stream-memory-job` posts the frame as an image paste right away (≤ 1 per stream per 90 min, dark-frame and registry checks, setting `ai_live_pastes_enabled`, default on). These pastes carry `metadata.live = true` and no VOD link.
- **Only good live clips, cut on sentences** — a live chat spike is not enough on its own. `server/ai/clip-score.js` scores the moment from 0 to 1: the spike (how far above the window's average and how big), how many people reacted, what was heard (laughter, cheering, a bang; music and speech alone count for nothing), how much was said, and the AI's yes or no (a no is a veto; with the AI off its part is neutral and both bars rise by 0.1). Below **0.40** no clip is cut, the log says why ("a weak chat spike (10 messages, 3.1× the average)"). From 0.40 the clip is made **unlisted**: the streamer can find it and make it public. Calibrated by the production audit (C01, `scripts/diagnostics/clip-audit-*.sql`: the old rule made 99.6 % of automatic clips public and 29 % were never watched in 30 days): a moment needs **evidence** to be kept at all (a spike of at least 3× the average and 10 messages, a strong sound, or 20 spoken words; the AI's yes alone is not enough), and it is cut **public** only at **0.70** or more **and** in the reacted bucket: at least 12 messages in the peak from at least 3 people (`chatReads.recentChatters`; unknown means unlisted). Every cut logs its bucket (A no chat, B 1–5, C 6–11, D 12+ with 3+ people), peak and chatters for the next audit. `server/ai/clip-trim.js` then moves the fixed 16 s before / 9 s after window onto the speech rows: the start goes back to where the sentence being spoken began (so the setup is in), at most 25 s before the moment; the end goes on to where the sentence ends, at most 20 s after, never past the recording's flushed edge. A snap that would make the clip shorter than 12 s or longer than 45 s is not taken. Both are pure, with their cases in `test/clip-score-trim.test.js`.
--- **VOD clips are scored too, and may be none** (item 151, C08). The VOD cutter (`clipVodMoment`, from the backfill and from the paste job's second beat) used to cut whatever the picker returned, public. Now the picker hands back the moment's **evidence**, gathered at that second from everything Live has (not the sample the prompt reads): the chat bucket there against the stream's own average per 30 s with the quiet stretches counted, how many people were in it (when Chat counts them), the sounds within 15 s, the words said within 20 s, and who picked it (`ai`, `chat`, `sound`, `people`, `scene`). The cutter runs it through the same `clip-score.js` as the live cutter: below the keep bar nothing is cut (the log says the score and why), a kept moment is **unlisted**, and only a reacted room (12+ messages from 3+ people, a score of 0.70 or more) is cut **public**, never above the recording's own visibility. A moment with no evidence is not cut. A clip never re-cuts a second a person already clipped (that footage is their clip already; it goes in the picker's avoid list), and with the AI off a clip needs a chat spike or a strong sound: a long scene note alone only ever makes a paste. The AI may answer that nothing is worth it (`{ abstain: true }` or `t: null`), and then no weaker signal is tried. The AI's own clips are no longer counted as "people clipped here" (`lookups.clipStartTimes`). Each cut logs its score, bucket, picker and policy (`vod-moment@2026-11-03`), and the daily Bench evaluation labels the AI's entry with that policy (`clip-evaluation.policyOf`), so Bench's accept rate tells this policy from the one before. Cases: `test/vod-moment-evidence.test.js`.
++- **VOD clips are scored too, and may be none** (item 151, C08). The VOD cutter (`clipVodMoment`, from the backfill and from the paste job's second beat) used to cut whatever the picker returned, public. Now the picker hands back the moment's **evidence**, gathered at that second from everything Live has (not the sample the prompt reads): the chat bucket there against the stream's own average per 30 s with the quiet stretches counted, how many people were in it (when Chat counts them), the sounds within 15 s, the words said within 20 s, and who picked it (`ai`, `chat`, `sound`, `people`, `scene`). The cutter runs it through the same `clip-score.js` as the live cutter: below the keep bar nothing is cut (the log says the score and why), a kept moment is **unlisted**, and only a reacted room (12+ messages from 3+ people, a score of 0.70 or more) is cut **public**, never above the recording's own visibility. A moment with no evidence is not cut. A clip never re-cuts a second a person already clipped (that footage is their clip already; it goes in the picker's avoid list), and with the AI off a clip needs a chat spike or a strong sound: a long scene note alone only ever makes a paste. The AI may answer that nothing is worth it (`{ abstain: true }` or `t: null`), and then no weaker signal is tried. The AI's own clips are no longer counted as "people clipped here" (`lookups.clipStartTimes`). A sound counts as evidence only when it is news in its stream (`clip-score.rareSounds`: at most 12 an hour; production on 2026-11-03 had thousands of "Explosion" and "Gunshot" readings across 200+ streams, while laughter was rare), for VOD moments over the stream and for live clips over the last hour; the AI still hears them all. Each cut logs its score, bucket, picker and policy (`vod-moment@2026-11-03`), and the daily Bench evaluation labels the AI's entry with that policy (`clip-evaluation.policyOf`), so Bench's accept rate tells this policy from the one before. Cases: `test/vod-moment-evidence.test.js`.
- **Fewer tokens** — VOD showcase scores are cached for 7 days (`home_hero_moments.rankCache`); a run only scores VODs it has never scored. Every `llm.complete` call keeps its `kind` so `ai_usage` shows exactly where the budget goes (`moment_vod_rank`, `moment_pick`, `moment_frame`, `stream_memory`, `auto_clip_confirm`).
## AI-inferred category
ddiff --git a/server/ai/ai-moments-job.js b/server/ai/ai-moments-job.js
iindex 65fbbd06..c190582e 100644
---- a/server/ai/ai-moments-job.js
++++ b/server/ai/ai-moments-job.js
@@@ -171,11 +171,14 @@ async function _momentContext(streamId, vodId) {
// The stream's average chat per 30 s, empty stretches counted (C07/C08: averaging only the busy buckets made every
// spike look small on a quiet channel and every lull look big on a busy one).
let chatAvg = 0;
++ const spanSec = started > 0 ? Math.max(0, ((ms(span.ended_at) || Date.now()) - started) / 1000) : 0;
try {
-- const secs = started > 0 ? ((ms(span.ended_at) || Date.now()) - started) / 1000 : 0;
-- if (secs >= 60) chatAvg = (Number((await chatReads.streamStats(streamId) || {}).messages) || 0) / (secs / 30);
++ if (spanSec >= 60) chatAvg = (Number((await chatReads.streamStats(streamId) || {}).messages) || 0) / (spanSec / 30);
} catch { /* chat optional */ }
-- return _inPart({ memories, transcript, said, clipTimes, spikes, sounds, heard, chatAvg }, vodId);
++ // A sound the stream hears all evening (a shooter's gunfire) is its background, not a moment: only the rare ones
++ // count as evidence or as a place to cut (clip-score.rareSounds).
++ const notable = clipScore.rareSounds(heard, spanSec);
++ return _inPart({ memories, transcript, said, clipTimes, spikes, sounds, heard: notable, chatAvg }, vodId);
}
/**
ddiff --git a/server/ai/auto-clip-job.js b/server/ai/auto-clip-job.js
iindex 16f3df22..ea8fe5cb 100644
---- a/server/ai/auto-clip-job.js
++++ b/server/ai/auto-clip-job.js
@@@ -80,11 +80,18 @@ async function _confirmLiveMoment(stream) {
// Sound events are a strong clip signal on their own — a burst of gunfire, an explosion or laughter is exactly
// the kind of thing viewers clip, and it is often the reason chat spiked in the first place.
let sounds = [];
++ let notable = [];
try {
-- const since = Math.max(0, ((Date.now() - new Date(stream.started_at + 'Z').getTime()) / 1000) - WINDOW_SEC);
++ const elapsed = Math.max(0, (Date.now() - new Date(stream.started_at + 'Z').getTime()) / 1000);
++ const since = Math.max(0, elapsed - WINDOW_SEC);
sounds = (await db.getTimeline(stream.id, { kind: 'sound', from: since, limit: 40 }) || [])
// Background music (audio-events.js spans) is the stream's bed, not a moment: never a clip signal.
.filter(e => Number(e.confidence || 0) >= 0.4 && e.label !== 'Music').slice(-12);
++ // The score counts only the sounds that are news in this stream: a shooter's gunfire all hour is its background
++ // (clip-score.rareSounds, over the last hour). The AI still hears them all.
++ const hour = await db.getTimeline(stream.id, { kind: 'sound', from: Math.max(0, elapsed - 3600), limit: 2000 }) || [];
++ const rare = new Set(clipScore.rareSounds(hour, Math.min(3600, elapsed)).map(e => String(e.label || '')));
++ notable = sounds.filter(e => rare.has(String(e.label || '')));
} catch { /* timeline optional */ }
// The decision rules are OpenVibe.AI's versioned template live.clips.confirm (WS-O task 2); Live sends what it saw,
// heard and read, with times so the model can line up speech, sounds and the chat spike.
@@@ -95,9 +102,9 @@ async function _confirmLiveMoment(stream) {
sounds: sounds.map(e => ({ t: Math.max(0, Number(e.start_sec) || 0), label: String(e.label || '').slice(0, 120), confidence: Number(e.confidence || 0) })),
chat: chat.slice(-30).map(c => _clean(c, 100)).filter(Boolean),
}, { target: { service: 'live', type: 'stream', id: String(stream.id) }, meter: { kind: 'auto_clip_confirm', role: 'legacy', ownerUserId: stream.user_id } });
-- if (!j) return { clip: false, transcript, sounds };
-- return { clip: j.clip === true, title: _clean(j.title, 80), desc: _clean(j.desc, 400), transcript, sounds };
-- } catch { return { clip: false, transcript, sounds }; }
++ if (!j) return { clip: false, transcript, sounds, notable };
++ return { clip: j.clip === true, title: _clean(j.title, 80), desc: _clean(j.desc, 400), transcript, sounds, notable };
++ } catch { return { clip: false, transcript, sounds, notable }; }
}
/** What was said around a stream second: the registry uses it so a static camera alone is not a repeat. */
@@@ -154,7 +161,7 @@ async function _checkLiveStream(stream) {
// The score: no clip beats a bad clip; a fair one is kept unlisted for the streamer; a clear one is public.
const near = _speechNear(verdict.transcript, streamSec, 20);
const chatters = await chatReads.recentChatters(streamId, BUCKET_SEC * 2).catch(() => null);
-- const rated = clipScore.score({ spike: { peak: peak.count, avg }, chatters, sounds: verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });
++ const rated = clipScore.score({ spike: { peak: peak.count, avg }, chatters, sounds: verdict.notable || verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });
if (rated.verdict === 'skip') { console.log(`[AutoClip] stream ${streamId}: skipped a weak moment (score ${rated.score}${rated.reasons.length ? `: ${rated.reasons.join('; ')}` : ''})`); return; }
const why = await registry.usedReason({ vod_id: rec.vodId, stream_id: streamId, offset: momentOffset, desc: verdict.desc, title, speech });
if (why) { console.log(`[AutoClip] stream ${streamId}: ${why}`); return; }
ddiff --git a/server/ai/clip-score.js b/server/ai/clip-score.js
iindex 4e5f6d0e..0e0e6978 100644
---- a/server/ai/clip-score.js
++++ b/server/ai/clip-score.js
@@@ -73,4 +73,18 @@ function score({ spike = {}, chatters = null, sounds = [], speechWords = 0, ai =
return { score: Math.round(total * 1000) / 1000, verdict, parts, reasons, bucket, evidence };
}
--module.exports = { score, BARS, WEIGHTS, STRONG_SOUND, EVIDENCE, REACTED };
++/**
++ * The sounds that are news in this stream (item 151, C08 follow-up). The audio classifier hears a shooter's explosions
++ * and gunfire all evening: in production (11-03) "Explosion" and "Gunshot, gunfire" at 0.6 or more were in the
++ * thousands across 200+ streams, while laughter was rare. A sound that comes back more than `perHour` times an hour in
++ * the stream is its background, not a moment, so it is no evidence. Rates are over at least a quarter hour.
++ */
++function rareSounds(events, spanSec, { perHour = 12 } = {}) {
++ const list = Array.isArray(events) ? events : [];
++ const hours = Math.max(0.25, (Number(spanSec) || 0) / 3600);
++ const count = new Map();
++ for (const e of list) { const k = String((e && e.label) || ''); count.set(k, (count.get(k) || 0) + 1); }
++ return list.filter((e) => (count.get(String((e && e.label) || '')) || 0) / hours <= perHour);
++}
++
++module.exports = { score, rareSounds, BARS, WEIGHTS, STRONG_SOUND, EVIDENCE, REACTED };
ddiff --git a/test/rare-sounds.test.js b/test/rare-sounds.test.js
nnew file mode 100644
iindex 00000000..1acdd857
---- /dev/null
++++ b/test/rare-sounds.test.js
@@@ -0,0 +1,50 @@
++'use strict';
++// A sound the stream hears all evening is its background, not a moment (item 151, C08 follow-up). In production
++// (2026-11-03) "Explosion" and "Gunshot, gunfire" at 0.6 or more ran into the thousands across 200+ streams while
++// laughter was rare, so a shooter's every explosion counted as "a strong sound" for the clip score. clip-score.rareSounds
++// keeps the labels heard at most 12 times an hour in the stream; the VOD picker and the live score use only those.
++const assert = require('assert');
++const db = require('../server/db/database');
++console.warn = () => {};
++
++(async () => {
++ const clipScore = require('../server/ai/clip-score');
++ const boom = (t) => ({ label: 'Explosion', confidence: 0.8, start_sec: t });
++ const laugh = (t) => ({ label: 'Laughter', confidence: 0.8, start_sec: t });
++
++ // ── Pure ──
++ const evening = [...Array.from({ length: 60 }, (_, i) => boom(i * 120)), laugh(3000)];
++ assert.deepStrictEqual(clipScore.rareSounds(evening, 7200).map((e) => e.label), ['Laughter'], '60 explosions in two hours are the game, one laugh is news');
++ assert.strictEqual(clipScore.rareSounds([boom(10), boom(20), boom(30)], 7200).length, 3, 'three in two hours are rare');
++ assert.strictEqual(clipScore.rareSounds([boom(10), boom(20), boom(30), boom(40)], 120).length, 0, 'a short span counts as a quarter hour: four is 16 an hour');
++ assert.deepStrictEqual(clipScore.rareSounds(null, 100), []);
++
++ // ── The VOD picker (AI off): the laugh is the moment, the explosions are not ──
++ await db.initDb();
++ const chatReads = require('../server/chat/chat-reads');
++ const media = require('../server/media-client');
++ const ai = require('../server/ai/ai-analysis');
++ const moments = require('../server/ai/ai-moments-job');
++ await require('../server/ai/moment-registry')._clear();
++ await db.run(`INSERT INTO users (id, username, display_name, email, password_hash, role, created_at) OVERRIDING SYSTEM VALUE
++ VALUES (31, 'rosa', 'rosa', 'rosa@x', 'x', 'streamer', '2025-01-01 00:00:00') ON CONFLICT DO NOTHING`);
++ const sid = (await db.get(`INSERT INTO streams (user_id, title, started_at, ended_at) VALUES (31, 'shooter night', '2026-11-01 10:00:00', '2026-11-01 12:00:00') RETURNING id`)).id;
++ ai.isEnabled = async () => false;
++ db.getStreamMemories = async () => [{ offset_seconds: 3000, description: 'The streamer laughs at a teammate falling off a ledge' }, { offset_seconds: 600, description: 'A firefight on a bridge' }];
++ db.getStreamTranscriptSegments = async () => [{ start: 2995, end: 3004, text: 'no way he just walked off the bridge oh my god' }];
++ db.getTimeline = async () => evening;
++ chatReads.spikeOffsets = async () => [];
++ chatReads.streamStats = async () => ({ messages: 0 });
++ media.listClips = async () => ({ clips: [] });
++ const v = { vod_id: 701, stream_id: sid, duration: 7200, title: 'Shooter night' };
++ const m = await moments.findBestMoment(v, { flavor: 'clip' });
++ assert.ok(m, 'a clip moment is found');
++ assert.strictEqual(m.offset, 3000, 'the laugh, not the first explosion');
++ assert.strictEqual(m.evidence.picked_by, 'sound');
++ assert.deepStrictEqual(m.evidence.sounds.map((e) => e.label), ['Laughter'], 'the explosions nearby are not evidence');
++
++ db.getTimeline = async () => evening.filter((e) => e.label === 'Explosion');
++ assert.strictEqual(await moments.findBestMoment(v, { flavor: 'clip' }), null, 'a stream of nothing but its background gives no clip');
++
++ console.log('rare sounds: all checks passed');
++})().catch((e) => { console.error(e); process.exit(1); });