openvibers/openvibe.live Commits
A sound the stream hears all evening is not a moment (item 151, C08 follow-up) (#582)
Changes
@@@ -214,7 +214,7 @@ Two jobs turn stream moments into content: `server/ai/ai-moments-job.js` (image- The paste job no longer clips the paste's own moment; it asks for a **second, different beat** from the same VOD for the clip. The clip backfill avoids every offset a paste used.- **Live pastes** — the stream-memory vision call now also answers "is this frame screenshot-worthy?" with a caption (no extra call). When it says yes, `stream-memory-job` posts the frame as an image paste right away (≤ 1 per stream per 90 min, dark-frame and registry checks, setting `ai_live_pastes_enabled`, default on). These pastes carry `metadata.live = true` and no VOD link.- **Only good live clips, cut on sentences** — a live chat spike is not enough on its own. `server/ai/clip-score.js` scores the moment from 0 to 1: the spike (how far above the window's average and how big), how many people reacted, what was heard (laughter, cheering, a bang; music and speech alone count for nothing), how much was said, and the AI's yes or no (a no is a veto; with the AI off its part is neutral and both bars rise by 0.1). Below **0.40** no clip is cut, the log says why ("a weak chat spike (10 messages, 3.1× the average)"). From 0.40 the clip is made **unlisted**: the streamer can find it and make it public. Calibrated by the production audit (C01, `scripts/diagnostics/clip-audit-*.sql`: the old rule made 99.6 % of automatic clips public and 29 % were never watched in 30 days): a moment needs **evidence** to be kept at all (a spike of at least 3× the average and 10 messages, a strong sound, or 20 spoken words; the AI's yes alone is not enough), and it is cut **public** only at **0.70** or more **and** in the reacted bucket: at least 12 messages in the peak from at least 3 people (`chatReads.recentChatters`; unknown means unlisted). Every cut logs its bucket (A no chat, B 1–5, C 6–11, D 12+ with 3+ people), peak and chatters for the next audit. `server/ai/clip-trim.js` then moves the fixed 16 s before / 9 s after window onto the speech rows: the start goes back to where the sentence being spoken began (so the setup is in), at most 25 s before the moment; the end goes on to where the sentence ends, at most 20 s after, never past the recording's flushed edge. A snap that would make the clip shorter than 12 s or longer than 45 s is not taken. Both are pure, with their cases in `test/clip-score-trim.test.js`.--- **VOD clips are scored too, and may be none** (item 151, C08). The VOD cutter (`clipVodMoment`, from the backfill and from the paste job's second beat) used to cut whatever the picker returned, public. Now the picker hands back the moment's **evidence**, gathered at that second from everything Live has (not the sample the prompt reads): the chat bucket there against the stream's own average per 30 s with the quiet stretches counted, how many people were in it (when Chat counts them), the sounds within 15 s, the words said within 20 s, and who picked it (`ai`, `chat`, `sound`, `people`, `scene`). The cutter runs it through the same `clip-score.js` as the live cutter: below the keep bar nothing is cut (the log says the score and why), a kept moment is **unlisted**, and only a reacted room (12+ messages from 3+ people, a score of 0.70 or more) is cut **public**, never above the recording's own visibility. A moment with no evidence is not cut. A clip never re-cuts a second a person already clipped (that footage is their clip already; it goes in the picker's avoid list), and with the AI off a clip needs a chat spike or a strong sound: a long scene note alone only ever makes a paste. The AI may answer that nothing is worth it (`{ abstain: true }` or `t: null`), and then no weaker signal is tried. The AI's own clips are no longer counted as "people clipped here" (`lookups.clipStartTimes`). Each cut logs its score, bucket, picker and policy (`vod-moment@2026-11-03`), and the daily Bench evaluation labels the AI's entry with that policy (`clip-evaluation.policyOf`), so Bench's accept rate tells this policy from the one before. Cases: `test/vod-moment-evidence.test.js`.++- **VOD clips are scored too, and may be none** (item 151, C08). The VOD cutter (`clipVodMoment`, from the backfill and from the paste job's second beat) used to cut whatever the picker returned, public. Now the picker hands back the moment's **evidence**, gathered at that second from everything Live has (not the sample the prompt reads): the chat bucket there against the stream's own average per 30 s with the quiet stretches counted, how many people were in it (when Chat counts them), the sounds within 15 s, the words said within 20 s, and who picked it (`ai`, `chat`, `sound`, `people`, `scene`). The cutter runs it through the same `clip-score.js` as the live cutter: below the keep bar nothing is cut (the log says the score and why), a kept moment is **unlisted**, and only a reacted room (12+ messages from 3+ people, a score of 0.70 or more) is cut **public**, never above the recording's own visibility. A moment with no evidence is not cut. A clip never re-cuts a second a person already clipped (that footage is their clip already; it goes in the picker's avoid list), and with the AI off a clip needs a chat spike or a strong sound: a long scene note alone only ever makes a paste. The AI may answer that nothing is worth it (`{ abstain: true }` or `t: null`), and then no weaker signal is tried. The AI's own clips are no longer counted as "people clipped here" (`lookups.clipStartTimes`). A sound counts as evidence only when it is news in its stream (`clip-score.rareSounds`: at most 12 an hour; production on 2026-11-03 had thousands of "Explosion" and "Gunshot" readings across 200+ streams, while laughter was rare), for VOD moments over the stream and for live clips over the last hour; the AI still hears them all. Each cut logs its score, bucket, picker and policy (`vod-moment@2026-11-03`), and the daily Bench evaluation labels the AI's entry with that policy (`clip-evaluation.policyOf`), so Bench's accept rate tells this policy from the one before. Cases: `test/vod-moment-evidence.test.js`.- **Fewer tokens** — VOD showcase scores are cached for 7 days (`home_hero_moments.rankCache`); a run only scores VODs it has never scored. Every `llm.complete` call keeps its `kind` so `ai_usage` shows exactly where the budget goes (`moment_vod_rank`, `moment_pick`, `moment_frame`, `stream_memory`, `auto_clip_confirm`).## AI-inferred category@@@ -171,11 +171,14 @@ async function _momentContext(streamId, vodId) {// The stream's average chat per 30 s, empty stretches counted (C07/C08: averaging only the busy buckets made every// spike look small on a quiet channel and every lull look big on a busy one).let chatAvg = 0;++ const spanSec = started > 0 ? Math.max(0, ((ms(span.ended_at) || Date.now()) - started) / 1000) : 0;try {-- const secs = started > 0 ? ((ms(span.ended_at) || Date.now()) - started) / 1000 : 0;-- if (secs >= 60) chatAvg = (Number((await chatReads.streamStats(streamId) || {}).messages) || 0) / (secs / 30);++ if (spanSec >= 60) chatAvg = (Number((await chatReads.streamStats(streamId) || {}).messages) || 0) / (spanSec / 30);} catch { /* chat optional */ }-- return _inPart({ memories, transcript, said, clipTimes, spikes, sounds, heard, chatAvg }, vodId);++ // A sound the stream hears all evening (a shooter's gunfire) is its background, not a moment: only the rare ones++ // count as evidence or as a place to cut (clip-score.rareSounds).++ const notable = clipScore.rareSounds(heard, spanSec);++ return _inPart({ memories, transcript, said, clipTimes, spikes, sounds, heard: notable, chatAvg }, vodId);}/**@@@ -80,11 +80,18 @@ async function _confirmLiveMoment(stream) {// Sound events are a strong clip signal on their own — a burst of gunfire, an explosion or laughter is exactly// the kind of thing viewers clip, and it is often the reason chat spiked in the first place.let sounds = [];++ let notable = [];try {-- const since = Math.max(0, ((Date.now() - new Date(stream.started_at + 'Z').getTime()) / 1000) - WINDOW_SEC);++ const elapsed = Math.max(0, (Date.now() - new Date(stream.started_at + 'Z').getTime()) / 1000);++ const since = Math.max(0, elapsed - WINDOW_SEC);sounds = (await db.getTimeline(stream.id, { kind: 'sound', from: since, limit: 40 }) || [])// Background music (audio-events.js spans) is the stream's bed, not a moment: never a clip signal..filter(e => Number(e.confidence || 0) >= 0.4 && e.label !== 'Music').slice(-12);++ // The score counts only the sounds that are news in this stream: a shooter's gunfire all hour is its background++ // (clip-score.rareSounds, over the last hour). The AI still hears them all.++ const hour = await db.getTimeline(stream.id, { kind: 'sound', from: Math.max(0, elapsed - 3600), limit: 2000 }) || [];++ const rare = new Set(clipScore.rareSounds(hour, Math.min(3600, elapsed)).map(e => String(e.label || '')));++ notable = sounds.filter(e => rare.has(String(e.label || '')));} catch { /* timeline optional */ }// The decision rules are OpenVibe.AI's versioned template live.clips.confirm (WS-O task 2); Live sends what it saw,// heard and read, with times so the model can line up speech, sounds and the chat spike.@@@ -95,9 +102,9 @@ async function _confirmLiveMoment(stream) {sounds: sounds.map(e => ({ t: Math.max(0, Number(e.start_sec) || 0), label: String(e.label || '').slice(0, 120), confidence: Number(e.confidence || 0) })),chat: chat.slice(-30).map(c => _clean(c, 100)).filter(Boolean),}, { target: { service: 'live', type: 'stream', id: String(stream.id) }, meter: { kind: 'auto_clip_confirm', role: 'legacy', ownerUserId: stream.user_id } });-- if (!j) return { clip: false, transcript, sounds };-- return { clip: j.clip === true, title: _clean(j.title, 80), desc: _clean(j.desc, 400), transcript, sounds };-- } catch { return { clip: false, transcript, sounds }; }++ if (!j) return { clip: false, transcript, sounds, notable };++ return { clip: j.clip === true, title: _clean(j.title, 80), desc: _clean(j.desc, 400), transcript, sounds, notable };++ } catch { return { clip: false, transcript, sounds, notable }; }}/** What was said around a stream second: the registry uses it so a static camera alone is not a repeat. */@@@ -154,7 +161,7 @@ async function _checkLiveStream(stream) {// The score: no clip beats a bad clip; a fair one is kept unlisted for the streamer; a clear one is public.const near = _speechNear(verdict.transcript, streamSec, 20);const chatters = await chatReads.recentChatters(streamId, BUCKET_SEC * 2).catch(() => null);-- const rated = clipScore.score({ spike: { peak: peak.count, avg }, chatters, sounds: verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });++ const rated = clipScore.score({ spike: { peak: peak.count, avg }, chatters, sounds: verdict.notable || verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });if (rated.verdict === 'skip') { console.log(`[AutoClip] stream ${streamId}: skipped a weak moment (score ${rated.score}${rated.reasons.length ? `: ${rated.reasons.join('; ')}` : ''})`); return; }const why = await registry.usedReason({ vod_id: rec.vodId, stream_id: streamId, offset: momentOffset, desc: verdict.desc, title, speech });if (why) { console.log(`[AutoClip] stream ${streamId}: ${why}`); return; }@@@ -73,4 +73,18 @@ function score({ spike = {}, chatters = null, sounds = [], speechWords = 0, ai =return { score: Math.round(total * 1000) / 1000, verdict, parts, reasons, bucket, evidence };}--module.exports = { score, BARS, WEIGHTS, STRONG_SOUND, EVIDENCE, REACTED };++/**++ * The sounds that are news in this stream (item 151, C08 follow-up). The audio classifier hears a shooter's explosions++ * and gunfire all evening: in production (11-03) "Explosion" and "Gunshot, gunfire" at 0.6 or more were in the++ * thousands across 200+ streams, while laughter was rare. A sound that comes back more than `perHour` times an hour in++ * the stream is its background, not a moment, so it is no evidence. Rates are over at least a quarter hour.++ */++function rareSounds(events, spanSec, { perHour = 12 } = {}) {++ const list = Array.isArray(events) ? events : [];++ const hours = Math.max(0.25, (Number(spanSec) || 0) / 3600);++ const count = new Map();++ for (const e of list) { const k = String((e && e.label) || ''); count.set(k, (count.get(k) || 0) + 1); }++ return list.filter((e) => (count.get(String((e && e.label) || '')) || 0) / hours <= perHour);++}++++module.exports = { score, rareSounds, BARS, WEIGHTS, STRONG_SOUND, EVIDENCE, REACTED };nnew file mode 100644@@@ -0,0 +1,50 @@++'use strict';++// A sound the stream hears all evening is its background, not a moment (item 151, C08 follow-up). In production++// (2026-11-03) "Explosion" and "Gunshot, gunfire" at 0.6 or more ran into the thousands across 200+ streams while++// laughter was rare, so a shooter's every explosion counted as "a strong sound" for the clip score. clip-score.rareSounds++// keeps the labels heard at most 12 times an hour in the stream; the VOD picker and the live score use only those.++const assert = require('assert');++const db = require('../server/db/database');++console.warn = () => {};++++(async () => {++ const clipScore = require('../server/ai/clip-score');++ const boom = (t) => ({ label: 'Explosion', confidence: 0.8, start_sec: t });++ const laugh = (t) => ({ label: 'Laughter', confidence: 0.8, start_sec: t });++++ // ── Pure ──++ const evening = [...Array.from({ length: 60 }, (_, i) => boom(i * 120)), laugh(3000)];++ assert.deepStrictEqual(clipScore.rareSounds(evening, 7200).map((e) => e.label), ['Laughter'], '60 explosions in two hours are the game, one laugh is news');++ assert.strictEqual(clipScore.rareSounds([boom(10), boom(20), boom(30)], 7200).length, 3, 'three in two hours are rare');++ assert.strictEqual(clipScore.rareSounds([boom(10), boom(20), boom(30), boom(40)], 120).length, 0, 'a short span counts as a quarter hour: four is 16 an hour');++ assert.deepStrictEqual(clipScore.rareSounds(null, 100), []);++++ // ── The VOD picker (AI off): the laugh is the moment, the explosions are not ──++ await db.initDb();++ const chatReads = require('../server/chat/chat-reads');++ const media = require('../server/media-client');++ const ai = require('../server/ai/ai-analysis');++ const moments = require('../server/ai/ai-moments-job');++ await require('../server/ai/moment-registry')._clear();++ await db.run(`INSERT INTO users (id, username, display_name, email, password_hash, role, created_at) OVERRIDING SYSTEM VALUE++ VALUES (31, 'rosa', 'rosa', 'rosa@x', 'x', 'streamer', '2025-01-01 00:00:00') ON CONFLICT DO NOTHING`);++ const sid = (await db.get(`INSERT INTO streams (user_id, title, started_at, ended_at) VALUES (31, 'shooter night', '2026-11-01 10:00:00', '2026-11-01 12:00:00') RETURNING id`)).id;++ ai.isEnabled = async () => false;++ db.getStreamMemories = async () => [{ offset_seconds: 3000, description: 'The streamer laughs at a teammate falling off a ledge' }, { offset_seconds: 600, description: 'A firefight on a bridge' }];++ db.getStreamTranscriptSegments = async () => [{ start: 2995, end: 3004, text: 'no way he just walked off the bridge oh my god' }];++ db.getTimeline = async () => evening;++ chatReads.spikeOffsets = async () => [];++ chatReads.streamStats = async () => ({ messages: 0 });++ media.listClips = async () => ({ clips: [] });++ const v = { vod_id: 701, stream_id: sid, duration: 7200, title: 'Shooter night' };++ const m = await moments.findBestMoment(v, { flavor: 'clip' });++ assert.ok(m, 'a clip moment is found');++ assert.strictEqual(m.offset, 3000, 'the laugh, not the first explosion');++ assert.strictEqual(m.evidence.picked_by, 'sound');++ assert.deepStrictEqual(m.evidence.sounds.map((e) => e.label), ['Laughter'], 'the explosions nearby are not evidence');++++ db.getTimeline = async () => evening.filter((e) => e.label === 'Explosion');++ assert.strictEqual(await moments.findBestMoment(v, { flavor: 'clip' }), null, 'a stream of nothing but its background gives no clip');++++ console.log('rare sounds: all checks passed');++})().catch((e) => { console.error(e); process.exit(1); });
@@ -214,7 +214,7 @@ Two jobs turn stream moments into content: `server/ai/ai-moments-job.js` (image
@@ -214,7 +214,7 @@ Two jobs turn stream moments into content: `server/ai/ai-moments-job.js` (image
- The paste job no longer clips the paste's own moment; it asks for a **second, different beat** from the same VOD for the clip. The clip backfill avoids every offset a paste used.
- The paste job no longer clips the paste's own moment; it asks for a **second, different beat** from the same VOD for the clip. The clip backfill avoids every offset a paste used.
- **Live pastes** — the stream-memory vision call now also answers "is this frame screenshot-worthy?" with a caption (no extra call). When it says yes, `stream-memory-job` posts the frame as an image paste right away (≤ 1 per stream per 90 min, dark-frame and registry checks, setting `ai_live_pastes_enabled`, default on). These pastes carry `metadata.live = true` and no VOD link.
- **Live pastes** — the stream-memory vision call now also answers "is this frame screenshot-worthy?" with a caption (no extra call). When it says yes, `stream-memory-job` posts the frame as an image paste right away (≤ 1 per stream per 90 min, dark-frame and registry checks, setting `ai_live_pastes_enabled`, default on). These pastes carry `metadata.live = true` and no VOD link.
- **Only good live clips, cut on sentences** — a live chat spike is not enough on its own. `server/ai/clip-score.js` scores the moment from 0 to 1: the spike (how far above the window's average and how big), how many people reacted, what was heard (laughter, cheering, a bang; music and speech alone count for nothing), how much was said, and the AI's yes or no (a no is a veto; with the AI off its part is neutral and both bars rise by 0.1). Below **0.40** no clip is cut, the log says why ("a weak chat spike (10 messages, 3.1× the average)"). From 0.40 the clip is made **unlisted**: the streamer can find it and make it public. Calibrated by the production audit (C01, `scripts/diagnostics/clip-audit-*.sql`: the old rule made 99.6 % of automatic clips public and 29 % were never watched in 30 days): a moment needs **evidence** to be kept at all (a spike of at least 3× the average and 10 messages, a strong sound, or 20 spoken words; the AI's yes alone is not enough), and it is cut **public** only at **0.70** or more **and** in the reacted bucket: at least 12 messages in the peak from at least 3 people (`chatReads.recentChatters`; unknown means unlisted). Every cut logs its bucket (A no chat, B 1–5, C 6–11, D 12+ with 3+ people), peak and chatters for the next audit. `server/ai/clip-trim.js` then moves the fixed 16 s before / 9 s after window onto the speech rows: the start goes back to where the sentence being spoken began (so the setup is in), at most 25 s before the moment; the end goes on to where the sentence ends, at most 20 s after, never past the recording's flushed edge. A snap that would make the clip shorter than 12 s or longer than 45 s is not taken. Both are pure, with their cases in `test/clip-score-trim.test.js`.
- **Only good live clips, cut on sentences** — a live chat spike is not enough on its own. `server/ai/clip-score.js` scores the moment from 0 to 1: the spike (how far above the window's average and how big), how many people reacted, what was heard (laughter, cheering, a bang; music and speech alone count for nothing), how much was said, and the AI's yes or no (a no is a veto; with the AI off its part is neutral and both bars rise by 0.1). Below **0.40** no clip is cut, the log says why ("a weak chat spike (10 messages, 3.1× the average)"). From 0.40 the clip is made **unlisted**: the streamer can find it and make it public. Calibrated by the production audit (C01, `scripts/diagnostics/clip-audit-*.sql`: the old rule made 99.6 % of automatic clips public and 29 % were never watched in 30 days): a moment needs **evidence** to be kept at all (a spike of at least 3× the average and 10 messages, a strong sound, or 20 spoken words; the AI's yes alone is not enough), and it is cut **public** only at **0.70** or more **and** in the reacted bucket: at least 12 messages in the peak from at least 3 people (`chatReads.recentChatters`; unknown means unlisted). Every cut logs its bucket (A no chat, B 1–5, C 6–11, D 12+ with 3+ people), peak and chatters for the next audit. `server/ai/clip-trim.js` then moves the fixed 16 s before / 9 s after window onto the speech rows: the start goes back to where the sentence being spoken began (so the setup is in), at most 25 s before the moment; the end goes on to where the sentence ends, at most 20 s after, never past the recording's flushed edge. A snap that would make the clip shorter than 12 s or longer than 45 s is not taken. Both are pure, with their cases in `test/clip-score-trim.test.js`.
-- **VOD clips are scored too, and may be none** (item 151, C08). The VOD cutter (`clipVodMoment`, from the backfill and from the paste job's second beat) used to cut whatever the picker returned, public. Now the picker hands back the moment's **evidence**, gathered at that second from everything Live has (not the sample the prompt reads): the chat bucket there against the stream's own average per 30 s with the quiet stretches counted, how many people were in it (when Chat counts them), the sounds within 15 s, the words said within 20 s, and who picked it (`ai`, `chat`, `sound`, `people`, `scene`). The cutter runs it through the same `clip-score.js` as the live cutter: below the keep bar nothing is cut (the log says the score and why), a kept moment is **unlisted**, and only a reacted room (12+ messages from 3+ people, a score of 0.70 or more) is cut **public**, never above the recording's own visibility. A moment with no evidence is not cut. A clip never re-cuts a second a person already clipped (that footage is their clip already; it goes in the picker's avoid list), and with the AI off a clip needs a chat spike or a strong sound: a long scene note alone only ever makes a paste. The AI may answer that nothing is worth it (`{ abstain: true }` or `t: null`), and then no weaker signal is tried. The AI's own clips are no longer counted as "people clipped here" (`lookups.clipStartTimes`). Each cut logs its score, bucket, picker and policy (`vod-moment@2026-11-03`), and the daily Bench evaluation labels the AI's entry with that policy (`clip-evaluation.policyOf`), so Bench's accept rate tells this policy from the one before. Cases: `test/vod-moment-evidence.test.js`.
+- **VOD clips are scored too, and may be none** (item 151, C08). The VOD cutter (`clipVodMoment`, from the backfill and from the paste job's second beat) used to cut whatever the picker returned, public. Now the picker hands back the moment's **evidence**, gathered at that second from everything Live has (not the sample the prompt reads): the chat bucket there against the stream's own average per 30 s with the quiet stretches counted, how many people were in it (when Chat counts them), the sounds within 15 s, the words said within 20 s, and who picked it (`ai`, `chat`, `sound`, `people`, `scene`). The cutter runs it through the same `clip-score.js` as the live cutter: below the keep bar nothing is cut (the log says the score and why), a kept moment is **unlisted**, and only a reacted room (12+ messages from 3+ people, a score of 0.70 or more) is cut **public**, never above the recording's own visibility. A moment with no evidence is not cut. A clip never re-cuts a second a person already clipped (that footage is their clip already; it goes in the picker's avoid list), and with the AI off a clip needs a chat spike or a strong sound: a long scene note alone only ever makes a paste. The AI may answer that nothing is worth it (`{ abstain: true }` or `t: null`), and then no weaker signal is tried. The AI's own clips are no longer counted as "people clipped here" (`lookups.clipStartTimes`). A sound counts as evidence only when it is news in its stream (`clip-score.rareSounds`: at most 12 an hour; production on 2026-11-03 had thousands of "Explosion" and "Gunshot" readings across 200+ streams, while laughter was rare), for VOD moments over the stream and for live clips over the last hour; the AI still hears them all. Each cut logs its score, bucket, picker and policy (`vod-moment@2026-11-03`), and the daily Bench evaluation labels the AI's entry with that policy (`clip-evaluation.policyOf`), so Bench's accept rate tells this policy from the one before. Cases: `test/vod-moment-evidence.test.js`.
- **Fewer tokens** — VOD showcase scores are cached for 7 days (`home_hero_moments.rankCache`); a run only scores VODs it has never scored. Every `llm.complete` call keeps its `kind` so `ai_usage` shows exactly where the budget goes (`moment_vod_rank`, `moment_pick`, `moment_frame`, `stream_memory`, `auto_clip_confirm`).
- **Fewer tokens** — VOD showcase scores are cached for 7 days (`home_hero_moments.rankCache`); a run only scores VODs it has never scored. Every `llm.complete` call keeps its `kind` so `ai_usage` shows exactly where the budget goes (`moment_vod_rank`, `moment_pick`, `moment_frame`, `stream_memory`, `auto_clip_confirm`).
## AI-inferred category
## AI-inferred category
@@ -171,11 +171,14 @@ async function _momentContext(streamId, vodId) {
@@ -171,11 +171,14 @@ async function _momentContext(streamId, vodId) {
// The stream's average chat per 30 s, empty stretches counted (C07/C08: averaging only the busy buckets made every
// The stream's average chat per 30 s, empty stretches counted (C07/C08: averaging only the busy buckets made every
// spike look small on a quiet channel and every lull look big on a busy one).
// spike look small on a quiet channel and every lull look big on a busy one).
let chatAvg = 0;
let chatAvg = 0;
+ const spanSec = started > 0 ? Math.max(0, ((ms(span.ended_at) || Date.now()) - started) / 1000) : 0;
try {
try {
- const secs = started > 0 ? ((ms(span.ended_at) || Date.now()) - started) / 1000 : 0;
- if (secs >= 60) chatAvg = (Number((await chatReads.streamStats(streamId) || {}).messages) || 0) / (secs / 30);
+ if (spanSec >= 60) chatAvg = (Number((await chatReads.streamStats(streamId) || {}).messages) || 0) / (spanSec / 30);
} catch { /* chat optional */ }
} catch { /* chat optional */ }
- return _inPart({ memories, transcript, said, clipTimes, spikes, sounds, heard, chatAvg }, vodId);
+ // A sound the stream hears all evening (a shooter's gunfire) is its background, not a moment: only the rare ones
+ // count as evidence or as a place to cut (clip-score.rareSounds).
+ const notable = clipScore.rareSounds(heard, spanSec);
+ return _inPart({ memories, transcript, said, clipTimes, spikes, sounds, heard: notable, chatAvg }, vodId);
}
}
/**
/**
@@ -80,11 +80,18 @@ async function _confirmLiveMoment(stream) {
@@ -80,11 +80,18 @@ async function _confirmLiveMoment(stream) {
// Sound events are a strong clip signal on their own — a burst of gunfire, an explosion or laughter is exactly
// Sound events are a strong clip signal on their own — a burst of gunfire, an explosion or laughter is exactly
// the kind of thing viewers clip, and it is often the reason chat spiked in the first place.
// the kind of thing viewers clip, and it is often the reason chat spiked in the first place.
let sounds = [];
let sounds = [];
+ let notable = [];
try {
try {
- const since = Math.max(0, ((Date.now() - new Date(stream.started_at + 'Z').getTime()) / 1000) - WINDOW_SEC);
+ const elapsed = Math.max(0, (Date.now() - new Date(stream.started_at + 'Z').getTime()) / 1000);
+ const since = Math.max(0, elapsed - WINDOW_SEC);
sounds = (await db.getTimeline(stream.id, { kind: 'sound', from: since, limit: 40 }) || [])
sounds = (await db.getTimeline(stream.id, { kind: 'sound', from: since, limit: 40 }) || [])
// Background music (audio-events.js spans) is the stream's bed, not a moment: never a clip signal.
// Background music (audio-events.js spans) is the stream's bed, not a moment: never a clip signal.
.filter(e => Number(e.confidence || 0) >= 0.4 && e.label !== 'Music').slice(-12);
.filter(e => Number(e.confidence || 0) >= 0.4 && e.label !== 'Music').slice(-12);
+ // The score counts only the sounds that are news in this stream: a shooter's gunfire all hour is its background
+ // (clip-score.rareSounds, over the last hour). The AI still hears them all.
+ const hour = await db.getTimeline(stream.id, { kind: 'sound', from: Math.max(0, elapsed - 3600), limit: 2000 }) || [];
+ const rare = new Set(clipScore.rareSounds(hour, Math.min(3600, elapsed)).map(e => String(e.label || '')));
+ notable = sounds.filter(e => rare.has(String(e.label || '')));
} catch { /* timeline optional */ }
} catch { /* timeline optional */ }
// The decision rules are OpenVibe.AI's versioned template live.clips.confirm (WS-O task 2); Live sends what it saw,
// The decision rules are OpenVibe.AI's versioned template live.clips.confirm (WS-O task 2); Live sends what it saw,
// heard and read, with times so the model can line up speech, sounds and the chat spike.
// heard and read, with times so the model can line up speech, sounds and the chat spike.
@@ -95,9 +102,9 @@ async function _confirmLiveMoment(stream) {
@@ -95,9 +102,9 @@ async function _confirmLiveMoment(stream) {
sounds: sounds.map(e => ({ t: Math.max(0, Number(e.start_sec) || 0), label: String(e.label || '').slice(0, 120), confidence: Number(e.confidence || 0) })),
sounds: sounds.map(e => ({ t: Math.max(0, Number(e.start_sec) || 0), label: String(e.label || '').slice(0, 120), confidence: Number(e.confidence || 0) })),
chat: chat.slice(-30).map(c => _clean(c, 100)).filter(Boolean),
chat: chat.slice(-30).map(c => _clean(c, 100)).filter(Boolean),
}, { target: { service: 'live', type: 'stream', id: String(stream.id) }, meter: { kind: 'auto_clip_confirm', role: 'legacy', ownerUserId: stream.user_id } });
}, { target: { service: 'live', type: 'stream', id: String(stream.id) }, meter: { kind: 'auto_clip_confirm', role: 'legacy', ownerUserId: stream.user_id } });
- if (!j) return { clip: false, transcript, sounds };
- return { clip: j.clip === true, title: _clean(j.title, 80), desc: _clean(j.desc, 400), transcript, sounds };
- } catch { return { clip: false, transcript, sounds }; }
+ if (!j) return { clip: false, transcript, sounds, notable };
+ return { clip: j.clip === true, title: _clean(j.title, 80), desc: _clean(j.desc, 400), transcript, sounds, notable };
+ } catch { return { clip: false, transcript, sounds, notable }; }
}
}
/** What was said around a stream second: the registry uses it so a static camera alone is not a repeat. */
/** What was said around a stream second: the registry uses it so a static camera alone is not a repeat. */
@@ -154,7 +161,7 @@ async function _checkLiveStream(stream) {
@@ -154,7 +161,7 @@ async function _checkLiveStream(stream) {
// The score: no clip beats a bad clip; a fair one is kept unlisted for the streamer; a clear one is public.
// The score: no clip beats a bad clip; a fair one is kept unlisted for the streamer; a clear one is public.
const near = _speechNear(verdict.transcript, streamSec, 20);
const near = _speechNear(verdict.transcript, streamSec, 20);
const chatters = await chatReads.recentChatters(streamId, BUCKET_SEC * 2).catch(() => null);
const chatters = await chatReads.recentChatters(streamId, BUCKET_SEC * 2).catch(() => null);
- const rated = clipScore.score({ spike: { peak: peak.count, avg }, chatters, sounds: verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });
+ const rated = clipScore.score({ spike: { peak: peak.count, avg }, chatters, sounds: verdict.notable || verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });
if (rated.verdict === 'skip') { console.log(`[AutoClip] stream ${streamId}: skipped a weak moment (score ${rated.score}${rated.reasons.length ? `: ${rated.reasons.join('; ')}` : ''})`); return; }
if (rated.verdict === 'skip') { console.log(`[AutoClip] stream ${streamId}: skipped a weak moment (score ${rated.score}${rated.reasons.length ? `: ${rated.reasons.join('; ')}` : ''})`); return; }
const why = await registry.usedReason({ vod_id: rec.vodId, stream_id: streamId, offset: momentOffset, desc: verdict.desc, title, speech });
const why = await registry.usedReason({ vod_id: rec.vodId, stream_id: streamId, offset: momentOffset, desc: verdict.desc, title, speech });
if (why) { console.log(`[AutoClip] stream ${streamId}: ${why}`); return; }
if (why) { console.log(`[AutoClip] stream ${streamId}: ${why}`); return; }
@@ -73,4 +73,18 @@ function score({ spike = {}, chatters = null, sounds = [], speechWords = 0, ai =
@@ -73,4 +73,18 @@ function score({ spike = {}, chatters = null, sounds = [], speechWords = 0, ai =
return { score: Math.round(total * 1000) / 1000, verdict, parts, reasons, bucket, evidence };
return { score: Math.round(total * 1000) / 1000, verdict, parts, reasons, bucket, evidence };
}
}
-module.exports = { score, BARS, WEIGHTS, STRONG_SOUND, EVIDENCE, REACTED };
+/**
+ * The sounds that are news in this stream (item 151, C08 follow-up). The audio classifier hears a shooter's explosions
+ * and gunfire all evening: in production (11-03) "Explosion" and "Gunshot, gunfire" at 0.6 or more were in the
+ * thousands across 200+ streams, while laughter was rare. A sound that comes back more than `perHour` times an hour in
+ * the stream is its background, not a moment, so it is no evidence. Rates are over at least a quarter hour.
+ */
+function rareSounds(events, spanSec, { perHour = 12 } = {}) {
+ const list = Array.isArray(events) ? events : [];
+ const hours = Math.max(0.25, (Number(spanSec) || 0) / 3600);
+ const count = new Map();
+ for (const e of list) { const k = String((e && e.label) || ''); count.set(k, (count.get(k) || 0) + 1); }
+ return list.filter((e) => (count.get(String((e && e.label) || '')) || 0) / hours <= perHour);
+}
+
+module.exports = { score, rareSounds, BARS, WEIGHTS, STRONG_SOUND, EVIDENCE, REACTED };
new file mode 100644
new file mode 100644
@@ -0,0 +1,50 @@
@@ -0,0 +1,50 @@
+'use strict';
+// A sound the stream hears all evening is its background, not a moment (item 151, C08 follow-up). In production
+// (2026-11-03) "Explosion" and "Gunshot, gunfire" at 0.6 or more ran into the thousands across 200+ streams while
+// laughter was rare, so a shooter's every explosion counted as "a strong sound" for the clip score. clip-score.rareSounds
+// keeps the labels heard at most 12 times an hour in the stream; the VOD picker and the live score use only those.
+const assert = require('assert');
+const db = require('../server/db/database');
+console.warn = () => {};
+
+(async () => {
+ const clipScore = require('../server/ai/clip-score');
+ const boom = (t) => ({ label: 'Explosion', confidence: 0.8, start_sec: t });
+ const laugh = (t) => ({ label: 'Laughter', confidence: 0.8, start_sec: t });
+
+ // ── Pure ──
+ const evening = [...Array.from({ length: 60 }, (_, i) => boom(i * 120)), laugh(3000)];
+ assert.deepStrictEqual(clipScore.rareSounds(evening, 7200).map((e) => e.label), ['Laughter'], '60 explosions in two hours are the game, one laugh is news');
+ assert.strictEqual(clipScore.rareSounds([boom(10), boom(20), boom(30)], 7200).length, 3, 'three in two hours are rare');
+ assert.strictEqual(clipScore.rareSounds([boom(10), boom(20), boom(30), boom(40)], 120).length, 0, 'a short span counts as a quarter hour: four is 16 an hour');
+ assert.deepStrictEqual(clipScore.rareSounds(null, 100), []);
+
+ // ── The VOD picker (AI off): the laugh is the moment, the explosions are not ──
+ await db.initDb();
+ const chatReads = require('../server/chat/chat-reads');
+ const media = require('../server/media-client');
+ const ai = require('../server/ai/ai-analysis');
+ const moments = require('../server/ai/ai-moments-job');
+ await require('../server/ai/moment-registry')._clear();
+ await db.run(`INSERT INTO users (id, username, display_name, email, password_hash, role, created_at) OVERRIDING SYSTEM VALUE
+ VALUES (31, 'rosa', 'rosa', 'rosa@x', 'x', 'streamer', '2025-01-01 00:00:00') ON CONFLICT DO NOTHING`);
+ const sid = (await db.get(`INSERT INTO streams (user_id, title, started_at, ended_at) VALUES (31, 'shooter night', '2026-11-01 10:00:00', '2026-11-01 12:00:00') RETURNING id`)).id;
+ ai.isEnabled = async () => false;
+ db.getStreamMemories = async () => [{ offset_seconds: 3000, description: 'The streamer laughs at a teammate falling off a ledge' }, { offset_seconds: 600, description: 'A firefight on a bridge' }];
+ db.getStreamTranscriptSegments = async () => [{ start: 2995, end: 3004, text: 'no way he just walked off the bridge oh my god' }];
+ db.getTimeline = async () => evening;
+ chatReads.spikeOffsets = async () => [];
+ chatReads.streamStats = async () => ({ messages: 0 });
+ media.listClips = async () => ({ clips: [] });
+ const v = { vod_id: 701, stream_id: sid, duration: 7200, title: 'Shooter night' };
+ const m = await moments.findBestMoment(v, { flavor: 'clip' });
+ assert.ok(m, 'a clip moment is found');
+ assert.strictEqual(m.offset, 3000, 'the laugh, not the first explosion');
+ assert.strictEqual(m.evidence.picked_by, 'sound');
+ assert.deepStrictEqual(m.evidence.sounds.map((e) => e.label), ['Laughter'], 'the explosions nearby are not evidence');
+
+ db.getTimeline = async () => evening.filter((e) => e.label === 'Explosion');
+ assert.strictEqual(await moments.findBestMoment(v, { flavor: 'clip' }), null, 'a stream of nothing but its background gives no clip');
+
+ console.log('rare sounds: all checks passed');
+})().catch((e) => { console.error(e); process.exit(1); });