openvibers/openvibe.live Commits

Automatic clips calibrated by the production audit: evidence first, public only when a crowd reacted (#446)

OP OpenVibers committed 15 days ago · 749be42718 · 5 files, +61 −10

C01 (Live#443) found the old rule made 99.6% of automatic clips public, 29% were never watched in 30 days, and neither the spike nor the audience predicted a watch. So clip-score now keeps a moment only with evidence (a spike of 3x the average and 10 messages, a strong sound, or 20 spoken words; never the AI's yes alone), cuts it public only at 0.70 and from the reacted bucket (12+ messages from 3+ people, counted by chatReads.recentChatters; unknown stays unlisted), and logs each cut's bucket, peak and chatters for the next audit.

Changes

ddiff --git a/docs/vods-and-clips.md b/docs/vods-and-clips.md
iindex ff94f668..af2eaaff 100644
---- a/docs/vods-and-clips.md
++++ b/docs/vods-and-clips.md
@@@ -213,7 +213,7 @@ Two jobs turn stream moments into content: `server/ai/ai-moments-job.js` (image
- **Flavoured picks** — `findBestMoment(vod, { flavor, avoid })`: `paste` asks for a frame that is striking on its own (a face, a gag, something odd on screen); `clip` asks for a beat that plays out over 25 s (a line, a reaction, a sound, chat exploding). The prompt lists the already-used timestamps and the model is told to stay away from them; if it still lands next to one, the objective signals (viewer clips, chat spikes, richest scene notes) pick a free spot.
- The paste job no longer clips the paste's own moment; it asks for a **second, different beat** from the same VOD for the clip. The clip backfill avoids every offset a paste used.
- **Live pastes** — the stream-memory vision call now also answers "is this frame screenshot-worthy?" with a caption (no extra call). When it says yes, `stream-memory-job` posts the frame as an image paste right away (≤ 1 per stream per 90 min, dark-frame and registry checks, setting `ai_live_pastes_enabled`, default on). These pastes carry `metadata.live = true` and no VOD link.
--- **Only good live clips, cut on sentences** — a live chat spike is not enough on its own. `server/ai/clip-score.js` scores the moment from 0 to 1: the spike (how far above the window's average and how big), how many people reacted, what was heard (laughter, cheering, a bang; music and speech alone count for nothing), how much was said, and the AI's yes or no (a no is a veto; with the AI off its part is neutral and both bars rise by 0.1). Below **0.40** no clip is cut, the log says why ("a weak chat spike (10 messages, 3.1× the average)"). From 0.40 the clip is made **unlisted**: the streamer can find it and make it public. Only **0.65** and up is cut **public**. `server/ai/clip-trim.js` then moves the fixed 16 s before / 9 s after window onto the speech rows: the start goes back to where the sentence being spoken began (so the setup is in), at most 25 s before the moment; the end goes on to where the sentence ends, at most 20 s after, never past the recording's flushed edge. A snap that would make the clip shorter than 12 s or longer than 45 s is not taken. Both are pure, with their cases in `test/clip-score-trim.test.js`.
++- **Only good live clips, cut on sentences** — a live chat spike is not enough on its own. `server/ai/clip-score.js` scores the moment from 0 to 1: the spike (how far above the window's average and how big), how many people reacted, what was heard (laughter, cheering, a bang; music and speech alone count for nothing), how much was said, and the AI's yes or no (a no is a veto; with the AI off its part is neutral and both bars rise by 0.1). Below **0.40** no clip is cut, the log says why ("a weak chat spike (10 messages, 3.1× the average)"). From 0.40 the clip is made **unlisted**: the streamer can find it and make it public. Calibrated by the production audit (C01, `scripts/diagnostics/clip-audit-*.sql`: the old rule made 99.6 % of automatic clips public and 29 % were never watched in 30 days): a moment needs **evidence** to be kept at all (a spike of at least 3× the average and 10 messages, a strong sound, or 20 spoken words; the AI's yes alone is not enough), and it is cut **public** only at **0.70** or more **and** in the reacted bucket: at least 12 messages in the peak from at least 3 people (`chatReads.recentChatters`; unknown means unlisted). Every cut logs its bucket (A no chat, B 1–5, C 6–11, D 12+ with 3+ people), peak and chatters for the next audit. `server/ai/clip-trim.js` then moves the fixed 16 s before / 9 s after window onto the speech rows: the start goes back to where the sentence being spoken began (so the setup is in), at most 25 s before the moment; the end goes on to where the sentence ends, at most 20 s after, never past the recording's flushed edge. A snap that would make the clip shorter than 12 s or longer than 45 s is not taken. Both are pure, with their cases in `test/clip-score-trim.test.js`.
- **Fewer tokens** — VOD showcase scores are cached for 7 days (`home_hero_moments.rankCache`); a run only scores VODs it has never scored. Every `llm.complete` call keeps its `kind` so `ai_usage` shows exactly where the budget goes (`moment_vod_rank`, `moment_pick`, `moment_frame`, `stream_memory`, `auto_clip_confirm`).
## AI-inferred category
ddiff --git a/server/ai/auto-clip-job.js b/server/ai/auto-clip-job.js
iindex 64ca201f..f6fb3840 100644
---- a/server/ai/auto-clip-job.js
++++ b/server/ai/auto-clip-job.js
@@@ -152,7 +152,8 @@ async function _checkLiveStream(stream) {
const speech = _speechNear(verdict.transcript, streamSec);
// The score: no clip beats a bad clip; a fair one is kept unlisted for the streamer; a clear one is public.
const near = _speechNear(verdict.transcript, streamSec, 20);
-- const rated = clipScore.score({ spike: { peak: peak.count, avg }, sounds: verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });
++ const chatters = await chatReads.recentChatters(streamId, BUCKET_SEC * 2).catch(() => null);
++ const rated = clipScore.score({ spike: { peak: peak.count, avg }, chatters, sounds: verdict.sounds, speechWords: near ? near.split(/\s+/).length : 0, ai: { clip: verdict.clip } });
if (rated.verdict === 'skip') { console.log(`[AutoClip] stream ${streamId}: skipped a weak moment (score ${rated.score}${rated.reasons.length ? `: ${rated.reasons.join('; ')}` : ''})`); return; }
const why = await registry.usedReason({ vod_id: rec.vodId, stream_id: streamId, offset: momentOffset, desc: verdict.desc, title, speech });
if (why) { console.log(`[AutoClip] stream ${streamId}: ${why}`); return; }
@@@ -169,7 +170,7 @@ async function _checkLiveStream(stream) {
if (clip) {
// deduplicated: Media handed back a clip someone already cut of this window. It stays in
// the log for the caps, but it is theirs, not the AI's (syncAutoClipFlags skips it).
-- await _logClip({ stream_id: streamId, vod_id: rec.vodId, start_time: clipStart, title, clip_id: clip.id, dedup: !!clip.deduplicated || undefined, desc: verdict.desc || '', speech, sig: _sceneSig(verdict.desc || title), score: rated.score, visibility: rated.verdict === 'public' ? 'public' : 'unlisted' });
++ await _logClip({ stream_id: streamId, vod_id: rec.vodId, start_time: clipStart, title, clip_id: clip.id, dedup: !!clip.deduplicated || undefined, desc: verdict.desc || '', speech, sig: _sceneSig(verdict.desc || title), score: rated.score, visibility: rated.verdict === 'public' ? 'public' : 'unlisted', bucket: rated.bucket, peak: peak.count, chatters: chatters == null ? undefined : chatters });
console.log(`[AutoClip] LIVE clip for stream ${streamId} @${Math.round(momentOffset)}s ("${title}") — spike ${peak.count} msgs, score ${rated.score} → ${rated.verdict}${cut.snapped ? ', on sentences' : ''}`);
}
} catch (e) {
ddiff --git a/server/ai/clip-score.js b/server/ai/clip-score.js
iindex bad7d465..4e5f6d0e 100644
---- a/server/ai/clip-score.js
++++ b/server/ai/clip-score.js
@@@ -11,15 +11,27 @@
* public — a clear moment.
*
* score({ spike: { peak, avg }, chatters, sounds: [{ label, confidence }], speechWords, ai: { clip } })
-- * → { score, verdict, parts, reasons }
++ * → { score, verdict, parts, reasons, bucket, evidence }
*
* The AI's "no" is a veto. With the AI off (ai.clip === null) its part is neutral and both bars rise, as the cutter's
* old no-AI rule required a stronger spike. Pure: no I/O, so the bars can be tuned against labelled moments.
++ *
++ * Calibrated by the production audit (C01, 2026-10-29: research-2026-10-29/clip-audit.md). The old rule made 99.6 % of
++ * automatic clips public, 29 % were never watched by a person in 30 days, and neither the spike nor the audience
++ * predicted a watch. So:
++ * - evidence first: a moment is kept only with a real spike (≥ 3× the average and ≥ 10 messages), a strong sound or
++ * ≥ 20 spoken words — never on the AI's yes alone;
++ * - public only from the reacted bucket: a score ≥ 0.70 and ≥ 12 messages in the peak from ≥ 3 people (unknown
++ * chatters are never public);
++ * - every cut says its bucket (A no chat · B 1–5 · C 6–11 · D ≥ 12 with ≥ 3 people) so the next audit can measure them.
*/
// keep 0.40 is where the old rule's borderline clips land (a 2.5× spike the AI agreed with): they are kept, unlisted;
--// public needs a clear moment. Calibrate both against C01's production audit.
--const BARS = Object.freeze({ keep: 0.4, public: 0.65, noAiRaise: 0.1 });
++// public needs a clear moment, in the reacted bucket (above).
++const BARS = Object.freeze({ keep: 0.4, public: 0.7, noAiRaise: 0.1 });
++// The evidence a kept moment needs, and the reacted bucket a public one needs (C01).
++const EVIDENCE = Object.freeze({ ratio: 3, peak: 10, soundConfidence: 0.6, words: 20 });
++const REACTED = Object.freeze({ peak: 12, people: 3 });
const WEIGHTS = Object.freeze({ spike: 0.35, breadth: 0.15, sound: 0.15, speech: 0.1, ai: 0.25 });
// Sounds people clip for: laughter, a crowd, a bang. Speech and music alone are not a moment.
const STRONG_SOUND = /laugh|cheer|applause|clap|crowd|scream|shout|explosion|gunshot|gunfire|boom|crash|siren|whistle|gasp/i;
@@@ -48,12 +60,17 @@ function score({ spike = {}, chatters = null, sounds = [], speechWords = 0, ai =
if (spikePart < 0.3) reasons.push(`a weak chat spike (${peak} messages, ${ratio.toFixed(1)}× the average)`);
if (chatters != null && chatters < 3) reasons.push(`only ${chatters} people reacted`);
if (loud.length) reasons.push(`heard: ${loud.map((s) => s.label).slice(0, 3).join(', ')}`);
++ const strongSound = loud.some((s) => Number(s.confidence) >= EVIDENCE.soundConfidence);
++ const evidence = (ratio >= EVIDENCE.ratio && peak >= EVIDENCE.peak) || strongSound || Number(speechWords) >= EVIDENCE.words;
++ const bucket = peak <= 0 ? 'A' : peak <= 5 ? 'B' : peak < REACTED.peak || !(chatters >= REACTED.people) ? 'C' : 'D';
let verdict;
if (ai.clip === false) verdict = 'skip';
-- else if (total >= BARS.public + raise) verdict = 'public';
++ else if (!evidence) { verdict = 'skip'; reasons.push('nothing but the AI\'s yes: no real spike, strong sound or speech'); }
++ else if (total >= BARS.public + raise && bucket === 'D') verdict = 'public';
else if (total >= BARS.keep + raise) verdict = 'unlisted';
else verdict = 'skip';
-- return { score: Math.round(total * 1000) / 1000, verdict, parts, reasons };
++ if (verdict === 'unlisted' && total >= BARS.public + raise && bucket !== 'D') reasons.push(chatters == null ? 'kept unlisted: how many people reacted is unknown' : `kept unlisted: ${peak} messages from ${chatters} people is not a crowd yet`);
++ return { score: Math.round(total * 1000) / 1000, verdict, parts, reasons, bucket, evidence };
}
--module.exports = { score, BARS, WEIGHTS, STRONG_SOUND };
++module.exports = { score, BARS, WEIGHTS, STRONG_SOUND, EVIDENCE, REACTED };
ddiff --git a/server/chat/chat-reads.js b/server/chat/chat-reads.js
iindex 2b49ea5d..fb798518 100644
---- a/server/chat/chat-reads.js
++++ b/server/chat/chat-reads.js
@@@ -479,6 +479,24 @@ function recentChatText(streamId, sinceSec = 120, limit = 40) {
}, () => []);
}
++/** How many different people wrote in a stream's chat in the last `sinceSec` seconds (the auto-clipper's "did a crowd
++ * react": a public clip needs ≥ 3). Counted by user id, else name; null when Chat does not answer. Cached like the rest. */
++function recentChatters(streamId, sinceSec = 30, limit = 200) {
++ const id = Number(streamId) || 0;
++ if (!id) return Promise.resolve(null);
++ const key = `rcp:${id}:${sinceSec}:${limit}`;
++ return load(key, async () => {
++ const out = await client.readMessages({ stream_id: id, limit });
++ if (!out || !Array.isArray(out.messages)) throw unavailable('recent messages');
++ const floor = Date.now() - Math.max(1, sinceSec) * 1000;
++ const people = new Set(out.messages
++ .filter((m) => m.message && Date.parse(String(m.timestamp).replace(' ', 'T') + 'Z') >= floor)
++ .map((m) => (m.user_id != null ? `u${m.user_id}` : m.username ? `n${String(m.username).toLowerCase()}` : null))
++ .filter(Boolean));
++ return people.size;
++ }, () => null);
++}
++
/** The busiest `bucketSec` time-buckets of a stream between `sinceMs` and `untilMs` (default now), as offsets from
* `sinceMs`: [{ offset, count }]. Chat answers at most 10,000 buckets, so a stream that ended gives its end: up to
* now, a stream from two months ago asked for 184,000 and got a 400 (2026-10-23). */
@@@ -735,7 +753,7 @@ module.exports = {
siteStats, siteStatsPeek, topChatters, homeSeries, homeSeriesPeek,
windowStats, windowStatsPeek, streamStats, streamStatsPeek, channelMaxId, channelMaxIdPeek, userMessageCountPeek,
messageById, channelMessages, channelMessagesPeek, userHistory, relayHistory, channelSamples, searchMessages,
-- liveChatBuckets, recentChatText, spikeOffsets, firstChat, firstChatPeek,
++ liveChatBuckets, recentChatText, recentChatters, spikeOffsets, firstChat, firstChatPeek,
pendingIp, relayUsers, relayUser, ttsOverride, ttsOverridePeek, isRelayUserHidden,
soundCount, soundByCommand, soundByCommandPeek, dmBlocked, pendingSounds, recordSoundAsset, invalidate,
_reset, _age, _size, _cacheMax, CACHE_TTL_MS, CACHE_KEEP_MS,
ddiff --git a/test/clip-score-trim.test.js b/test/clip-score-trim.test.js
iindex 790e76d0..024493ca 100644
---- a/test/clip-score-trim.test.js
++++ b/test/clip-score-trim.test.js
@@@ -29,6 +29,21 @@ const { trim } = require('../server/ai/clip-trim');
assert.strictEqual(music.parts.sound, 0, 'music and speech are not a moment by themselves');
assert.strictEqual(score({ spike: { peak: 20, avg: 5 }, ai: { clip: true } }).parts.breadth, score({ spike: { peak: 20, avg: 5 }, ai: { clip: true } }).parts.spike, 'unknown chatters follow the spike');
assert.strictEqual(score().verdict, 'skip', 'nothing is nothing');
++
++ // Calibrated by the C01 audit: evidence first, public only from the reacted bucket, every cut says its bucket.
++ const yesOnly = score({ spike: { peak: 9, avg: 4 }, chatters: 6, sounds: [{ label: 'Laughter', confidence: 0.4 }], speechWords: 12, ai: { clip: true } });
++ assert.strictEqual(yesOnly.verdict, 'skip', `the AI's yes alone keeps nothing (${JSON.stringify(yesOnly)})`);
++ assert.ok(yesOnly.reasons.some((r) => /nothing but the AI/.test(r)) && yesOnly.evidence === false);
++ const loud = score({ spike: { peak: 9, avg: 4 }, chatters: 6, sounds: [{ label: 'Explosion', confidence: 0.9 }], speechWords: 0, ai: { clip: true } });
++ assert.ok(loud.evidence && loud.verdict !== 'skip', 'a strong sound is evidence on its own');
++ const unknownCrowd = score({ spike: { peak: 34, avg: 5 }, sounds: [{ label: 'Laughter', confidence: 0.9 }], speechWords: 35, ai: { clip: true } });
++ assert.strictEqual(unknownCrowd.verdict, 'unlisted', 'without knowing how many reacted, never public');
++ assert.ok(unknownCrowd.reasons.some((r) => /how many people reacted is unknown/.test(r)));
++ const fewPeople = score({ spike: { peak: 34, avg: 5 }, chatters: 2, sounds: [{ label: 'Laughter', confidence: 0.9 }], speechWords: 35, ai: { clip: true } });
++ assert.strictEqual(fewPeople.verdict, 'unlisted', 'two excited people are not a crowd');
++ assert.strictEqual(strong.bucket, 'D');
++ assert.deepStrictEqual([score({ spike: { peak: 0 } }).bucket, score({ spike: { peak: 4 } }).bucket, score({ spike: { peak: 8 }, chatters: 9 }).bucket, score({ spike: { peak: 20 }, chatters: 2 }).bucket], ['A', 'B', 'C', 'C']);
++ assert.strictEqual(BARS.public, 0.7);
}
// ── Trimming ──