openvibers/openvibe.ai Commits

live.moments.verify: a claim missing from the lines is unsupported, not unsure (#281)

OP OpenVibers committed 10 days ago · 9d4c4a0fc2 · 2 files, +2 −1

The production probe after #280: a 'Dragon Boss Down' title over lines about tripping on
a curb came back unsure, the dragon named only in 'why'. The prompt now says a claim the
lines never mention is unsupported and listed, and unsure is only for too few lines.

Changes

ddiff --git a/server/workflows/live.js b/server/workflows/live.js
iindex 666494c..6f60e39 100644
---- a/server/workflows/live.js
++++ b/server/workflows/live.js
@@@ -253,7 +253,7 @@ const templates = [
              },
          },
          system_prompt: '',
--        user_prompt: 'A clip is about to be published from the livestream "{{title}}". It is about the moment at [{{at}}].\n\nPROPOSED TITLE: {{clip_title}}\nPROPOSED DESCRIPTION: {{clip_desc}}\n\nEverything known about the minute around it, each line timestamped [m:ss]:\nSCENE NOTES:\n{{timeline}}\n\nSPEECH:\n{{transcript}}\n\nSOUNDS:\n{{sounds}}\n\nCHAT: {{chat}}\n\nCheck the title and the description against these lines only. Answer "supported" when everything they say happens in the lines; "unsupported" when they claim something the lines contradict or never mention (an event, an object, a reaction, a person); "unsure" when the lines are too thin to tell. A funny or punchy wording is fine; an invented event is not. Never guess. Return STRICT JSON only: {"verdict": "supported" | "unsupported" | "unsure", "unsupported": ["<each claim the lines do not back>"], "why": "<a few words>"}',
++        user_prompt: 'A clip is about to be published from the livestream "{{title}}". It is about the moment at [{{at}}].\n\nPROPOSED TITLE: {{clip_title}}\nPROPOSED DESCRIPTION: {{clip_desc}}\n\nEverything known about the minute around it, each line timestamped [m:ss]:\nSCENE NOTES:\n{{timeline}}\n\nSPEECH:\n{{transcript}}\n\nSOUNDS:\n{{sounds}}\n\nCHAT: {{chat}}\n\nCheck the title and the description against these lines only. Answer "supported" when everything they say happens in the lines. Answer "unsupported" when they name an event, an object, an action, a reaction or a person that the lines contradict or never mention, and list each such claim in "unsupported": a claim missing from the lines is unsupported, not unsure. Answer "unsure" only when there are too few lines to judge at all. A funny or punchy wording is fine; an invented event is not. Never guess. Return STRICT JSON only: {"verdict": "supported" | "unsupported" | "unsure", "unsupported": ["<each claim the lines do not back>"], "why": "<a few words>"}',
          default_route: 'live.legacy', owner: 'live', metadata: { json_name: 'moment_verify' },
      },
      {
ddiff --git a/test/live-templates.test.js b/test/live-templates.test.js
iindex 2b1df31..d956bfc 100644
---- a/test/live-templates.test.js
++++ b/test/live-templates.test.js
@@@ -83,6 +83,7 @@ t.test('the verifier (C09) reads the minute around the moment and only answers f
      assert.ok(!/far away/.test(out), 'only the minute around the moment');
      assert.match(out, /CHAT: 22 messages in the busiest 30 s \(the stream averages 3\) from 7 people/);
      assert.ok(!/\{\{|\$\{/.test(out), 'nothing left unrendered');
++    assert.match(out, /a claim missing from the lines is unsupported, not unsure\. Answer "unsure" only when there are too few lines to judge at all/, 'a missing event is unsupported (11-03: a dragon fight came back unsure)');
      assert.deepStrictEqual(PREPARE['live.moments.verify']({ clip_title: 'x', t: 10, timeline: [{ t: 500, text: 'later' }] }), { output: { verdict: 'unsure', unsupported: [], why: 'nothing recorded near the moment to check against' } }, 'nothing to check: unsure, no model');
      const post = POSTPROCESS['live.moments.verify'];
      assert.deepStrictEqual(post({ json: { verdict: 'supported', unsupported: [], why: 'he cheers' } }), { verdict: 'supported', unsupported: [], why: 'he cheers' });