openvibers/openvibe.actor Commits

Actor: answers backed by tool results are compared against them by OpenAI's stronger checker (nano failed a true one and passed a misquote) (#57)

OP OpenVibers committed 30 days ago · 9d49526728 · 6 files, +30 −9

After #56 the checker saw the real tool results, but the production proof still failed. OpenAI's smallest checker (gpt-5-nano) failed a correct answer, saying it was "inconsistent with the required JSON format", having read the JSON tool results as a format the answer should follow.

Tried on the host against the real models, with the proof's answer and tool results:
- **Delimited sections and a plain statement** ("the answer is prose for a person, never JSON; only your reply is JSON"): nano then passed the true answer 3 of 3. It also passed an answer that misquoted the file it had been shown.
- **gpt-5-mini** passed the true answer 3 of 3 and failed the misquote, naming the difference.
- **DeepSeek** did the same, but it is the runtime's own family.

So:
- The checker's input has delimited sections, and its prompt says the answer is prose, never JSON.
- It compares every fact a tool result covers (a file's contents, a program's output, names, counts) and fails on any difference.
- An answer backed by tool results is read by OpenAI's stronger checker (`openaiGrounded`, `ACTOR_OPENAI_GROUNDED_CHECK_MODEL`, default gpt-5-mini, with its own rate card). The check stays cross-family, as ADR-044 §5 asks. Answers without tool results keep gpt-5-nano.

Tests: `test/computers.test.js` checks the sections, the prompt, the stronger checker for grounded answers only, and that private mode stays local. The full suite passes (29/29), including the existing cross-family assertion in `tasks.test.js`.

Changes

ddiff --git a/.env.example b/.env.example
iindex 0617b52..57c02c2 100644
---- a/.env.example
++++ b/.env.example
@@@ -37,6 +37,8 @@ ACTOR_DEEPSEEK_API_KEY=
ACTOR_OPENAI_API_KEY=
# ACTOR_OPENAI_MODEL=gpt-5-mini
# ACTOR_OPENAI_CHECK_MODEL=gpt-5-nano
++# The checker for answers backed by tool results (it compares them; the nano model did not):
++# ACTOR_OPENAI_GROUNDED_CHECK_MODEL=gpt-5-mini
# The open model on this server (private mode): an OpenAI-compatible server such as llama.cpp.
ACTOR_LOCAL_LLM_URL=
ACTOR_LOCAL_LLM_MODEL=
ddiff --git a/server/agents/adapters.js b/server/agents/adapters.js
iindex 3a82e05..908954d 100644
---- a/server/agents/adapters.js
++++ b/server/agents/adapters.js
@@@ -108,6 +108,8 @@ function createAdapters({ config, rates, fetchImpl = globalThis.fetch, log = con
const checkers = {
deepseek: deepseekQuick,
openai: p.openai.apiKey ? chatClient({ baseUrl: p.openai.baseUrl, apiKey: p.openai.apiKey, model: p.openai.checkModel || 'gpt-5-nano', rates: rates.openaiCheck, fetchImpl }) : null,
++ // An answer backed by tool results: compared against them by a model that does it reliably (agents/check.js).
++ openaiGrounded: p.openai.apiKey ? chatClient({ baseUrl: p.openai.baseUrl, apiKey: p.openai.apiKey, model: p.openai.groundedCheckModel || 'gpt-5-mini', rates: rates.openaiGroundedCheck, fetchImpl }) : null,
local,
};
ddiff --git a/server/agents/catalog.js b/server/agents/catalog.js
iindex 5aa9910..677da52 100644
---- a/server/agents/catalog.js
++++ b/server/agents/catalog.js
@@@ -155,6 +155,7 @@ const RATES = {
deepseek: { input_usd_per_mtok: 0.30, cached_usd_per_mtok: 0.006, output_usd_per_mtok: 1.20 },
openai: { input_usd_per_mtok: 0.25, cached_usd_per_mtok: 0.025, output_usd_per_mtok: 2.00, per_call_usd: 0.01 },
openaiCheck: { input_usd_per_mtok: 0.05, cached_usd_per_mtok: 0.005, output_usd_per_mtok: 0.40 },
++ openaiGroundedCheck: { input_usd_per_mtok: 0.25, cached_usd_per_mtok: 0.025, output_usd_per_mtok: 2.00 },
local: { input_usd_per_mtok: 0, output_usd_per_mtok: 0 },
};
ddiff --git a/server/agents/check.js b/server/agents/check.js
iindex 034d738..f6376fa 100644
---- a/server/agents/check.js
++++ b/server/agents/check.js
@@@ -16,20 +16,27 @@ const FAMILY = { 'openvibe-runtime': 'deepseek', 'openai-agent': 'openai', 'open
const CHECKER_FOR = { deepseek: ['openai', 'deepseek'], openai: ['deepseek', 'openai'], local: ['local'], codes: ['deepseek', 'openai'] };
const SYSTEM = `You check an AI agent's answer before a person sees it. Reply with JSON only: {"ok": true|false, "reason": "<one short sentence>"}.
++Only your own reply is JSON. The answer is prose meant to be read by a person (often Markdown) and is never expected to be JSON; tool results are JSON because services return JSON.
ok is true when the answer addresses the task and nothing in it is clearly invented, contradictory or unsafe. It does not have to be perfect.
ok is false when the answer is empty, refuses without a good reason, answers a different question, contradicts itself, or states something that is plainly impossible.
The agent may have searched the web or run tools: do not fail an answer only because it states recent facts, versions, prices or dates you cannot verify yourself, especially when it cites a source for them.
The agent may have made the person's own robot act through OpenVibe.Bot: an answer that reports what the robot did, or why it stopped or was refused, addresses the task.
The agent may have used the person's own computer through OpenVibe.Bot (listed or read files, ran a program, wrote a file): an answer that reports what it found, what a program printed, or why the computer refused, addresses the task.
--When the tool results the agent received are given below, they are real: judge the answer against them, and fail it only when it contradicts them or claims what they do not show.
++When the tool results the agent received are given, they are real. Compare every fact in the answer that a tool result covers (a file's contents, a program's output, names, counts, numbers): if any of them differs from the tool result, ok is false and the reason names it. Otherwise do not fail the answer for acting on the person's systems: the tool results show it did.
The agent may have run a coding task on the person's own machine through OpenVibe.Codes: an answer that reports what changed (a summary, the files, a patch), or that the machine is still working on it with a link to follow it, addresses the task.
When the task is a greeting, small talk, a test message or too vague to act on, a short friendly reply, a summary of what the agent can do or a clarifying question is a good answer: ok is true.`;
--/** Which checkers may read this agent's answer, best first: another family, then the same family. */
--function checkersFor(agentId, checkers, { mode }) {
++/**
++ * Which checkers may read this agent's answer, best first: another family, then the same family. An answer backed by
++ * tool results is checked by comparing the two, which OpenAI's smallest model did not do (it passed an answer that
++ * misquoted a file it had been shown, 2026-10-14): for those, OpenAI's stronger checker (`openaiGrounded`, gpt-5-mini by
++ * default) reads it when one is configured, so the check stays cross-family.
++ */
++function checkersFor(agentId, checkers, { mode, grounded = false }) {
const family = FAMILY[agentId];
const order = mode === 'private' ? ['local'] : (CHECKER_FOR[family] || []);
-- return order.filter((f) => checkers[f]).map((f) => ({ family: f, chat: checkers[f], cross: f !== family }));
++ const pick = (f) => (grounded && f === 'openai' && checkers.openaiGrounded ? checkers.openaiGrounded : checkers[f]);
++ return order.filter((f) => pick(f)).map((f) => ({ family: f, chat: pick(f), cross: f !== family }));
}
const checkerFor = (agentId, checkers, opts) => checkersFor(agentId, checkers, opts)[0] || null;
@@@ -38,7 +45,7 @@ const checkerFor = (agentId, checkers, opts) => checkersFor(agentId, checkers, o
* passes the turn to the next; only when none gives a verdict is the result unavailable (and never delivered).
*/
async function check({ task, answer, sources = [], evidence = [], agentId, mode, checkers, signal }) {
-- const list = checkersFor(agentId, checkers, { mode });
++ const list = checkersFor(agentId, checkers, { mode, grounded: evidence.length > 0 });
if (!list.length) return { ok: false, unavailable: true, reason: 'no model is available to check this answer', cost_usd: 0 };
let cost = 0;
let last = 'no checker gave a verdict';
@@@ -53,11 +60,11 @@ async function check({ task, answer, sources = [], evidence = [], agentId, mode,
room -= line.length;
seen.push(line);
}
-- const tools = seen.length ? `\n\nTool results the agent received (real):\n${seen.join('\n')}` : '';
++ const tools = seen.length ? `\n\n=== Tool results the agent received (real, returned by OpenVibe services) ===\n${seen.join('\n')}\n=== End of tool results ===` : '';
for (const c of list) {
try {
const r = await c.chat({
-- messages: [{ role: 'system', content: SYSTEM }, { role: 'user', content: `Task:\n${String(task).slice(0, 6000)}\n\nAnswer:\n${String(answer).slice(0, 12000)}${cited}${tools}` }],
++ messages: [{ role: 'system', content: SYSTEM }, { role: 'user', content: `=== Task ===\n${String(task).slice(0, 6000)}\n\n=== The agent's answer (prose to the person) ===\n${String(answer).slice(0, 12000)}\n=== End of answer ===${cited}${tools}` }],
json: true, maxTokens: 160, signal,
});
cost += r.cost_usd || 0;
ddiff --git a/server/config.js b/server/config.js
iindex 9e9fa9d..23e5764 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -58,7 +58,7 @@ function load(env = process.env) {
// listed as unavailable, with the reason; nothing is invented. Keys are the operator's.
providers: {
deepseek: { apiKey: env.ACTOR_DEEPSEEK_API_KEY || '', baseUrl: trim(env.ACTOR_DEEPSEEK_BASE_URL || 'https://api.deepseek.com'), model: env.ACTOR_DEEPSEEK_MODEL || 'deepseek-flash' },
-- openai: { apiKey: env.ACTOR_OPENAI_API_KEY || '', baseUrl: trim(env.ACTOR_OPENAI_BASE_URL || 'https://api.openai.com/v1'), model: env.ACTOR_OPENAI_MODEL || 'gpt-5-mini', checkModel: env.ACTOR_OPENAI_CHECK_MODEL || 'gpt-5-nano' },
++ openai: { apiKey: env.ACTOR_OPENAI_API_KEY || '', baseUrl: trim(env.ACTOR_OPENAI_BASE_URL || 'https://api.openai.com/v1'), model: env.ACTOR_OPENAI_MODEL || 'gpt-5-mini', checkModel: env.ACTOR_OPENAI_CHECK_MODEL || 'gpt-5-nano', groundedCheckModel: env.ACTOR_OPENAI_GROUNDED_CHECK_MODEL || 'gpt-5-mini' },
local: { url: trim(env.ACTOR_LOCAL_LLM_URL || ''), model: env.ACTOR_LOCAL_LLM_MODEL || '' },
},
// The OpenVibe services the runtime acts through.
ddiff --git a/test/computers.test.js b/test/computers.test.js
iindex 42814cf..cb2c930 100644
---- a/test/computers.test.js
++++ b/test/computers.test.js
@@@ -122,10 +122,19 @@ const as = { requester: `user:${ME}` };
assert.strictEqual(v.ok, true);
const [system, user] = seen[0];
assert.match(system.content, /person's own computer through OpenVibe\.Bot/);
-- assert.match(user.content, /Tool results the agent received \(real\):\ncomputer_run: \{"exit_code":0,"stdout":"Linux/);
++ assert.match(user.content, /=== Tool results the agent received \(real, returned by OpenVibe services\) ===\ncomputer_run: \{"exit_code":0,"stdout":"Linux/);
++ assert.match(system.content, /never expected to be JSON/);
assert.ok(user.content.length < 6000 + 6000 + 2000, 'the evidence is capped');
await checkAnswer({ task: 't', answer: 'a', agentId: 'openvibe-runtime', mode: 'balanced', checkers: { openai: chat } });
assert.doesNotMatch(seen[1][1].content, /Tool results/, 'no tools, no section');
++ const { checkersFor } = require('../server/agents/check');
++ const strong = async () => ({ text: '{"ok":true}' });
++ const both = { openai: chat, deepseek: chat, openaiGrounded: strong };
++ const grounded = checkersFor('openvibe-runtime', both, { mode: 'balanced', grounded: true });
++ assert.deepStrictEqual(grounded.map((c) => [c.family, c.cross]), [['openai', true], ['deepseek', false]], 'still another family first');
++ assert.strictEqual(grounded[0].chat, strong, 'grounded: OpenAI\'s stronger checker');
++ assert.strictEqual(checkersFor('openvibe-runtime', both, { mode: 'balanced' })[0].chat, chat, 'otherwise the cheapest');
++ assert.deepStrictEqual(checkersFor('openvibe-runtime', { ...both, local: chat }, { mode: 'private', grounded: true }).map((c) => c.family), ['local'], 'private stays on OpenVibe hardware');
});
await check('off without a Bot URL or a client secret', async () => {