openvibers/openvibe.ai Commits

Exchange usage: count only runs a provider actually answered (#320)

OP OpenVibers committed 7 days ago · 15f752883a · 2 files, +8 −4

* Exchange catalog: speed and uptime from the runs that happened

The catalog read provider_stats_daily keyed by the registry's model name, so the resolved
upstream id every run records (gpt-5-nano-2025-08-07) never matched the row (gpt-5-nano):
one stray failure showed 0% uptime and a 25.6 s typical answer while thousands of real
answers sat in runs. It now folds the window's runs and answering attempts itself: uptime
is succeeded/(succeeded+failed), requests are real runs, cached answers follow the run they
reused and are counted apart, and p50/p95 use the receipt's TTFT where recorded else the
answering attempt's total latency. No data stays null, never 0.

The public usage read now comes from the finished runs too, so a product's models are the
provider and model that actually answered, and a cached or failed run is not usage.

* Exchange usage: count only runs a provider actually answered

A workflow that finished without an AI call (provider_key null, zero tokens, zero cost) was
counted as a product request and named a null model. Usage now names only real answers, so
every product model is one that actually answered.

* Merge remote-tracking branch 'origin/main' into ai/exchange-stats-2

# Conflicts:
#	server/exchange/usage.js
#	test/exchange-read.test.js

Changes

ddiff --git a/server/exchange/usage.js b/server/exchange/usage.js
iindex 6e66fb8..bf0f068 100644
---- a/server/exchange/usage.js
++++ b/server/exchange/usage.js
@@@ -14,9 +14,10 @@ const dayOf = (ms) => new Date(ms).toISOString().slice(0, 10);
  
  const NUM_COLUMNS = 'COUNT(*)::bigint AS requests, SUM(tokens_in)::bigint AS tokens_in, SUM(tokens_out)::bigint AS tokens_out, SUM(cost_usd)::double precision AS cost_usd';
  const DAY = "to_char(created_at::timestamptz AT TIME ZONE 'UTC', 'YYYY-MM-DD')";
--// A real answer: the provider was called and answered. A cached run called nobody; a failed one
--// answered nothing.
--const REAL = "status = 'succeeded' AND synthetic = 0";
++// A real answer: a provider was called and answered. A cached run called nobody, a failed one
++// answered nothing, and a workflow that finished without an AI call (provider_key null: zero tokens,
++// zero cost) has no model to attribute.
++const REAL = "status = 'succeeded' AND synthetic = 0 AND provider_key IS NOT NULL";
  
  const shape = (r) => ({
      requests: Number(r.requests) || 0,
ddiff --git a/test/exchange-read.test.js b/test/exchange-read.test.js
iindex 1aaf82b..07cd1f8 100644
---- a/test/exchange-read.test.js
++++ b/test/exchange-read.test.js
@@@ -129,9 +129,11 @@ t.test('usage: day, model and product aggregates from runs, and never an attribu
      for (let i = 0; i < 2; i++) await insertRun({ requester: 'live', provider: 'deepseek', model: 'deepseek-chat', created: at(1), tokensIn: 100, tokensOut: 20, cost: 0.0002 });
      for (let i = 0; i < 3; i++) await insertRun({ requester: 'wiki', provider: 'deepseek', model: 'deepseek-chat', tokensIn: 100, tokensOut: 20, cost: 0.0002 });
      await insertRun({ type: 'app', requester: 'app_ABC', provider: 'deepseek', model: 'deepseek-chat', tokensIn: 100, tokensOut: 20, cost: 0.0002 });
--    // Not usage: a cache hit called no provider, and a failed run answered nothing.
++    // Not usage: a cache hit called no provider, a failed run answered nothing, and a workflow that
++    // finished without an AI call has no model to attribute.
      await insertRun({ status: 'cached', requester: 'live', model: 'gpt-5-nano-2025-08-07', cachedFrom: 'run_source', tokensIn: 9999, tokensOut: 9999, cost: 9 });
      await insertRun({ status: 'failed', requester: 'live', tokensIn: 9999, tokensOut: 9999, cost: 9 });
++    await insertRun({ requester: 'live', tokensIn: 9999, tokensOut: 9999, cost: 9 });
  
      const r = await request(h.base, 'GET', '/api/v1/exchange/usage?days=7');
      assert.strictEqual(r.status, 200, r.text);
@@@ -146,6 +148,7 @@ t.test('usage: day, model and product aggregates from runs, and never an attribu
      assert.strictEqual(live.models.length, 2);
      assert.strictEqual(live.models[0].model, 'gpt-5-nano-2025-08-07', 'the model that actually answered');
      assert.strictEqual(live.models[0].requests, 5);
++    assert.ok(live.models.every((m) => m.model), 'a product names only models that actually answered');
      assert.ok(!r.body.by_product.some((p) => p.product.includes('app_')), 'only services are products');
      assert.ok(!r.text.includes('SECRETPERSON') && !r.text.includes('usr_'), 'no person in the aggregate');
      assert.ok(r.body.by_model.some((m) => m.model === 'gpt-5-nano-2025-08-07' && m.requests === 5));