openvibers/openvibe.compare Commits

Show measured AI speed and reliability beside model prices (#21)

OP OpenVibers committed 5 h ago · d41f0d5440 · 8 files, +106 −18

* Show measured AI speed and reliability beside model prices

* Measured note: link Report's open AI performance data (the AI comparison pages move to Compare)

Changes

ddiff --git a/AGENTS.md b/AGENTS.md
iindex 15769ec..d91aeb2 100644
---- a/AGENTS.md
++++ b/AGENTS.md
@@@ -29,7 +29,7 @@ schema with no API in v1).
## Who it reads
--Six feeds in [server/data.js](server/data.js) `ENDPOINTS`, and the compute book in
++Seven feeds in [server/data.js](server/data.js) `ENDPOINTS`, and the compute book in
[server/compute.js](server/compute.js) `SOURCE`:
| Feed | What |
@@@ -39,6 +39,7 @@ Six feeds in [server/data.js](server/data.js) `ENDPOINTS`, and the compute book
| `bench.openvibe.services/api/v1/runs?kind=codes.patch` / `codes.ticket` | verified Bench runs (OpenVibe.Bench) |
| `openvibe.codes/api/v1/harnesses` | coding harness catalog (OpenVibe.Codes) |
| `billing.openvibe.network/api/v1/plans` | published plans (OpenVibe.Billing) |
++| `openvibe.report/api/v1/ai/performance?group=model&days=30` | measured AI time to first token, tokens per second and success rate, public model rows with a 20-call floor (OpenVibe.Report) |
| `openvibe.exchange/api/compute/prices` | reviewed compute tariffs per provider and region (OpenVibe.Exchange) |
It does not call AI, Live, Network or any authenticated API. Nothing here writes anywhere.
@@@ -48,7 +49,7 @@ It does not call AI, Live, Network or any authenticated API. Nothing here writes
- **Entry:** [server/index.js](server/index.js) — listens on `PORT || 5280`, graceful SIGTERM.
- **App/routes:** [server/app.js](server/app.js) — security headers, static `public/`, `/shared`,
health/ready, `/release.json`, `/api/v1`, pages, 404 and 503.
--- **Data:** [server/data.js](server/data.js) — fetches the five feeds (8 s timeout), builds the three
++- **Data:** [server/data.js](server/data.js) — fetches the seven feeds (8 s timeout; 3 s for Report), builds the three
comparisons, coalesces concurrent reads and caches for ten minutes; a failed feed becomes `null`.
- **Compute:** [server/compute.js](server/compute.js) normalizes Exchange's tariffs and derives the
month (730 h) and GPU GB-hour prices; [server/compute-views.js](server/compute-views.js) renders the
@@@ -98,6 +99,9 @@ file and `test/compute.test.js` plus the repository checks the harness adds.
and observation dates are labelled; no price is estimated.
- Arena quality: at least ten judged games and **exact** model identity (`r.model === id`); no fuzzy
joins, because a fuzzy join silently compares a different model revision.
++- Measured AI columns: Report's public model rows over 30 days, matched by the full thing id or the
++ offer's provider model name. Report groups by model name across providers, so the values describe the
++ model, not one provider's serving; under 20 calls or a failed feed is a dash, never zero.
- Coding agents: only `provenance === 'verified'` runs; benchmark version and model form separate
criteria; newest verified run per harness/cohort.
- Plans: packages and memberships have separate units and are never ranked together.
ddiff --git a/README.md b/README.md
iindex 93eafd1..3d3ce5b 100644
---- a/README.md
++++ b/README.md
@@@ -3,7 +3,7 @@ Source-linked comparisons for anything
Compare v1 serves public comparisons at https://openvibe.compare. Run `npm ci`, then `npm start`; check with `ov test`.
--Data: Exchange price-book market and its reviewed compute tariffs, Bench public arena/verified runs, Codes harness catalog, Billing public plans. Reads are coalesced and cached for ten minutes. Price provenance retains the provider's checked date; a board without an update timestamp labels its date as the observation date. Failed feeds produce explicit unavailable states. There is no private data or paid execution. Compute prices are read from OpenVibe.Exchange's `GET /api/compute/prices`; Compare adds only derived arithmetic (per month at 24/7, per GPU GB-hour), never a copied tariff.
++Data: Exchange price-book market and its reviewed compute tariffs, Report's public AI performance rows (30 days, model group), Bench public arena/verified runs, Codes harness catalog, Billing public plans. Reads are coalesced and cached for ten minutes. Price provenance retains the provider's checked date; a board without an update timestamp labels its date as the observation date. Failed feeds produce explicit unavailable states. The AI models comparison shows Report's measured time to first token p50, tokens per second p50 and success rate beside each provider's price; a model under 20 calls or with Report down is a dash, never zero. There is no private data or paid execution. Compute prices are read from OpenVibe.Exchange's `GET /api/compute/prices`; Compare adds only derived arithmetic (per month at 24/7, per GPU GB-hour), never a copied tariff.
Pages: `/`, `/comparisons/:slug`, `/things/:id` (URI-encoded identity), and `/compute` (filters by kind, GPU model, region and interruptible; sorts by price per hour, per GPU GB-hour and per month at 24/7; a two- or three-offer side-by-side). JSON at `/api/v1/comparisons[/slug]` and `/api/v1/compute`. Pages negotiate `Accept: text/markdown`. Dataset JSON-LD, sitemap and llms.txt are generated from the same data. Desktop tables and phone cards share ranking logic; compute cards run to 899 px and its table starts at 900 px. Weight URL parameters are `w.<criterion>`; all-zero weights leave things unranked. Builder and version-history fields are reserved, with no write API in v1.
ddiff --git a/public/compare.css b/public/compare.css
iindex 67b1f92..6fb11b7 100644
---- a/public/compare.css
++++ b/public/compare.css
@@@ -7,6 +7,8 @@ main button,main summary{cursor:pointer}
main .ovk-card{padding:1rem;margin:1rem 0;border:1px solid currentColor;border-radius:.5rem}
main dl>div{padding:.5rem 0}main dd{margin:.25rem 0}
.desktop-table{display:none}
++.measured-note{margin:1rem 0;max-width:75ch}
++.ovm-unknown{color:var(--text-muted,#8b96ad)}
main table{border-collapse:separate;border-spacing:0;width:100%}
main th,main td{padding:.75rem;text-align:left;border-bottom:1px solid currentColor;vertical-align:top}
main pre{white-space:pre-wrap;overflow-wrap:anywhere}.evidence a{overflow-wrap:anywhere}
ddiff --git a/public/comparison.js b/public/comparison.js
iindex 457202f..4bd17cc 100644
---- a/public/comparison.js
++++ b/public/comparison.js
@@@ -4,6 +4,7 @@
const controls=[...document.querySelectorAll('[data-weight]')];
const input=()=>Object.fromEntries(controls.map(el=>[el.dataset.weight,Number(el.value)]));
const scoreText=n=>n===null?'Unranked':n.toFixed(1);
++ const words=(k,v)=>k&&k.format&&window.OVMetrics?window.OVMetrics.format[k.format](v):null;
let ranked=CompareRank.rank(c,input());
function order(ids){for(const id of ids)for(const parent of [document.getElementById('cards'),document.getElementById('rows')]){const el=[...parent.children].find(e=>e.dataset.thing===id);if(el)parent.append(el);}}
function update(){
@@@ -19,14 +20,14 @@
const ids=c.things.map(t=>t.id).sort((a,b)=>{const x=value(a),y=value(b);if(x==null)return y==null?0:1;if(y==null)return -1;return descending?y-x:x-y;});order(ids);
document.querySelectorAll('thead th').forEach(th=>th.removeAttribute('aria-sort'));b.closest('th').setAttribute('aria-sort',descending?'descending':'ascending');
}));
-- function evidence(r){const d=document.createElement('details'),s=document.createElement('summary'),a=document.createElement('a');s.textContent=String(r.value);a.href=r.source.url;a.textContent=r.source.label;d.append(s,document.createTextNode(`${r.as_of.slice(0,10)} · `),a);return d;}
++ function evidence(r,k){const d=document.createElement('details'),s=document.createElement('summary'),a=document.createElement('a');s.textContent=words(k,r.value)||String(r.value);a.href=r.source.url;a.textContent=r.source.label;d.append(s,document.createTextNode(`${r.as_of.slice(0,10)} · `),a);return d;}
function chart(){
const box=document.getElementById('chart');box.replaceChildren();const mode=document.getElementById('chart-mode').value;if(mode==='none')return;
if(mode==='bars')for(const k of c.criteria){
const h=document.createElement('h3');h.textContent=`${k.label} (${k.unit})`;box.append(h);
const rs=c.results.filter(r=>r.criterion===k.id),max=Math.max(1,...rs.map(r=>Math.abs(r.value)));
if(!rs.length){const p=document.createElement('p');p.textContent='Not measured';box.append(p);}
-- for(const r of rs){const div=document.createElement('div');div.className='chart-row';const meter=document.createElement('meter');meter.min=Math.min(0,...rs.map(r=>r.value));meter.max=max;meter.value=r.value;meter.setAttribute('aria-label',`${c.things.find(t=>t.id===r.thing).name}: ${r.value} ${k.unit}`);div.append(document.createTextNode(c.things.find(t=>t.id===r.thing).name+' '),meter,evidence(r));box.append(div);}
++ for(const r of rs){const div=document.createElement('div');div.className='chart-row';const meter=document.createElement('meter');meter.min=Math.min(0,...rs.map(r=>r.value));meter.max=max;meter.value=r.value;meter.setAttribute('aria-label',`${c.things.find(t=>t.id===r.thing).name}: ${words(k,r.value)||`${r.value} ${k.unit}`}`);div.append(document.createTextNode(c.things.find(t=>t.id===r.thing).name+' '),meter,evidence(r,k));box.append(div);}
}
if(mode==='scatter'){
const points=c.things.map(t=>({t,x:c.results.find(r=>r.thing===t.id&&r.criterion==='output'),y:c.results.find(r=>r.thing===t.id&&r.criterion==='quality')})).filter(p=>p.x&&p.y);
ddiff --git a/server/app.js b/server/app.js
iindex 6697085..837402e 100644
---- a/server/app.js
++++ b/server/app.js
@@@ -48,7 +48,7 @@ function createApp({getData=createData(),getCompute=createCompute()}={}) {
const cs=(await getData()).filter(c=>c.things.some(t=>t.id===req.params.id));if(!cs.length)return res.sendStatus(404);
const t=cs[0].things.find(t=>t.id===req.params.id);res.vary('Accept');
if(req.accepts(['html','text/markdown'])==='text/markdown')return res.type('text/markdown').send(cs.map(c=>v.markdown({...c,things:[t]})).join('\n'));
-- res.send(v.page(t.name,`<h1>${v.esc(t.name)}</h1><p>${v.esc(t.maker)}</p><a href="${v.esc(t.url)}">Original listing</a>${cs.map(c=>`<h2><a href="/comparisons/${c.slug}">${v.esc(c.title)}</a></h2><dl>${c.criteria.map(k=>`<dt>${v.esc(k.label)} (${v.esc(k.unit)})</dt><dd>${v.number(c.results.find(r=>r.thing===t.id && r.criterion===k.id))}</dd>`).join('')}</dl><p>${v.esc(c.method)}</p>`).join('')}`,req.path));
++ res.send(v.page(t.name,`<h1>${v.esc(t.name)}</h1><p>${v.esc(t.maker)}</p><a href="${v.esc(t.url)}">Original listing</a>${cs.map(c=>`<h2><a href="/comparisons/${c.slug}">${v.esc(c.title)}</a></h2><dl>${c.criteria.map(k=>`<dt>${v.esc(k.label)} (${v.esc(k.unit)})</dt><dd>${v.number(c.results.find(r=>r.thing===t.id && r.criterion===k.id),k)}</dd>`).join('')}</dl><p>${v.esc(c.method)}</p>`).join('')}`,req.path));
}));
app.use((_q,r)=>r.status(404).send('Not found'));
app.use((_e,_q,r,_n)=>r.status(503).json({error:'compare.unavailable'}));
ddiff --git a/server/data.js b/server/data.js
iindex 1220551..1f67990 100644
---- a/server/data.js
++++ b/server/data.js
@@@ -6,7 +6,10 @@ const ENDPOINTS = {
tickets: 'https://bench.openvibe.services/api/v1/runs?kind=codes.ticket&limit=100',
harnesses: 'https://openvibe.codes/api/v1/harnesses',
plans: 'https://billing.openvibe.network/api/v1/plans',
++ perf: 'https://openvibe.report/api/v1/ai/performance?group=model&days=30',
};
++// Report is one call of several: a slow answer must not hold the whole page.
++const TIMEOUT_MS = {perf: 3000};
const METHOD = 'Weights use min–max normalization within this comparison (higher or lower as labelled), then a weighted mean on a 0–100 scale. Equal values receive 50. Criteria with no measurements are excluded. A thing missing any selected, measured criterion is unranked; missing is never zero. All-zero weights give no ranking. Scores express these preferences, not statistical significance. Dates are source dates or explicitly labelled observation dates. No costs or benchmark results are estimated. Different benchmark versions are separate criteria. Builder and version history are reserved for a later release.';
const criterion = (id,label,unit,better,weight,group) => ({id,label,unit,better,weight,group});
const date = v => v && Number.isFinite(Date.parse(v)) ? new Date(v).toISOString() : null;
@@@ -17,9 +20,28 @@ function result(c, thing, key, value, at, label, url) {
function base(slug,title,kind,summary,at) {
return {slug,title,kind,summary,things:[],criteria:[],results:[],method:METHOD,sources:[],updated_at:at,versions:[],availability:[]};
}
++// Report's public model rows are grouped by the model's own name (provider/model in Compare's price book), with a
++// floor of 20 calls. The match is exact: the full id first, then the offer's provider model name.
++function measuredRows(feed) {
++ const rows=new Map();
++ for(const r of feed?.rows || []) {
++ if(!r || typeof r.key!=='string' || !r.key || !(Number(r.calls)>=20)) continue;
++ rows.set(r.key,r);
++ }
++ return rows;
++}
++function measuredOf(rows,id,o) {
++ return rows.get(id) || (o.provider_model ? rows.get(o.provider_model) : null) || null;
++}
function build(feeds, at) {
const models=base('ai-models','AI models','ai.models','Provider prices and measured Bench results. Unmeasured quality stays unknown.',at);
-- models.criteria=[criterion('input','Input price','USD / 1M tokens','lower',40,'Price'),criterion('output','Output price','USD / 1M tokens','lower',40,'Price'),criterion('quality','Arena quality','Elo','higher',20,'Quality'),criterion('speed','Median response time','ms','lower',0,'Speed')];
++ models.criteria=[criterion('input','Input price','USD / 1M tokens','lower',40,'Price'),criterion('output','Output price','USD / 1M tokens','lower',40,'Price'),
++ {...criterion('ttft','Time to first token p50','ms','lower',0,'Measured'),format:'ms'},
++ {...criterion('tps','Tokens per second p50','tok/s','higher',0,'Measured'),format:'tps'},
++ {...criterion('success','Success rate','%','higher',0,'Measured'),format:'rate'},
++ criterion('quality','Arena quality','Elo','higher',20,'Quality'),criterion('speed','Median response time','ms','lower',0,'Speed')];
++ const perf=measuredRows(feeds.perf);
++ const perfAt=date(feeds.perf?.range?.to ? `${feeds.perf.range.to}T00:00:00.000Z` : at) || at;
const offers=new Map();
for(const o of feeds.market?.offers || []) {
if(o.kind !== 'chat' || !o.model || o.price?.currency !== 'USD' || o.price?.per !== '1M tokens' || typeof o.price.in !== 'number' || typeof o.price.out !== 'number' || !date(o.price_checked_at) || !/^https:\/\//.test(o.price_source || '')) continue;
@@@ -30,6 +52,14 @@ function build(feeds, at) {
models.things.push({id,name:o.name || id,maker:o.maker || id.split('/')[0],url:o.price_source,icon:'model'});
result(models,id,'input',o.price.in,o.price_checked_at,`${o.provider_name} price book`,o.price_source);
result(models,id,'output',o.price.out,o.price_checked_at,`${o.provider_name} price book`,o.price_source);
++ const m=measuredOf(perf,id,o);
++ if(m) {
++ const seenAt=date(m.range?.to ? `${m.range.to}T00:00:00.000Z` : perfAt);
++ const label=`Report AI performance; ${Number(m.calls).toLocaleString('en-US')} calls in 30 days`;
++ result(models,id,'ttft',m.ttft_ms?.p50,seenAt,label,ENDPOINTS.perf);
++ result(models,id,'tps',m.tps?.p50,seenAt,label,ENDPOINTS.perf);
++ result(models,id,'success',m.success_rate,seenAt,label,ENDPOINTS.perf);
++ }
// Exact identity only: fuzzy joins can silently compare a different model revision.
const q=(feeds.quality?.board?.rows || []).find(r => r.model === id || r.subject === id);
if(q && q.games >= 10) {
@@@ -37,7 +67,7 @@ function build(feeds, at) {
result(models,id,'speed',q.median_ms == null ? null : Number(q.median_ms),q.updated_at || at,'Bench chat.arena median total response time; observation date when absent',ENDPOINTS.quality);
}
}
-- models.method += ' Prices use the cheapest dated offer by input + output price; both prices come from that same provider. Arena quality requires at least ten judged games and exact model identity. Speed is total response time, not tokens per second.';
++ models.method += ' Prices use the cheapest dated offer by input + output price; both prices come from that same provider. Arena quality requires at least ten judged games and exact model identity. Speed is total response time, not tokens per second. The three measured columns are Report\'s public model rows over 30 days, matched by the model’s full id or its provider model name: time to first token p50, output tokens per second after the first token p50, and success rate. Report groups by model name across providers, so they describe the model, not this provider\'s serving of it. A model with fewer than 20 calls in 30 days, or with Report unavailable, shows a dash; nothing is estimated.';
const agents=base('coding-agents','AI coding agents','ai.coding-agents','Coding harnesses from Codes, with verified public Bench runs where available.',at);
const hs=feeds.harnesses?.harnesses || [];
agents.things=hs.map(h => ({id:h.id,name:h.name,maker:h.provider,url:`https://openvibe.codes/harnesses#${encodeURIComponent(h.id)}`,icon:'code'}));
@@@ -62,7 +92,7 @@ function build(feeds, at) {
if(p.kind==='package') { result(plans,p.id,'package-price',p.price_cents/100,p.as_of,label,ENDPOINTS.plans); result(plans,p.id,'vibes',p.vibes,p.as_of,label,ENDPOINTS.plans); }
if(p.kind==='membership') {result(plans,p.id,'membership-price',p.price_cents/100,p.as_of,label,ENDPOINTS.plans);result(plans,p.id,'period',p.period_days,p.as_of,label,ENDPOINTS.plans);}
}
-- for(const [c,names] of [[models,['market','quality']],[agents,['runs','tickets','harnesses']],[plans,['plans']]]) {
++ for(const [c,names] of [[models,['market','quality','perf']],[agents,['runs','tickets','harnesses']],[plans,['plans']]]) {
c.sources=names.map(n=>({label:n,url:ENDPOINTS[n]}));
c.availability=names.filter(n=>!feeds[n]).map(n=>`${n} is unavailable; no values have been filled in.`);
c.updated_at=c.results.length ? c.results.map(r=>r.as_of).sort().at(-1) : at;
@@@ -77,7 +107,7 @@ function createData({fetchImpl=fetch,now=Date.now,ttl=600000}={}) {
pending=(async()=>{
const feeds={};
await Promise.all(Object.entries(ENDPOINTS).map(async([key,url])=>{
-- try {const r=await fetchImpl(url,{signal:AbortSignal.timeout(8000),headers:{Accept:'application/json'}});if(!r.ok) throw Error('upstream');feeds[key]=await r.json();} catch {feeds[key]=null;}
++ try {const r=await fetchImpl(url,{signal:AbortSignal.timeout(TIMEOUT_MS[key]||8000),headers:{Accept:'application/json'}});if(!r.ok) throw Error('upstream');feeds[key]=await r.json();} catch {feeds[key]=null;}
}));
cached=build(feeds,new Date(now()).toISOString()); expires=now()+ttl; return cached;
})().finally(()=>{pending=null;});
ddiff --git a/server/views.js b/server/views.js
iindex 78b2ea2..fe7a4a5 100644
---- a/server/views.js
++++ b/server/views.js
@@@ -2,21 +2,31 @@
const kit=require('openvibe-shared/kit');
const shell=require('openvibe-shared/shell');
const serve=require('openvibe-shared/serve');
++const metrics=require('openvibe-shared/ov-metrics');
const {rank,weights}=require('../public/rank');
const esc=v=>String(v ?? '').replace(/[&<>"']/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
const json=v=>JSON.stringify(v).replace(/</g,'\\u003c');
const thingPath=id=>`/things/${encodeURIComponent(id)}`;
--function number(r) {return r ? `<details class="evidence"><summary>${esc(r.value.toLocaleString('en-US',{maximumFractionDigits:4}))}</summary><time datetime="${esc(r.as_of)}">${esc(r.as_of.slice(0,10))}</time> · <a href="${esc(r.source.url)}">${esc(r.source.label)}</a></details>` : '<span>Not measured</span>';}
++// The words are Shared's, so Compare, Report and Exchange write 240 ms, 45.2 tok/s and 98.0% the same way.
++const FORMAT={ms:metrics.format.ms,tps:metrics.format.tps,rate:metrics.format.rate};
++const word=(k,v)=>k&&FORMAT[k.format]?FORMAT[k.format](v):null;
++function number(r,k) {
++ const f=k&&FORMAT[k.format];
++ if(!r) return f?`<span class="ovm-unknown">${esc(f(null))}</span>`:'<span>Not measured</span>';
++ const text=f?f(r.value):r.value.toLocaleString('en-US',{maximumFractionDigits:4});
++ return `<details class="evidence"><summary>${esc(text)}</summary><time datetime="${esc(r.as_of)}">${esc(r.as_of.slice(0,10))}</time> · <a href="${esc(r.source.url)}">${esc(r.source.label)}</a></details>`;
++}
function score(n,c){return n===null?'Unranked':`<details><summary>${n.toFixed(1)}</summary>Derived from the dated results below, using <a href="#method">this method</a>. Updated ${esc(c.updated_at.slice(0,10))}.</details>`;}
function comparison(c,input={}) {
const ranked=rank(c,input), w=weights(c,input);
-- const cell=(t,k)=>number(c.results.find(r=>r.thing===t.id && r.criterion===k.id));
++ const cell=(t,k)=>number(c.results.find(r=>r.thing===t.id && r.criterion===k.id),k);
return `${kit.head({title:c.title,sub:c.summary,icon:'compare',sticky:false})}${c.availability.map(s=>`<p role="status">${esc(s)}</p>`).join('')}
<p>Updated <time>${esc(c.updated_at.slice(0,10))}</time> · <a href="/api/v1/comparisons/${c.slug}">JSON data</a></p>
-- <fieldset><legend>What matters to you</legend>${c.criteria.map(k=>`<label class="weight">${esc(k.label)} <input type="range" min="0" max="100" value="${w[k.id]}" data-weight="${esc(k.id)}"><output>${w[k.id]}</output></label>`).join('')}</fieldset>
++ <fieldset><legend>What matters to you</legend>${c.criteria.filter(k=>!k.format).map(k=>`<label class="weight">${esc(k.label)} <input type="range" min="0" max="100" value="${w[k.id]}" data-weight="${esc(k.id)}"><output>${w[k.id]}</output></label>`).join('')}</fieldset>
<p id="rank-note">Missing evidence stays unranked. Scores change with your weights; they are not a claim of statistical significance.</p>
-- <div class="phone-cards" id="cards">${ranked.map(({thing:t,score:n})=>`<article class="ovk-card" data-thing="${esc(t.id)}"><h2><a href="${thingPath(t.id)}">${esc(t.name)}</a></h2><p>Overall <span data-score>${score(n,c)}</span></p><dl>${c.criteria.map(k=>`<div><dt>${esc(k.label)} (${esc(k.unit)})</dt><dd>${cell(t,k)}</dd></div>`).join('')}</dl></article>`).join('')}</div>
-- <div class="desktop-table"><table><caption>${esc(c.title)}; select a heading to sort</caption><thead><tr><th scope="col">Thing</th><th scope="col"><button data-sort="overall">Overall</button></th>${c.criteria.map(k=>`<th scope="col"><button data-sort="${esc(k.id)}">${esc(k.label)} (${esc(k.unit)})</button></th>`).join('')}</tr></thead><tbody id="rows">${ranked.map(({thing:t,score:n})=>`<tr data-thing="${esc(t.id)}"><th scope="row"><a href="${thingPath(t.id)}">${esc(t.name)}</a></th><td data-score>${score(n,c)}</td>${c.criteria.map(k=>`<td>${cell(t,k)}</td>`).join('')}</tr>`).join('')}</tbody></table></div>
++ <div class="phone-cards" id="cards">${ranked.map(({thing:t,score:n})=>`<article class="ovk-card" data-thing="${esc(t.id)}"><h2><a href="${thingPath(t.id)}">${esc(t.name)}</a></h2><p>Overall <span data-score>${score(n,c)}</span></p><dl>${c.criteria.map(k=>`<div><dt>${esc(k.label)} (${esc(k.unit)})</dt><dd data-criterion="${esc(k.id)}">${cell(t,k)}</dd></div>`).join('')}</dl></article>`).join('')}</div>
++ <div class="desktop-table"><table><caption>${esc(c.title)}; select a heading to sort</caption><thead><tr><th scope="col">Thing</th><th scope="col"><button data-sort="overall">Overall</button></th>${c.criteria.map(k=>`<th scope="col"><button data-sort="${esc(k.id)}">${esc(k.label)} (${esc(k.unit)})</button></th>`).join('')}</tr></thead><tbody id="rows">${ranked.map(({thing:t,score:n})=>`<tr data-thing="${esc(t.id)}"><th scope="row"><a href="${thingPath(t.id)}">${esc(t.name)}</a></th><td data-score>${score(n,c)}</td>${c.criteria.map(k=>`<td data-criterion="${esc(k.id)}">${cell(t,k)}</td>`).join('')}</tr>`).join('')}</tbody></table></div>
++ ${c.kind==='ai.models'?'<p class="measured-note">Measured on OpenVibe.AI\'s real traffic; a model shows after 20 calls in 30 days. The numbers come from OpenVibe.Report&rsquo;s <a href="https://openvibe.report/developers">open AI performance data</a>.</p>':''}
<label>Chart <select id="chart-mode"><option value="none">Hidden</option><option value="bars">Bars by criterion</option>${c.kind==='ai.models'?'<option value="scatter">Price against quality</option>':''}</select></label><div id="chart" aria-live="polite"></div>
<section id="method"><h2>Method</h2><p>${esc(c.method)}</p><h3>Sources</h3><ul>${c.sources.map(s=>`<li><a href="${esc(s.url)}">${esc(s.label)}</a></li>`).join('')}</ul></section>
<script type="application/json" id="comparison-data">${json(c)}</script><script src="/rank.js" defer></script><script src="/comparison.js" defer></script>`;
@@@ -39,6 +49,6 @@ function home(all,compute,q='',task='balanced',budget='') {
<form><label>Search comparisons <input name="q" value="${esc(q)}" type="search"></label><button class="ovk-btn ovk-btn--quiet">Search</button></form>
<div class="comparison-list">${computeCard(compute,q)}${all.filter(c=>(c.title+' '+c.summary).toLowerCase().includes(q.toLowerCase())).map(c=>`<article class="ovk-card"><h2><a href="/comparisons/${c.slug}">${esc(c.title)}</a></h2><p>${esc(c.summary)}</p>${rank(c).filter(r=>r.score!==null).slice(0,3).map(r=>`<div>${esc(r.thing.name)} <meter min="0" max="100" value="${r.score}">${r.score.toFixed(1)}</meter> ${score(r.score,c)}</div>`).join('') || '<p>No comparable measurements yet.</p>'}<p>${c.things.length} things · ${c.results.length} dated results</p></article>`).join('')}</div><section id="method"><h2>How picks are ranked</h2><p>${esc(c?.method || '')}</p></section>`;
}
--function markdown(c){return `# ${c.title}\n\n${c.summary}\n\nUpdated: ${c.updated_at}\n\n`+c.things.map(t=>`## ${t.name}\n\n`+c.criteria.map(k=>{const r=c.results.find(r=>r.thing===t.id&&r.criterion===k.id);return `- ${k.label}: ${r?`${r.value} ${k.unit} — ${r.as_of} — [${r.source.label}](${r.source.url})`:'Not measured'}`;}).join('\n')).join('\n\n')+`\n\n## Method\n\n${c.method}\n`;}
--function page(title,body,path='/',ld=[]) {return shell.page({name:'OpenVibe.Compare',siteName:'OpenVibe.Compare',title,description:'Compare things with dated sources, transparent methods and your own priorities.',canonical:`https://openvibe.compare${path}`,robots:'index, follow',jsonLd:ld,navbar:{service:'compare',apiBase:'https://openvibe.network',links:[{label:'Comparisons',href:'/'},{label:'Compute',href:'/compute'}]},footer:{service:'compare',variant:'full',mount:'#ov-footer'},head:`<link rel="stylesheet" href="${serve.url('kit.css')}"><link rel="stylesheet" href="/compare.css"><script src="/compare-account.js" defer></script>`,body:`<div id="navbar-mount"></div><main id="main" class="ovk-container">${body}</main>`});}
++function markdown(c){return `# ${c.title}\n\n${c.summary}\n\nUpdated: ${c.updated_at}\n\n`+c.things.map(t=>`## ${t.name}\n\n`+c.criteria.map(k=>{const r=c.results.find(r=>r.thing===t.id&&r.criterion===k.id);if(!r)return `- ${k.label}: ${word(k,null)||'Not measured'}`;const shown=word(k,r.value)||`${r.value} ${k.unit}`;return `- ${k.label}: ${shown} — ${r.as_of} — [${r.source.label}](${r.source.url})`;}).join('\n')).join('\n\n')+`\n\n## Method\n\n${c.method}\n`;}
++function page(title,body,path='/',ld=[]) {return shell.page({name:'OpenVibe.Compare',siteName:'OpenVibe.Compare',title,description:'Compare things with dated sources, transparent methods and your own priorities.',canonical:`https://openvibe.compare${path}`,robots:'index, follow',jsonLd:ld,navbar:{service:'compare',apiBase:'https://openvibe.network',links:[{label:'Comparisons',href:'/'},{label:'Compute',href:'/compute'}]},footer:{service:'compare',variant:'full',mount:'#ov-footer'},head:`<link rel="stylesheet" href="${serve.url('kit.css')}"><link rel="stylesheet" href="/compare.css"><script src="${serve.url('ov-metrics.js')}" defer></script><script src="/compare-account.js" defer></script>`,body:`<div id="navbar-mount"></div><main id="main" class="ovk-container">${body}</main>`});}
module.exports={comparison,home,page,markdown,esc,thingPath,number,computeCard};
ddiff --git a/test/compare.test.js b/test/compare.test.js
iindex 3891aa0..9b969cc 100644
---- a/test/compare.test.js
++++ b/test/compare.test.js
@@@ -3,7 +3,7 @@ const test=require('node:test'),assert=require('node:assert/strict');
const {build,createData,ENDPOINTS}=require('../server/data');
const {rank}=require('../public/rank');
const at='2026-11-10T00:00:00.000Z';
--const fixture={market:{offers:[1,2,3].map(n=>({model:`maker/m${n}`,name:`Model ${n}`,kind:'chat',price:{in:n,out:n*2,currency:'USD',per:'1M tokens'},price_source:'https://example.com/prices',price_checked_at:at}))},harnesses:{harnesses:[{id:'codex',name:'Codex',provider:'openai'}]},runs:{runs:[]},plans:{plans:[]},quality:{board:{rows:[]}}};
++const fixture={market:{offers:[1,2,3].map(n=>({model:`maker/m${n}`,provider_model:`m${n}`,name:`Model ${n}`,kind:'chat',price:{in:n,out:n*2,currency:'USD',per:'1M tokens'},price_source:'https://example.com/prices',price_checked_at:at}))},harnesses:{harnesses:[{id:'codex',name:'Codex',provider:'openai'}]},runs:{runs:[]},plans:{plans:[]},quality:{board:{rows:[]}}};
const data=build(fixture,at);
test('every measurement is finite, dated, sourced and references a known thing and criterion',()=>{
for(const c of data)for(const r of c.results){assert(Number.isFinite(r.value));assert(Number.isFinite(Date.parse(r.as_of)));assert.match(r.source.url,/^https:/);assert(r.source.label);assert(c.things.some(t=>t.id===r.thing));assert(c.criteria.some(k=>k.id===r.criterion));}
@@@ -41,3 +41,44 @@ test('upstream adapters preserve cohorts, observation dates and unknown quality'
f.plans.plans=[{id:'vibes-100',name:'100 Vibes',kind:'package',vibes:100,price_cents:150,as_of:at,url:'https://openvibe.network/vibes'},{id:'membership',name:'Membership',kind:'membership',price_cents:499,period_days:30,as_of:at,url:'https://openvibe.vip'}];
const [models,agents,plans]=build(f,at);assert(!models.results.some(r=>r.criterion==='quality'));assert.equal(models.results.find(r=>r.criterion==='speed').value,12);assert.equal(agents.results.length,1);assert.equal(agents.results[0].value,70);assert.equal(agents.results[0].as_of,at);assert.match(agents.results[0].source.label,/1.2/);assert.equal(plans.results.find(r=>r.criterion==='package-price').value,1.5);assert.equal(plans.results.find(r=>r.criterion==='membership-price').value,4.99);
});
++
++const measuredFeed={group:'model',range:{from:'2026-10-15',to:'2026-11-13'},rows:[
++ {group:'model',key:'m1',calls:30,success_rate:0.981,ttft_ms:{p50:240,p90:500,p95:800,n:30},tps:{p50:45.2,p90:70,p95:90,n:30}},
++ {group:'model',key:'m2',calls:19,success_rate:0.5,ttft_ms:{p50:100,p90:200,p95:300,n:19},tps:{p50:10,p90:20,p95:30,n:19}}
++]};
++test('measured AI columns come from Report public model rows with Shared words and a dash when absent',()=>{
++ const f=structuredClone(fixture);f.perf=measuredFeed;
++ const [models]=build(f,at);
++ const get=(id,k)=>models.results.find(r=>r.thing===id&&r.criterion===k);
++ assert.equal(get('maker/m1','ttft').value,240);assert.equal(get('maker/m1','tps').value,45.2);assert.equal(get('maker/m1','success').value,0.981);
++ assert.equal(get('maker/m1','ttft').source.url,ENDPOINTS.perf);assert.equal(get('maker/m1','ttft').as_of,'2026-11-13T00:00:00.000Z');assert.match(get('maker/m1','ttft').source.label,/30 calls in 30 days/);
++ assert.equal(get('maker/m2','ttft'),undefined,'19 calls is under the public floor');
++ assert.equal(models.results.filter(r=>r.criterion==='ttft').length,1);
++ assert.deepEqual(models.criteria.slice(0,5).map(c=>c.id),['input','output','ttft','tps','success']);
++ assert.deepEqual(models.criteria.slice(2,5).map(c=>c.format),['ms','tps','rate']);
++ const html=require('../server/views').comparison(models);
++ assert.match(html,/240 ms/);assert.match(html,/45\.2 tok\/s/);assert.match(html,/98\.1%/);
++ assert.match(html,/<td data-criterion="ttft"><span class="ovm-unknown">—<\/span><\/td>/);assert.match(html,/data-sort="ttft"/);
++ assert.match(html,/Measured on OpenVibe\.AI's real traffic; a model shows after 20 calls in 30 days/);
++ assert.match(html,/https:\/\/openvibe\.report\/developers/);
++ const md=require('../server/views').markdown(models);
++ assert.match(md,/- Time to first token p50: 240 ms — 2026-11-13/);assert.match(md,/- Time to first token p50: —/);
++});
++test('Report down leaves measured dashes and the page answers 200',async()=>{
++ const {createApp}=require('../server/app');
++ const down=createData({fetchImpl:async url=>{
++ if(url===ENDPOINTS.perf) throw Error('offline');
++ const key=Object.entries(ENDPOINTS).find(([,v])=>v===url)[0];
++ return {ok:true,json:async()=>fixture[key]};
++ }});
++ const server=createApp({getData:down}).listen(0,'127.0.0.1');await new Promise(r=>server.once('listening',r));
++ const base=`http://127.0.0.1:${server.address().port}`;
++ try {
++ const r=await fetch(base+'/comparisons/ai-models');assert.equal(r.status,200);
++ const html=await r.text();
++ assert.match(html,/<td data-criterion="ttft"><span class="ovm-unknown">—<\/span><\/td>/);
++ assert.match(html,/<td data-criterion="tps"><span class="ovm-unknown">—<\/span><\/td>/);
++ assert.match(html,/<td data-criterion="success"><span class="ovm-unknown">—<\/span><\/td>/);
++ assert.match(html,/perf is unavailable; no values have been filled in/);
++ }finally{await new Promise(r=>server.close(r));}
++});