openvibers/openvibe.ai Commits

media.analyze, local-first media analysis (roadmap WS-O task 5). A new step kind, media_analysis (server/workflows/media-analysis.js), behind the workflow media.analyze (server/workflows/media.js, namespace media; Live may run it through the default AI_NS_FALLBACK). A VOD, clip or audio file is fetched like any media input: allow-listed hosts only, into media-tmp beside the database (never /tmp, which is memory on the production host), capped at 4 GB and at that disk's free space less a 3 GB reserve, refused with media.no_space before downloading. Everything runs on this host, one analysis at a time, at nice 15. (1) Signals come from one FFmpeg pass, video decoded at keyframes only: scene changes (scdet), black and frozen picture, silence, and loudness (integrated, range, the typical level, i.e. the median audible second, since one burst drags the gated integrated level up, and loud moments at least 6 LU above it). (2) Scenes come from the changes. (3) Speech is whisper.cpp through live.stt. (4) Highlights are windows scored by loud, scene changes and speech, each with its reasons and transcript excerpt. (5) The overview comes from a local model (new provider `local`, any OpenAI-compatible server on this host, AI_LOCAL_LLM_URL, route media.local), else it is extractive from the measured facts. A paid provider (route media.paid) writes it only with allow_paid and an active quota with workflow_prefix media.paid and a max_cost_usd while today's media.* spend is under it; that quota gates only this call (quotas.paidBudget), never runs. overview.source names the writer, gaps list what is missing, and the output is not cached. test/media-analysis.test.js: the log parser; a real run on a 28 s fixture (cuts, black, still picture, silence, a loud burst) with a fake local model that sees only the facts; the local model down -> extractive and said; paid refused without allow_paid, without a budget and past it, used within it; too little disk refused before any download. workflows.test runs it on the fixture too (both skip the runs without ffmpeg). npm test 19/19.

OP OpenVibers committed 27 Sept 2026 · 67fb2a4125 · 15 files, +698 −17


Changes

ddiff --git a/.env.example b/.env.example
iindex 42dfa00..27eb021 100644
---- a/.env.example
++++ b/.env.example
@@@ -70,6 +70,19 @@ AI_MODEL=
#AI_FALLBACK_MODEL=
# Optional local HTTP seam provider (POST {operation, request} -> {text, json, usage})
#AI_HTTP_SEAM_URL=http://127.0.0.1:4799/complete
++# A local model server speaking the OpenAI API (llama.cpp llama-server, Ollama /v1): provider `local`, route
++# media.local, used by media.analyze's overview. No key; never a paid provider.
++#AI_LOCAL_LLM_URL=http://127.0.0.1:8090/v1
++#AI_LOCAL_LLM_MODEL=qwen2.5-1.5b-instruct
++#AI_LOCAL_LLM_TIMEOUT_MS=180000
++# media.analyze: file cap (also bounded by free temp disk minus the reserve), download timeout, scene threshold
++#AI_MEDIA_ANALYSIS_DIR=/var/lib/openvibe-ai/media-tmp (default: media-tmp beside AI_DB_PATH; never a tmpfs)
++#AI_MEDIA_ANALYSIS_MAX_BYTES=4294967296
++#AI_MEDIA_ANALYSIS_DISK_RESERVE_BYTES=3221225472
++#AI_MEDIA_ANALYSIS_FETCH_TIMEOUT_MS=1200000
++#AI_MEDIA_SCENE_THRESHOLD=10
++#FFMPEG_BIN=ffmpeg
++#FFPROBE_BIN=ffprobe
# Let routes fall back to the deterministic stub when every real provider fails. Default: on outside
# production, off in production (synthetic output must never stand in for a real answer there).
#AI_STUB_FALLBACK=false
ddiff --git a/README.md b/README.md
iindex 8f179e4..e2ef503 100644
---- a/README.md
++++ b/README.md
@@@ -158,6 +158,7 @@ definition, but never over an admin's version.
| `anthropic` | Messages API with cached system blocks and forced-tool JSON. Ported from Live's `llm.js`. | same, with `AI_PROVIDER=anthropic` |
| `http` | The local HTTP seam: `POST {operation, …}` → `{text, json, usage}` | `AI_HTTP_SEAM_URL` or the admin API |
| `whisper` | whisper.cpp on this host, with Live's VAD, hallucination filter, multilingual model and live/batch lanes | `WHISPER_*` — Live's names |
++| `openai`, key `local` | A model server on this host speaking the OpenAI API (llama.cpp's `llama-server`, Ollama's `/v1`): no key, never paid; route `media.local` | `AI_LOCAL_LLM_URL`, `AI_LOCAL_LLM_MODEL`, `AI_LOCAL_LLM_TIMEOUT_MS` |
Routing tries the route's primary, then each fallback; a provider that is disabled, missing
credentials, circuit-open or unable to do the operation is **skipped and logged**; every call has a
@@@ -181,6 +182,25 @@ stub joins every route as a last resort outside production only (`AI_STUB_FALLBA
(prompts moved verbatim), plus passthrough workflows for the features whose prompts Live still
renders (`live.viewers.*`, `live.chat.insight`, `live.moments.*`, `live.hero.slogans`,
`live.arena.*`, …). See `docs/migration.md`.
++- **Media** — `media.analyze` (roadmap WS-O task 5), local-first analysis of a VOD, clip or audio file by `media_url`
++ or MediaRef, fetched like any media input (allow-listed hosts) into `media-tmp` beside the database
++ (`AI_MEDIA_ANALYSIS_DIR`; never `/tmp`, which is memory on the production host). The file cap is
++ `AI_MEDIA_ANALYSIS_MAX_BYTES` (4 GB), also bounded by that disk's free space minus
++ `AI_MEDIA_ANALYSIS_DISK_RESERVE_BYTES` (3 GB). Otherwise the run is refused with `media.no_space` before downloading. Everything runs on this host, one analysis at a time, at nice 15:
++ 1. **Signals**, one FFmpeg pass with video decoded at keyframes only: scene changes (`scdet`, threshold
++ `AI_MEDIA_SCENE_THRESHOLD`), black and frozen picture, silence, and loudness. Loudness has the integrated level
++ and range, the typical level (the median audible second) and loud moments at least 6 LU above it.
++ 2. **Scenes** from the scene changes (at most 300).
++ 3. **Speech**: whisper.cpp through `live.stt`.
++ 4. **Highlights**: windows scored by loud, scene changes and speech, each with its reasons and transcript excerpt.
++ 5. **An overview** from the local model (`media.local`), else extractive from the measured facts. A paid provider
++ (`media.paid`, the shared provider) writes it only with `allow_paid: true` **and** an active quota with
++ `workflow_prefix` `media.paid` and `max_cost_usd` > 0, while today's `media.*` spend is under it. That quota
++ gates only this call, never whole runs.
++
++ `overview.source` says which (`local_model`, `paid`, `extractive`), and `gaps` lists what is missing (no local
++ model, no speech-to-text, no budget). The output is not cached: callers keep it. Live may run it by the default
++ `AI_NS_FALLBACK`.
- **Direct operations** — `ai.chat|generate|summarize|classify|extract|enrich|embed`.
## Guarantees and where they are tested
@@@ -199,6 +219,7 @@ stub joins every route as a last resort outside production only (`AI_STUB_FALLBA
| Token and capability denial, namespaces, no secret values in any response, runs private to the requester | `test/auth.test.js` |
| Operator console: SSO + PKCE, staff-only (anonymous → sign-in, non-staff 403), the staff → AI capability map, CSRF on every write, no script/secret on any page, noindex/robots, failed-run filters, audit rows from API and console changes | `test/console.test.js` |
| Ported adapters against fake OpenAI/Anthropic servers, stub determinism, whisper filter, templates | `test/providers.test.js` |
++| `media.analyze` on a real 28 s fixture: signals, scenes and loud moments measured, highlights with evidence, the local model's overview from the facts only; the local model down → extractive and said; paid only with `allow_paid` and a `media.paid` budget, never past it; too little disk refused before downloading | `test/media-analysis.test.js` |
| Import from a Live snapshot: dry run, holds, idempotent re-run | `test/import.test.js` |
| Capability proposals are valid contracts documents matching what is enforced | `test/proposals.test.js` |
@@@ -228,7 +249,8 @@ Deployed: `/opt/openvibe.ai`, env `/etc/openvibe/ai.env` (0600), unit `deploy/sy
## Not done yet
- `ai.run.*` events to OpenVibe.Events (and their schemas in Contracts); per-actor (BYO) provider
-- secrets; moving Live's local transcription (whisper) and the passthrough prompts into AI templates;
++ secrets; Live's VOD and clip overviews calling `media.analyze` instead of extracting frames and transcribing
++ themselves; moving the passthrough prompts into AI templates;
removing Network's fallback to Live's `/internal/ai/site-copy`; a public server-rendered status page.
- The operator console is built but not reachable yet: it needs the vhost below installed (which
replaces the Sites placeholder), `https://ai.openvibe.network/auth/callback` listed as a redirect URI
ddiff --git a/server/config.js b/server/config.js
iindex 6000b36..3f4ae57 100644
---- a/server/config.js
++++ b/server/config.js
@@@ -28,9 +28,10 @@ const trimUrl = (v) => String(v || '').trim().replace(/\/+$/, '');
* AI_NS_FALLBACK: namespaces for a service token that carries no `ns` claim, as
* `service=ns|ns,service=ns`. A service not listed gets `<service>.*`; `none` turns the fallback off
* (an ns-less token then runs nothing). The default covers Live, whose Network grant has no
-- * namespaces: its own workflows, plus network.site_copy for its /internal/ai/site-copy fallback.
++ * namespaces: its own workflows, plus network.site_copy for its /internal/ai/site-copy fallback and
++ * media.analyze for its VOD and clip overviews.
*/
--const DEFAULT_NS_FALLBACK = 'live=live.*|network.site_copy';
++const DEFAULT_NS_FALLBACK = 'live=live.*|network.site_copy|media.analyze';
function nsFallback(v) {
const raw = v == null || v === '' ? DEFAULT_NS_FALLBACK : String(v).trim();
if (/^(none|off|0|false)$/i.test(raw)) return { derive: false, fallback: {} };
@@@ -118,6 +119,21 @@ function load(env = process.env) {
},
// Configurable local HTTP seam: POST {operation, request} JSON to this URL.
httpSeamUrl: trimUrl(env.AI_HTTP_SEAM_URL),
++ // A local model server speaking the OpenAI API (llama.cpp's llama-server, Ollama's /v1): the provider `local`,
++ // route media.local (roadmap WS-O task 5). No key; never a paid provider.
++ localLlm: { url: trimUrl(env.AI_LOCAL_LLM_URL), model: String(env.AI_LOCAL_LLM_MODEL || '').trim(), timeoutMs: int(env.AI_LOCAL_LLM_TIMEOUT_MS, 180000) },
++ // media.analyze (server/workflows/media-analysis.js): the file cap (also bounded by free disk minus a reserve),
++ // the download timeout, the scene-change threshold (scdet, 0-100) and the tools.
++ mediaAnalysis: {
++ // On disk beside the database, not os.tmpdir(): a tmpfs /tmp is memory, and a VOD is gigabytes.
++ workDir: env.AI_MEDIA_ANALYSIS_DIR || require('path').join(require('path').dirname(env.AI_DB_PATH || './data/ai.db'), 'media-tmp'),
++ maxBytes: int(env.AI_MEDIA_ANALYSIS_MAX_BYTES, 4 * 1024 * 1024 * 1024),
++ reserveBytes: int(env.AI_MEDIA_ANALYSIS_DISK_RESERVE_BYTES, 3 * 1024 * 1024 * 1024),
++ fetchTimeoutMs: int(env.AI_MEDIA_ANALYSIS_FETCH_TIMEOUT_MS, 20 * 60000),
++ sceneThreshold: float(env.AI_MEDIA_SCENE_THRESHOLD, 10),
++ ffmpeg: env.FFMPEG_BIN || 'ffmpeg',
++ ffprobe: env.FFPROBE_BIN || 'ffprobe',
++ },
// Let routes fall back to the deterministic stub when every real provider fails.
// Off in production: synthetic output must never stand in for a real answer there.
stubFallback: bool(env.AI_STUB_FALLBACK, !isProduction),
ddiff --git a/server/fetcher.js b/server/fetcher.js
iindex ac7acd7..f72e727 100644
---- a/server/fetcher.js
++++ b/server/fetcher.js
@@@ -197,12 +197,12 @@ function createFetcher(config, { transport = null } = {}) {
}
/** Media input -> a temp file path (caller deletes it). */
-- async function loadMediaToFile(input, { signal } = {}) {
++ async function loadMediaToFile(input, { signal, maxBytes, timeoutMs, dir } = {}) {
const url = input.media_url || (input.media ? mediaRefUrl(input.media) : null);
if (!url) throw new AiError(422, 'input.invalid', 'media needs media_url or media (MediaRef)');
-- const file = path.join(os.tmpdir(), `openvibe-ai-media-${Date.now()}-${Math.floor(Math.random() * 1e6)}`);
++ const file = path.join(dir || os.tmpdir(), `openvibe-ai-media-${Date.now()}-${Math.floor(Math.random() * 1e6)}`);
try {
-- await fetchUrl(url, { signal, toFile: file });
++ await fetchUrl(url, { signal, toFile: file, ...(maxBytes ? { maxBytes } : {}), ...(timeoutMs ? { timeoutMs } : {}) });
} catch (e) {
try { fs.unlinkSync(file); } catch { /* */ }
throw e;
ddiff --git a/server/index.js b/server/index.js
iindex a3f30ca..be76daa 100644
---- a/server/index.js
++++ b/server/index.js
@@@ -30,7 +30,7 @@ async function start({ config, clock = { now: () => Date.now() }, fetchImpl = gl
const cache = createCache(db, { clock });
const pool = createProviderPool({ db, registry, config, clock, fetchImpl, env, log });
const fetcher = createFetcher(config);
-- const engine = createEngine({ registry, pool, fetcher });
++ const engine = createEngine({ registry, pool, fetcher, quotas, config, log });
// ai.run.* to OpenVibe.Events through the outbox (server/events.js); before recovery, so its failures are announced.
require('./events').init(db, { log });
// ai.preferences (read) and ai.usage_summary (written): Network user modules (server/user-modules.js).
ddiff --git a/server/quota.js b/server/quota.js
iindex 367d1db..87ad514 100644
---- a/server/quota.js
++++ b/server/quota.js
@@@ -155,7 +155,21 @@ function createQuotas(db, { clock = { now: () => Date.now() }, registry } = {})
});
}
-- return { reserve, account, list, upsert, usage, counters, WINDOWS };
++ /**
++ * A cost budget that gates one kind of paid call rather than whole runs (media.analyze's paid overview, WS-O task 5):
++ * the active quotas whose workflow_prefix is `prefix` (a name no workflow key starts with, so reserve() never
++ * applies them) and whose max_cost_usd is above 0. The tightest cap counts, against today's (UTC) spend of the
++ * workflows starting with `spendPrefix`. null when there is none: no budget, no paid call.
++ */
++ function paidBudget(prefix, spendPrefix) {
++ const rows = db.prepare("SELECT max_cost_usd FROM quotas WHERE status = 'active' AND workflow_prefix = ? AND max_cost_usd > 0").all(prefix);
++ if (!rows.length) return null;
++ const day = new Date(clock.now()).toISOString().slice(0, 10);
++ const spent = db.prepare("SELECT COALESCE(SUM(cost_usd), 0) AS c FROM usage_daily WHERE day = ? AND substr(workflow_key, 1, ?) = ?").get(day, spendPrefix.length, spendPrefix).c;
++ return { max_cost_usd: Math.min(...rows.map((r) => r.max_cost_usd)), spent, window: 'day' };
++ }
++
++ return { reserve, account, list, upsert, usage, counters, paidBudget, WINDOWS };
}
module.exports = { createQuotas, WINDOWS };
ddiff --git a/server/registry.js b/server/registry.js
iindex 7cf26f3..48c72c6 100644
---- a/server/registry.js
++++ b/server/registry.js
@@@ -292,8 +292,8 @@ function createRegistry(db, { clock = { now: () => Date.now() }, env = process.e
schemas.assertSchema(w.output_schema, 'output_schema');
if (!Array.isArray(w.steps) || !w.steps.length) throw new AiError(422, 'ai.invalid', 'workflow needs at least one step');
for (const s of w.steps) {
-- if (!s || !['llm', 'passthrough', 'transcribe', 'embed'].includes(s.kind)) throw new AiError(422, 'ai.invalid', 'step kind must be llm, passthrough, transcribe or embed');
-- if (s.kind === 'llm' && !(s.template && latestAny('templates', s.template))) throw new AiError(422, 'ai.invalid', `step references unknown template ${s.template}`);
++ if (!s || !['llm', 'passthrough', 'transcribe', 'embed', 'media_analysis'].includes(s.kind)) throw new AiError(422, 'ai.invalid', 'step kind must be llm, passthrough, transcribe, embed or media_analysis');
++ if ((s.kind === 'llm' || (s.kind === 'media_analysis' && s.template)) && !(s.template && latestAny('templates', s.template))) throw new AiError(422, 'ai.invalid', `step references unknown template ${s.template}`);
}
if (!['none', 'private', 'service'].includes(w.cache_mode)) throw new AiError(422, 'ai.invalid', 'cache_mode must be none, private or service');
if (!LIFECYCLE.includes(w.status)) throw new AiError(422, 'ai.invalid', `workflow status must be one of ${LIFECYCLE.join(', ')}`);
ddiff --git a/server/workflows/engine.js b/server/workflows/engine.js
iindex d408eac..a499839 100644
---- a/server/workflows/engine.js
++++ b/server/workflows/engine.js
@@@ -10,6 +10,8 @@
* routed by role (route_prefix.role) or a fixed route
* transcribe fetch the media (allow-listed hosts only) and run speech-to-text
* embed embeddings
++ * media_analysis local-first media analysis: FFmpeg signals, scenes, local speech-to-text, highlights and
++ * an overview from a local model (or extractive; paid only with a budget). See media-analysis.js.
*
* The final output is validated against the workflow's output schema. An empty or invalid answer
* fails the run (output.empty / output.invalid); nothing is ever filled in. input.sources become
@@@ -94,7 +96,8 @@ function normMessages(messages, user) {
return out;
}
--function createEngine({ registry, pool, fetcher }) {
++function createEngine({ registry, pool, fetcher, quotas = null, config = {}, spawnImpl, log = console }) {
++ const mediaAnalysis = require('./media-analysis').createMediaAnalysis({ registry, pool, fetcher, quotas, config, spawnImpl, log });
function routeFor(key) {
const r = registry.resolveRoute(key);
if (!r) throw new AiError(503, 'route.unavailable', `no route ${key}`);
@@@ -202,7 +205,7 @@ function createEngine({ registry, pool, fetcher }) {
return { output: { vectors, dimensions: vectors[0] ? vectors[0].length : 0 }, exec, route };
}
-- const STEP = { llm: llmStep, passthrough: passthroughStep, transcribe: transcribeStep, embed: embedStep };
++ const STEP = { llm: llmStep, passthrough: passthroughStep, transcribe: transcribeStep, embed: embedStep, media_analysis: mediaAnalysis.step };
/**
* Run a workflow. ctx: { signal, logRequest, debugRaw, inputHash }
ddiff --git a/server/workflows/media-analysis.js b/server/workflows/media-analysis.js
nnew file mode 100644
iindex 0000000..84c8fae
---- /dev/null
++++ b/server/workflows/media-analysis.js
@@@ -0,0 +1,359 @@
++'use strict';
++/**
++ * Local-first media analysis (roadmap WS-O task 5): the `media_analysis` step behind the workflow media.analyze.
++ *
++ * 1. Signals: one FFmpeg pass over the fetched file (nice 15; video decoded at keyframes only and scaled small):
++ * scene changes (scdet), black and frozen picture (blackdetect, freezedetect), silence (silencedetect) and
++ * loudness (ebur128: the programme's integrated level and range, the momentary level per second).
++ * 2. Scenes: the scene changes, as scenes with a start and an end.
++ * 3. Speech: whisper.cpp on this host (route live.stt), when it is installed.
++ * 4. Highlights: 30-second windows scored by loudness above the programme's typical level (the median second: one
++ * loud burst would drag the gated integrated level up with it), scene changes and speech.
++ * Each carries its reasons and its transcript excerpt: nothing is asserted without the evidence for it.
++ * 5. Overview: a local model (route media.local, AI_LOCAL_LLM_URL) when one is configured, else extractive (the
++ * signals and the transcript, no model). A paid provider (route media.paid) only when the caller allows it
++ * (allow_paid) AND a quota gives it a daily cost budget (workflow_prefix `media.paid`, max_cost_usd > 0) that
++ * today's media.* spend has not used up. The output says which one wrote the overview, and lists what is missing.
++ *
++ * One analysis runs at a time on this host; the others wait their turn.
++ */
++const fs = require('fs');
++const path = require('path');
++const { spawn } = require('child_process');
++const { AiError } = require('../util');
++const { render } = require('../templates');
++
++const WINDOW = 30; // highlight window, seconds
++const LOUD_LU = 6; // "loud": the momentary level this far above the programme's typical (median) level
++const LIMIT = { changes: 300, scenes: 300, black: 200, frozen: 200, silence: 500, peaks: 50, minutes: 600, highlights: 10, segments: 5000, text: 200000 };
++const r2 = (n) => Math.round(n * 100) / 100;
++const clock = (s) => { s = Math.max(0, Math.round(s)); const h = Math.floor(s / 3600), m = Math.floor((s % 3600) / 60), x = s % 60; return (h ? `${h}:${String(m).padStart(2, '0')}` : String(m)) + `:${String(x).padStart(2, '0')}`; };
++
++/** A line-by-line parser of the filters' log (a long VOD logs tens of MB, so it is never buffered whole). */
++function createSignalParser() {
++ const out = { scene_changes: [], black: [], frozen: [], silence: [], integrated: null, range: null };
++ const perSecond = new Map(); // second -> the loudest momentary level in it
++ let silenceStart = null; let freezeStart = null; let summary = false;
++ function line(l) {
++ let m;
++ if ((m = /lavfi\.scd\.score:\s*([\d.]+),\s*lavfi\.scd\.time:\s*([\d.]+)/.exec(l))) out.scene_changes.push({ t: r2(+m[2]), score: r2(+m[1]) });
++ else if ((m = /black_start:\s*([\d.]+)\s+black_end:\s*([\d.]+)/.exec(l))) out.black.push({ start: r2(+m[1]), end: r2(+m[2]) });
++ else if ((m = /freezedetect\.freeze_start:\s*([\d.]+)/.exec(l))) freezeStart = +m[1];
++ else if ((m = /freezedetect\.freeze_end:\s*([\d.]+)/.exec(l))) { if (freezeStart != null) out.frozen.push({ start: r2(freezeStart), end: r2(+m[1]) }); freezeStart = null; }
++ else if ((m = /silence_start:\s*(-?[\d.]+)/.exec(l))) silenceStart = Math.max(0, +m[1]);
++ else if ((m = /silence_end:\s*([\d.]+)/.exec(l))) { out.silence.push({ start: r2(silenceStart == null ? 0 : silenceStart), end: r2(+m[1]) }); silenceStart = null; }
++ else if ((m = /\bt:\s*([\d.]+)\s+TARGET:.*?\bM:\s*(-?[\d.]+)/.exec(l))) {
++ const sec = Math.floor(+m[1]); const v = +m[2];
++ if (v > -70) perSecond.set(sec, Math.max(perSecond.has(sec) ? perSecond.get(sec) : -Infinity, v));
++ } else if (/ebur128/.test(l) && /Summary:/.test(l)) summary = true;
++ else if (summary && (m = /^\s*I:\s*(-?[\d.]+)\s*LUFS/.exec(l))) out.integrated = +m[1];
++ else if (summary && (m = /^\s*LRA:\s*([\d.]+)\s*LU\b/.exec(l))) { out.range = +m[1]; summary = false; }
++ }
++ /** duration: what was analysed; an open silence or freeze runs to its end. */
++ function finish(duration) {
++ if (silenceStart != null && duration > silenceStart) out.silence.push({ start: r2(silenceStart), end: r2(duration) });
++ if (freezeStart != null && duration > freezeStart) out.frozen.push({ start: r2(freezeStart), end: r2(duration) });
++ const seconds = [...perSecond.entries()].sort((a, b) => a[0] - b[0]);
++ const integrated = out.integrated != null && out.integrated > -70 ? out.integrated : null;
++ const levels = seconds.map(([, v]) => v).filter((v) => v > -55).sort((a, b) => a - b); // audible seconds only
++ const typical = levels.length ? levels[Math.floor(levels.length / 2)] : null;
++ // Loud moments: seconds well above the programme's typical level, merged when under 3 s apart.
++ const peaks = [];
++ if (typical != null) {
++ for (const [s, v] of seconds) {
++ if (v < typical + LOUD_LU || v < -40) continue;
++ const last = peaks[peaks.length - 1];
++ if (last && s - last.end <= 3) { last.end = s + 1; last.lufs = Math.max(last.lufs, v); } else peaks.push({ start: s, end: s + 1, lufs: v });
++ }
++ }
++ const minutes = [];
++ for (const [s, v] of seconds) { const i = Math.floor(s / 60); if (i < LIMIT.minutes) minutes[i] = minutes[i] == null ? v : Math.max(minutes[i], v); }
++ for (let i = 0; i < minutes.length; i++) if (minutes[i] === undefined) minutes[i] = null;
++ return {
++ scene_changes: out.scene_changes.slice(0, 20000),
++ black: out.black.slice(0, LIMIT.black), frozen: out.frozen.slice(0, LIMIT.frozen), silence: out.silence.slice(0, LIMIT.silence),
++ loudness: {
++ integrated_lufs: integrated, range_lu: integrated != null ? out.range : null, typical_lufs: typical == null ? null : r2(typical),
++ peaks: peaks.sort((a, b) => b.lufs - a.lufs).slice(0, LIMIT.peaks).sort((a, b) => a.start - b.start).map((p) => ({ start: p.start, end: p.end, lufs: r2(p.lufs) })),
++ per_minute: minutes.map((v) => (v == null ? null : r2(v))),
++ },
++ _perSecond: perSecond,
++ };
++ }
++ return { line, finish };
++}
++
++function parseSignals(text, duration) {
++ const p = createSignalParser();
++ String(text).split('\n').forEach(p.line);
++ return p.finish(duration);
++}
++
++/** Scene changes -> scenes. More changes than the limit: the strongest become the boundaries. */
++function scenesOf(changes, duration) {
++ let cuts = changes.filter((c) => c.t > 0.5 && c.t < duration - 0.5);
++ if (cuts.length > LIMIT.scenes - 1) cuts = cuts.slice().sort((a, b) => b.score - a.score).slice(0, LIMIT.scenes - 1);
++ const at = [0, ...cuts.map((c) => c.t).sort((a, b) => a - b), duration];
++ const out = [];
++ for (let i = 0; i < at.length - 1; i++) if (at[i + 1] - at[i] >= 1 || !out.length) out.push({ start: r2(at[i]), end: r2(at[i + 1]) });
++ return out;
++}
++
++function overlap(a0, a1, b0, b1) { return Math.max(0, Math.min(a1, b1) - Math.max(a0, b0)); }
++
++/** Overlapping spans merged, so a second that is both black and silent counts once. */
++function union(spans) {
++ const out = [];
++ for (const x of spans.slice().sort((a, b) => a.start - b.start)) {
++ const last = out[out.length - 1];
++ if (last && x.start <= last.end) last.end = Math.max(last.end, x.end); else out.push({ start: x.start, end: x.end });
++ }
++ return out;
++}
++
++/** Windows (30 s, or a quarter of a short clip; every half window) scored by loudness, scene changes and speech; the best that do not overlap. */
++function highlightsOf({ duration, signals, perSecond, segments }) {
++ const I = signals.loudness.typical_lufs;
++ const dead = union([...signals.black, ...signals.frozen, ...signals.silence]);
++ const W = Math.max(5, Math.min(WINDOW, duration / 4));
++ const cands = [];
++ for (let s = 0; s < Math.max(duration - W / 2, 0.001); s += W / 2) {
++ const e = Math.min(duration, s + W);
++ const len = e - s;
++ if (len < Math.min(5, W)) break;
++ const reasons = []; let score = 0;
++ if (I != null && perSecond) {
++ let max = -Infinity;
++ for (let t = Math.floor(s); t < e; t++) if (perSecond.has(t)) max = Math.max(max, perSecond.get(t));
++ if (max >= I + LOUD_LU && max > -40) { score += Math.min(3, (max - I) / LOUD_LU); reasons.push('loud'); }
++ }
++ const cuts = signals.scene_changes.filter((c) => c.t >= s && c.t < e).length;
++ if (cuts >= 2) { score += Math.min(2, cuts * 0.5); reasons.push('scene changes'); }
++ const words = segments.reduce((n, g) => n + (overlap(s, e, g.start, g.end) > 0 ? g.text.split(/\s+/).filter(Boolean).length : 0), 0);
++ if (words / len >= 1.5) { score += 1; reasons.push('speech'); }
++ const still = dead.reduce((n, d) => n + overlap(s, e, d.start, d.end), 0);
++ if (still / len > 0.5 || !score) continue;
++ const excerpt = segments.filter((g) => overlap(s, e, g.start, g.end) > 0).map((g) => g.text).join(' ').replace(/\s+/g, ' ').trim().slice(0, 300);
++ cands.push({ start: r2(s), end: r2(e), score: r2(score), reasons, excerpt });
++ }
++ const picked = [];
++ for (const c of cands.sort((a, b) => b.score - a.score || a.start - b.start)) {
++ if (picked.length >= LIMIT.highlights) break;
++ if (!picked.some((p) => overlap(p.start, p.end, c.start, c.end) > 0)) picked.push(c);
++ }
++ return picked.sort((a, b) => a.start - b.start);
++}
++
++function speechRatio(segments, duration) {
++ if (!duration) return null;
++ let t = 0; let last = 0;
++ for (const g of segments.slice().sort((a, b) => a.start - b.start)) { const s = Math.max(g.start, last); if (g.end > s) { t += g.end - s; last = g.end; } }
++ return r2(Math.min(1, t / duration));
++}
++
++/** What the model (or the extractive overview) works from: facts only, in plain lines. */
++function factsOf({ duration, streams, signals, scenes, highlights, ratio, transcript }) {
++ const lines = [`Length analysed: ${clock(duration)}.`, `Streams: ${[streams.video && 'video', streams.audio && 'audio'].filter(Boolean).join(' and ') || 'none'}.`];
++ if (streams.video) lines.push(`Scenes: ${scenes.length}${signals.black.length ? `; black picture ${signals.black.length} time(s)` : ''}${signals.frozen.length ? `; frozen picture ${signals.frozen.length} time(s)` : ''}.`);
++ if (streams.audio) {
++ const quiet = signals.silence.reduce((n, x) => n + (x.end - x.start), 0);
++ lines.push(`Silence: ${Math.round((quiet / Math.max(duration, 1)) * 100)}% of the time.${signals.loudness.integrated_lufs != null ? ` Programme loudness ${signals.loudness.integrated_lufs} LUFS.` : ''}`);
++ }
++ if (ratio != null) lines.push(`Speech: ${Math.round(ratio * 100)}% of the time (${transcript.language}).`);
++ if (signals.loudness.peaks.length) lines.push(`Loudest moments: ${signals.loudness.peaks.slice().sort((a, b) => b.lufs - a.lufs).slice(0, 5).map((p) => clock(p.start)).join(', ')}.`);
++ return lines;
++}
++
++function extractive(facts, highlights) {
++ const parts = [facts.join(' ')];
++ const quoted = highlights.filter((h) => h.excerpt).slice(0, 3);
++ if (quoted.length) parts.push(`Highlights: ${quoted.map((h) => `${clock(h.start)}–${clock(h.end)} (${h.reasons.join(', ')}): "${h.excerpt.slice(0, 160)}"`).join('; ')}.`);
++ else if (highlights.length) parts.push(`Highlights: ${highlights.slice(0, 5).map((h) => `${clock(h.start)}–${clock(h.end)} (${h.reasons.join(', ')})`).join('; ')}.`);
++ return parts.join(' ').slice(0, 1500);
++}
++
++function createMediaAnalysis({ registry, pool, fetcher, quotas, config = {}, spawnImpl = spawn, log = console }) {
++ const ma = config.mediaAnalysis || {};
++ const ffmpeg = ma.ffmpeg || 'ffmpeg';
++ const ffprobe = ma.ffprobe || 'ffprobe';
++ let turn = Promise.resolve(); // one analysis at a time
++ const workDir = ma.workDir || require('os').tmpdir();
++ let pruned = false;
++ /** The work directory, made on first use; files a crash left behind (older than a day) are removed then. */
++ function work() {
++ fs.mkdirSync(workDir, { recursive: true, mode: 0o700 });
++ if (!pruned) {
++ pruned = true;
++ for (const f of fs.readdirSync(workDir)) {
++ if (!f.startsWith('openvibe-ai-media-')) continue;
++ const p = path.join(workDir, f);
++ try { if (Date.now() - fs.statSync(p).mtimeMs > 86400000) fs.unlinkSync(p); } catch { /* */ }
++ }
++ }
++ return workDir;
++ }
++
++ function run(bin, args, { signal, timeoutMs, onLine }) {
++ return new Promise((resolve, reject) => {
++ let ch; let out = ''; let buf = '';
++ try { ch = spawnImpl('nice', ['-n', '15', bin, ...args], { stdio: ['ignore', 'pipe', 'pipe'] }); } catch (e) { return reject(new AiError(503, 'media.tool_unavailable', `${bin}: ${e.message}`)); }
++ const kill = () => { try { ch.kill('SIGKILL'); } catch { /* */ } };
++ const timer = setTimeout(kill, timeoutMs);
++ if (signal) signal.addEventListener('abort', kill, { once: true });
++ ch.stdout.on('data', (c) => { if (out.length < 1e6) out += c; });
++ ch.stderr.on('data', (c) => {
++ if (!onLine) return;
++ buf += c;
++ let i;
++ while ((i = buf.indexOf('\n')) >= 0) { onLine(buf.slice(0, i).replace(/\r$/, '')); buf = buf.slice(i + 1); }
++ });
++ ch.on('error', (e) => { clearTimeout(timer); reject(new AiError(503, 'media.tool_unavailable', `${bin}: ${e.message}`)); });
++ ch.on('close', (code) => {
++ clearTimeout(timer);
++ if (buf && onLine) onLine(buf);
++ if (signal && signal.aborted) return reject(new AiError(499, 'run.cancelled', 'cancelled'));
++ if (code !== 0) return reject(new AiError(422, 'media.unreadable', `${bin} could not read the media (exit ${code})`));
++ resolve(out);
++ });
++ });
++ }
++
++ async function probe(file, signal) {
++ const raw = await run(ffprobe, ['-v', 'error', '-show_entries', 'format=duration:stream=codec_type', '-of', 'json', file], { signal, timeoutMs: 60000 });
++ let j; try { j = JSON.parse(raw); } catch { throw new AiError(422, 'media.unreadable', 'ffprobe gave no answer'); }
++ const types = (j.streams || []).map((s) => s.codec_type);
++ return { duration: Number(j.format && j.format.duration) || 0, video: types.includes('video'), audio: types.includes('audio') };
++ }
++
++ async function signalsOf(file, { duration, streams, seconds, signal }) {
++ const graph = [];
++ if (streams.video) graph.push(`[0:v:0]scale=160:-2,scdet=threshold=${ma.sceneThreshold || 10},blackdetect=d=1:pix_th=0.10,freezedetect=n=-60dB:d=4[vo]`);
++ if (streams.audio) graph.push('[0:a:0]silencedetect=n=-35dB:d=2,ebur128[ao]');
++ if (!graph.length) return null;
++ const args = ['-hide_banner', '-nostats', '-nostdin'];
++ if (seconds) args.push('-t', String(seconds));
++ if (streams.video) args.push('-skip_frame', 'nokey');
++ args.push('-i', file, '-filter_complex', graph.join(';'));
++ if (streams.video) args.push('-map', '[vo]', '-f', 'null', '-');
++ if (streams.audio) args.push('-map', '[ao]', '-f', 'null', '-');
++ const parser = createSignalParser();
++ // Keyframe decoding is quick; allow a minute plus a quarter of the programme, at most 30 minutes.
++ await run(ffmpeg, args, { signal, onLine: parser.line, timeoutMs: Math.min(30 * 60000, 60000 + duration * 250) });
++ return parser.finish(duration);
++ }
++
++ /** A quota with workflow_prefix `media.paid` and a cost cap: today's media.* spend must be under it. */
++ function paidBudget() {
++ return quotas && quotas.paidBudget ? quotas.paidBudget('media.paid', 'media.') : null;
++ }
++
++ /** Refuse before downloading when the temp disk could not hold the file (a full disk stops every service). */
++ function roomFor(dir) {
++ let free = Infinity;
++ try { const st = fs.statfsSync(dir); free = st.bavail * st.bsize; } catch { /* no statfs: trust the byte cap */ }
++ const reserve = ma.reserveBytes == null ? 3 * 1024 ** 3 : ma.reserveBytes;
++ const room = free - reserve;
++ if (room < 64 * 1024 ** 2) throw new AiError(503, 'media.no_space', 'not enough free disk to analyse media right now');
++ return Math.min(ma.maxBytes || 4 * 1024 ** 3, room);
++ }
++
++ async function overview({ step, input, wf, run: runRow, ctx, facts, highlights, transcript, gaps }) {
++ const template = registry.activeTemplate(step.template || 'media.analyze.overview');
++ const vars = {
++ facts: facts.join('\n'),
++ highlights: highlights.map((h) => `${clock(h.start)}–${clock(h.end)} [${h.reasons.join(', ')}] ${h.excerpt}`).join('\n') || '(none)',
++ transcript: (transcript.text || '').slice(0, 8000) || '(no transcript)',
++ };
++ const call = async (routeKey) => {
++ const route = registry.resolveRoute(routeKey);
++ if (!route || route.disabled) return { missing: true };
++ const system = render(template.system_prompt, vars).trim();
++ const req = { system: [{ text: system, cache: true }], messages: [{ role: 'user', content: render(template.user_prompt, vars) }], image: null, json: null, maxTokens: 350, temperature: 0.3, timeoutMs: route.timeout_ms || 120000, cacheKey: `${wf.key}:${wf.version}` };
++ const exec = await pool.execute(route, 'summarize', req, { ...ctx, routeKey: route.key, routeVersion: route.version, promptHash: null });
++ const text = String(exec.result.text || '').trim();
++ if (!text) throw new AiError(502, 'output.empty', 'the model answered with nothing');
++ return { text: text.slice(0, 1500), exec, route };
++ };
++ if (template) {
++ try {
++ const local = await call(step.local_route || 'media.local');
++ if (!local.missing) return { overview: { text: local.text, source: 'local_model', provider: local.exec.provider, model: local.exec.model }, exec: local.exec, route: local.route };
++ gaps.push('No local model is configured (AI_LOCAL_LLM_URL), so a model did not write this overview.');
++ } catch (e) {
++ gaps.push(`The local model did not answer (${e.code || e.message}).`);
++ }
++ if (input.allow_paid) {
++ const budget = paidBudget();
++ if (!budget) gaps.push('Paid providers are off for media analysis: no quota gives media.paid a cost budget.');
++ else if (budget.spent >= budget.max_cost_usd) gaps.push(`Today's paid budget for media analysis ($${budget.max_cost_usd}) is spent.`);
++ else {
++ try {
++ const paid = await call(step.paid_route || 'media.paid');
++ if (!paid.missing) return { overview: { text: paid.text, source: 'paid', provider: paid.exec.provider, model: paid.exec.model }, exec: paid.exec, route: paid.route };
++ gaps.push('No paid route (media.paid) is configured.');
++ } catch (e) { gaps.push(`The paid provider did not answer (${e.code || e.message}).`); }
++ }
++ }
++ } else gaps.push('The overview template media.analyze.overview is missing.');
++ return { overview: { text: extractive(facts, highlights), source: 'extractive' } };
++ }
++
++ async function step(stepDef, input, wf, runRow, ctx) {
++ const mine = turn.then(() => analyse(stepDef, input, wf, runRow, ctx));
++ turn = mine.catch(() => {});
++ return mine;
++ }
++
++ async function analyse(stepDef, input, wf, runRow, ctx) {
++ const gaps = [];
++ const dir = work();
++ const { file } = await fetcher.loadMediaToFile(input, { signal: ctx.signal, maxBytes: roomFor(dir), timeoutMs: ma.fetchTimeoutMs, dir });
++ try {
++ const info = await probe(file, ctx.signal);
++ if (!info.video && !info.audio) throw new AiError(422, 'media.unreadable', 'the file has no audio or video stream');
++ const seconds = input.seconds ? Math.min(input.seconds, info.duration || input.seconds) : 0;
++ const duration = r2(seconds || info.duration);
++ const streams = { video: info.video, audio: info.audio };
++ const sig = await signalsOf(file, { duration, streams, seconds, signal: ctx.signal });
++ const perSecond = sig._perSecond; delete sig._perSecond;
++ const signals = { ...sig, scene_changes: sig.scene_changes.slice().sort((a, b) => b.score - a.score).slice(0, LIMIT.changes).sort((a, b) => a.t - b.t) };
++ const scenes = streams.video ? scenesOf(sig.scene_changes, duration) : [];
++ if (!streams.video) gaps.push('No video stream: no scenes, black or frozen picture.');
++ if (!streams.audio) gaps.push('No audio stream: no loudness, silence or speech.');
++
++ let transcript = { available: false, language: input.language || 'en', text: '', segments: [] };
++ let sttExec = null;
++ if (streams.audio) {
++ const route = registry.resolveRoute(stepDef.stt_route || 'live.stt');
++ if (!route || route.disabled) gaps.push('No speech-to-text route (live.stt): no transcript.');
++ else {
++ try {
++ sttExec = await pool.execute(route, 'transcribe', { filePath: file, language: input.language || 'en', seconds: seconds || 0, offsetSec: 0, timeoutMs: 3600000 }, { ...ctx, routeKey: route.key, routeVersion: route.version, promptHash: null });
++ const r = sttExec.result;
++ const segments = (r.segments || []).slice(0, LIMIT.segments).map((g) => ({ start: r2(g.start), end: r2(g.end), text: String(g.text || '').slice(0, 2000) }));
++ transcript = { available: true, language: r.language || input.language || 'en', text: String(r.text || '').slice(0, LIMIT.text), segments };
++ } catch (e) {
++ if (ctx.signal && ctx.signal.aborted) throw e;
++ gaps.push(`Speech-to-text did not run (${e.code || e.message}): no transcript.`);
++ }
++ }
++ }
++ const ratio = transcript.available ? speechRatio(transcript.segments, duration) : null;
++ const highlights = highlightsOf({ duration, signals, perSecond, segments: transcript.segments });
++ const facts = factsOf({ duration, streams, signals, scenes, highlights, ratio, transcript });
++ const ov = await overview({ step: stepDef, input, wf, run: runRow, ctx, facts, highlights, transcript, gaps });
++ const output = {
++ duration_seconds: r2(info.duration), analyzed_seconds: duration, streams,
++ signals, scenes, transcript, speech_ratio: ratio, highlights, overview: ov.overview, gaps: gaps.slice(0, 20),
++ };
++ // The run's provider: the model that wrote the overview, else the speech-to-text engine that ran.
++ return { output, exec: ov.exec || sttExec || null, route: ov.route || null };
++ } finally {
++ try { fs.unlinkSync(file); } catch { /* */ }
++ }
++ }
++
++ return { step };
++}
++
++module.exports = { createMediaAnalysis, createSignalParser, parseSignals, scenesOf, highlightsOf, speechRatio, extractive, clock };
ddiff --git a/server/workflows/media.js b/server/workflows/media.js
nnew file mode 100644
iindex 0000000..6d2f533
---- /dev/null
++++ b/server/workflows/media.js
@@@ -0,0 +1,80 @@
++'use strict';
++/**
++ * Media workflows (roadmap WS-O task 5).
++ *
++ * media.analyze local-first analysis of a VOD, clip or audio file (server/workflows/media-analysis.js): FFmpeg
++ * signals, scenes, local speech-to-text, highlights with their evidence, and an overview from a local
++ * model, or extractive; a paid provider only when the caller allows it and a budget covers it.
++ * Media owns media; Live calls it for its VOD and clip overviews (AI_NS_FALLBACK default).
++ */
++const { MEDIA_REF } = require('./common-schemas');
++
++const NUM = { type: 'number' };
++const SPAN = { type: 'object', additionalProperties: false, required: ['start', 'end'], properties: { start: NUM, end: NUM } };
++const ARR = (items, max) => ({ type: 'array', items, maxItems: max });
++const STR = (n) => ({ type: 'string', maxLength: n });
++
++const templates = [
++ {
++ key: 'media.analyze.overview', name: 'Media: overview from signals and a transcript',
++ description: 'The overview step of media.analyze, for a local model (media.local) or, with a budget, a paid one (media.paid).',
++ input_schema: { type: 'object' }, output_schema: { type: 'object' },
++ system_prompt: 'You describe a video or audio recording for its creator, from facts a machine measured and from a speech transcript. Use only what is given. Do not guess what is on screen beyond what the transcript says, and never invent names, numbers or events. Write 2 to 4 plain sentences (at most 600 characters): what the recording is about and what happens in it, mentioning the highlight times that matter. No markup, no emoji, no preamble.',
++ user_prompt: 'Measured facts:\n{{facts}}\n\nHighlights (time, why, what was said):\n{{highlights}}\n\nTranscript:\n{{transcript}}',
++ default_route: 'media.local', owner: 'media', metadata: {},
++ },
++];
++
++const workflows = [
++ {
++ key: 'media.analyze', name: 'Local-first media analysis: signals, scenes, speech, highlights, overview', namespace: 'media',
++ description: 'FFmpeg signals (scene changes, black and frozen picture, silence, loudness), scenes, whisper.cpp speech-to-text and scored highlights, all on this host; the overview from a local model, else extractive. Paid providers only with allow_paid and a media.paid cost budget.',
++ input_schema: {
++ type: 'object', additionalProperties: false, required: [],
++ properties: {
++ media: MEDIA_REF, media_url: STR(2048),
++ language: { type: 'string', pattern: '^(auto|[a-z]{2})$' },
++ seconds: { type: 'integer', minimum: 0, maximum: 36000, description: 'analyse at most this many seconds from the start (0: all)' },
++ allow_paid: { type: 'boolean', description: 'a paid provider may write the overview when no local model can and a budget allows it' },
++ },
++ },
++ output_schema: {
++ type: 'object', additionalProperties: false,
++ required: ['duration_seconds', 'analyzed_seconds', 'streams', 'signals', 'scenes', 'transcript', 'speech_ratio', 'highlights', 'overview', 'gaps'],
++ properties: {
++ duration_seconds: NUM, analyzed_seconds: NUM,
++ streams: { type: 'object', additionalProperties: false, required: ['video', 'audio'], properties: { video: { type: 'boolean' }, audio: { type: 'boolean' } } },
++ signals: {
++ type: 'object', additionalProperties: false, required: ['scene_changes', 'black', 'frozen', 'silence', 'loudness'],
++ properties: {
++ scene_changes: ARR({ type: 'object', additionalProperties: false, required: ['t', 'score'], properties: { t: NUM, score: NUM } }, 300),
++ black: ARR(SPAN, 200), frozen: ARR(SPAN, 200), silence: ARR(SPAN, 500),
++ loudness: {
++ type: 'object', additionalProperties: false, required: ['integrated_lufs', 'range_lu', 'typical_lufs', 'peaks', 'per_minute'],
++ properties: {
++ integrated_lufs: { type: ['number', 'null'] }, range_lu: { type: ['number', 'null'] }, typical_lufs: { type: ['number', 'null'], description: 'the median second: the programme\'s usual level' },
++ peaks: ARR({ type: 'object', additionalProperties: false, required: ['start', 'end', 'lufs'], properties: { start: NUM, end: NUM, lufs: NUM } }, 50),
++ per_minute: ARR({ type: ['number', 'null'] }, 600),
++ },
++ },
++ },
++ },
++ scenes: ARR(SPAN, 300),
++ transcript: {
++ type: 'object', additionalProperties: false, required: ['available', 'language', 'text', 'segments'],
++ properties: { available: { type: 'boolean' }, language: STR(8), text: { type: 'string', maxLength: 200000 }, segments: ARR({ type: 'object', additionalProperties: false, required: ['start', 'end', 'text'], properties: { start: NUM, end: NUM, text: { type: 'string' } } }, 5000) },
++ },
++ speech_ratio: { type: ['number', 'null'], minimum: 0, maximum: 1 },
++ highlights: ARR({ type: 'object', additionalProperties: false, required: ['start', 'end', 'score', 'reasons', 'excerpt'], properties: { start: NUM, end: NUM, score: NUM, reasons: ARR({ type: 'string', enum: ['loud', 'scene changes', 'speech'] }, 3), excerpt: STR(300) } }, 10),
++ overview: { type: 'object', additionalProperties: false, required: ['text', 'source'], properties: { text: STR(1500), source: { type: 'string', enum: ['local_model', 'paid', 'extractive'] }, provider: { type: ['string', 'null'] }, model: { type: ['string', 'null'] } } },
++ gaps: ARR(STR(300), 20),
++ },
++ },
++ steps: [{ kind: 'media_analysis', stt_route: 'live.stt', template: 'media.analyze.overview', local_route: 'media.local', paid_route: 'media.paid' }],
++ // Not cached: a result made while the local model was down (or before a budget) must not outlive that, and
++ // callers keep what they asked for (Live's vod_ai_state/clip_ai_state).
++ cache_mode: 'none',
++ },
++];
++
++module.exports = { templates, workflows };
ddiff --git a/server/workflows/seed.js b/server/workflows/seed.js
iindex 6979972..54e5120 100644
---- a/server/workflows/seed.js
++++ b/server/workflows/seed.js
@@@ -14,6 +14,7 @@
* request rate for every service
*/
const live = require('./live');
++const media = require('./media');
const core = require('./core');
const products = require('./products');
const { resolveSecret } = require('../util');
@@@ -53,6 +54,10 @@ function providerRecords(config, env) {
const kind = config.fallback.kind === 'anthropic' ? 'anthropic' : 'openai';
out.push({ key: 'fallback', display_name: 'Fallback provider', kind, status: 'active', base_url: config.fallback.baseUrl, auth_mode: resolveSecret(config.fallback.apiKeyRef, env) ? (kind === 'anthropic' ? 'x-api-key' : 'bearer') : 'none', secret_ref: config.fallback.apiKeyRef, default_model: config.fallback.model || null, capabilities: [], timeout_ms: 30000, priority: 50, metadata: { configured_by: 'AI_FALLBACK_*' } });
}
++ if (config.localLlm && config.localLlm.url) {
++ // A local model server (llama.cpp, Ollama) speaking the OpenAI API: no key, never paid (WS-O task 5).
++ out.push({ key: 'local', display_name: 'Local model (OpenAI-compatible server on this host)', kind: 'openai', status: 'active', base_url: config.localLlm.url, auth_mode: 'none', secret_ref: null, default_model: config.localLlm.model || 'local', capabilities: ['chat', 'generate', 'summarize', 'classify', 'extract'], timeout_ms: config.localLlm.timeoutMs, priority: 5, metadata: { configured_by: 'AI_LOCAL_LLM_URL', local: true, paid: false } });
++ }
if (config.httpSeamUrl) {
out.push({ key: 'http-seam', display_name: 'Local HTTP seam', kind: 'http', status: 'active', base_url: config.httpSeamUrl, auth_mode: 'none', default_model: null, capabilities: [], timeout_ms: 30000, priority: 60, metadata: { configured_by: 'AI_HTTP_SEAM_URL' } });
}
@@@ -89,14 +94,17 @@ function seed({ registry, quotas, config, env = process.env, db }) {
registry.seedVersioned('route', 'default.chat', { primary: { provider: 'shared', model: null }, fallbacks, options: {}, max_output_tokens: 800, response_format: 'text', timeout_ms: 30000, alias_of: null });
registry.seedVersioned('route', 'default.json', { primary: { provider: 'shared', model: null }, fallbacks, options: { temperature: 0.3 }, max_output_tokens: 2400, response_format: 'json', timeout_ms: 60000, alias_of: null });
registry.seedVersioned('route', 'default.embedding', { primary: { provider: 'shared', model: env.AI_EMBEDDING_MODEL || 'text-embedding-3-small' }, fallbacks: [], options: {}, max_output_tokens: null, response_format: 'text', timeout_ms: 30000, alias_of: null });
++ // media.analyze (WS-O task 5): the local model when there is one; the paid route is used only under a media.paid budget.
++ if (config.localLlm && config.localLlm.url) registry.seedVersioned('route', 'media.local', { primary: { provider: 'local', model: config.localLlm.model || null }, fallbacks: [], options: { temperature: 0.3 }, max_output_tokens: 350, response_format: 'text', timeout_ms: config.localLlm.timeoutMs, alias_of: null });
++ registry.seedVersioned('route', 'media.paid', { primary: { provider: 'shared', model: config.shared.roleModels.summary || null }, fallbacks, options: { temperature: 0.3 }, max_output_tokens: 350, response_format: 'text', timeout_ms: 60000, alias_of: null });
for (const key of HISTORICAL_ROUTES) registry.seedVersioned('route', key, { primary: { provider: 'shared', model: null }, fallbacks: [], options: {}, max_output_tokens: null, response_format: 'json', timeout_ms: null, alias_of: 'default.json' });
// Templates, then workflows (workflows reference templates)
-- for (const t of [...live.templates, ...core.templates, ...products.templates]) {
++ for (const t of [...live.templates, ...core.templates, ...products.templates, ...media.templates]) {
const { key, ...def } = t;
registry.seedVersioned('template', key, { description: null, default_route: null, owner: 'ai', visibility: 'internal', metadata: {}, ...def });
}
-- for (const w of [...live.workflows, ...core.workflows, ...products.workflows]) {
++ for (const w of [...live.workflows, ...core.workflows, ...products.workflows, ...media.workflows]) {
const { key, ...def } = w;
registry.seedVersioned('workflow', key, { description: null, default_route: null, cache_ttl_sec: null, metadata: {}, ...def });
}
ddiff --git a/test/auth.test.js b/test/auth.test.js
iindex c931476..037ade0 100644
---- a/test/auth.test.js
++++ b/test/auth.test.js
@@@ -70,7 +70,7 @@ t.test('namespaces fail closed: a token without ns runs nothing outside its own
denied(await run(token('games', ['ai.run.create'], { ns: null }), 'ai.generate', { prompt: 'hi' }), 'svc:games without ns -> ai.generate');
denied(await run(token('games', ['ai.run.create'], { ns: [] }), 'wiki.generate_page', { title: 'x', sources: [{ source_type: 'web.page' }] }), 'svc:games with ns [] -> wiki.*');
assert.notStrictEqual((await run(token('games', ['ai.run.create'], { ns: null }), 'games.generate_lore', {})).status, 403, 'svc:games -> games.*');
-- // Live (Network grants it no namespaces): live.* and network.site_copy by the default AI_NS_FALLBACK, nothing else.
++ // Live (Network grants it no namespaces): live.*, network.site_copy and media.analyze by the default AI_NS_FALLBACK, nothing else.
const live = token('live', ['ai.run.create'], { ns: null });
assert.strictEqual((await request(h.base, 'POST', '/api/v1/runs?wait=3000', { tok: live, body: { workflow: 'live.translate', input: { text: 'bonjour tout le monde', from: 'fr', to: 'en' } } })).status, 201);
assert.notStrictEqual((await run(live, 'network.site_copy', { sites: [], links: [] })).status, 403, 'svc:live -> network.site_copy');
@@@ -90,9 +90,11 @@ t.test('AI_NS_FALLBACK and AI_NS_REQUIRED', () => {
const { namespaceAllowed } = require('../server/auth');
const svc = (id, ns) => ({ sub: `svc:${id}`, ns });
const d = load({}).namespaces;
-- assert.deepStrictEqual(d, { required: true, derive: true, fallback: { live: ['live.*', 'network.site_copy'] } });
++ assert.deepStrictEqual(d, { required: true, derive: true, fallback: { live: ['live.*', 'network.site_copy', 'media.analyze'] } });
assert.strictEqual(namespaceAllowed(svc('live', []), 'live.translate', d), true);
assert.strictEqual(namespaceAllowed(svc('live', []), 'network.site_copy', d), true);
++ assert.strictEqual(namespaceAllowed(svc('live', []), 'media.analyze', d), true, 'Live\'s VOD and clip overviews (WS-O task 5)');
++ assert.strictEqual(namespaceAllowed(svc('live', []), 'media.other', d), false, 'that one workflow, not media.*');
assert.strictEqual(namespaceAllowed(svc('live', []), 'ai.generate', d), false);
assert.strictEqual(namespaceAllowed(svc('news', undefined), 'news.summarize_story', d), true);
assert.strictEqual(namespaceAllowed({ sub: 'svc:live' }, 'wiki.generate_page', d), false);
ddiff --git a/test/fixtures/media-analysis.mp4 b/test/fixtures/media-analysis.mp4
nnew file mode 100644
iindex 0000000..89b58a4
BBinary files /dev/null and b/test/fixtures/media-analysis.mp4 differ
ddiff --git a/test/media-analysis.test.js b/test/media-analysis.test.js
nnew file mode 100644
iindex 0000000..1f8e1dc
---- /dev/null
++++ b/test/media-analysis.test.js
@@@ -0,0 +1,159 @@
++'use strict';
++// media.analyze, local-first media analysis (roadmap WS-O task 5), on test/fixtures/media-analysis.mp4: 28 s with cuts
++// at 8, 16 and 20 s, black picture 16-20 s, a still picture 8-16 s, silence 10-16 s, and a loud burst at 16-18 s over
++// a quiet tone.
++// - the log parser reads scdet, blackdetect, freezedetect, silencedetect and ebur128 lines (no FFmpeg needed);
++// - a real run: signals, scenes, loud moments measured against the typical level, highlights with their evidence,
++// and the overview from the local model (media.local), which sees only the measured facts and the transcript;
++// - the local model down: an extractive overview that says so; paid only with allow_paid AND a media.paid budget,
++// never past it; the paid call is the run's provider;
++// - too little free disk refuses before downloading.
++// The runs need ffprobe/ffmpeg on PATH; without them those parts are skipped (the parser part always runs).
++const assert = require('assert');
++const fs = require('fs');
++const path = require('path');
++const nodeHttp = require('http');
++const { spawnSync } = require('child_process');
++const { boot, request, token, suite, seamServer, ALL } = require('./helpers');
++const ma = require('../server/workflows/media-analysis');
++
++const HAS_FFMPEG = spawnSync('ffprobe', ['-version']).status === 0 && spawnSync('ffmpeg', ['-version']).status === 0;
++const FIXTURE = fs.readFileSync(path.join(__dirname, 'fixtures', 'media-analysis.mp4'));
++const LOG = [
++ '[scdet @ 0x1] lavfi.scd.score: 20.854, lavfi.scd.time: 8',
++ '[silencedetect @ 0x2] silence_start: 9.99975',
++ '[freezedetect @ 0x3] lavfi.freezedetect.freeze_start: 9',
++ '[silencedetect @ 0x2] silence_end: 16.0001 | silence_duration: 6.00031',
++ '[freezedetect @ 0x3] lavfi.freezedetect.freeze_end: 16',
++ '[blackdetect @ 0x4] black_start:16 black_end:20 black_duration:4',
++ ...Array.from({ length: 30 }, (_, i) => `[Parsed_ebur128_5 @ 0x5] t: ${(i + 0.5).toFixed(1)} TARGET:-23 LUFS M: ${i === 17 ? '-9.5' : i >= 10 && i < 16 ? '-120.7' : '-35.6'} S: -35.7 I: -35.7 LUFS LRA: 20.0 LU`),
++ '[silencedetect @ 0x2] silence_start: 25.9999',
++ '[Parsed_ebur128_5 @ 0x5] Summary:',
++ ' Integrated loudness:',
++ ' I: -10.1 LUFS',
++ ' Threshold: -20.2 LUFS',
++ ' Loudness range:',
++ ' LRA: 25.7 LU',
++];
++
++const t = suite('media-analysis');
++let media; let mediaBase;
++const tok = token('live', ALL);
++const run = (h, input) => request(h.base, 'POST', '/api/v1/runs?wait=60000', { tok, body: { workflow: 'media.analyze', input: { media_url: `${mediaBase}/fixture.mp4`, language: 'en', ...input } } });
++const completion = (content) => ({ id: 'x', object: 'chat.completion', model: 'fake-model', choices: [{ index: 0, message: { role: 'assistant', content }, finish_reason: 'stop' }], usage: { prompt_tokens: 50, completion_tokens: 20 } });
++
++t.test('the parser reads every filter\'s lines; loud means well above the typical second', async () => {
++ const s = ma.parseSignals(LOG.join('\n'), 28);
++ assert.deepStrictEqual(s.scene_changes, [{ t: 8, score: 20.85 }]);
++ assert.deepStrictEqual(s.black, [{ start: 16, end: 20 }]);
++ assert.deepStrictEqual(s.frozen, [{ start: 9, end: 16 }]);
++ assert.deepStrictEqual(s.silence, [{ start: 10, end: 16 }, { start: 26, end: 28 }], 'an open silence runs to the end');
++ assert.strictEqual(s.loudness.integrated_lufs, -10.1);
++ assert.strictEqual(s.loudness.range_lu, 25.7);
++ assert.strictEqual(s.loudness.typical_lufs, -35.6, 'the median audible second, not the gated integrated level one burst dragged up');
++ assert.deepStrictEqual(s.loudness.peaks, [{ start: 17, end: 18, lufs: -9.5 }]);
++ assert.deepStrictEqual(ma.scenesOf([{ t: 8, score: 20 }, { t: 16, score: 31 }, { t: 20, score: 50 }], 28), [{ start: 0, end: 8 }, { start: 8, end: 16 }, { start: 16, end: 20 }, { start: 20, end: 28 }]);
++ const many = Array.from({ length: 400 }, (_, i) => ({ t: i * 10 + 5, score: i % 7 }));
++ assert.strictEqual(ma.scenesOf(many, 4010).length, 300, 'at most 300 scenes; the strongest changes are the boundaries');
++ assert.strictEqual(ma.speechRatio([{ start: 0, end: 10, text: 'a' }, { start: 5, end: 12, text: 'b' }], 24), 0.5, 'overlapping speech counts once');
++ assert.strictEqual(ma.clock(3725), '1:02:05');
++});
++
++t.test('a real run: signals, scenes, highlights with evidence, and the local model\'s overview', async () => {
++ if (!HAS_FFMPEG) return console.log(' (skipped: no ffmpeg/ffprobe)');
++ media = nodeHttp.createServer((req, res) => { res.writeHead(req.url === '/fixture.mp4' ? 200 : 404, { 'Content-Type': 'video/mp4' }); res.end(req.url === '/fixture.mp4' ? FIXTURE : ''); });
++ await new Promise((r) => media.listen(0, '127.0.0.1', r));
++ mediaBase = `http://127.0.0.1:${media.address().port}`;
++ const local = await seamServer(() => completion('A short test recording: a quiet tone, a pause, then a loud burst at 0:16.'));
++ const h = await boot({ env: { OV_MEDIA_INTERNAL_URL: mediaBase, AI_LOCAL_LLM_URL: `${local.url}/v1`, AI_LOCAL_LLM_MODEL: 'qwen-test' } });
++ try {
++ const r = await run(h, {});
++ assert.strictEqual(r.status, 201, r.text.slice(0, 300));
++ const out = r.body.run.output;
++ assert.strictEqual(r.body.run.status, 'succeeded', JSON.stringify(r.body.run.error));
++ assert.ok(Math.abs(out.duration_seconds - 28) < 0.2);
++ assert.deepStrictEqual(out.streams, { video: true, audio: true });
++ assert.deepStrictEqual(out.signals.scene_changes.map((c) => c.t), [8, 16, 20]);
++ assert.deepStrictEqual(out.signals.black, [{ start: 16, end: 20 }]);
++ assert.deepStrictEqual(out.scenes.map((s) => [s.start, Math.round(s.end)]), [[0, 8], [8, 16], [16, 20], [20, 28]]);
++ assert.ok(out.signals.silence.some((s) => Math.round(s.start) === 10 && Math.round(s.end) === 16));
++ assert.ok(out.signals.loudness.typical_lufs < -30 && out.signals.loudness.integrated_lufs > -15, 'one burst lifts the integrated level, not the typical one');
++ assert.ok(out.signals.loudness.peaks.some((p) => p.start >= 16 && p.start <= 18), 'the burst is a loud moment');
++ assert.ok(out.highlights.some((x) => x.reasons.includes('loud') && x.start <= 18 && x.end >= 17), JSON.stringify(out.highlights));
++ // The model is what the server says it ran; the request asked for the configured one.
++ assert.deepStrictEqual(out.overview, { text: 'A short test recording: a quiet tone, a pause, then a loud burst at 0:16.', source: 'local_model', provider: 'local', model: 'fake-model' });
++ assert.strictEqual(r.body.run.provenance.model, 'fake-model');
++ const prompt = JSON.stringify(local.last.messages);
++ assert.match(prompt, /Scenes: 4/, 'the model sees the measured facts');
++ assert.match(prompt, /Loudest moments: 0:1[6-8]/);
++ assert.strictEqual(local.last.model, 'qwen-test');
++ assert.ok(!r.body.run.output.gaps.some((g) => /local model/.test(g)));
++ const work = path.join(h.dir, 'media-tmp');
++ assert.ok(fs.existsSync(work), 'the work directory is beside the database (not a tmpfs /tmp)');
++ assert.deepStrictEqual(fs.readdirSync(work).filter((f) => f.startsWith('openvibe-ai-media-')), [], 'the downloaded file is removed');
++ } finally { await h.stop(); await local.close(); }
++});
++
++t.test('local model down: extractive, and it says so; paid only with allow_paid and a budget, never past it', async () => {
++ if (!HAS_FFMPEG) return console.log(' (skipped: no ffmpeg/ffprobe)');
++ const local = await seamServer(() => ({ status: 500, body: { error: 'model not loaded' } }));
++ const paid = await seamServer(() => completion('Paid overview of the recording.'));
++ const h = await boot({ env: {
++ OV_MEDIA_INTERNAL_URL: mediaBase, AI_LOCAL_LLM_URL: `${local.url}/v1`, AI_STUB_FALLBACK: '0',
++ AI_PROVIDER: 'openai', AI_BASE_URL: `${paid.url}/v1`, AI_API_KEY: 'paid-key', AI_MODEL: 'paid-model', AI_PROVIDER_RETRIES: '0',
++ } });
++ try {
++ let r = await run(h, {});
++ let out = r.body.run.output;
++ assert.strictEqual(r.body.run.status, 'succeeded', JSON.stringify(r.body.run.error));
++ assert.strictEqual(out.overview.source, 'extractive');
++ assert.match(out.overview.text, /Length analysed: 0:28\. Streams: video and audio\. Scenes: 4/);
++ assert.ok(out.gaps.some((g) => /The local model did not answer/.test(g)), JSON.stringify(out.gaps));
++ assert.ok(out.gaps.some((g) => /Speech-to-text did not run/.test(g)), 'no whisper here, and no stub standing in for it');
++ assert.strictEqual(out.transcript.available, false);
++ assert.strictEqual(paid.calls, 0, 'no paid call without allow_paid');
++
++ r = await run(h, { allow_paid: true });
++ out = r.body.run.output;
++ assert.strictEqual(out.overview.source, 'extractive');
++ assert.ok(out.gaps.some((g) => /no quota gives media\.paid a cost budget/.test(g)));
++ assert.strictEqual(paid.calls, 0, 'no paid call without a budget');
++
++ h.quotas.upsert({ scope_type: 'global', scope_id: '*', window: 'day', max_cost_usd: 0.5, workflow_prefix: 'media.paid' }, { actor: 'test' });
++ r = await run(h, { allow_paid: true });
++ out = r.body.run.output;
++ assert.deepStrictEqual([out.overview.source, out.overview.text], ['paid', 'Paid overview of the recording.'], JSON.stringify(out.gaps));
++ assert.strictEqual(paid.calls, 1);
++ assert.strictEqual(paid.last.model, 'paid-model');
++ assert.strictEqual(r.body.run.provenance.model, 'fake-model', 'the paid call is the run\'s provider');
++ const other = await request(h.base, 'POST', '/api/v1/runs?wait=10000', { tok, body: { workflow: 'ai.summarize', input: { text: 'SQLite is small.', max_words: 20 } } });
++ assert.strictEqual(other.status, 201, `the media.paid budget does not gate other workflows: ${other.text.slice(0, 300)}`);
++
++ const day = new Date().toISOString().slice(0, 10);
++ const callsBefore = paid.calls;
++ h.db.prepare("INSERT INTO usage_daily (day, requester, attribution, workflow_key, provider_key, model_key, requests, cost_usd) VALUES (?, 'service:live', '', 'media.analyze', 'shared', 'paid-model', 1, 0.75)").run(day);
++ r = await run(h, { allow_paid: true });
++ out = r.body.run.output;
++ assert.strictEqual(r.body.run.status, 'succeeded', 'a spent budget refuses the paid call, not the analysis');
++ assert.strictEqual(out.overview.source, 'extractive');
++ assert.ok(out.gaps.some((g) => /paid budget for media analysis \(\$0\.5\) is spent/.test(g)));
++ assert.strictEqual(paid.calls, callsBefore, 'nothing paid past the budget');
++ } finally { await h.stop(); await local.close(); await paid.close(); }
++});
++
++t.test('too little free disk refuses before downloading', async () => {
++ if (!HAS_FFMPEG) return console.log(' (skipped: no ffmpeg/ffprobe)');
++ let fetched = 0;
++ const counting = nodeHttp.createServer((req, res) => { fetched++; res.writeHead(200, { 'Content-Type': 'video/mp4' }); res.end(FIXTURE); });
++ await new Promise((r) => counting.listen(0, '127.0.0.1', r));
++ const base = `http://127.0.0.1:${counting.address().port}`;
++ const h = await boot({ env: { OV_MEDIA_INTERNAL_URL: base, AI_MEDIA_ANALYSIS_DISK_RESERVE_BYTES: String(1024 ** 5) } });
++ try {
++ const r = await request(h.base, 'POST', '/api/v1/runs?wait=20000', { tok, body: { workflow: 'media.analyze', input: { media_url: `${base}/fixture.mp4` } } });
++ assert.strictEqual(r.body.run.status, 'failed');
++ assert.strictEqual(r.body.run.error.code, 'media.no_space');
++ assert.strictEqual(fetched, 0);
++ } finally { await h.stop(); counting.closeAllConnections?.(); counting.close(); if (media) { media.closeAllConnections?.(); media.close(); } }
++});
++
++t.run();
ddiff --git a/test/workflows.test.js b/test/workflows.test.js
iindex 030086f..366ad5a 100644
---- a/test/workflows.test.js
++++ b/test/workflows.test.js
@@@ -23,6 +23,7 @@ const EXAMPLES = {
'live.media.overview': { frames: ['a desk', 'a keyboard'], transcript: 'hello chat' },
'live.stream.recap': { facts: { streamer: 'Ann', title: 'Rust night', duration: '2h 1m', viewers: { peak: 4, avg: 2 } } },
'live.media.transcribe': { media_url: '__MEDIA__/v/1', language: 'en' },
++ 'media.analyze': { media_url: '__MEDIA__/fixture.mp4', language: 'en' },
'network.site_copy': { sites: [{ id: 'live', name: 'OpenVibe.Live', what: 'live streaming' }], links: [{ id: 'tools', name: 'Tools', about: 'online tools' }] },
'ai.chat': { messages: [{ role: 'user', content: 'hello' }] },
'ai.generate': { prompt: 'Write a haiku about SQLite.' },
@@@ -48,15 +49,18 @@ const EXAMPLES = {
for (const k of new Set(Object.values(KIND_TO_WORKFLOW).concat('live.complete'))) EXAMPLES[k] = PASS;
const t = suite('workflows');
++const HAS_FFMPEG = require('child_process').spawnSync('ffprobe', ['-version']).status === 0;
let h;
let media;
const tok = token('live', ALL);
t.test('boot seeds every required workflow key', async () => {
-- media = nodeHttp.createServer((req, res) => { res.writeHead(200, { 'Content-Type': 'video/mp4' }); res.end(Buffer.alloc(2048, 1)); });
++ const fixture = require('fs').readFileSync(require('path').join(__dirname, 'fixtures', 'media-analysis.mp4'));
++ media = nodeHttp.createServer((req, res) => { res.writeHead(200, { 'Content-Type': 'video/mp4' }); res.end(req.url === '/fixture.mp4' ? fixture : Buffer.alloc(2048, 1)); });
await new Promise(r => media.listen(0, '127.0.0.1', r));
const mediaBase = `http://127.0.0.1:${media.address().port}`;
EXAMPLES['live.media.transcribe'].media_url = `${mediaBase}/v/1`;
++ EXAMPLES['media.analyze'].media_url = `${mediaBase}/fixture.mp4`;
h = await boot({ env: { OV_MEDIA_INTERNAL_URL: mediaBase } });
const keys = h.registry.listWorkflows().map(w => w.key);
for (const k of ['wiki.generate_space', 'wiki.generate_page', 'blog.draft_post', 'news.summarize_story', 'news.compare_perspectives', 'reviews.summarize_entity', 'deals.enrich_deal',
@@@ -82,6 +86,7 @@ t.test('every workflow has compiling, versioned input and output schemas and an
t.test('every workflow runs end to end on the stub and returns schema-valid output', async () => {
for (const w of h.registry.listWorkflows()) {
++ if (w.key === 'media.analyze' && !HAS_FFMPEG) continue; // test/media-analysis.test.js says so
const r = await request(h.base, 'POST', '/api/v1/runs?wait=10000', { tok, body: { workflow: w.key, input: EXAMPLES[w.key] } });
assert.strictEqual(r.status, 201, `${w.key}: ${r.status} ${r.text.slice(0, 300)}`);
const run = r.body.run;