openvibers/openvibe.tools Commits
Tools: the legacy limiters are deleted (the guard enforces since 2026-10-07); TOOLS_GUARD defaults to enforce. (#15)
Changes
@@@ -264,7 +264,7 @@ reload reattaches.address key), at most`TOOLS_JOBS_MAX_QUEUED` queued jobs in all (default 200) and `TOOLS_DISK_BUDGET_MB` under `data/jobs` (default8192) before a submit answers `503 tools.busy` with `Retry-After` (through the guard, below), plus the guard's-- quotas and the satellites' older burst and processing limits on submit.++ quotas on submit.- **Retention.** Finished jobs expire after 1 hour (browser sessions) or 24 hours (signed-in people, principals);the pruner deletes the row, its events, its files and its Media objects. It never touches a job that has areference (above). If Media will not delete a result object (a retention hold, `409 media.object.held`), cannot@@@ -380,12 +380,10 @@ One module used by the gateway and every satellite, driven by each tool's descriCalls with a Bearer token, calls without those cookies and requests with neither header pass. Checked on therun API, the job routes' writes, `/api/info`, `/api/probe` and the webhook bins' create anddelete; `403 tools.origin.refused`, recorded as reason `origin`.--- **Mode.** `TOOLS_GUARD=report` (default) records and counts what it would refuse and refuses nothing, and the-- apps' older express-rate-limit limiters stay in force (now keyed by the resolved caller, after sign-in is read).-- `TOOLS_GUARD=enforce` refuses and the older limiters step aside. (On `/api/v1/…` an older limiter's refusal is-- problem+json `tools.quota.exceeded`, `scope: "legacy"`, with `Retry-After`; job runs through the run API pass the-- same burst and processing limiters as job submits.) Hard limits apply in both modes: the pixel limit,-- ffmpeg's whitelists and duration cap, upload sniffing, the port-scan cap and the per-target throttle.++- **Mode.** `TOOLS_GUARD=enforce` (default, production) refuses what is past a limit with `429`++ problem+json `tools.quota.exceeded`. `TOOLS_GUARD=report` (local debugging) records and counts what it++ would refuse and refuses nothing. Hard limits apply in both modes: the pixel limit, ffmpeg's whitelists++ and duration cap, upload sniffing, the port-scan cap and the per-target throttle.Default quotas (units = descriptor `cost` per run; `perMinute` / `burst` / `perDay`, 0 = none):@@@ -407,7 +405,7 @@ Environment: `TOOLS_GUARD`, `TOOLS_GUARD_LIMITS` (JSON merged into the table, e.## Per-actor limits (`apps/_shared/actor-limits.js`)Roadmap WS-R task 4, openvibe-sdk/limits, one limiter per app, only where the guard has no per-caller--limit of its own. The guard, its quotas, the older limiters and `TOOLS_GUARD` are unchanged; these limits++limit of its own. The guard, its quotas and `TOOLS_GUARD` are unchanged; these limitsread the guard's resolved caller (`guard.caller`): a person as `user:usr_…`, a service or app by itsprincipal, a browser session or nobody by the guard's hashed address key (`ip:<HMAC>`, never a rawaddress). Past a limit: `429` problem+json `rate_limited` with `Retry-After`, one `[Limits]` log line and@@@ -659,7 +657,7 @@ with `DATABASE_DIRECT_URL` set for the cutover; it prints a per-table count andAfter the move the apps read `DATABASE_URL` / `DATABASE_DIRECT_URL` (PostgreSQL, via PgBouncer) and`VALKEY_URL` (Valkey); `guard_salt`, `guard_day` and the minute/burst buckets live in Valkey (a throttle,never authoritative), while `guard_abuse`, the job store and the analytics tables stay in PostgreSQL.--`TOOLS_GUARD` (default `report`) and its meaning are unchanged.++`TOOLS_GUARD` (default `enforce`) and its meaning are unchanged.State (plan T8 landed 2026-10-02; production serves the PostgreSQL-only release): every app opens the one `tools`database (`apps/_shared/db.js`, `DATABASE_URL` else an embedded PGlite database in development;@@@ -2,8 +2,8 @@// ═══════════════════════════════════════════════════════════════// Per-actor rate limits (roadmap WS-R task 4; openvibe-sdk/limits), one limiter per app, only where// the guard (./guard) has no per-caller limit of its own. The guard is not changed: its tiered quotas--// (quotaClass × tier × cost, day allowances), the older express-rate-limit limiters, the per-target--// throttle and TOOLS_GUARD's report/enforce mode stay exactly as they are. These limits read the++// (quotaClass × tier × cost, day allowances), the per-target throttle and TOOLS_GUARD's report/enforce++// mode stay exactly as they are. These limits read the// guard's resolved caller (guard.caller) and add://// registry reads GET/HEAD /api/v1/tools[/:id[/schema]] by a signed-in person or a third-party@@@ -15,7 +15,7 @@// backstop job submits and retries (POST /api/v1/jobs, /api/v1/jobs/:id/retry) and runs// (POST /api/v1/tools/:id/run): a ceiling ABOVE the guard's quotas, so it never// decides a normal caller's allowance (the guard does) and still stops a runaway--// one while the guard only reports (TOOLS_GUARD=report, the default).++// one while the guard only reports (TOOLS_GUARD=report, for local debugging).// admin the gateway's /api/internal/analytics (the Network admin's analytics page).//// Counted: a person as user:usr_…, a service or app by its principal (svc:…, app:…), a browser session@@@ -8,7 +8,7 @@// const guard = createGuard({ app: 'img', dataDir, db, valkey, contracts, specs, networkUrl, … });// app.set('trust proxy', TRUST_PROXY); // ./ip.js: one loopback hop// app.use(guard.identify); // req.user (Network sign-in, aud checked)--// app.use('/api/', legacyLimiter, guard.apiQuota); // every /api/ request (tools-api)++// app.use('/api/', guard.apiQuota); // every /api/ request (tools-api)// app.post('/api/v1/tools/:id/run', guard.toolQuota(hostTool), upload, guard.admitUpload(toolOf), guard.heavy(), …)//// What it does:@@@ -30,11 +30,10 @@// own hosts (custom domains included) or the request's own host (CSRF); calls with a// Bearer token and calls without those cookies do not need it. 403 tools.origin.refused//--// Mode: TOOLS_GUARD=report (default) logs and counts what it would refuse and refuses nothing, except--// the hard limits that apply in both modes: the image pixel limit, ffmpeg's protocol and format--// whitelists and duration cap, upload sniffing, the port-scan cap and the per-target throttle.--// TOOLS_GUARD=enforce refuses. The apps' older per-route limiters keep working in report mode and step--// aside in enforce mode (guard.enforcing), where the tiered quotas replace them.++// Mode: TOOLS_GUARD=enforce (default, production) refuses what is past a limit. TOOLS_GUARD=report++// (local debugging) logs and counts what it would refuse and refuses nothing, except the hard limits++// that apply in both modes: the image pixel limit, ffmpeg's protocol and format whitelists and++// duration cap, upload sniffing, the port-scan cap and the per-target throttle.// ═══════════════════════════════════════════════════════════════const fs = require('fs');@@@ -97,7 +96,7 @@ function createGuard(o = {}) {const log = o.log || console;const now = o.now || Date.now;const appName = o.app || 'tools';-- const mode = String(env.TOOLS_GUARD || 'report').trim().toLowerCase() === 'enforce' ? 'enforce' : 'report';++ const mode = String(env.TOOLS_GUARD || 'enforce').trim().toLowerCase() === 'enforce' ? 'enforce' : 'report';const enforcing = mode === 'enforce';const bounds = limits.bounds(env);const table = limits.quotas(env, log);@@@ -207,9 +206,8 @@ function createGuard(o = {}) {async function charge(req, res, { quotaClass, cost, tool: toolId }) {const c = caller(req, res);const r = await quotas.check(c, { quotaClass, cost }, { report: !enforcing });-- // In report mode an older limiter that ran on this request (req.rateLimit) is the one in force:-- // its RateLimit-* headers stay. Otherwise the guard's describe the allowance closest to running out.-- if (enforcing || !req.rateLimit) setHeaders(res, r);++ // The guard's RateLimit-* headers describe the allowance closest to running out.++ setHeaders(res, r);if (r.ok) return true;if (enforcing && (c.tier === 'anonymous' || c.tier === 'session') && challenge.required(req, c, { reason: 'quota', tool: toolId })) {if (await challenge.verify(req, c)) return true;@@@ -427,47 +425,7 @@ function createGuard(o = {}) {return false;}-- // ── Older limiters, and tool errors ──────────────────────-- /**-- * One of the apps' older express-rate-limit limiters, keyed by the resolved caller: a signed-in-- * person or a principal by who they are (their higher number applies — these limiters used to run-- * before sign-in was read), everyone else by address (IPv6 /64, hashed). Skipped in enforce mode.-- * @param {Function} rateLimit require('express-rate-limit')-- */-- function legacyLimiter(rateLimit, { windowMs, anonymous, signedIn = anonymous, message, headers = true }) {-- const signed = (c) => c.tier === 'user' || c.tier === 'service';-- return rateLimit({-- windowMs,-- max: (req) => (signed(caller(req)) ? signedIn : anonymous),-- standardHeaders: headers,-- legacyHeaders: false,-- keyGenerator: (req) => { const c = caller(req); return signed(c) || c.tier === 'sandbox' ? c.key : c.ipKey; },-- skip: (req) => enforcing || isProbe(req), // liveness and readiness probes are never limited-- message: { error: message },-- // The platform API (/api/v1/…) answers like the guard does: problem+json tools.quota.exceeded-- // with Retry-After; the older routes keep their { error } shape.-- handler: (req, res, _next, options) => {-- const reset = req.rateLimit && req.rateLimit.resetTime ? new Date(req.rateLimit.resetTime).getTime() : 0;-- const retry = Math.max(1, Math.ceil(((reset || (now() + windowMs)) - now()) / 1000));-- res.setHeader('Retry-After', String(retry));-- if (/^\/api\/v1\//.test(String(req.originalUrl || req.url || ''))) {-- const c = caller(req);-- return sendProblem(req, res, 429, 'tools.quota.exceeded', message, { scope: 'legacy', tier: c.tier, retry_after: retry }, o.contracts);-- }-- return res.status(options.statusCode).json(options.message);-- },-- });-- }-- /** The three limiters an upload app had (LEGACY numbers in ./limits.js). */-- function legacyLimiters(rateLimit, numbers) {-- const n = numbers || limits.LEGACY[appName];-- return {-- apiLimiter: legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: n.api[0], signedIn: n.api[1], message: 'Too many requests. Please try again later.' }),-- processLimiter: legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: n.process[0], signedIn: n.process[1], message: 'Processing rate limit reached. Sign in for higher limits or wait a moment.' }),-- burstLimiter: legacyLimiter(rateLimit, { windowMs: 5_000, anonymous: n.burst, signedIn: n.burst, headers: false, message: 'Too many requests in quick succession. Please slow down.' }),-- };-- }--++ // ── Tool errors ──────────────────────────────────────────/*** A tool refused its input on a hard limit (err.guardReason: 'pixels', 'ffmpeg.format',* 'ffmpeg.duration', 'timeout'): record it and answer its problem. → true when answered.@@@ -503,7 +461,7 @@ function createGuard(o = {}) {checkFiles, admitUpload, admitJob, uploadsOf, discardUploads,heavy, sync, jobsBusy,target, portScan, targets, ports, normalizeTarget,-- legacyLimiter, legacyLimiters, toolRefused,++ toolRefused,originOk, originCheck, allowedOrigin, tokens,attachMetrics, prune,close() { if (pruneTimer) clearInterval(pruneTimer); if (keys.stop) keys.stop(); store.close(); },@@@ -64,15 +64,6 @@ const QUOTAS = {// dropping the cookie starts a new session, never a new allowance.const SESSION_IP_SHARE = 3;--// The apps' older per-route limiters (express-rate-limit, requests per window per address, or per--// person when signed in). They stay in force in report mode and step aside with TOOLS_GUARD=enforce,--// where the tiered quotas above take over. [anonymous or session, signed in or service].--const LEGACY = {-- img: { api: [60, 120], process: [10, 30], burst: 4 },-- audio: { api: [60, 120], process: [6, 20], burst: 3 },-- docs: { api: [60, 120], process: [10, 30], burst: 4 },--};--function envInt(name, fallback, env = process.env) {const v = parseInt(env[name], 10);return Number.isFinite(v) && v >= 0 ? v : fallback;@@@ -128,4 +119,4 @@ function quotas(env = process.env, log = console) {}}--module.exports = { TIERS, QUOTAS, SESSION_IP_SHARE, LEGACY, bounds, quotas, merge };++module.exports = { TIERS, QUOTAS, SESSION_IP_SHARE, bounds, quotas, merge };@@@ -59,7 +59,7 @@ const REPLAY_TTL_MS = 15 * 60 * 1000;const isRunPath = (p) => RUN_RE.test(String(p || ''));--/** Run Express middleware in order (the apps' older limiters) → true when every one called next(). */++/** Run Express middleware in order (a route's limiters) → true when every one called next(). */async function passes(list, req, res) {for (const mw of list || []) {const ok = await new Promise((resolve) => {@@@ -454,7 +454,7 @@ function createRunApi(o) {// The tool's quota class and cost (recorded for the job admission, so it is not charged twice).req._ovCharged = { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id };if (!(await guard.charge(req, res, { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id }))) return undefined;-- // A satellite's older per-route limiters count job runs like its job submits (report mode: in force).++ // The satellite's limiters (the actor backstop, the tool's quota) count a run like a job submit.if (mode === 'job' && o.jobs.limiters && !(await passes(o.jobs.limiters, req, res))) return undefined;const multipart = req.is && req.is('multipart/form-data');if (multipart && mode !== 'job') throw new Refusal(400, 'tools.run.invalid', `${d.id} takes no files: send JSON (tools.run-request@1).`);@@@ -572,7 +572,7 @@ function createSiblingFetch({ self, ports, fetchImpl = fetch }) {* @param {Function} o.multer @param {string} o.uploadsDir* @param {Function} o.Ajv @param {Function} [o.addFormats]* @param {() => object} o.ports satellite ports (./satellites.js)-- * @param {Function[]} [o.limiters] the app's older burst and processing limiters (as on its job submits)++ * @param {Function[]} [o.limiters] the app's limiters (as on its job submits: the actor backstop, the tool's quota)*/function satelliteRunApi(o) {const uploadsDir = path.resolve(o.uploadsDir);@@@ -13,7 +13,6 @@ const cookieParser = require('cookie-parser');const path = require('path');const fs = require('fs');--const rateLimit = require('express-rate-limit');const config = require('./config');const { resolveContext, DOMAIN_MAP } = require('./domain-map');const { getTool, listTools } = require('./tools');@@@ -40,8 +39,8 @@ const { requireInternalAccess } = require('../../_shared/internal-token');// ── Guard (apps/_shared/guard) ───────────────────────────────// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the--// abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse. Every audio--// tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).++// abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would++// refuse. Every audio tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).const SPECS = require('./descriptors').SPECS;// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-audio' });@@@ -51,7 +50,6 @@ const guard = createGuard({app: 'audio', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,});--const { apiLimiter, processLimiter, burstLimiter } = guard.legacyLimiters(rateLimit);/** The tool a request is for before its body is read: the host's own, else the operation it defaults to. */const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('audio.process', req.ctx.defaultOp || 'convert', null) || { id: null }).id);/** The tool a parsed request runs: its `tool` (operation) on this host. */@@@ -123,12 +121,12 @@ app.use(exceptRegistry(cors({credentials: true,}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));--// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──++// ── Who is asking, then the guard's quotas ───────────────────app.use(guard.identify);--app.use('/api/', apiLimiter, guard.apiQuota);++app.use('/api/', guard.apiQuota);// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's audio tools ──--// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'ffmpeg' ? !!which(process.env.FFMPEG_PATH || 'ffmpeg') : true)) });// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).@@@ -189,7 +187,7 @@ app.get('/api/tools', (_req, res) => {});// ── Probe / Metadata Endpoint ────────────────────────────────--app.post('/api/probe', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {++app.post('/api/probe', guard.originCheck(), guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {try {// Hardened like every run: local files only, audio demuxers only.const { info, meta } = await jobContext.run(limitsFor('metadata'), async () => ({ info: await probe(req.file.path), meta: await readMetadata(req.file.path) }));@@@ -210,7 +208,7 @@ const jobs = jobsRuntime.setupJobs({getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,define: defineJobs,receive: uploadAny,-- limiters: [actorLimits.backstop('tools.job.create'), burstLimiter, processLimiter, guard.toolQuota(hostTool)],++ limiters: [actorLimits.backstop('tools.job.create'), guard.toolQuota(hostTool)],jobTool: (req, type, input) => { const d = guard.toolForJob('audio.process', String((input && input.tool) || req.ctx.defaultOp || 'convert'), req.ctx.toolId); return d ? d.id : null; },defaults(req, input) {const out = { ...input };@@@ -227,7 +225,7 @@ const jobs = jobsRuntime.setupJobs({const runApi = satelliteRunApi({app: 'audio', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),-- limiters: [actorLimits.backstop('tools.tool.run'), burstLimiter, processLimiter],++ limiters: [actorLimits.backstop('tools.tool.run')],});app.use(runApi.handle);@@@ -22,13 +22,13 @@ async function waitFor(app, id, cookie, pred, what, ms = 30000) {const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();if (pred(job)) return job;if (Date.now() - t0 > ms) throw new Error(`timed out waiting for ${what}: ${JSON.stringify(job)}`);-- await sleep(150); // the older /api/ limiter allows an address 60 requests a minute++ await sleep(150); // the guard's tools-api quota allows an address 120 requests a minute}}(async () => {const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-audio-'));-- const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output') };++ const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: 'report' }; // a session-less submit is the point here; enforce would refuse it 401// Three minutes of tone: small to upload, a couple of seconds to re-encode.const input = path.join(data, 'tone.mp3');execFileSync('ffmpeg', ['-loglevel', 'error', '-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=180', '-ac', '1', '-b:a', '16k', input]);@@@ -21,7 +21,7 @@ const seconds = (file) => parseFloat(execFileSync('ffprobe', ['-v', 'error', '-s(async () => {const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-audio-merge-'));process.env.DATA_DIR = data; // the tool's temp files go here, not into the app directory-- const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output') };++ const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: 'report' }; // the merge job is submitted without a session; enforce would refuse it 401// Different formats, rates and channel counts: a 1 s mono MP3 at 44.1 kHz and a 0.5 s stereo WAV at 22.05 kHz.const mp3 = path.join(data, 'one.mp3'), wav = path.join(data, 'two.wav');execFileSync('ffmpeg', ['-loglevel', 'error', '-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=1', '-ac', '1', '-ar', '44100', mp3]);@@@ -14,7 +14,6 @@ const path = require('path');const fs = require('fs');const fsp = require('fs/promises');--const rateLimit = require('express-rate-limit');const config = require('./config');const { resolveContext, DOMAIN_MAP } = require('./domain-map');const { getTool, listTools, BINARIES } = require('./tools');@@@ -40,8 +39,8 @@ const { requireInternalAccess } = require('../../_shared/internal-token');// ── Guard (apps/_shared/guard) ───────────────────────────────// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the--// abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse. Every PDF--// tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).++// abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would++// refuse. Every PDF tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).const SPECS = require('./descriptors').SPECS;// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-docs' });@@@ -51,7 +50,6 @@ const guard = createGuard({app: 'docs', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,});--const { apiLimiter, processLimiter, burstLimiter } = guard.legacyLimiters(rateLimit);/** The tool a request is for before its body is read: the host's own (the hubs: merge, the first). */const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('docs.process', req.ctx.defaultOp || 'merge', null) || { id: null }).id);/** The tool a parsed request runs: its `tool` (operation) on this host. */@@@ -139,12 +137,12 @@ app.use(exceptRegistry(cors({credentials: true,}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));--// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──++// ── Who is asking, then the guard's quotas ───────────────────app.use(guard.identify);--app.use('/api/', apiLimiter, guard.apiQuota);++app.use('/api/', guard.apiQuota);// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's PDF tools ──--// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.const pdfPrograms = require('./tools/pdf');const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (pdfPrograms[p] ? pdfPrograms[p].available() : true)) });@@@ -218,7 +216,7 @@ app.get('/api/tools', (_req, res) => {});// ── PDF Info (no file mutation) ──────────────────────────────--app.post('/api/info', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {++app.post('/api/info', guard.originCheck(), guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {try {const tool = getTool('metadata');const [buffer] = await readUploads([req.file]);@@@ -237,7 +235,7 @@ const jobs = jobsRuntime.setupJobs({getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,define: (system) => defineJobs(system, pool),receive: uploadAny,-- limiters: [actorLimits.backstop('tools.job.create'), burstLimiter, processLimiter, guard.toolQuota(hostTool)],++ limiters: [actorLimits.backstop('tools.job.create'), guard.toolQuota(hostTool)],jobTool: (req, type, input) => { const op = String((input && input.tool) || req.ctx.defaultOp || ''); const d = op ? guard.toolForJob('docs.process', op, req.ctx.toolId) : null; return d ? d.id : null; },defaults(req, input) {const out = { ...input };@@@ -254,7 +252,7 @@ const jobs = jobsRuntime.setupJobs({const runApi = satelliteRunApi({app: 'docs', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),-- limiters: [actorLimits.backstop('tools.tool.run'), burstLimiter, processLimiter],++ limiters: [actorLimits.backstop('tools.tool.run')],});app.use(runApi.handle);@@@ -24,7 +24,7 @@ async function makePdf(pages) {const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-guard-'));const uploads = path.join(data, 'uploads');const app = await startApp('docs', { DATA_DIR: data, UPLOADS_DIR: uploads, OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: mode });-- // Each call from its own address, so the older burst limiter (4 in 5 s per address) stays out of it.++ // Each call from its own address, so the per-address quota never sees two calls in a row.let n = 0;const post = (id, files, input, cookie) => {const f = new FormData();@@@ -29,7 +29,7 @@ async function finished(app, id, cookie) {(async () => {const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-'));-- const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output') };++ const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: 'report' }; // submits here carry no session; enforce would refuse them 401const a = await pdf(2, 'First'), b = await pdf(3);let app;try {@@@ -143,7 +143,7 @@ const is503 = (e) => e.status === 503 && e.code === 'tools.unavailable' && /bein// ── As a real process without qpdf: the API answers 503 and the page is told ──const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-tools-'));-- const app = await startApp('docs', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), QPDF_PATH: '/nonexistent/qpdf' });++ const app = await startApp('docs', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), QPDF_PATH: '/nonexistent/qpdf', TOOLS_GUARD: 'report' }); // no session here: enforce would answer 401 before the tool's 503try {const fd = new FormData();fd.append('type', 'docs.process');@@@ -6,7 +6,6 @@ require('dotenv').config();const express = require('express');const cors = require('cors');const helmet = require('helmet');--const rateLimit = require('express-rate-limit');const path = require('path');const http = require('http');const fs = require('fs');@@@ -23,7 +22,7 @@ const { createGuard, TRUST_PROXY } = require('../../_shared/guard');// ── Guard (apps/_shared/guard) ───────────────────────────────// Who is asking (Network sign-in with aud openvibe.tools, else the address), the tools-api quota and// the abuse log (data/guard.db). The data lookups are counted by maps (tools-map), which gets the--// visitor's address from here. TOOLS_GUARD=report (default) records what it would refuse.++// visitor's address from here. TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-food' });@@@ -94,11 +93,10 @@ app.use(helmet({// Sign-in first, so limits know who is asking; everyone else counts by address (IPv6 /64).app.use(cookieParser());app.use(guard.identify);--app.use(guard.legacyLimiter(rateLimit, { windowMs: 60000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }));app.use('/api/', guard.apiQuota);// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's food finder ──--// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.// The food finder is built on the maps backend; its descriptor lives with the maps app's.const toolRegistry = createLocalRegistry({ specs: require('../../maps/server/descriptors').SPECS.filter(s => s.id === 'food') });@@@ -1,6 +1,6 @@'use strict';// Food proxies every API call to maps. It used to call maps without the visitor's address, so maps'--// per-IP limit (30/min) was one bucket shared by every food visitor (127.0.0.1). Now food forwards++// per-caller quota was one bucket shared by every food visitor (127.0.0.1). Now food forwards// X-Forwarded-For / X-Real-IP from req.ip, and maps believes X-Forwarded-For from one loopback hop only.const assert = require('assert');const fs = require('fs');@@@ -12,10 +12,10 @@ const { startApp } = require('../../_shared/test/spawn');const food = await startApp('food', { MAPS_API: maps.base }, undefined, { readyPath: '/api/ready' });try {const ask = (ip) => fetch(`${food.base}/api/foods?search=rice`, { headers: { 'X-Forwarded-For': ip } });-- // One visitor uses up maps' budget…++ // One visitor uses up maps' tools-map quota (descriptor maps: cost 2; anonymous burst 40 units = 20 calls)…let limited = null;for (let i = 0; i < 40 && !limited; i++) { const r = await ask('203.0.113.1'); if (r.status === 429) limited = i; else assert.strictEqual(r.status, 200); }-- assert.strictEqual(limited, 30, 'maps limits that visitor after 30 requests a minute');++ assert.ok(limited >= 19 && limited <= 21, `maps limits that visitor after its burst of 20 (got ${limited})`);// …and another visitor is not affected.const other = await ask('203.0.113.2');assert.strictEqual(other.status, 200, 'a second visitor has their own bucket');@@@ -46,8 +46,9 @@ MEDIA_PUBLIC_URL=https://openvibe.media#TOOLS_LIMITS_MINUTE=120#TOOLS_LIMITS_HOUR=3000--# The shared guard's mode (apps/_shared/guard/index.js): today's default is 'report' (log only); T8 step 2 flips this to enforce.--#TOOLS_GUARD=enforce++# The shared guard's mode (apps/_shared/guard/index.js): 'enforce' (default, production) refuses what is++# past a limit; set 'report' for local debugging to log what it would refuse instead.++#TOOLS_GUARD=report# ── Satellites: jobs (img, audio, docs read the same tools.env) ──#TOOLS_JOBS_CONCURRENCY=2@@@ -57,7 +57,8 @@ let auth = null; // created below; the key check reads it at request time// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else// the address), the tools-api quota, the net and dev tools' quotas by descriptor, the per-target// throttle, the port-scan cap, webhook bin caps and the abuse log (data/guard.db). The Network key is--// the OAuth client's (loaded and retried by it). TOOLS_GUARD=report (default) records what it would refuse.++// the OAuth client's (loaded and retried by it). TOOLS_GUARD=enforce (default) refuses; report records++// what it would refuse.// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-gateway' });const toolsValkey = require('../../_shared/db').openToolsValkey({ createValkey: require('openvibe-sdk/valkey').createValkey });@@@ -191,17 +192,17 @@ app.use(exceptRegistry(cors({exposedHeaders: ['Location', 'Idempotent-Replayed', 'Retry-After', 'RateLimit-Limit', 'RateLimit-Remaining', 'RateLimit-Reset', 'X-OpenVibe-Request-Id', 'Deprecation', 'Sunset', 'Link'],}), { isFirstParty: isAllowedOrigin }));--// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──++// ── Who is asking, then the guard's quotas (sign-in first, so a signed-in tier applies) ──app.use(guard.identify);// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).app.use(require('../../_shared/usage').usagePages({ snapshot: toolRegistry.snapshot }));--app.use('/api/', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 120, signedIn: 240, message: 'Too many requests. Please try again later.' }), guard.apiQuota);++app.use('/api/', guard.apiQuota);app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, keyGenerator: (req) => guard.caller(req).ipKey }));// ── Tool registry (ADR-027, capability tools.tool.read) ──────// GET /api/v1/tools (tools.tool-list@1), /api/v1/tools/:id (tools.tool@1), /api/v1/tools/:id/schema.// Every tool, every family; satellite hosts answer the same routes for their own tools. Counted by--// the /api/ limiter above. /api/catalog.json stays as it is (Network reads it).++// the /api/ quota above. /api/catalog.json stays as it is (Network reads it).// Per-actor limits (apps/_shared/actor-limits.js, roadmap WS-R task 4): signed-in registry reads, a backstop above the// guard's quotas on runs and on the jobs facade's submits, and the admin analytics route.const actorLimits = require('../../_shared/actor-limits').createToolsLimits({ app: 'gateway', createActorLimiter: require('openvibe-sdk/limits').createActorLimiter, guard, registry: obs.registry });@@@ -307,7 +308,7 @@ app.get('/api/brand', (_req, res) => res.json(BRAND));// which owns pastes (roadmap Wave 5). The visitor's Network JWT goes along: Community verifies it and// files writes under the person's canonical subject, so no site has to translate ids any more. The// client address goes along too, for Community's anonymous-write limits.--app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 120, signedIn: 240, message: 'Too many requests. Please try again later.' }), async (req, res) => {++app.use('/api/pastes', async (req, res) => {try {const target = `${config.communityUrl}/api/pastes${req.url}`;// The path is the visitor's: "/api/pastes/../../internal/x" (or %2e%2e) resolves to another of@@@ -347,7 +348,7 @@ app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonym// our pages (the /in URL a bin receives on is anyone's to call).const { createWebhookRouter } = require('./dev/routes');const devOrigin = guard.originCheck();--app.use('/api/dev/webhook', (req, res, next) => (/^\/bins\/[^/]+\/in(\/|$)/.test(req.path) ? next() : devOrigin(req, res, next)), guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), createWebhookRouter({ guard }));++app.use('/api/dev/webhook', (req, res, next) => (/^\/bins\/[^/]+\/in(\/|$)/.test(req.path) ? next() : devOrigin(req, res, next)), createWebhookRouter({ guard }));// ── Host-header subdomain routing ────────────────────────────function subdomainOf(req) {@@@ -13,7 +13,6 @@ const cookieParser = require('cookie-parser');const path = require('path');const fs = require('fs');--const rateLimit = require('express-rate-limit');const config = require('./config');const { resolveContext, DOMAIN_MAP } = require('./domain-map');const { getTool, listTools } = require('./tools');@@@ -40,7 +39,7 @@ const { requireInternalAccess } = require('../../_shared/internal-token');// ── Guard (apps/_shared/guard) ───────────────────────────────// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the--// abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse.++// abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.const SPECS = require('./descriptors').SPECS;// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4): guard_abuse in// PostgreSQL, guard_salt/guard_day and the quota buckets in Valkey. `ready` resolves when the schema is in.@@@ -51,7 +50,6 @@ const guard = createGuard({app: 'img', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,});--const { apiLimiter, processLimiter, burstLimiter } = guard.legacyLimiters(rateLimit);/** The tool a request is for before its body is read: the host's own, else the operation it defaults to. */const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('img.process', req.ctx.defaultOp || 'convert', null) || { id: null }).id);/** The tool a parsed request runs: its `tool` (operation) on this host. */@@@ -129,12 +127,12 @@ app.use(exceptRegistry(cors({credentials: true,}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));--// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──++// ── Who is asking, then the guard's quotas ───────────────────app.use(guard.identify);--app.use('/api/', apiLimiter, guard.apiQuota);++app.use('/api/', guard.apiQuota);// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's image tools ──--// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'heif-dec' ? codec.heif.available() : true)) });// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).@@@ -204,7 +202,7 @@ const jobs = jobsRuntime.setupJobs({getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,define: (system) => defineJobs(system, pool),receive: uploadSingle,-- limiters: [actorLimits.backstop('tools.job.create'), burstLimiter, processLimiter, guard.toolQuota(hostTool)],++ limiters: [actorLimits.backstop('tools.job.create'), guard.toolQuota(hostTool)],jobTool: (req, type, input) => { const d = guard.toolForJob('img.process', String((input && input.tool) || req.ctx.defaultOp || 'convert'), req.ctx.toolId); return d ? d.id : null; },defaults(req, input) {const out = { ...input };@@@ -221,7 +219,7 @@ const jobs = jobsRuntime.setupJobs({const runApi = satelliteRunApi({app: 'img', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),-- limiters: [actorLimits.backstop('tools.tool.run'), burstLimiter, processLimiter],++ limiters: [actorLimits.backstop('tools.tool.run')],});app.use(runApi.handle);@@@ -36,7 +36,7 @@ const { startApp } = require('../../_shared/test/spawn');// ── Through the app ──const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-img-guard-'));-- const app = await startApp('img', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_MAX_INPUT_PIXELS: '10000' });++ const app = await startApp('img', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_MAX_INPUT_PIXELS: '10000', TOOLS_GUARD: 'report' });const run = (buf, name, type) => {const f = new FormData();f.append('input', JSON.stringify({ format: 'png' }));@@@ -9,7 +9,6 @@const express = require('express');const cors = require('cors');const helmet = require('helmet');--const rateLimit = require('express-rate-limit');const path = require('path');const fs = require('fs');const NodeCache = require('node-cache');@@@ -30,7 +29,7 @@ const { createPacer, Busy } = require('../../_shared/guard/semaphore');// ── Guard (apps/_shared/guard) ───────────────────────────────// Who is asking (Network sign-in with aud openvibe.tools, else the address — the food app forwards its// visitor's), the tools-map quota on the data routes (descriptor maps: cost 2 a call) and the abuse log--// (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse.++// (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-maps' });@@@ -94,13 +93,12 @@ app.use(express.json());// Rate limits: sign-in first, so limits know who is asking; everyone else counts by address (IPv6 /64).app.use(guard.identify);--const apiLimiter = guard.legacyLimiter(rateLimit, { windowMs: 60 * 1000, anonymous: 30, signedIn: 60, message: 'Too many requests — slow down, traveler.' });--app.use('/api/', apiLimiter, guard.apiQuota);++app.use('/api/', guard.apiQuota);// The map's data lookups (OpenStreetMap, weather, food banks…): the tools-map quota, by descriptor.app.use(['/api/geocode', '/api/search', '/api/weather', '/api/terrain', '/api/food-banks', '/api/stores', '/api/foods', '/api/meal-plan'], guard.toolQuota('maps'));// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's survival map ──--// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS.filter(s => s.id === 'maps') });// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).@@@ -4,7 +4,6 @@ const express = require('express');const cors = require('cors');const helmet = require('helmet');const cookieParser = require('cookie-parser');--const rateLimit = require('express-rate-limit');const path = require('path');const fs = require('fs');@@@ -20,7 +19,8 @@ const { createGuard, TRUST_PROXY } = require('../../_shared/guard');// ── Guard (apps/_shared/guard) ───────────────────────────────// Who is asking (Network sign-in with aud openvibe.tools, else the address), the tools-api quota on--// /api/ and the abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse.++// /api/ and the abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what++// it would refuse.const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-text' });@@@ -89,13 +89,12 @@ app.use(exceptRegistry(cors({credentials: true,})));--// ── Who is asking, then rate limits ──────────────────────────++// ── Who is asking, then the guard's quotas ───────────────────app.use(guard.identify);--app.use(guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 200, signedIn: 400, message: 'Too many requests. Please try again later.' }));app.use('/api/', guard.apiQuota);// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's text and logo tools ──--// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS });// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).@@@ -7,7 +7,7 @@ Free YouTube video and audio downloader at `yt.openvibe.tools`.- Video: Best, 1080p, 720p, 480p, 360p, WebM- Audio: MP3, M4A, OPUS, FLAC- Real-time SSE progress streaming--- Tiered rate limiting (anon: 5/hr, authed: 20/hr)++- Rate limiting and quotas through the shared guard (`tools-download`: a download costs 50; `tools-api` for every `/api/` call)## Stack@@@ -33,12 +33,6 @@ module.exports = {cleanupInterval: 5 * 60 * 1000, // every 5 min},-- // Rate limits (per user/IP)-- rateLimit: {-- anonPerHour: 5,-- authedPerHour: 20,-- },--// PathsdataDir: process.env.DATA_DIR || 'data',downloadsDir: process.env.DOWNLOADS_DIR || 'data/downloads',@@@ -9,7 +9,6 @@ const express = require('express');const cors = require('cors');const helmet = require('helmet');const cookieParser = require('cookie-parser');--const rateLimit = require('express-rate-limit');const path = require('path');const fs = require('fs');@@@ -28,7 +27,7 @@ const { requireInternalAccess } = require('../../_shared/internal-token');// ── Guard (apps/_shared/guard) ───────────────────────────────// Who is asking (Network sign-in with aud openvibe.tools, the browser session, else the address),// the tools-download quota (a download costs 50, descriptor yt) and the abuse log (data/guard.db).--// TOOLS_GUARD=report (default) records what it would refuse; the older limiters below stay until enforce.++// TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-yt' });const analytics = new AnalyticsTrackerPg(toolsDb.db, 'openvibe-yt', { retention: { days: 30 } }); // ADR-021; PostgreSQL (plan T8)@@@ -105,18 +104,14 @@ app.use(exceptRegistry(cors({})));// ── Rate Limiting ────────────────────────────────────────────--// Progress is one SSE connection, or a poll every 1–3 s when SSE is unavailable; the poll must--// not eat the general budget (60/min would cut a download off after a minute).// Sign-in is read first, so limits know who is asking; everyone else counts by address (IPv6 by /64).--const isStatusRoute = (req) => req.method === 'GET' && /^\/status\/[a-f0-9]+(\/stream)?$/.test(req.path);++// The guard's tools-api quota covers every /api/ call, the progress polls included.app.use(guard.identify);--const apiLimiter = guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' });--app.use('/api/', (req, res, next) => (isStatusRoute(req) ? next() : apiLimiter(req, res, (err) => (err ? next(err) : guard.apiQuota(req, res, next)))));--app.use('/api/status/', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 240, signedIn: 240, message: 'Too many requests. Please try again later.' }));++app.use('/api/', guard.apiQuota);app.use('/api/', (_req, res, next) => { res.setHeader('Cache-Control', 'no-store'); next(); });// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's YouTube downloader ──--// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The++// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'yt-dlp' ? !!which(config.ytdlpPath) : true)) });// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).@@@ -130,10 +125,9 @@ app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));// ── Analytics Middleware ─────────────────────────────────────app.use(analytics.middleware());--// Downloads: the older hourly limit (5 anonymous, 20 signed in) until enforce mode, and the guard's--// tools-download quota (descriptor yt: cost 50) in both. A download belongs to whoever started it (the--// browser session cookie, minted on the start, or a sign-in): its status, progress stream, file and--// cancel answer 404 to anyone else, the same as for an id that does not exist.++// Downloads: the guard's tools-download quota (descriptor yt: cost 50). A download belongs to whoever++// started it (the browser session cookie, minted on the start, or a sign-in): its status, progress++// stream, file and cancel answer 404 to anyone else, the same as for an id that does not exist.const owners = new Map(); // download id → ownerconst ownsDownload = (req, res, id) => {const owner = owners.get(String(id || ''));@@@ -141,7 +135,6 @@ const ownsDownload = (req, res, id) => {return !!owner && !!who.owner && owner === who.owner;};setInterval(() => { for (const id of owners.keys()) if (!downloader.getStatus(id) && !downloader.getFile(id)) owners.delete(id); }, 10 * 60_000).unref();--const downloadLimiter = guard.legacyLimiter(rateLimit, { windowMs: 60 * 60 * 1000, anonymous: config.rateLimit.anonPerHour, signedIn: config.rateLimit.authedPerHour, message: 'Download limit reached. Sign in for more downloads or wait an hour.' });// ── Hosts ────────────────────────────────────────────────────// Pages are only rendered for hosts this tool serves (or that the gateway vouches for with@@@ -176,7 +169,7 @@ app.post('/api/info', async (req, res) => {});// Start download--app.post('/api/download', downloadLimiter, guard.toolQuota('yt'), async (req, res) => {++app.post('/api/download', guard.toolQuota('yt'), async (req, res) => {const { url, quality, title } = req.body;if (!url) return res.status(400).json({ error: 'URL is required' });if (!downloader.isValidUrl(url)) return res.status(400).json({ error: 'Only YouTube URLs are supported' });
@@ -264,7 +264,7 @@ reload reattaches.
@@ -264,7 +264,7 @@ reload reattaches.
address key), at most
address key), at most
`TOOLS_JOBS_MAX_QUEUED` queued jobs in all (default 200) and `TOOLS_DISK_BUDGET_MB` under `data/jobs` (default
`TOOLS_JOBS_MAX_QUEUED` queued jobs in all (default 200) and `TOOLS_DISK_BUDGET_MB` under `data/jobs` (default
8192) before a submit answers `503 tools.busy` with `Retry-After` (through the guard, below), plus the guard's
8192) before a submit answers `503 tools.busy` with `Retry-After` (through the guard, below), plus the guard's
- quotas and the satellites' older burst and processing limits on submit.
+ quotas on submit.
- **Retention.** Finished jobs expire after 1 hour (browser sessions) or 24 hours (signed-in people, principals);
- **Retention.** Finished jobs expire after 1 hour (browser sessions) or 24 hours (signed-in people, principals);
the pruner deletes the row, its events, its files and its Media objects. It never touches a job that has a
the pruner deletes the row, its events, its files and its Media objects. It never touches a job that has a
reference (above). If Media will not delete a result object (a retention hold, `409 media.object.held`), cannot
reference (above). If Media will not delete a result object (a retention hold, `409 media.object.held`), cannot
@@ -380,12 +380,10 @@ One module used by the gateway and every satellite, driven by each tool's descri
@@ -380,12 +380,10 @@ One module used by the gateway and every satellite, driven by each tool's descri
Calls with a Bearer token, calls without those cookies and requests with neither header pass. Checked on the
Calls with a Bearer token, calls without those cookies and requests with neither header pass. Checked on the
run API, the job routes' writes, `/api/info`, `/api/probe` and the webhook bins' create and
run API, the job routes' writes, `/api/info`, `/api/probe` and the webhook bins' create and
delete; `403 tools.origin.refused`, recorded as reason `origin`.
delete; `403 tools.origin.refused`, recorded as reason `origin`.
-- **Mode.** `TOOLS_GUARD=report` (default) records and counts what it would refuse and refuses nothing, and the
- apps' older express-rate-limit limiters stay in force (now keyed by the resolved caller, after sign-in is read).
- `TOOLS_GUARD=enforce` refuses and the older limiters step aside. (On `/api/v1/…` an older limiter's refusal is
- problem+json `tools.quota.exceeded`, `scope: "legacy"`, with `Retry-After`; job runs through the run API pass the
- same burst and processing limiters as job submits.) Hard limits apply in both modes: the pixel limit,
- ffmpeg's whitelists and duration cap, upload sniffing, the port-scan cap and the per-target throttle.
+- **Mode.** `TOOLS_GUARD=enforce` (default, production) refuses what is past a limit with `429`
+ problem+json `tools.quota.exceeded`. `TOOLS_GUARD=report` (local debugging) records and counts what it
+ would refuse and refuses nothing. Hard limits apply in both modes: the pixel limit, ffmpeg's whitelists
+ and duration cap, upload sniffing, the port-scan cap and the per-target throttle.
Default quotas (units = descriptor `cost` per run; `perMinute` / `burst` / `perDay`, 0 = none):
Default quotas (units = descriptor `cost` per run; `perMinute` / `burst` / `perDay`, 0 = none):
@@ -407,7 +405,7 @@ Environment: `TOOLS_GUARD`, `TOOLS_GUARD_LIMITS` (JSON merged into the table, e.
@@ -407,7 +405,7 @@ Environment: `TOOLS_GUARD`, `TOOLS_GUARD_LIMITS` (JSON merged into the table, e.
## Per-actor limits (`apps/_shared/actor-limits.js`)
## Per-actor limits (`apps/_shared/actor-limits.js`)
Roadmap WS-R task 4, openvibe-sdk/limits, one limiter per app, only where the guard has no per-caller
Roadmap WS-R task 4, openvibe-sdk/limits, one limiter per app, only where the guard has no per-caller
-limit of its own. The guard, its quotas, the older limiters and `TOOLS_GUARD` are unchanged; these limits
+limit of its own. The guard, its quotas and `TOOLS_GUARD` are unchanged; these limits
read the guard's resolved caller (`guard.caller`): a person as `user:usr_…`, a service or app by its
read the guard's resolved caller (`guard.caller`): a person as `user:usr_…`, a service or app by its
principal, a browser session or nobody by the guard's hashed address key (`ip:<HMAC>`, never a raw
principal, a browser session or nobody by the guard's hashed address key (`ip:<HMAC>`, never a raw
address). Past a limit: `429` problem+json `rate_limited` with `Retry-After`, one `[Limits]` log line and
address). Past a limit: `429` problem+json `rate_limited` with `Retry-After`, one `[Limits]` log line and
@@ -659,7 +657,7 @@ with `DATABASE_DIRECT_URL` set for the cutover; it prints a per-table count and
@@ -659,7 +657,7 @@ with `DATABASE_DIRECT_URL` set for the cutover; it prints a per-table count and
After the move the apps read `DATABASE_URL` / `DATABASE_DIRECT_URL` (PostgreSQL, via PgBouncer) and
After the move the apps read `DATABASE_URL` / `DATABASE_DIRECT_URL` (PostgreSQL, via PgBouncer) and
`VALKEY_URL` (Valkey); `guard_salt`, `guard_day` and the minute/burst buckets live in Valkey (a throttle,
`VALKEY_URL` (Valkey); `guard_salt`, `guard_day` and the minute/burst buckets live in Valkey (a throttle,
never authoritative), while `guard_abuse`, the job store and the analytics tables stay in PostgreSQL.
never authoritative), while `guard_abuse`, the job store and the analytics tables stay in PostgreSQL.
-`TOOLS_GUARD` (default `report`) and its meaning are unchanged.
+`TOOLS_GUARD` (default `enforce`) and its meaning are unchanged.
State (plan T8 landed 2026-10-02; production serves the PostgreSQL-only release): every app opens the one `tools`
State (plan T8 landed 2026-10-02; production serves the PostgreSQL-only release): every app opens the one `tools`
database (`apps/_shared/db.js`, `DATABASE_URL` else an embedded PGlite database in development;
database (`apps/_shared/db.js`, `DATABASE_URL` else an embedded PGlite database in development;
@@ -2,8 +2,8 @@
@@ -2,8 +2,8 @@
// ═══════════════════════════════════════════════════════════════
// ═══════════════════════════════════════════════════════════════
// Per-actor rate limits (roadmap WS-R task 4; openvibe-sdk/limits), one limiter per app, only where
// Per-actor rate limits (roadmap WS-R task 4; openvibe-sdk/limits), one limiter per app, only where
// the guard (./guard) has no per-caller limit of its own. The guard is not changed: its tiered quotas
// the guard (./guard) has no per-caller limit of its own. The guard is not changed: its tiered quotas
-// (quotaClass × tier × cost, day allowances), the older express-rate-limit limiters, the per-target
-// throttle and TOOLS_GUARD's report/enforce mode stay exactly as they are. These limits read the
+// (quotaClass × tier × cost, day allowances), the per-target throttle and TOOLS_GUARD's report/enforce
+// mode stay exactly as they are. These limits read the
// guard's resolved caller (guard.caller) and add:
// guard's resolved caller (guard.caller) and add:
//
//
// registry reads GET/HEAD /api/v1/tools[/:id[/schema]] by a signed-in person or a third-party
// registry reads GET/HEAD /api/v1/tools[/:id[/schema]] by a signed-in person or a third-party
@@ -15,7 +15,7 @@
@@ -15,7 +15,7 @@
// backstop job submits and retries (POST /api/v1/jobs, /api/v1/jobs/:id/retry) and runs
// backstop job submits and retries (POST /api/v1/jobs, /api/v1/jobs/:id/retry) and runs
// (POST /api/v1/tools/:id/run): a ceiling ABOVE the guard's quotas, so it never
// (POST /api/v1/tools/:id/run): a ceiling ABOVE the guard's quotas, so it never
// decides a normal caller's allowance (the guard does) and still stops a runaway
// decides a normal caller's allowance (the guard does) and still stops a runaway
-// one while the guard only reports (TOOLS_GUARD=report, the default).
+// one while the guard only reports (TOOLS_GUARD=report, for local debugging).
// admin the gateway's /api/internal/analytics (the Network admin's analytics page).
// admin the gateway's /api/internal/analytics (the Network admin's analytics page).
//
//
// Counted: a person as user:usr_…, a service or app by its principal (svc:…, app:…), a browser session
// Counted: a person as user:usr_…, a service or app by its principal (svc:…, app:…), a browser session
@@ -8,7 +8,7 @@
@@ -8,7 +8,7 @@
// const guard = createGuard({ app: 'img', dataDir, db, valkey, contracts, specs, networkUrl, … });
// const guard = createGuard({ app: 'img', dataDir, db, valkey, contracts, specs, networkUrl, … });
// app.set('trust proxy', TRUST_PROXY); // ./ip.js: one loopback hop
// app.set('trust proxy', TRUST_PROXY); // ./ip.js: one loopback hop
// app.use(guard.identify); // req.user (Network sign-in, aud checked)
// app.use(guard.identify); // req.user (Network sign-in, aud checked)
-// app.use('/api/', legacyLimiter, guard.apiQuota); // every /api/ request (tools-api)
+// app.use('/api/', guard.apiQuota); // every /api/ request (tools-api)
// app.post('/api/v1/tools/:id/run', guard.toolQuota(hostTool), upload, guard.admitUpload(toolOf), guard.heavy(), …)
// app.post('/api/v1/tools/:id/run', guard.toolQuota(hostTool), upload, guard.admitUpload(toolOf), guard.heavy(), …)
//
//
// What it does:
// What it does:
@@ -30,11 +30,10 @@
@@ -30,11 +30,10 @@
// own hosts (custom domains included) or the request's own host (CSRF); calls with a
// own hosts (custom domains included) or the request's own host (CSRF); calls with a
// Bearer token and calls without those cookies do not need it. 403 tools.origin.refused
// Bearer token and calls without those cookies do not need it. 403 tools.origin.refused
//
//
-// Mode: TOOLS_GUARD=report (default) logs and counts what it would refuse and refuses nothing, except
-// the hard limits that apply in both modes: the image pixel limit, ffmpeg's protocol and format
-// whitelists and duration cap, upload sniffing, the port-scan cap and the per-target throttle.
-// TOOLS_GUARD=enforce refuses. The apps' older per-route limiters keep working in report mode and step
-// aside in enforce mode (guard.enforcing), where the tiered quotas replace them.
+// Mode: TOOLS_GUARD=enforce (default, production) refuses what is past a limit. TOOLS_GUARD=report
+// (local debugging) logs and counts what it would refuse and refuses nothing, except the hard limits
+// that apply in both modes: the image pixel limit, ffmpeg's protocol and format whitelists and
+// duration cap, upload sniffing, the port-scan cap and the per-target throttle.
// ═══════════════════════════════════════════════════════════════
// ═══════════════════════════════════════════════════════════════
const fs = require('fs');
const fs = require('fs');
@@ -97,7 +96,7 @@ function createGuard(o = {}) {
@@ -97,7 +96,7 @@ function createGuard(o = {}) {
const log = o.log || console;
const log = o.log || console;
const now = o.now || Date.now;
const now = o.now || Date.now;
const appName = o.app || 'tools';
const appName = o.app || 'tools';
- const mode = String(env.TOOLS_GUARD || 'report').trim().toLowerCase() === 'enforce' ? 'enforce' : 'report';
+ const mode = String(env.TOOLS_GUARD || 'enforce').trim().toLowerCase() === 'enforce' ? 'enforce' : 'report';
const enforcing = mode === 'enforce';
const enforcing = mode === 'enforce';
const bounds = limits.bounds(env);
const bounds = limits.bounds(env);
const table = limits.quotas(env, log);
const table = limits.quotas(env, log);
@@ -207,9 +206,8 @@ function createGuard(o = {}) {
@@ -207,9 +206,8 @@ function createGuard(o = {}) {
async function charge(req, res, { quotaClass, cost, tool: toolId }) {
async function charge(req, res, { quotaClass, cost, tool: toolId }) {
const c = caller(req, res);
const c = caller(req, res);
const r = await quotas.check(c, { quotaClass, cost }, { report: !enforcing });
const r = await quotas.check(c, { quotaClass, cost }, { report: !enforcing });
- // In report mode an older limiter that ran on this request (req.rateLimit) is the one in force:
- // its RateLimit-* headers stay. Otherwise the guard's describe the allowance closest to running out.
- if (enforcing || !req.rateLimit) setHeaders(res, r);
+ // The guard's RateLimit-* headers describe the allowance closest to running out.
+ setHeaders(res, r);
if (r.ok) return true;
if (r.ok) return true;
if (enforcing && (c.tier === 'anonymous' || c.tier === 'session') && challenge.required(req, c, { reason: 'quota', tool: toolId })) {
if (enforcing && (c.tier === 'anonymous' || c.tier === 'session') && challenge.required(req, c, { reason: 'quota', tool: toolId })) {
if (await challenge.verify(req, c)) return true;
if (await challenge.verify(req, c)) return true;
@@ -427,47 +425,7 @@ function createGuard(o = {}) {
@@ -427,47 +425,7 @@ function createGuard(o = {}) {
return false;
return false;
}
}
- // ── Older limiters, and tool errors ──────────────────────
- /**
- * One of the apps' older express-rate-limit limiters, keyed by the resolved caller: a signed-in
- * person or a principal by who they are (their higher number applies — these limiters used to run
- * before sign-in was read), everyone else by address (IPv6 /64, hashed). Skipped in enforce mode.
- * @param {Function} rateLimit require('express-rate-limit')
- */
- function legacyLimiter(rateLimit, { windowMs, anonymous, signedIn = anonymous, message, headers = true }) {
- const signed = (c) => c.tier === 'user' || c.tier === 'service';
- return rateLimit({
- windowMs,
- max: (req) => (signed(caller(req)) ? signedIn : anonymous),
- standardHeaders: headers,
- legacyHeaders: false,
- keyGenerator: (req) => { const c = caller(req); return signed(c) || c.tier === 'sandbox' ? c.key : c.ipKey; },
- skip: (req) => enforcing || isProbe(req), // liveness and readiness probes are never limited
- message: { error: message },
- // The platform API (/api/v1/…) answers like the guard does: problem+json tools.quota.exceeded
- // with Retry-After; the older routes keep their { error } shape.
- handler: (req, res, _next, options) => {
- const reset = req.rateLimit && req.rateLimit.resetTime ? new Date(req.rateLimit.resetTime).getTime() : 0;
- const retry = Math.max(1, Math.ceil(((reset || (now() + windowMs)) - now()) / 1000));
- res.setHeader('Retry-After', String(retry));
- if (/^\/api\/v1\//.test(String(req.originalUrl || req.url || ''))) {
- const c = caller(req);
- return sendProblem(req, res, 429, 'tools.quota.exceeded', message, { scope: 'legacy', tier: c.tier, retry_after: retry }, o.contracts);
- }
- return res.status(options.statusCode).json(options.message);
- },
- });
- }
- /** The three limiters an upload app had (LEGACY numbers in ./limits.js). */
- function legacyLimiters(rateLimit, numbers) {
- const n = numbers || limits.LEGACY[appName];
- return {
- apiLimiter: legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: n.api[0], signedIn: n.api[1], message: 'Too many requests. Please try again later.' }),
- processLimiter: legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: n.process[0], signedIn: n.process[1], message: 'Processing rate limit reached. Sign in for higher limits or wait a moment.' }),
- burstLimiter: legacyLimiter(rateLimit, { windowMs: 5_000, anonymous: n.burst, signedIn: n.burst, headers: false, message: 'Too many requests in quick succession. Please slow down.' }),
- };
- }
-
+ // ── Tool errors ──────────────────────────────────────────
/**
/**
* A tool refused its input on a hard limit (err.guardReason: 'pixels', 'ffmpeg.format',
* A tool refused its input on a hard limit (err.guardReason: 'pixels', 'ffmpeg.format',
* 'ffmpeg.duration', 'timeout'): record it and answer its problem. → true when answered.
* 'ffmpeg.duration', 'timeout'): record it and answer its problem. → true when answered.
@@ -503,7 +461,7 @@ function createGuard(o = {}) {
@@ -503,7 +461,7 @@ function createGuard(o = {}) {
checkFiles, admitUpload, admitJob, uploadsOf, discardUploads,
checkFiles, admitUpload, admitJob, uploadsOf, discardUploads,
heavy, sync, jobsBusy,
heavy, sync, jobsBusy,
target, portScan, targets, ports, normalizeTarget,
target, portScan, targets, ports, normalizeTarget,
- legacyLimiter, legacyLimiters, toolRefused,
+ toolRefused,
originOk, originCheck, allowedOrigin, tokens,
originOk, originCheck, allowedOrigin, tokens,
attachMetrics, prune,
attachMetrics, prune,
close() { if (pruneTimer) clearInterval(pruneTimer); if (keys.stop) keys.stop(); store.close(); },
close() { if (pruneTimer) clearInterval(pruneTimer); if (keys.stop) keys.stop(); store.close(); },
@@ -64,15 +64,6 @@ const QUOTAS = {
@@ -64,15 +64,6 @@ const QUOTAS = {
// dropping the cookie starts a new session, never a new allowance.
// dropping the cookie starts a new session, never a new allowance.
const SESSION_IP_SHARE = 3;
const SESSION_IP_SHARE = 3;
-// The apps' older per-route limiters (express-rate-limit, requests per window per address, or per
-// person when signed in). They stay in force in report mode and step aside with TOOLS_GUARD=enforce,
-// where the tiered quotas above take over. [anonymous or session, signed in or service].
-const LEGACY = {
- img: { api: [60, 120], process: [10, 30], burst: 4 },
- audio: { api: [60, 120], process: [6, 20], burst: 3 },
- docs: { api: [60, 120], process: [10, 30], burst: 4 },
-};
-
function envInt(name, fallback, env = process.env) {
function envInt(name, fallback, env = process.env) {
const v = parseInt(env[name], 10);
const v = parseInt(env[name], 10);
return Number.isFinite(v) && v >= 0 ? v : fallback;
return Number.isFinite(v) && v >= 0 ? v : fallback;
@@ -128,4 +119,4 @@ function quotas(env = process.env, log = console) {
@@ -128,4 +119,4 @@ function quotas(env = process.env, log = console) {
}
}
}
}
-module.exports = { TIERS, QUOTAS, SESSION_IP_SHARE, LEGACY, bounds, quotas, merge };
+module.exports = { TIERS, QUOTAS, SESSION_IP_SHARE, bounds, quotas, merge };
@@ -59,7 +59,7 @@ const REPLAY_TTL_MS = 15 * 60 * 1000;
@@ -59,7 +59,7 @@ const REPLAY_TTL_MS = 15 * 60 * 1000;
const isRunPath = (p) => RUN_RE.test(String(p || ''));
const isRunPath = (p) => RUN_RE.test(String(p || ''));
-/** Run Express middleware in order (the apps' older limiters) → true when every one called next(). */
+/** Run Express middleware in order (a route's limiters) → true when every one called next(). */
async function passes(list, req, res) {
async function passes(list, req, res) {
for (const mw of list || []) {
for (const mw of list || []) {
const ok = await new Promise((resolve) => {
const ok = await new Promise((resolve) => {
@@ -454,7 +454,7 @@ function createRunApi(o) {
@@ -454,7 +454,7 @@ function createRunApi(o) {
// The tool's quota class and cost (recorded for the job admission, so it is not charged twice).
// The tool's quota class and cost (recorded for the job admission, so it is not charged twice).
req._ovCharged = { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id };
req._ovCharged = { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id };
if (!(await guard.charge(req, res, { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id }))) return undefined;
if (!(await guard.charge(req, res, { quotaClass: d.quotaClass, cost: d.cost || 1, tool: d.id }))) return undefined;
- // A satellite's older per-route limiters count job runs like its job submits (report mode: in force).
+ // The satellite's limiters (the actor backstop, the tool's quota) count a run like a job submit.
if (mode === 'job' && o.jobs.limiters && !(await passes(o.jobs.limiters, req, res))) return undefined;
if (mode === 'job' && o.jobs.limiters && !(await passes(o.jobs.limiters, req, res))) return undefined;
const multipart = req.is && req.is('multipart/form-data');
const multipart = req.is && req.is('multipart/form-data');
if (multipart && mode !== 'job') throw new Refusal(400, 'tools.run.invalid', `${d.id} takes no files: send JSON (tools.run-request@1).`);
if (multipart && mode !== 'job') throw new Refusal(400, 'tools.run.invalid', `${d.id} takes no files: send JSON (tools.run-request@1).`);
@@ -572,7 +572,7 @@ function createSiblingFetch({ self, ports, fetchImpl = fetch }) {
@@ -572,7 +572,7 @@ function createSiblingFetch({ self, ports, fetchImpl = fetch }) {
* @param {Function} o.multer @param {string} o.uploadsDir
* @param {Function} o.multer @param {string} o.uploadsDir
* @param {Function} o.Ajv @param {Function} [o.addFormats]
* @param {Function} o.Ajv @param {Function} [o.addFormats]
* @param {() => object} o.ports satellite ports (./satellites.js)
* @param {() => object} o.ports satellite ports (./satellites.js)
- * @param {Function[]} [o.limiters] the app's older burst and processing limiters (as on its job submits)
+ * @param {Function[]} [o.limiters] the app's limiters (as on its job submits: the actor backstop, the tool's quota)
*/
*/
function satelliteRunApi(o) {
function satelliteRunApi(o) {
const uploadsDir = path.resolve(o.uploadsDir);
const uploadsDir = path.resolve(o.uploadsDir);
@@ -13,7 +13,6 @@ const cookieParser = require('cookie-parser');
@@ -13,7 +13,6 @@ const cookieParser = require('cookie-parser');
const path = require('path');
const path = require('path');
const fs = require('fs');
const fs = require('fs');
-const rateLimit = require('express-rate-limit');
const config = require('./config');
const config = require('./config');
const { resolveContext, DOMAIN_MAP } = require('./domain-map');
const { resolveContext, DOMAIN_MAP } = require('./domain-map');
const { getTool, listTools } = require('./tools');
const { getTool, listTools } = require('./tools');
@@ -40,8 +39,8 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
@@ -40,8 +39,8 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
// ── Guard (apps/_shared/guard) ───────────────────────────────
// ── Guard (apps/_shared/guard) ───────────────────────────────
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the
// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the
-// abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse. Every audio
-// tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).
+// abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would
+// refuse. Every audio tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).
const SPECS = require('./descriptors').SPECS;
const SPECS = require('./descriptors').SPECS;
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-audio' });
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-audio' });
@@ -51,7 +50,6 @@ const guard = createGuard({
@@ -51,7 +50,6 @@ const guard = createGuard({
app: 'audio', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,
app: 'audio', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,
issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,
issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,
});
});
-const { apiLimiter, processLimiter, burstLimiter } = guard.legacyLimiters(rateLimit);
/** The tool a request is for before its body is read: the host's own, else the operation it defaults to. */
/** The tool a request is for before its body is read: the host's own, else the operation it defaults to. */
const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('audio.process', req.ctx.defaultOp || 'convert', null) || { id: null }).id);
const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('audio.process', req.ctx.defaultOp || 'convert', null) || { id: null }).id);
/** The tool a parsed request runs: its `tool` (operation) on this host. */
/** The tool a parsed request runs: its `tool` (operation) on this host. */
@@ -123,12 +121,12 @@ app.use(exceptRegistry(cors({
@@ -123,12 +121,12 @@ app.use(exceptRegistry(cors({
credentials: true,
credentials: true,
}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
-// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
+// ── Who is asking, then the guard's quotas ───────────────────
app.use(guard.identify);
app.use(guard.identify);
-app.use('/api/', apiLimiter, guard.apiQuota);
+app.use('/api/', guard.apiQuota);
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's audio tools ──
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's audio tools ──
-// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'ffmpeg' ? !!which(process.env.FFMPEG_PATH || 'ffmpeg') : true)) });
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'ffmpeg' ? !!which(process.env.FFMPEG_PATH || 'ffmpeg') : true)) });
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
@@ -189,7 +187,7 @@ app.get('/api/tools', (_req, res) => {
@@ -189,7 +187,7 @@ app.get('/api/tools', (_req, res) => {
});
});
// ── Probe / Metadata Endpoint ────────────────────────────────
// ── Probe / Metadata Endpoint ────────────────────────────────
-app.post('/api/probe', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {
+app.post('/api/probe', guard.originCheck(), guard.toolQuota('metadata'), uploadSingle, guard.admitUpload(() => 'metadata'), async (req, res) => {
try {
try {
// Hardened like every run: local files only, audio demuxers only.
// Hardened like every run: local files only, audio demuxers only.
const { info, meta } = await jobContext.run(limitsFor('metadata'), async () => ({ info: await probe(req.file.path), meta: await readMetadata(req.file.path) }));
const { info, meta } = await jobContext.run(limitsFor('metadata'), async () => ({ info: await probe(req.file.path), meta: await readMetadata(req.file.path) }));
@@ -210,7 +208,7 @@ const jobs = jobsRuntime.setupJobs({
@@ -210,7 +208,7 @@ const jobs = jobsRuntime.setupJobs({
getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,
getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,
define: defineJobs,
define: defineJobs,
receive: uploadAny,
receive: uploadAny,
- limiters: [actorLimits.backstop('tools.job.create'), burstLimiter, processLimiter, guard.toolQuota(hostTool)],
+ limiters: [actorLimits.backstop('tools.job.create'), guard.toolQuota(hostTool)],
jobTool: (req, type, input) => { const d = guard.toolForJob('audio.process', String((input && input.tool) || req.ctx.defaultOp || 'convert'), req.ctx.toolId); return d ? d.id : null; },
jobTool: (req, type, input) => { const d = guard.toolForJob('audio.process', String((input && input.tool) || req.ctx.defaultOp || 'convert'), req.ctx.toolId); return d ? d.id : null; },
defaults(req, input) {
defaults(req, input) {
const out = { ...input };
const out = { ...input };
@@ -227,7 +225,7 @@ const jobs = jobsRuntime.setupJobs({
@@ -227,7 +225,7 @@ const jobs = jobsRuntime.setupJobs({
const runApi = satelliteRunApi({
const runApi = satelliteRunApi({
app: 'audio', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
app: 'audio', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
- limiters: [actorLimits.backstop('tools.tool.run'), burstLimiter, processLimiter],
+ limiters: [actorLimits.backstop('tools.tool.run')],
});
});
app.use(runApi.handle);
app.use(runApi.handle);
@@ -22,13 +22,13 @@ async function waitFor(app, id, cookie, pred, what, ms = 30000) {
@@ -22,13 +22,13 @@ async function waitFor(app, id, cookie, pred, what, ms = 30000) {
const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();
const job = await (await fetch(`${app.base}/api/v1/jobs/${id}`, { headers: { cookie } })).json();
if (pred(job)) return job;
if (pred(job)) return job;
if (Date.now() - t0 > ms) throw new Error(`timed out waiting for ${what}: ${JSON.stringify(job)}`);
if (Date.now() - t0 > ms) throw new Error(`timed out waiting for ${what}: ${JSON.stringify(job)}`);
- await sleep(150); // the older /api/ limiter allows an address 60 requests a minute
+ await sleep(150); // the guard's tools-api quota allows an address 120 requests a minute
}
}
}
}
(async () => {
(async () => {
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-audio-'));
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-audio-'));
- const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output') };
+ const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: 'report' }; // a session-less submit is the point here; enforce would refuse it 401
// Three minutes of tone: small to upload, a couple of seconds to re-encode.
// Three minutes of tone: small to upload, a couple of seconds to re-encode.
const input = path.join(data, 'tone.mp3');
const input = path.join(data, 'tone.mp3');
execFileSync('ffmpeg', ['-loglevel', 'error', '-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=180', '-ac', '1', '-b:a', '16k', input]);
execFileSync('ffmpeg', ['-loglevel', 'error', '-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=180', '-ac', '1', '-b:a', '16k', input]);
@@ -21,7 +21,7 @@ const seconds = (file) => parseFloat(execFileSync('ffprobe', ['-v', 'error', '-s
@@ -21,7 +21,7 @@ const seconds = (file) => parseFloat(execFileSync('ffprobe', ['-v', 'error', '-s
(async () => {
(async () => {
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-audio-merge-'));
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-audio-merge-'));
process.env.DATA_DIR = data; // the tool's temp files go here, not into the app directory
process.env.DATA_DIR = data; // the tool's temp files go here, not into the app directory
- const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output') };
+ const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: 'report' }; // the merge job is submitted without a session; enforce would refuse it 401
// Different formats, rates and channel counts: a 1 s mono MP3 at 44.1 kHz and a 0.5 s stereo WAV at 22.05 kHz.
// Different formats, rates and channel counts: a 1 s mono MP3 at 44.1 kHz and a 0.5 s stereo WAV at 22.05 kHz.
const mp3 = path.join(data, 'one.mp3'), wav = path.join(data, 'two.wav');
const mp3 = path.join(data, 'one.mp3'), wav = path.join(data, 'two.wav');
execFileSync('ffmpeg', ['-loglevel', 'error', '-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=1', '-ac', '1', '-ar', '44100', mp3]);
execFileSync('ffmpeg', ['-loglevel', 'error', '-y', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=1', '-ac', '1', '-ar', '44100', mp3]);
@@ -14,7 +14,6 @@ const path = require('path');
@@ -14,7 +14,6 @@ const path = require('path');
const fs = require('fs');
const fs = require('fs');
const fsp = require('fs/promises');
const fsp = require('fs/promises');
-const rateLimit = require('express-rate-limit');
const config = require('./config');
const config = require('./config');
const { resolveContext, DOMAIN_MAP } = require('./domain-map');
const { resolveContext, DOMAIN_MAP } = require('./domain-map');
const { getTool, listTools, BINARIES } = require('./tools');
const { getTool, listTools, BINARIES } = require('./tools');
@@ -40,8 +39,8 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
@@ -40,8 +39,8 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
// ── Guard (apps/_shared/guard) ───────────────────────────────
// ── Guard (apps/_shared/guard) ───────────────────────────────
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the
// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the
-// abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse. Every PDF
-// tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).
+// abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would
+// refuse. Every PDF tool needs a browser session, a sign-in or a token (descriptor auth.anonymous false).
const SPECS = require('./descriptors').SPECS;
const SPECS = require('./descriptors').SPECS;
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-docs' });
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-docs' });
@@ -51,7 +50,6 @@ const guard = createGuard({
@@ -51,7 +50,6 @@ const guard = createGuard({
app: 'docs', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,
app: 'docs', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,
issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,
issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,
});
});
-const { apiLimiter, processLimiter, burstLimiter } = guard.legacyLimiters(rateLimit);
/** The tool a request is for before its body is read: the host's own (the hubs: merge, the first). */
/** The tool a request is for before its body is read: the host's own (the hubs: merge, the first). */
const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('docs.process', req.ctx.defaultOp || 'merge', null) || { id: null }).id);
const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('docs.process', req.ctx.defaultOp || 'merge', null) || { id: null }).id);
/** The tool a parsed request runs: its `tool` (operation) on this host. */
/** The tool a parsed request runs: its `tool` (operation) on this host. */
@@ -139,12 +137,12 @@ app.use(exceptRegistry(cors({
@@ -139,12 +137,12 @@ app.use(exceptRegistry(cors({
credentials: true,
credentials: true,
}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
-// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
+// ── Who is asking, then the guard's quotas ───────────────────
app.use(guard.identify);
app.use(guard.identify);
-app.use('/api/', apiLimiter, guard.apiQuota);
+app.use('/api/', guard.apiQuota);
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's PDF tools ──
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's PDF tools ──
-// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
const pdfPrograms = require('./tools/pdf');
const pdfPrograms = require('./tools/pdf');
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (pdfPrograms[p] ? pdfPrograms[p].available() : true)) });
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (pdfPrograms[p] ? pdfPrograms[p].available() : true)) });
@@ -218,7 +216,7 @@ app.get('/api/tools', (_req, res) => {
@@ -218,7 +216,7 @@ app.get('/api/tools', (_req, res) => {
});
});
// ── PDF Info (no file mutation) ──────────────────────────────
// ── PDF Info (no file mutation) ──────────────────────────────
-app.post('/api/info', guard.originCheck(), burstLimiter, processLimiter, guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {
+app.post('/api/info', guard.originCheck(), guard.toolQuota(hostTool), uploadSingle, guard.admitUpload(hostTool), guard.heavy(hostTool), async (req, res) => {
try {
try {
const tool = getTool('metadata');
const tool = getTool('metadata');
const [buffer] = await readUploads([req.file]);
const [buffer] = await readUploads([req.file]);
@@ -237,7 +235,7 @@ const jobs = jobsRuntime.setupJobs({
@@ -237,7 +235,7 @@ const jobs = jobsRuntime.setupJobs({
getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,
getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,
define: (system) => defineJobs(system, pool),
define: (system) => defineJobs(system, pool),
receive: uploadAny,
receive: uploadAny,
- limiters: [actorLimits.backstop('tools.job.create'), burstLimiter, processLimiter, guard.toolQuota(hostTool)],
+ limiters: [actorLimits.backstop('tools.job.create'), guard.toolQuota(hostTool)],
jobTool: (req, type, input) => { const op = String((input && input.tool) || req.ctx.defaultOp || ''); const d = op ? guard.toolForJob('docs.process', op, req.ctx.toolId) : null; return d ? d.id : null; },
jobTool: (req, type, input) => { const op = String((input && input.tool) || req.ctx.defaultOp || ''); const d = op ? guard.toolForJob('docs.process', op, req.ctx.toolId) : null; return d ? d.id : null; },
defaults(req, input) {
defaults(req, input) {
const out = { ...input };
const out = { ...input };
@@ -254,7 +252,7 @@ const jobs = jobsRuntime.setupJobs({
@@ -254,7 +252,7 @@ const jobs = jobsRuntime.setupJobs({
const runApi = satelliteRunApi({
const runApi = satelliteRunApi({
app: 'docs', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
app: 'docs', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
- limiters: [actorLimits.backstop('tools.tool.run'), burstLimiter, processLimiter],
+ limiters: [actorLimits.backstop('tools.tool.run')],
});
});
app.use(runApi.handle);
app.use(runApi.handle);
@@ -24,7 +24,7 @@ async function makePdf(pages) {
@@ -24,7 +24,7 @@ async function makePdf(pages) {
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-guard-'));
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-guard-'));
const uploads = path.join(data, 'uploads');
const uploads = path.join(data, 'uploads');
const app = await startApp('docs', { DATA_DIR: data, UPLOADS_DIR: uploads, OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: mode });
const app = await startApp('docs', { DATA_DIR: data, UPLOADS_DIR: uploads, OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: mode });
- // Each call from its own address, so the older burst limiter (4 in 5 s per address) stays out of it.
+ // Each call from its own address, so the per-address quota never sees two calls in a row.
let n = 0;
let n = 0;
const post = (id, files, input, cookie) => {
const post = (id, files, input, cookie) => {
const f = new FormData();
const f = new FormData();
@@ -29,7 +29,7 @@ async function finished(app, id, cookie) {
@@ -29,7 +29,7 @@ async function finished(app, id, cookie) {
(async () => {
(async () => {
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-'));
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-'));
- const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output') };
+ const env = { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_GUARD: 'report' }; // submits here carry no session; enforce would refuse them 401
const a = await pdf(2, 'First'), b = await pdf(3);
const a = await pdf(2, 'First'), b = await pdf(3);
let app;
let app;
try {
try {
@@ -143,7 +143,7 @@ const is503 = (e) => e.status === 503 && e.code === 'tools.unavailable' && /bein
@@ -143,7 +143,7 @@ const is503 = (e) => e.status === 503 && e.code === 'tools.unavailable' && /bein
// ── As a real process without qpdf: the API answers 503 and the page is told ──
// ── As a real process without qpdf: the API answers 503 and the page is told ──
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-tools-'));
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-docs-tools-'));
- const app = await startApp('docs', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), QPDF_PATH: '/nonexistent/qpdf' });
+ const app = await startApp('docs', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), QPDF_PATH: '/nonexistent/qpdf', TOOLS_GUARD: 'report' }); // no session here: enforce would answer 401 before the tool's 503
try {
try {
const fd = new FormData();
const fd = new FormData();
fd.append('type', 'docs.process');
fd.append('type', 'docs.process');
@@ -6,7 +6,6 @@ require('dotenv').config();
@@ -6,7 +6,6 @@ require('dotenv').config();
const express = require('express');
const express = require('express');
const cors = require('cors');
const cors = require('cors');
const helmet = require('helmet');
const helmet = require('helmet');
-const rateLimit = require('express-rate-limit');
const path = require('path');
const path = require('path');
const http = require('http');
const http = require('http');
const fs = require('fs');
const fs = require('fs');
@@ -23,7 +22,7 @@ const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
@@ -23,7 +22,7 @@ const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
// ── Guard (apps/_shared/guard) ───────────────────────────────
// ── Guard (apps/_shared/guard) ───────────────────────────────
// Who is asking (Network sign-in with aud openvibe.tools, else the address), the tools-api quota and
// Who is asking (Network sign-in with aud openvibe.tools, else the address), the tools-api quota and
// the abuse log (data/guard.db). The data lookups are counted by maps (tools-map), which gets the
// the abuse log (data/guard.db). The data lookups are counted by maps (tools-map), which gets the
-// visitor's address from here. TOOLS_GUARD=report (default) records what it would refuse.
+// visitor's address from here. TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.
const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';
const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-food' });
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-food' });
@@ -94,11 +93,10 @@ app.use(helmet({
@@ -94,11 +93,10 @@ app.use(helmet({
// Sign-in first, so limits know who is asking; everyone else counts by address (IPv6 /64).
// Sign-in first, so limits know who is asking; everyone else counts by address (IPv6 /64).
app.use(cookieParser());
app.use(cookieParser());
app.use(guard.identify);
app.use(guard.identify);
-app.use(guard.legacyLimiter(rateLimit, { windowMs: 60000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }));
app.use('/api/', guard.apiQuota);
app.use('/api/', guard.apiQuota);
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's food finder ──
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's food finder ──
-// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// The food finder is built on the maps backend; its descriptor lives with the maps app's.
// The food finder is built on the maps backend; its descriptor lives with the maps app's.
const toolRegistry = createLocalRegistry({ specs: require('../../maps/server/descriptors').SPECS.filter(s => s.id === 'food') });
const toolRegistry = createLocalRegistry({ specs: require('../../maps/server/descriptors').SPECS.filter(s => s.id === 'food') });
@@ -1,6 +1,6 @@
@@ -1,6 +1,6 @@
'use strict';
'use strict';
// Food proxies every API call to maps. It used to call maps without the visitor's address, so maps'
// Food proxies every API call to maps. It used to call maps without the visitor's address, so maps'
-// per-IP limit (30/min) was one bucket shared by every food visitor (127.0.0.1). Now food forwards
+// per-caller quota was one bucket shared by every food visitor (127.0.0.1). Now food forwards
// X-Forwarded-For / X-Real-IP from req.ip, and maps believes X-Forwarded-For from one loopback hop only.
// X-Forwarded-For / X-Real-IP from req.ip, and maps believes X-Forwarded-For from one loopback hop only.
const assert = require('assert');
const assert = require('assert');
const fs = require('fs');
const fs = require('fs');
@@ -12,10 +12,10 @@ const { startApp } = require('../../_shared/test/spawn');
@@ -12,10 +12,10 @@ const { startApp } = require('../../_shared/test/spawn');
const food = await startApp('food', { MAPS_API: maps.base }, undefined, { readyPath: '/api/ready' });
const food = await startApp('food', { MAPS_API: maps.base }, undefined, { readyPath: '/api/ready' });
try {
try {
const ask = (ip) => fetch(`${food.base}/api/foods?search=rice`, { headers: { 'X-Forwarded-For': ip } });
const ask = (ip) => fetch(`${food.base}/api/foods?search=rice`, { headers: { 'X-Forwarded-For': ip } });
- // One visitor uses up maps' budget…
+ // One visitor uses up maps' tools-map quota (descriptor maps: cost 2; anonymous burst 40 units = 20 calls)…
let limited = null;
let limited = null;
for (let i = 0; i < 40 && !limited; i++) { const r = await ask('203.0.113.1'); if (r.status === 429) limited = i; else assert.strictEqual(r.status, 200); }
for (let i = 0; i < 40 && !limited; i++) { const r = await ask('203.0.113.1'); if (r.status === 429) limited = i; else assert.strictEqual(r.status, 200); }
- assert.strictEqual(limited, 30, 'maps limits that visitor after 30 requests a minute');
+ assert.ok(limited >= 19 && limited <= 21, `maps limits that visitor after its burst of 20 (got ${limited})`);
// …and another visitor is not affected.
// …and another visitor is not affected.
const other = await ask('203.0.113.2');
const other = await ask('203.0.113.2');
assert.strictEqual(other.status, 200, 'a second visitor has their own bucket');
assert.strictEqual(other.status, 200, 'a second visitor has their own bucket');
@@ -46,8 +46,9 @@ MEDIA_PUBLIC_URL=https://openvibe.media
@@ -46,8 +46,9 @@ MEDIA_PUBLIC_URL=https://openvibe.media
#TOOLS_LIMITS_MINUTE=120
#TOOLS_LIMITS_MINUTE=120
#TOOLS_LIMITS_HOUR=3000
#TOOLS_LIMITS_HOUR=3000
-# The shared guard's mode (apps/_shared/guard/index.js): today's default is 'report' (log only); T8 step 2 flips this to enforce.
-#TOOLS_GUARD=enforce
+# The shared guard's mode (apps/_shared/guard/index.js): 'enforce' (default, production) refuses what is
+# past a limit; set 'report' for local debugging to log what it would refuse instead.
+#TOOLS_GUARD=report
# ── Satellites: jobs (img, audio, docs read the same tools.env) ──
# ── Satellites: jobs (img, audio, docs read the same tools.env) ──
#TOOLS_JOBS_CONCURRENCY=2
#TOOLS_JOBS_CONCURRENCY=2
@@ -57,7 +57,8 @@ let auth = null; // created below; the key check reads it at request time
@@ -57,7 +57,8 @@ let auth = null; // created below; the key check reads it at request time
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// the address), the tools-api quota, the net and dev tools' quotas by descriptor, the per-target
// the address), the tools-api quota, the net and dev tools' quotas by descriptor, the per-target
// throttle, the port-scan cap, webhook bin caps and the abuse log (data/guard.db). The Network key is
// throttle, the port-scan cap, webhook bin caps and the abuse log (data/guard.db). The Network key is
-// the OAuth client's (loaded and retried by it). TOOLS_GUARD=report (default) records what it would refuse.
+// the OAuth client's (loaded and retried by it). TOOLS_GUARD=enforce (default) refuses; report records
+// what it would refuse.
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-gateway' });
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-gateway' });
const toolsValkey = require('../../_shared/db').openToolsValkey({ createValkey: require('openvibe-sdk/valkey').createValkey });
const toolsValkey = require('../../_shared/db').openToolsValkey({ createValkey: require('openvibe-sdk/valkey').createValkey });
@@ -191,17 +192,17 @@ app.use(exceptRegistry(cors({
@@ -191,17 +192,17 @@ app.use(exceptRegistry(cors({
exposedHeaders: ['Location', 'Idempotent-Replayed', 'Retry-After', 'RateLimit-Limit', 'RateLimit-Remaining', 'RateLimit-Reset', 'X-OpenVibe-Request-Id', 'Deprecation', 'Sunset', 'Link'],
exposedHeaders: ['Location', 'Idempotent-Replayed', 'Retry-After', 'RateLimit-Limit', 'RateLimit-Remaining', 'RateLimit-Reset', 'X-OpenVibe-Request-Id', 'Deprecation', 'Sunset', 'Link'],
}), { isFirstParty: isAllowedOrigin }));
}), { isFirstParty: isAllowedOrigin }));
-// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
+// ── Who is asking, then the guard's quotas (sign-in first, so a signed-in tier applies) ──
app.use(guard.identify);
app.use(guard.identify);
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
app.use(require('../../_shared/usage').usagePages({ snapshot: toolRegistry.snapshot }));
app.use(require('../../_shared/usage').usagePages({ snapshot: toolRegistry.snapshot }));
-app.use('/api/', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 120, signedIn: 240, message: 'Too many requests. Please try again later.' }), guard.apiQuota);
+app.use('/api/', guard.apiQuota);
app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, keyGenerator: (req) => guard.caller(req).ipKey }));
app.use('/auth/', rateLimit({ windowMs: 15 * 60_000, max: 60, keyGenerator: (req) => guard.caller(req).ipKey }));
// ── Tool registry (ADR-027, capability tools.tool.read) ──────
// ── Tool registry (ADR-027, capability tools.tool.read) ──────
// GET /api/v1/tools (tools.tool-list@1), /api/v1/tools/:id (tools.tool@1), /api/v1/tools/:id/schema.
// GET /api/v1/tools (tools.tool-list@1), /api/v1/tools/:id (tools.tool@1), /api/v1/tools/:id/schema.
// Every tool, every family; satellite hosts answer the same routes for their own tools. Counted by
// Every tool, every family; satellite hosts answer the same routes for their own tools. Counted by
-// the /api/ limiter above. /api/catalog.json stays as it is (Network reads it).
+// the /api/ quota above. /api/catalog.json stays as it is (Network reads it).
// Per-actor limits (apps/_shared/actor-limits.js, roadmap WS-R task 4): signed-in registry reads, a backstop above the
// Per-actor limits (apps/_shared/actor-limits.js, roadmap WS-R task 4): signed-in registry reads, a backstop above the
// guard's quotas on runs and on the jobs facade's submits, and the admin analytics route.
// guard's quotas on runs and on the jobs facade's submits, and the admin analytics route.
const actorLimits = require('../../_shared/actor-limits').createToolsLimits({ app: 'gateway', createActorLimiter: require('openvibe-sdk/limits').createActorLimiter, guard, registry: obs.registry });
const actorLimits = require('../../_shared/actor-limits').createToolsLimits({ app: 'gateway', createActorLimiter: require('openvibe-sdk/limits').createActorLimiter, guard, registry: obs.registry });
@@ -307,7 +308,7 @@ app.get('/api/brand', (_req, res) => res.json(BRAND));
@@ -307,7 +308,7 @@ app.get('/api/brand', (_req, res) => res.json(BRAND));
// which owns pastes (roadmap Wave 5). The visitor's Network JWT goes along: Community verifies it and
// which owns pastes (roadmap Wave 5). The visitor's Network JWT goes along: Community verifies it and
// files writes under the person's canonical subject, so no site has to translate ids any more. The
// files writes under the person's canonical subject, so no site has to translate ids any more. The
// client address goes along too, for Community's anonymous-write limits.
// client address goes along too, for Community's anonymous-write limits.
-app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 120, signedIn: 240, message: 'Too many requests. Please try again later.' }), async (req, res) => {
+app.use('/api/pastes', async (req, res) => {
try {
try {
const target = `${config.communityUrl}/api/pastes${req.url}`;
const target = `${config.communityUrl}/api/pastes${req.url}`;
// The path is the visitor's: "/api/pastes/../../internal/x" (or %2e%2e) resolves to another of
// The path is the visitor's: "/api/pastes/../../internal/x" (or %2e%2e) resolves to another of
@@ -347,7 +348,7 @@ app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonym
@@ -347,7 +348,7 @@ app.use('/api/pastes', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonym
// our pages (the /in URL a bin receives on is anyone's to call).
// our pages (the /in URL a bin receives on is anyone's to call).
const { createWebhookRouter } = require('./dev/routes');
const { createWebhookRouter } = require('./dev/routes');
const devOrigin = guard.originCheck();
const devOrigin = guard.originCheck();
-app.use('/api/dev/webhook', (req, res, next) => (/^\/bins\/[^/]+\/in(\/|$)/.test(req.path) ? next() : devOrigin(req, res, next)), guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' }), createWebhookRouter({ guard }));
+app.use('/api/dev/webhook', (req, res, next) => (/^\/bins\/[^/]+\/in(\/|$)/.test(req.path) ? next() : devOrigin(req, res, next)), createWebhookRouter({ guard }));
// ── Host-header subdomain routing ────────────────────────────
// ── Host-header subdomain routing ────────────────────────────
function subdomainOf(req) {
function subdomainOf(req) {
@@ -13,7 +13,6 @@ const cookieParser = require('cookie-parser');
@@ -13,7 +13,6 @@ const cookieParser = require('cookie-parser');
const path = require('path');
const path = require('path');
const fs = require('fs');
const fs = require('fs');
-const rateLimit = require('express-rate-limit');
const config = require('./config');
const config = require('./config');
const { resolveContext, DOMAIN_MAP } = require('./domain-map');
const { resolveContext, DOMAIN_MAP } = require('./domain-map');
const { getTool, listTools } = require('./tools');
const { getTool, listTools } = require('./tools');
@@ -40,7 +39,7 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
@@ -40,7 +39,7 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
// ── Guard (apps/_shared/guard) ───────────────────────────────
// ── Guard (apps/_shared/guard) ───────────────────────────────
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// Who is asking (Network sign-in with aud openvibe.tools, service tokens, the browser session, else
// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the
// the address), tiered quotas by each tool's descriptor, upload sniffing, the sync semaphore and the
-// abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse.
+// abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.
const SPECS = require('./descriptors').SPECS;
const SPECS = require('./descriptors').SPECS;
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4): guard_abuse in
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4): guard_abuse in
// PostgreSQL, guard_salt/guard_day and the quota buckets in Valkey. `ready` resolves when the schema is in.
// PostgreSQL, guard_salt/guard_day and the quota buckets in Valkey. `ready` resolves when the schema is in.
@@ -51,7 +50,6 @@ const guard = createGuard({
@@ -51,7 +50,6 @@ const guard = createGuard({
app: 'img', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,
app: 'img', db: toolsDb.db, valkey: toolsValkey, contracts, specs: SPECS,
issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,
issuer: config.networkUrl, networkUrl: config.networkUrl, networkInternalUrl: config.networkInternalUrl, publicKeyFiles: config.publicKeyPaths,
});
});
-const { apiLimiter, processLimiter, burstLimiter } = guard.legacyLimiters(rateLimit);
/** The tool a request is for before its body is read: the host's own, else the operation it defaults to. */
/** The tool a request is for before its body is read: the host's own, else the operation it defaults to. */
const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('img.process', req.ctx.defaultOp || 'convert', null) || { id: null }).id);
const hostTool = (req) => (guard.tool(req.ctx.toolId) ? req.ctx.toolId : (guard.toolForJob('img.process', req.ctx.defaultOp || 'convert', null) || { id: null }).id);
/** The tool a parsed request runs: its `tool` (operation) on this host. */
/** The tool a parsed request runs: its `tool` (operation) on this host. */
@@ -129,12 +127,12 @@ app.use(exceptRegistry(cors({
@@ -129,12 +127,12 @@ app.use(exceptRegistry(cors({
credentials: true,
credentials: true,
}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
}), { isFirstParty: (origin) => /^https:\/\/([a-z0-9-]+\.)?openvibe\.tools$/.test(origin) || (process.env.NODE_ENV === 'development' && /^http:\/\/(localhost|127\.0\.0\.1):\d+$/.test(origin)) }));
-// ── Who is asking, then rate limits (sign-in first, so a signed-in tier applies) ──
+// ── Who is asking, then the guard's quotas ───────────────────
app.use(guard.identify);
app.use(guard.identify);
-app.use('/api/', apiLimiter, guard.apiQuota);
+app.use('/api/', guard.apiQuota);
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's image tools ──
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's image tools ──
-// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'heif-dec' ? codec.heif.available() : true)) });
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'heif-dec' ? codec.heif.available() : true)) });
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
@@ -204,7 +202,7 @@ const jobs = jobsRuntime.setupJobs({
@@ -204,7 +202,7 @@ const jobs = jobsRuntime.setupJobs({
getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,
getPublicKey: guard.keys.get, issuer: config.networkUrl, guard,
define: (system) => defineJobs(system, pool),
define: (system) => defineJobs(system, pool),
receive: uploadSingle,
receive: uploadSingle,
- limiters: [actorLimits.backstop('tools.job.create'), burstLimiter, processLimiter, guard.toolQuota(hostTool)],
+ limiters: [actorLimits.backstop('tools.job.create'), guard.toolQuota(hostTool)],
jobTool: (req, type, input) => { const d = guard.toolForJob('img.process', String((input && input.tool) || req.ctx.defaultOp || 'convert'), req.ctx.toolId); return d ? d.id : null; },
jobTool: (req, type, input) => { const d = guard.toolForJob('img.process', String((input && input.tool) || req.ctx.defaultOp || 'convert'), req.ctx.toolId); return d ? d.id : null; },
defaults(req, input) {
defaults(req, input) {
const out = { ...input };
const out = { ...input };
@@ -221,7 +219,7 @@ const jobs = jobsRuntime.setupJobs({
@@ -221,7 +219,7 @@ const jobs = jobsRuntime.setupJobs({
const runApi = satelliteRunApi({
const runApi = satelliteRunApi({
app: 'img', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
app: 'img', guard, contracts, snapshot: toolRegistry.snapshot, system: () => jobs,
multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
multer: require('multer'), uploadsDir: path.resolve(config.uploadsDir), ...ajvFrom(require), ports: () => satellitePorts(),
- limiters: [actorLimits.backstop('tools.tool.run'), burstLimiter, processLimiter],
+ limiters: [actorLimits.backstop('tools.tool.run')],
});
});
app.use(runApi.handle);
app.use(runApi.handle);
@@ -36,7 +36,7 @@ const { startApp } = require('../../_shared/test/spawn');
@@ -36,7 +36,7 @@ const { startApp } = require('../../_shared/test/spawn');
// ── Through the app ──
// ── Through the app ──
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-img-guard-'));
const data = fs.mkdtempSync(path.join(os.tmpdir(), 'ov-img-guard-'));
- const app = await startApp('img', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_MAX_INPUT_PIXELS: '10000' });
+ const app = await startApp('img', { DATA_DIR: data, UPLOADS_DIR: path.join(data, 'uploads'), OUTPUT_DIR: path.join(data, 'output'), TOOLS_MAX_INPUT_PIXELS: '10000', TOOLS_GUARD: 'report' });
const run = (buf, name, type) => {
const run = (buf, name, type) => {
const f = new FormData();
const f = new FormData();
f.append('input', JSON.stringify({ format: 'png' }));
f.append('input', JSON.stringify({ format: 'png' }));
@@ -9,7 +9,6 @@
@@ -9,7 +9,6 @@
const express = require('express');
const express = require('express');
const cors = require('cors');
const cors = require('cors');
const helmet = require('helmet');
const helmet = require('helmet');
-const rateLimit = require('express-rate-limit');
const path = require('path');
const path = require('path');
const fs = require('fs');
const fs = require('fs');
const NodeCache = require('node-cache');
const NodeCache = require('node-cache');
@@ -30,7 +29,7 @@ const { createPacer, Busy } = require('../../_shared/guard/semaphore');
@@ -30,7 +29,7 @@ const { createPacer, Busy } = require('../../_shared/guard/semaphore');
// ── Guard (apps/_shared/guard) ───────────────────────────────
// ── Guard (apps/_shared/guard) ───────────────────────────────
// Who is asking (Network sign-in with aud openvibe.tools, else the address — the food app forwards its
// Who is asking (Network sign-in with aud openvibe.tools, else the address — the food app forwards its
// visitor's), the tools-map quota on the data routes (descriptor maps: cost 2 a call) and the abuse log
// visitor's), the tools-map quota on the data routes (descriptor maps: cost 2 a call) and the abuse log
-// (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse.
+// (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.
const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';
const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-maps' });
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-maps' });
@@ -94,13 +93,12 @@ app.use(express.json());
@@ -94,13 +93,12 @@ app.use(express.json());
// Rate limits: sign-in first, so limits know who is asking; everyone else counts by address (IPv6 /64).
// Rate limits: sign-in first, so limits know who is asking; everyone else counts by address (IPv6 /64).
app.use(guard.identify);
app.use(guard.identify);
-const apiLimiter = guard.legacyLimiter(rateLimit, { windowMs: 60 * 1000, anonymous: 30, signedIn: 60, message: 'Too many requests — slow down, traveler.' });
-app.use('/api/', apiLimiter, guard.apiQuota);
+app.use('/api/', guard.apiQuota);
// The map's data lookups (OpenStreetMap, weather, food banks…): the tools-map quota, by descriptor.
// The map's data lookups (OpenStreetMap, weather, food banks…): the tools-map quota, by descriptor.
app.use(['/api/geocode', '/api/search', '/api/weather', '/api/terrain', '/api/food-banks', '/api/stores', '/api/foods', '/api/meal-plan'], guard.toolQuota('maps'));
app.use(['/api/geocode', '/api/search', '/api/weather', '/api/terrain', '/api/food-banks', '/api/stores', '/api/foods', '/api/meal-plan'], guard.toolQuota('maps'));
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's survival map ──
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's survival map ──
-// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS.filter(s => s.id === 'maps') });
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS.filter(s => s.id === 'maps') });
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
@@ -4,7 +4,6 @@ const express = require('express');
@@ -4,7 +4,6 @@ const express = require('express');
const cors = require('cors');
const cors = require('cors');
const helmet = require('helmet');
const helmet = require('helmet');
const cookieParser = require('cookie-parser');
const cookieParser = require('cookie-parser');
-const rateLimit = require('express-rate-limit');
const path = require('path');
const path = require('path');
const fs = require('fs');
const fs = require('fs');
@@ -20,7 +19,8 @@ const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
@@ -20,7 +19,8 @@ const { createGuard, TRUST_PROXY } = require('../../_shared/guard');
// ── Guard (apps/_shared/guard) ───────────────────────────────
// ── Guard (apps/_shared/guard) ───────────────────────────────
// Who is asking (Network sign-in with aud openvibe.tools, else the address), the tools-api quota on
// Who is asking (Network sign-in with aud openvibe.tools, else the address), the tools-api quota on
-// /api/ and the abuse log (data/guard.db). TOOLS_GUARD=report (default) records what it would refuse.
+// /api/ and the abuse log (data/guard.db). TOOLS_GUARD=enforce (default) refuses; report records what
+// it would refuse.
const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';
const NETWORK_URL = process.env.OV_NETWORK_URL || 'https://openvibe.network';
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-text' });
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-text' });
@@ -89,13 +89,12 @@ app.use(exceptRegistry(cors({
@@ -89,13 +89,12 @@ app.use(exceptRegistry(cors({
credentials: true,
credentials: true,
})));
})));
-// ── Who is asking, then rate limits ──────────────────────────
+// ── Who is asking, then the guard's quotas ───────────────────
app.use(guard.identify);
app.use(guard.identify);
-app.use(guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 200, signedIn: 400, message: 'Too many requests. Please try again later.' }));
app.use('/api/', guard.apiQuota);
app.use('/api/', guard.apiQuota);
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's text and logo tools ──
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's text and logo tools ──
-// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS });
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS });
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
@@ -7,7 +7,7 @@ Free YouTube video and audio downloader at `yt.openvibe.tools`.
@@ -7,7 +7,7 @@ Free YouTube video and audio downloader at `yt.openvibe.tools`.
- Video: Best, 1080p, 720p, 480p, 360p, WebM
- Video: Best, 1080p, 720p, 480p, 360p, WebM
- Audio: MP3, M4A, OPUS, FLAC
- Audio: MP3, M4A, OPUS, FLAC
- Real-time SSE progress streaming
- Real-time SSE progress streaming
-- Tiered rate limiting (anon: 5/hr, authed: 20/hr)
+- Rate limiting and quotas through the shared guard (`tools-download`: a download costs 50; `tools-api` for every `/api/` call)
## Stack
## Stack
@@ -33,12 +33,6 @@ module.exports = {
@@ -33,12 +33,6 @@ module.exports = {
cleanupInterval: 5 * 60 * 1000, // every 5 min
cleanupInterval: 5 * 60 * 1000, // every 5 min
},
},
- // Rate limits (per user/IP)
- rateLimit: {
- anonPerHour: 5,
- authedPerHour: 20,
- },
-
// Paths
// Paths
dataDir: process.env.DATA_DIR || 'data',
dataDir: process.env.DATA_DIR || 'data',
downloadsDir: process.env.DOWNLOADS_DIR || 'data/downloads',
downloadsDir: process.env.DOWNLOADS_DIR || 'data/downloads',
@@ -9,7 +9,6 @@ const express = require('express');
@@ -9,7 +9,6 @@ const express = require('express');
const cors = require('cors');
const cors = require('cors');
const helmet = require('helmet');
const helmet = require('helmet');
const cookieParser = require('cookie-parser');
const cookieParser = require('cookie-parser');
-const rateLimit = require('express-rate-limit');
const path = require('path');
const path = require('path');
const fs = require('fs');
const fs = require('fs');
@@ -28,7 +27,7 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
@@ -28,7 +27,7 @@ const { requireInternalAccess } = require('../../_shared/internal-token');
// ── Guard (apps/_shared/guard) ───────────────────────────────
// ── Guard (apps/_shared/guard) ───────────────────────────────
// Who is asking (Network sign-in with aud openvibe.tools, the browser session, else the address),
// Who is asking (Network sign-in with aud openvibe.tools, the browser session, else the address),
// the tools-download quota (a download costs 50, descriptor yt) and the abuse log (data/guard.db).
// the tools-download quota (a download costs 50, descriptor yt) and the abuse log (data/guard.db).
-// TOOLS_GUARD=report (default) records what it would refuse; the older limiters below stay until enforce.
+// TOOLS_GUARD=enforce (default) refuses; report records what it would refuse.
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
// The one `tools` database and the shared Valkey (plan T8, decisions 3 and 4).
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-yt' });
const toolsDb = require('../../_shared/db').openToolsDb({ createDb: require('openvibe-sdk/db').createDb, service: 'tools-yt' });
const analytics = new AnalyticsTrackerPg(toolsDb.db, 'openvibe-yt', { retention: { days: 30 } }); // ADR-021; PostgreSQL (plan T8)
const analytics = new AnalyticsTrackerPg(toolsDb.db, 'openvibe-yt', { retention: { days: 30 } }); // ADR-021; PostgreSQL (plan T8)
@@ -105,18 +104,14 @@ app.use(exceptRegistry(cors({
@@ -105,18 +104,14 @@ app.use(exceptRegistry(cors({
})));
})));
// ── Rate Limiting ────────────────────────────────────────────
// ── Rate Limiting ────────────────────────────────────────────
-// Progress is one SSE connection, or a poll every 1–3 s when SSE is unavailable; the poll must
-// not eat the general budget (60/min would cut a download off after a minute).
// Sign-in is read first, so limits know who is asking; everyone else counts by address (IPv6 by /64).
// Sign-in is read first, so limits know who is asking; everyone else counts by address (IPv6 by /64).
-const isStatusRoute = (req) => req.method === 'GET' && /^\/status\/[a-f0-9]+(\/stream)?$/.test(req.path);
+// The guard's tools-api quota covers every /api/ call, the progress polls included.
app.use(guard.identify);
app.use(guard.identify);
-const apiLimiter = guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 60, signedIn: 120, message: 'Too many requests. Please try again later.' });
-app.use('/api/', (req, res, next) => (isStatusRoute(req) ? next() : apiLimiter(req, res, (err) => (err ? next(err) : guard.apiQuota(req, res, next)))));
-app.use('/api/status/', guard.legacyLimiter(rateLimit, { windowMs: 60_000, anonymous: 240, signedIn: 240, message: 'Too many requests. Please try again later.' }));
+app.use('/api/', guard.apiQuota);
app.use('/api/', (_req, res, next) => { res.setHeader('Cache-Control', 'no-store'); next(); });
app.use('/api/', (_req, res, next) => { res.setHeader('Cache-Control', 'no-store'); next(); });
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's YouTube downloader ──
// ── Tool registry (ADR-027): GET /api/v1/tools[/:id[/schema]] for this app's YouTube downloader ──
-// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the limiter above. The
+// Public (Access-Control-Allow-Origin *), cacheable (ETag), counted by the quota above. The
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
// gateway (openvibe.tools) answers the same routes for every tool and reads this list for status.
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'yt-dlp' ? !!which(config.ytdlpPath) : true)) });
const toolRegistry = createLocalRegistry({ specs: require('./descriptors').SPECS, statusOf: requiresStatus((p) => (p === 'yt-dlp' ? !!which(config.ytdlpPath) : true)) });
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
// A signed-in person's page view of a tool goes into their tools.usage module (the launchers' recent tools).
@@ -130,10 +125,9 @@ app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
@@ -130,10 +125,9 @@ app.use(createToolsApi({ snapshot: toolRegistry.snapshot }));
// ── Analytics Middleware ─────────────────────────────────────
// ── Analytics Middleware ─────────────────────────────────────
app.use(analytics.middleware());
app.use(analytics.middleware());
-// Downloads: the older hourly limit (5 anonymous, 20 signed in) until enforce mode, and the guard's
-// tools-download quota (descriptor yt: cost 50) in both. A download belongs to whoever started it (the
-// browser session cookie, minted on the start, or a sign-in): its status, progress stream, file and
-// cancel answer 404 to anyone else, the same as for an id that does not exist.
+// Downloads: the guard's tools-download quota (descriptor yt: cost 50). A download belongs to whoever
+// started it (the browser session cookie, minted on the start, or a sign-in): its status, progress
+// stream, file and cancel answer 404 to anyone else, the same as for an id that does not exist.
const owners = new Map(); // download id → owner
const owners = new Map(); // download id → owner
const ownsDownload = (req, res, id) => {
const ownsDownload = (req, res, id) => {
const owner = owners.get(String(id || ''));
const owner = owners.get(String(id || ''));
@@ -141,7 +135,6 @@ const ownsDownload = (req, res, id) => {
@@ -141,7 +135,6 @@ const ownsDownload = (req, res, id) => {
return !!owner && !!who.owner && owner === who.owner;
return !!owner && !!who.owner && owner === who.owner;
};
};
setInterval(() => { for (const id of owners.keys()) if (!downloader.getStatus(id) && !downloader.getFile(id)) owners.delete(id); }, 10 * 60_000).unref();
setInterval(() => { for (const id of owners.keys()) if (!downloader.getStatus(id) && !downloader.getFile(id)) owners.delete(id); }, 10 * 60_000).unref();
-const downloadLimiter = guard.legacyLimiter(rateLimit, { windowMs: 60 * 60 * 1000, anonymous: config.rateLimit.anonPerHour, signedIn: config.rateLimit.authedPerHour, message: 'Download limit reached. Sign in for more downloads or wait an hour.' });
// ── Hosts ────────────────────────────────────────────────────
// ── Hosts ────────────────────────────────────────────────────
// Pages are only rendered for hosts this tool serves (or that the gateway vouches for with
// Pages are only rendered for hosts this tool serves (or that the gateway vouches for with
@@ -176,7 +169,7 @@ app.post('/api/info', async (req, res) => {
@@ -176,7 +169,7 @@ app.post('/api/info', async (req, res) => {
});
});
// Start download
// Start download
-app.post('/api/download', downloadLimiter, guard.toolQuota('yt'), async (req, res) => {
+app.post('/api/download', guard.toolQuota('yt'), async (req, res) => {
const { url, quality, title } = req.body;
const { url, quality, title } = req.body;
if (!url) return res.status(400).json({ error: 'URL is required' });
if (!url) return res.status(400).json({ error: 'URL is required' });
if (!downloader.isValidUrl(url)) return res.status(400).json({ error: 'Only YouTube URLs are supported' });
if (!downloader.isValidUrl(url)) return res.status(400).json({ error: 'Only YouTube URLs are supported' });