Commits

main · README.md

Job fencing (WS-P lifecycle follow-up): succeed() and fail() matched status = 'running' only, with no owner token, so a worker that had lost its job (lease run out, recovery requeued it, another claim took it) could still overwrite it; safe only while one process runs the queue. Now a claim writes a fresh random lease_token on the row and hands it to the worker; renew, saveCheckpoint (with its alsoInTx writes, rolled back), succeed, fail and markCancelled match the id AND that token. A stale holder is refused, not applied: logged ([Jobs] refused a stale <action> of <id> ...), counted (queue.staleStats(), worker status stale_refused, media_job_stale_completions_total{action} on /metrics) and emitted on the queue bus; its heartbeat aborts the handler (media.job.lease_lost). Recovery takes a job over only with the token it read and, past boot, only while the lease is still expired; rows claimed before the column (NULL token) are still recovered at boot. Migration is additive only: media_jobs.lease_token TEXT in schema.sql and migrateColumns (ADD COLUMN); an older release ignores it. test/jobs-fencing.test.js: the column added to an older database; two claimants, the stale one's heartbeat, checkpoint, succeed, fail and cancel refused and counted while the current holder's result stands; a declined takeover; the worker aborting a job whose lease it lostOpenVibers · 26 Sept 2026 a348ea7
Waveform and preview-sprite variants as jobs (WS-G task 13): object.waveform (PNG, showwavespic) and object.sprite (JPEG sheet from one fast seek per frame, layout in its metadata) make private asset objects recorded as the source's waveform / sprite variant; no_audio / no_video are permanent; a remote-only source over MEDIA_PREVIEW_REMOTE_MAX_MB gets no waveformOpenVibers · 25 Sept 2026 f9c89e1
Retention holds (WS-G task 10), on the existing media_holds (no second table): a note column (added to existing databases), placed_by/placed_at in every answer (created_by/created_at kept), and staff routes on the app-key admin router: GET/POST /api/v1/:app/admin/storage/holds (by object_id, legacy ref, vod_id or clip_id; reason required; a row with no object yet is projected first; refused to calls acting for a user) and POST /holds/:id/release (also DELETE); every placement and release is an [Admin] log line and the row keeps who and when. Clips follow their VOD: a clip cut from a held VOD is held through clip_of or its row's vod_id, even before it has an object (database.js heldSql, one rule for model.isHeld and the new _v2 delete triggers, which replace the first-version ones in one transaction; idx_clips_object). Newly guarded paths: moveToHot refuses a held VOD (it flipped the row to local and dropped the R2 copy; a clip cut then uses the cloud copy), finalize settles a held empty or zero-byte recording as failed and keeps its row and file instead of deleting them, the boot junk sweep skips held rows, the paste DELETE answers 409 and the paste bulk delete skips held screenshots. v2 GET /:id/holds lists inherited_holds. test/retention-holds.test.jsOpenVibers · 25 Sept 2026 a63bb3a
Storage orphan report (WS-G task 8), report only: server/vod/orphans-report.js buildStorageReport compares what storage holds with what the database names and lists files under the data directories and B2/R2 keys that no row names (with hints: a recorder name whose VOD row is gone, bytes of a deleted object, an R2 copy left after its VOD went back to B2, parked vods-orphans/ keys, parts of a session that no longer exists, a temp of an upload that never finished), copies the database records for live objects that are not there (no local file, a key absent from its bucket's listing), and multipart uploads left open (media_uploads sessions, expired ones flagged, and the buckets' own via ListMultipartUploads, new read-only vodStorage.listMultipartUploads). Live thumbnails and temps younger than an hour are counted as expected. It never deletes, moves or writes a file, an object or a row; it writes <data>/reports/storage-orphans-<time>.json. The job storage.orphans.scan runs it every MEDIA_ORPHAN_SCAN_DAYS (30; 0 = never) from the jobs worker (off in a restore drill) under the system app _media: no apps row, no media.job events, and POST /api/v2/:app/jobs refuses it (403 media.job.forbidden). On demand: scripts/vods-orphans-report.js --storage [--no-remote]. test/storage-orphans.test.js snapshots files, tables and buckets before and after, and its fake S3 refuses and counts any writeOpenVibers · 25 Sept 2026 95a449d
Readiness levels (WS-G task 5): every media object answers metadata, bytes_verified and playable, read from the object row and its copies' recorded checks (server/objects/readiness.js). bytes_verified needs a copy that is present with a verified_at (the local file found, a B2/R2 HEAD with the right size, a v2 upload hashed on receipt; sha256 against content_hash where the verification job checks it) and no checksum contradicting content_hash; a B2 copy nobody has checked yet does not count. playable adds a finished vod or clip (lifecycle ready: not recording, not still cutting, not failed or deleted); the size invariant does not gate it, as in code it only refuses oversized v2 uploads. v2 object answers and the v1 VOD/clip answers carry readiness (webhook payloads do not; existing fields unchanged). The /v and /c watch pages offer a player, download link, video tags and VideoObject only when playable, and otherwise say why (still being recorded, still processing, not available) and are noindex; a clip still being cut gets that page instead of a 404, its bytes stay a 404. The byte routes are unchanged (Live's DVR reads recordings through them). The sitemap lists only playable pages (readiness.playableSql). test/readiness.test.jsOpenVibers · 25 Sept 2026 8b1abf0
R2 tier decisions are recorded and explainable: every promotion to and demotion from the R2 popularity cache gets a media_tier_decisions row with the inputs it saw (view_count, last_accessed_at, provider, recording, hold, size, created_at), the R2 thresholds in force each with its source (default or a media_settings override), the reason in words, who asked (sweep, admin, clip, drill, manual) and the outcome (done, already, refused, failed, with the error). Before, a tier move only flipped media_locations and wrote a console line. The sweep states its rule per VOD (view_count 25 >= r2MinViews 20 and last accessed within r2RecentAccessDays 3; idle longer than r2MaxIdleDays 14), admin moves name the app, a clip cut that fetches an R2 VOD home (which drops the R2 copy) is logged as a demotion to local, and the eviction drill's evict and re-warm are logged as such. Staff read the policy through the app's key: GET /api/v1/:app/admin/storage/tiers/policy (each threshold with value, default and source, the promote and demote rules, provider status, the app's 24-hour counts and 20 latest decisions) and GET /tiers/decisions (filters vod_id, action, outcome; paged by before_id), both app-scoped and read-only; /metrics gains media_tier_decisions_24h{action,outcome}. The thresholds are unchanged (the test pins them). test/tier-decisions.test.js runs the sweep, admin moves, a hot-fetch and a drill step against a fake S3 through the real SDK.OpenVibers · 24 Sept 2026 7114433
Events media.object.deleted and media.object.visibility_changed through the outbox, so consumers (Search, Live, Community) learn that an object went away or stopped being public (Media emitted neither). Payloads are minimal: media.object.deleted { object_id, app_id, kind, legacy_ref, deleted_at }, media.object.visibility_changed { object_id, app_id, kind, legacy_ref, visibility, previous_visibility, changed_at }; subject object <id>, internal, important, actor service:media, times ISO 8601 UTC, legacy_ref null for native objects, never the title, owner, storage keys, sizes or metadata. Events only, no webhook; sandbox tenants produce none. Triggers on media_objects stage one media_object_changes row per deletion (lifecycle to deleted) and per visibility change (not for an object that is deleted) inside the transaction that makes it, whichever path made it: the v1 visibility routes through projection sync, the inherited row-delete triggers, a v2 soft delete, operator SQL. recordObjectChanges() turns staged rows into outbox envelopes and removes them, and must run inside a transaction: Media calls it at the end of its own transactions (projection sync, announce(), soft delete), so those events commit with their change and a rolled-back change has none; the relay drains the rest on its interval. With the outbox off the staged rows are dropped hourly. The outbox test now expects the media.object.deleted that a finalize row delete queues with media.vod.failed. test/object-events.test.js covers each path, the payload shape, rollback, the drain and sandboxes. Contract schemas for both types are being added in OpenVibe.Contracts v0.34.0 (payload shape in the C1 handoff).OpenVibers · 24 Sept 2026 039dc48
Crawler basics at the openvibe.media origin: /robots.txt (openvibe-shared seo.robotsTxt: AI and search crawlers named; /api/, /auth/, /internal/, /o/, /metrics, /release-metrics and the /live/ dev-data endpoints disallowed; the sitemap), /sitemap.xml and /llms.txt, each cached an hour. The sitemap lists the media index and the watch pages Media is the canonical home of: public, ready VODs and clips of apps other than Live, with their thumbnails as image entries. Live's items are left out on purpose: their watch pages here are noindex with the canonical on openvibe.live/vod and /clip, which Live's own sitemap lists, and a sitemap URL marked noindex is an error for search engines. AI clips, private, unlisted, developer-sandbox, recording, quarantined and failed items are never listed. The sitemap and the page share one predicate (pages.watchIndexable), so every listed page renders index, follow. Watch pages: AI clips (auto_generated) are now noindex, follow for every app, labelled AI clip, attributed to no person (creator Organization OpenVibe AI instead of a Person author) and isBasedOn their source VOD, as on Live; the schema.org VideoObject is emitted on public pages only (none on unlisted or private ones). test/crawl.test.js.OpenVibers · 24 Sept 2026 a90d0ed
RTMP pull SSRF: POST /vods/:id/ingest/rtmp makes ffmpeg connect to the URL it is given, and any rtmp:// or rtmps:// URL was accepted, so an app key (or a leaked one) could point Media's ffmpeg at any host and port it can reach. The recorder now pulls only from allow-listed RTMP servers: MEDIA_RTMP_PULL_ALLOW, a host:port list defaulting to Live's RTMP server on this host (127.0.0.1:1935, localhost:1935, [::1]:1935; a URL without a port means 1935, or 443 for rtmps). The URL must be a plain rtmp[s]://host[:port]/path[?query]: no user info, backslash, whitespace or control characters, nothing a URL parser and ffmpeg could read differently, and alternative spellings of loopback (0x7f.0.0.1, 127.1) are not on the list. The scheme is handed to ffmpeg in lower case. A refused URL answers 400 (was 409) and starts no ffmpeg. test/rtmp-allowlist.test.js covers allowed and refused URLs (metadata address, other local ports, parser differentials, injected arguments) and the route.OpenVibers · 24 Sept 2026 cb17f29
trust proxy is loopback only (was true): req.ip is the X-Forwarded-For that the local nginx set when a request came through it, and the socket address for anyone else, so a caller reaching port 4100 directly (HOST defaults to 0.0.0.0) can no longer choose its IP. The view counter read CF-Connecting-IP / X-Forwarded-For itself, from any peer; it now keys on req.ip through server/client-ip.js, like paste limits and the live-frame rate limit. The unused getRequesterIp header reader in public/routes.js is gone. test/trust-proxy.test.js checks the setting on an app (a loopback proxy's forwarded address is believed, a non-loopback peer's X-Forwarded-For and CF-Connecting-IP are not), and boots the real server: forwarded clients get their own live-frame budgets, while a direct caller rotating X-Forwarded-For is rate-limited as one address (it fails with trust proxy true). README: client addresses, and the new jobs and scripts in the layout.OpenVibers · 24 Sept 2026 47b4349
VOD durations are measured, never estimated: finalize stores what the file says and vods.duration_source says where it came from. probe = ffprobe's container duration of the finished file; remux = the last packet time of a stream-copy pass (the finalize remux reports it, or a separate pass into the null muxer when the remux gave none), used when the header has no duration or claims far more than the packets (1.5x + 30 s: the header is corrupt); unknown = nothing measurable, stored as 0 with health_status needs_review (probe_failed or inflated_duration) and hidden, where the old code kept the wall clock. The recorder's own start time only bounds a value (longer than 1.5x its run + 30 s is refused); an orphan finalized after a restart has no bound and is never measured as now - created_at (it used to be, so a boot-time finalize of a 2-day-old row could store 2 days). The short, corrupt and stat-failure outcomes now also replace the recorder's live elapsed-time estimate (they left it in duration_seconds), as does the recorder's failed-finalize quarantine. A later finalize that measures a VOD lifts the quarantine a failed attempt left and restores the owner's visibility. Merge/remux failures are recorded as health issues (merge_failed, remux_failed) instead of being ignored. The health job's duration repair and the GET /vods/:id self-heal record source probe. test/vod-duration.test.js covers the choice, every outcome and a real ffmpeg orphan.OpenVibers · 24 Sept 2026 cc84bf1
openvibe-shared v1.5.1 (Track R release manifest), openvibe-contracts v0.33.0, openvibe-sdk v0.5.0OpenVibers · 23 Sept 2026 0c27f0b
Restore drills: MEDIA_DRILL=1 makes Media a read-only instance on a restored copy of media.db, so `ovhost drill media` can boot one next to production. It refuses to start unless DB_PATH is set and outside the checkout and /opt/openvibe.media, HOST is loopback and PORT is not 4100, before the database is opened. It then starts only its HTTP server: no app seeding or JWKS refresh, no tiering sweep, health job, junk sweep, clip re-cuts, copy verification, jobs worker, disk guardian, thumbnail cleanup, object purge, backfill, orphan-recording finalize or Events relay, and webhooks are never sent (their URLs in the apps table are production's). It writes no file and creates no directory, and never opens a file path from the database: every byte route (/v and /c bytes, /t, /a, /f, /o, paste screenshots, live frames, legacy thumbnails) answers 503 while watch pages and /browse still render. Everything but GET/HEAD/OPTIONS answers 403, /auth/* 503, and net/fetch/child_process/dgram/listen are guarded so nothing leaves the process (only git runs: no ffmpeg or ffprobe). /api/ready drops the storage and remote-tier checks and says mode: drill. test/drill-mode.test.js boots the real server that way against a copy whose app has a webhook URL and checks each pointOpenVibers · 23 Sept 2026 6e74eb0
Job events carry a projection, not the job: media.job.<transition> payloads are queue.jobEvent (id, app_id, object_id, type, status, attempts, max_attempts, error_code, cancel_requested, has_result, run_after, decided_at, created_at, updated_at, started_at, finished_at), times in ISO 8601 UTC or nullOpenVibers · 23 Sept 2026 ad2217b
Wave 4: media job system, multipart and presigned uploads, R2 eviction drillOpenVibers · 23 Sept 2026 d006749
Media's Events outbox row is written in the state change's transaction: webhooks.announce() runs the change (VOD ready/quarantine/delete, clip ready/failed, object ready + invariant row) and events.record() in ONE SQLite transaction, then wakes the relay and sends the webhook, so an outcome is never lost (committed without its event) or a phantom (event for a rolled-back change); a failed outbox insert rolls the change back. The thumbnail is generated before the ready commit so the payload still carries it. Webhook bodies carry event_id = the durable event's id, so an app reading both paths (Live's webhook → Events transition) handles each outcome once. Lifecycle events are priority important (roadmap §6.3; storage.recovered stays low). test/outbox-transaction.test.js covers announce, outbox-insert failure, change failure, and the clip-jobs and finalize call sitesOpenVibers · 23 Sept 2026 cbf8410
Scheduled copy verification: every 10 min verify the 50 least recently verified ready objects (local stat + sha256 up to 64 MB, B2/R2 HEAD + size), record verdicts on media_locations, media_verifications and media_verify_runs, restore a missing remote copy from a verified-good local copy (2 per run; corrupt remote only with MEDIA_VERIFY_REPAIR_CORRUPT=1), never delete anything; media_objects_no_good_copy on /metrics and an optional object_copies check in /api/ready (degraded, never not-ready); scripts/no-good-copy-report.js lists those objects with owners, references and locations, read-onlyOpenVibers · 23 Sept 2026 e42af9e
Docs: STATUS.json and README match production (2026-09-23)OpenVibers · 23 Sept 2026 22ba4aa
Private items answer exactly like missing ones: /v and /c returned 403 "This media is private", which confirmed that a private recording (or, through the basename form of /v, a file name) exists; now the same 404 and body as an unknown id, for the bytes and the watch page. Same rule on /v/:id/transcript.json (legacy rows with no visibility and is_public 0 count as private, clips-only sources are hidden like on /v), the legacy /api/thumbnails redirect (no longer hands out a private row's current thumbnail URL), /p/:slug/raw (no screenshot bounce or Community redirect that only real private pastes got), /o/:id (a deleted private object said 410 before the signature check), and the v1 detail routes when an app acts for a user via X-OV-User-Id (GET /vods/:id, /clips/:id, cutting a clip from someone else's private VOD, GET /pastes/:slug, which also no longer counts a view before refusing). test/private-not-found.test.js compares every refused answer with the missing-id answerOpenVibers · 23 Sept 2026 dfb2172
Developer-project tenants: no inline HTML/SVG from /f, race-free files quota, retained deletes countOpenVibers · 23 Sept 2026 20189e9
Developer-project tenants (Wave 20, ADR-014): app tokens get a Media tenant per projectOpenVibers · 23 Sept 2026 bfc9e1b
Truthful GET /api/ready and loopback-only GET /metrics (openvibe-shared v1.3.0)OpenVibers · 23 Sept 2026 e5525bd
openvibe-shared is the pinned OpenVibe.Shared v1.0.0 release (vendor/ is gone); openvibe-contracts v0.7.0OpenVibers · 23 Sept 2026 76f3a44
Canonical object model (Wave 4): media_objects with locations, relationships, variants, jobs, retention holds and the public-size invariant; vods/clips/files/screenshot pastes become projections via object_idOpenVibers · 23 Sept 2026 cced10f
README: visitor sign-in and legal pagesOpenVibers · 18 Sept 2026 8810861
VOD storage: make the B2 drain self-monitoring — free-space budget, fast re-runs, bounded uploads, stall alertsOpenVibers · 29 Aug 2026 0dd8bd4
Dev data APIs: transcripts, AI timelines, chat insightOpenVibers · 18 Aug 2026 934dba3
Offline card as real JPEG bytesOpenVibers · 18 Aug 2026 970ebfa
Live frame API: slug + @username selectors, styled OFFLINE cardOpenVibers · 18 Aug 2026 1598ae7
Live frame API: GET /live/:msid/frame.jpgOpenVibers · 18 Aug 2026 1be94ae

More commits