openvibers/openvibe.live Commits

docs/vods-and-clips.md: what used to read the frozen tables and what answers now, the Media and Community API additions that would make the moved lookups exact (batch by stream ids, created_after, owner-subject and metadata filters for pastes, a public paste count), and C-73 step 3, the procedure to drop vods, clips, pastes, paste_likes and paste_comments 30 days after this deploy: ship the code that stops creating them, back up (online .backup plus a .dump of the five tables, checksummed), reconcile read-only against media.db and community.db (every missing count 0, nothing else references them), drop children first with foreign keys off in one transaction, verify through a restart, rollback from the dump. docs/architecture.md points there

OP OpenVibers committed 23 Sept 2026 · bc2f2a628f · 2 files, +198 −2





Changes
ddiff --git a/docs/architecture.md b/docs/architecture.md
iindex e2309730..e909bd45 100644
---- a/docs/architecture.md
++++ b/docs/architecture.md
@@@ -11,7 +11,9 @@ OpenVibe.Live is a self-hosted live streaming platform, one of the OpenVibe serv
  - **FFmpeg** — recording hand-off, restreams, thumbnails, audio capture for transcription
  - **OpenVibe.Network** — SSO/OAuth2 (RS256 JWTs), OpenCoins wallet, shared browser modules (`openvibe-shared`, a pinned OpenVibe.Shared release served at `/shared/*` from `node_modules`)
  - **OpenVibe.Media** — VODs, clips, pastes, thumbnails and files. Live proxies to it
--  (`server/media-client.js`, `server/media-proxy/`); the local `vods`/`clips`/`pastes` tables are frozen, read-only.
++  (`server/media-client.js`, `server/media-proxy/`); the local `vods`/`clips`/`pastes` tables are frozen: nothing reads or
++  writes them ([vods-and-clips.md](vods-and-clips.md#live-reads-nothing-from-the-frozen-tables)), and they are dropped by
++  [this procedure](vods-and-clips.md#dropping-the-frozen-tables).
  - **OpenVibe.Community** — pastes (`server/pastes-client.js`) and VOD/clip comments: `/api/comments` is an adapter over Community comment threads
    (`server/comments-client.js`, [vods-and-clips.md](vods-and-clips.md#comments-are-openvibecommunity-threads)); the local `comments` table is frozen, read-only.
  
ddiff --git a/docs/vods-and-clips.md b/docs/vods-and-clips.md
iindex 3b054a72..ec8a679d 100644
---- a/docs/vods-and-clips.md
++++ b/docs/vods-and-clips.md
@@@ -2,7 +2,9 @@
  
  > Written before the media split. Storage, cutting, thumbnails and files now live in OpenVibe.Media;
  > Live proxies to it (`server/media-proxy/`) and starts/stops recordings (`server/streaming/recorder.js`).
--> The local `vods`/`clips` tables are frozen and read-only. Implementation details below may be out of date.
++> Pastes live in OpenVibe.Community. Live's old local `vods`/`clips`/`pastes` tables are frozen: nothing
++> writes or reads them ([below](#live-reads-nothing-from-the-frozen-tables)), and they are dropped by
++> [this procedure](#dropping-the-frozen-tables). Implementation details below may be out of date.
  
  ## Recording System
  
@@@ -90,3 +92,195 @@ Two jobs turn stream moments into content: `server/ai/ai-moments-job.js` (image
  Go-live used to default every stream to `irl`, so every AI prompt (AI viewers, streamer overviews, VOD overviews, the Arena, sound detection) assumed everyone is an IRL streamer. Now the category selector defaults to **Auto** (stored as `NULL`), and the stream-memory rollup — the same summarise call that already runs — returns `{ overview, category, tags }` with `category` from the fixed taxonomy (`outdoors, travel, building, music, gaming, robot, desktop, irl, other`). It is stored on `streams.ai_category` / `streams.ai_tags` and the channel inherits the latest read (`channels.ai_category`).
  
  Everything reads the effective value: listing queries return `COALESCE(ai_category, category) AS category` (the self-selected value is still there as `chosen_category`), `stream_category` on VODs/clips, the AI viewers' session block ("judged from the stream itself"), streamer overviews, sound detection and the Arena all prefer `ai_category`. A streamer can still pick a category manually; the AI read wins once it exists.
++
++## Live reads nothing from the frozen tables
++
++Live's local `vods`, `clips`, `pastes`, `paste_likes` and `paste_comments` tables stopped at the Media
++split: nothing recorded, clipped or pasted since then was ever in them, and OpenVibe.Media (VODs and clips,
++same ids) and OpenVibe.Community (pastes, by `legacy_media_id` or slug) hold the old rows too. Since
++register item C-73 step 2, no Live code reads them. `test/frozen-tables.test.js` fails on any write or read
++of these tables in `server/` or `scripts/` (SQL text, split over lines too, and `table: '…'` registries). Its
++allow-list of kept reads is empty. A read that has to stay goes into that list with a reason, and into this
++page.
++
++What used to read them, and what answers now (most of it in `server/media-proxy/lookups.js`):
++
++| Page or job | Now asks | Hidden items |
++|---|---|---|
++| Recently ended streams (`/api/streams/recent`): VOD id, thumbnail, duration | Media, per stream (cached 5 min) | never: public VODs only |
++| Recently online cards: slot thumbnail | Media `GET /vods/latest-thumbs` (the route already did) | never |
++| Channel page badges: pastes, clips taken | Community (pastes), Media (clips, `hide_self`) | owner and staff only |
++| Channel offline screen: top VOD and clip per week, month, all time | Media, newest 100 plus most-viewed | never |
++| Workspace session history: each session's VOD | Media by slot (`managed_stream_id`) | owner only (private and recording included) |
++| Setup hub: "Publish a paste" | Community, as the person | the person's own |
++| Settings: avatar history | Community, as the person (screenshots tagged `metadata.kind = 'avatar'`) | the person's own |
++| AI timeline (channel AI tab): each session's VOD link | Media by user, public only; not cached while Media is down | never |
++| Transcript view: the stream's VOD | Media (the old local fallback is gone) | never |
++| Home hero, kiosk, SEO snapshot: VOD, clip, hour and paste counts | Media `/stats`; pastes from Community `/api/pastes/admin/stats` when it owns them | counts only |
++| Home "over time" charts for VODs, clips, hours | Media `/stats/series/*` (the route already did) | counts only |
++| Admin dashboard: VOD totals | Media list totals (null when Media is down) | staff |
++| Admin AI explorer and the streamer overview job: VODs, clips, pastes | Media and Community | staff: any visibility (the overview job reads public VODs) |
++| AI moments and auto-clip jobs: VOD pool, VOD source, where viewers clipped | Media (no local fallback: Media down means no run) | internal signal only |
++| Stream analytics: clips per stream | Media, written into `stream_analytics.clips_created` right after the stream ends | internal |
++| Arena portraits: VOD thumbnails | Media, public VODs | never |
++
++Removed from `server/db/database.js`: 45 exported helpers that had no caller (VOD health, public VOD and clip
++lists, the old paste API, paste comments and likes, filtered channel lists), the 11 whose callers now ask Media
++or Community, and the boot-time migrations that still rewrote rows of the frozen `vods`/`clips` tables (short
++overviews, transcript status) along with the indexes that only served the removed listings. The streamer
++overview job now only considers streamers with stream memories; the frozen `vods` table was its other signal,
++and its rows moved to Media at the split.
++
++### API additions that would help (not blocking)
++
++- **Media: `GET /vods?stream_ids=1,2,3`** (a batch filter). `/api/streams/recent` asks once per stream now, up
++  to 100 calls on a cold cache.
++- **Media: `created_after` on `GET /vods` and `GET /clips`.** The offline screen's week and month windows are
++  cut from the newest 100 items; a streamer with more than 100 in a month loses that month's oldest.
++- **Media: `order=peak_viewers` for real.** It is an alias of `views` in Media. Live has `streams.peak_viewers`
++  but cannot sort Media's list by it.
++- **Community: list by owner subject** (`GET /api/pastes?owner=usr_…`). Live lists a person's pastes by
++  username, which relies on Community's projection cache knowing that username.
++- **Community: a metadata filter** (`GET /api/pastes?metadata.kind=avatar`). Avatar history lists up to 200 of
++  the person's screenshots and keeps the avatar-tagged ones.
++- **Community: a public paste count.** The hero reads Community's staff-only `/api/pastes/admin/stats` with
++  Live's `community.paste.moderate` grant. Media's `/stats` and `/stats/series/pastes` still count Media's
++  frozen paste copies; the hero's paste chart does too.
++
++## Dropping the frozen tables
++
++Register item C-73 step 3. **Start no earlier than 30 days after the deploy that contains C-73 step 2**
++(the commit that made `test/frozen-tables.test.js` refuse reads), and only if every deploy since then passed
++that test. Nothing in the procedure needs Live to stop. It drops five tables from Live's database:
++`vods`, `clips`, `pastes`, `paste_likes`, `paste_comments`. Their indexes go with them. Live's `comments` table
++is not part of this (OpenVibe.Community's comment import reads it).
++
++Paths on the production host (check `DB_PATH` in `/etc/openvibe/live.env` first):
++
++```bash
++LIVE_DB=/opt/openvibe.live/data/live.db              # -> /opt/openvibe.live/shared/data/live.db
++MEDIA_DB=/opt/openvibe.media/data/media.db
++COMMUNITY_DB=/var/lib/openvibe-community/community.db
++STAMP=$(date -u +%Y%m%d)
++BACKUP_DIR=/opt/backups
++```
++
++### 1. Ship the code that stops creating the tables
++
++One Live commit, deployed before anything is dropped. Without it, the next boot recreates the five tables empty.
++
++- `server/db/schema.sql`: remove `CREATE TABLE vods`, `CREATE TABLE clips` and the indexes
++  `idx_vods_user_id`, `idx_clips_stream_id`, `idx_clips_user_id`.
++- `server/db/database.js` `initDb()`: remove every block that creates or alters these tables. That is the `vods`
++  column adds (thumbnail_url through is_recording, storage_provider/storage_key, ai_overview/ai_transcript/
++  ai_analyzed_at, visibility), the `clips` column adds (storage_provider/storage_key, the AI columns,
++  clip_notified/clip_notify_at, visibility, auto_generated), the two visibility backfills, and the `pastes`,
++  `paste_likes` and `paste_comments` blocks (tables, indexes, the copies/likes/is_nsfw columns). The
++  `idx_vods_*`/`idx_clips_*` listing and transcript indexes already left the boot code in step 2; the ones
++  production still has go with their tables.
++- `server/db/migrations.js`: leave `002_pastes_ai_columns` in place. Its ledger row exists in production;
++  with the table gone, `up` returns `DEFER` and changes nothing.
++- `test/frozen-tables.test.js`: empty `ALLOWED_WRITES` (the backfills are gone), and add a check that
++  `initDb()` on a new database creates none of the five tables.
++
++Deploying this changes no data: the existing tables stay where they are, and nothing reads, writes, creates or
++alters them.
++
++### 2. Back up
++
++```bash
++sudo mkdir -p "$BACKUP_DIR"
++# The whole database, online (safe while Live runs):
++sudo sqlite3 "$LIVE_DB" ".timeout 10000" ".backup '$BACKUP_DIR/live-pre-c73-drop-$STAMP.db'"
++# The five tables alone, as SQL that recreates them:
++sudo sqlite3 -readonly "$LIVE_DB" ".dump vods clips pastes paste_likes paste_comments" \
++  | gzip -9 | sudo tee "$BACKUP_DIR/live-frozen-tables-$STAMP.sql.gz" > /dev/null
++sudo sqlite3 -readonly "$BACKUP_DIR/live-pre-c73-drop-$STAMP.db" "PRAGMA integrity_check"     # ok
++sudo sha256sum "$BACKUP_DIR"/live-pre-c73-drop-$STAMP.db "$BACKUP_DIR"/live-frozen-tables-$STAMP.sql.gz \
++  | sudo tee "$BACKUP_DIR/live-c73-$STAMP.sha256"
++```
++
++Keep both files for at least a year, and copy them off the host with the usual backups. They are the only
++copy of any row that step 3 finds missing upstream.
++
++### 3. Reconcile (read-only)
++
++```bash
++sudo sqlite3 -readonly "$LIVE_DB" <<SQL
++.headers on
++SELECT 'vods' AS t, COUNT(*) AS n FROM vods UNION ALL SELECT 'clips', COUNT(*) FROM clips
++UNION ALL SELECT 'pastes', COUNT(*) FROM pastes UNION ALL SELECT 'paste_likes', COUNT(*) FROM paste_likes
++UNION ALL SELECT 'paste_comments', COUNT(*) FROM paste_comments;
++ATTACH '$MEDIA_DB' AS m;
++ATTACH '$COMMUNITY_DB' AS c;
++SELECT 'vods missing in Media' AS t, COUNT(*) AS n FROM main.vods v
++  WHERE NOT EXISTS (SELECT 1 FROM m.vods x WHERE x.id = v.id AND x.app_id = 'live');
++SELECT 'clips missing in Media' AS t, COUNT(*) AS n FROM main.clips k
++  WHERE NOT EXISTS (SELECT 1 FROM m.clips x WHERE x.id = k.id AND x.app_id = 'live');
++SELECT 'pastes missing in Community' AS t, COUNT(*) AS n FROM main.pastes p
++  WHERE NOT EXISTS (SELECT 1 FROM c.pastes x WHERE x.legacy_media_id = p.id OR x.slug = p.slug);
++SELECT 'paste_likes missing in Media' AS t, COUNT(*) AS n FROM main.paste_likes l
++  WHERE NOT EXISTS (SELECT 1 FROM m.paste_likes x WHERE x.paste_id = l.paste_id AND x.user_id = l.user_id);
++SELECT 'paste_comments missing in Media' AS t, COUNT(*) AS n FROM main.paste_comments pc
++  WHERE NOT EXISTS (SELECT 1 FROM m.paste_comments x WHERE x.id = pc.id);
++-- Anything else in Live's schema that still points at the five tables (foreign keys, views, triggers):
++SELECT m2.name AS referencing_table, f."table" AS frozen FROM sqlite_master m2, pragma_foreign_key_list(m2.name) f
++  WHERE m2.type = 'table' AND f."table" IN ('vods','clips','pastes','paste_likes','paste_comments')
++    AND m2.name NOT IN ('vods','clips','pastes','paste_likes','paste_comments');
++SELECT type, name FROM sqlite_master WHERE type IN ('view','trigger')
++  AND (sql LIKE '%vods%' OR sql LIKE '%clips%' OR sql LIKE '%pastes%' OR sql LIKE '%paste_likes%' OR sql LIKE '%paste_comments%');
++SQL
++```
++
++The media split dropped these tables from `live.db` and `schema.sql` recreated them empty, so the first query
++most likely returns 0 for all five. Whatever it returns, **every "missing" count must be 0, and the last two
++queries must return no rows.** Otherwise stop: export the missing rows to Media or Community first (and log
++them in the register), or fix what still references the tables, then run step 3 again.
++
++### 4. Drop
++
++Children first, foreign keys off, in one transaction. Live keeps running: it has no statement on these tables.
++
++```bash
++sudo sqlite3 "$LIVE_DB" <<'SQL'
++.timeout 10000
++PRAGMA foreign_keys = OFF;
++BEGIN IMMEDIATE;
++DROP TABLE IF EXISTS paste_comments;
++DROP TABLE IF EXISTS paste_likes;
++DROP TABLE IF EXISTS clips;
++DROP TABLE IF EXISTS pastes;
++DROP TABLE IF EXISTS vods;
++COMMIT;
++PRAGMA foreign_key_check;
++SQL
++```
++
++If the tables held many rows, reclaim the space later with `VACUUM` at a quiet hour. It locks the database
++while it runs.
++
++### 5. Verify
++
++```bash
++sudo sqlite3 -readonly "$LIVE_DB" "SELECT name FROM sqlite_master WHERE name IN ('vods','clips','pastes','paste_likes','paste_comments')"   # nothing
++sudo systemctl restart openvibe-live            # socket activation: no 502s; boot ~6 s
++curl -fsS http://127.0.0.1:3000/api/ready
++sudo sqlite3 -readonly "$LIVE_DB" "SELECT name FROM sqlite_master WHERE name IN ('vods','clips','pastes','paste_likes','paste_comments')"   # still nothing
++sudo journalctl -u openvibe-live --since '-10 min' | grep -iE 'no such table|SQLITE_ERROR' || echo clean
++for p in /api/streams/recent /api/streams/recently-online /api/streams/recent-vods /api/home/hero; do
++  curl -fsS -o /dev/null -w "%{http_code} $p\n" "http://127.0.0.1:3000$p"; done
++```
++
++Then open a channel page (Videos, Clips, Pastes and AI tabs), the home page, and the dashboard's session history.
++
++### Rollback
++
++Nothing in Live reads the tables, so a rollback is only ever needed to get rows back. Recreate the five tables
++from the dump (it holds their `CREATE TABLE`, the rows and the indexes):
++
++```bash
++gunzip -c "$BACKUP_DIR/live-frozen-tables-$STAMP.sql.gz" | sudo sqlite3 "$LIVE_DB"
++```
++
++`live-pre-c73-drop-$STAMP.db` is the second copy. Afterwards, record the date, the counts from step 3 and the
++backup paths under C-73 in OpenVibe.Host's `docs/compatibility-register.md`.