openvibers/openvibe.meme Commits
Meme v2: a judge for the candidates, and the encyclopedia as its base (#33)
Changes
@@@ -36,12 +36,47 @@ MEME_EVENTS_URL=#MEME_EVENTS_ENDPOINT=http://127.0.0.1:5120/internal/events#MEME_EVENTS_INTERVAL_MS=5000--# The public APIs the importer and the citation re-verifier read. No service grant is used in v1.++# The public APIs the importer and the citation re-verifier read. No service grant is used for them.OV_CHAT_URL=https://openvibe.chatOV_LIVE_URL=https://openvibe.liveOV_VIDEO_URL=https://openvibe.video#MEME_UPSTREAM_TIMEOUT_MS=8000++# OpenVibe.AI: the judge and the encyclopedia's writing pass. Meme calls ai.generate as the service `meme` with a++# Network client-credentials token (audience openvibe.ai, scope ai.run.create); Network grants the ai.generate++# namespace. Meme never names a provider — AI's routing decides.++OV_AI_INTERNAL_URL=http://127.0.0.1:4700++#MEME_AI_AUDIENCE=openvibe.ai++#MEME_AI_SCOPE=ai.run.create++#MEME_AI_WAIT_MS=55000++#MEME_AI_TIMEOUT_MS=120000++#MEME_AI_POLL_MS=1500++#MEME_AI_MAX_TOKENS=1400++++# The judge: it scores a candidate from its own citations and writes the entry. Only what clears the bar is++# published; everything else stays hidden for staff with its score.++#MEME_JUDGE_INTERVAL_MS=1800000++#MEME_JUDGE_MIN_SCORE=0.75++#MEME_JUDGE_MIN_USES=3++#MEME_JUDGE_MIN_DAYS=2++#MEME_JUDGE_MIN_CHANNELS=2++#MEME_JUDGE_MAX_PER_RUN=10++#MEME_JUDGE_CONTEXT_LINES=6++++# The encyclopedia and the templates: Wikipedia (CC BY-SA), Wikidata (CC0) and Wikimedia Commons (each file under++# its own licence). Read-only public APIs; a site whose terms forbid it is never scraped.++#MEME_WIKIPEDIA_URL=https://en.wikipedia.org++#MEME_WIKIDATA_URL=https://www.wikidata.org++#MEME_COMMONS_URL=https://commons.wikimedia.org++#MEME_WIKI_USER_AGENT=OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])++#MEME_WIKI_TIMEOUT_MS=20000++#MEME_WIKI_MIN_INTERVAL_MS=250++#MEME_ENCYCLOPEDIA_LIMIT=400++#MEME_ENCYCLOPEDIA_LISTS=List_of_Internet_phenomena++#MEME_ENCYCLOPEDIA_CATEGORIES=Internet memes,Memes++#MEME_ENCYCLOPEDIA_MIN_SCORE=0.75++#MEME_TEMPLATES_LIMIT=40++# The candidate importer: every 30 minutes, hidden candidates only, off under MEME_DRILL.#MEME_IMPORTER_INTERVAL_MS=1800000#MEME_IMPORTER_MIN_USES=2@@@ -19,6 +19,17 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta- **Candidates stay hidden.** `server/importer.js` reads public Chat/Live APIs only, finds a 3–6 word phrase used atleast twice on two different days, attaches citations and inserts the entry `hidden`. It never publishes. A staffperson (Network role `global_mod` or above) publishes it with `POST /api/v1/entries/:slug/publish`.++- **A judge writes the entry.** `server/ai/judge.js` asks OpenVibe.AI to score each candidate from its own++ citations — with the chat lines and transcript lines around each quote — and to write the entry from that material++ only: a title in natural case, a short summary, where it started, how people use it and two or three example uses.++ The score and the reason are stored with the entry.++- **Only what clears the bar is published.** Score 0.75 or more, used at least three times across at least two days++ or two channels, nothing denied, nothing about anyone's looks, the entry complete and a live origin citation.++ Everything else stays hidden for staff, with its score, on `GET /api/v1/staff/candidates`. Every decision is a++ status event.++- **The encyclopedia.** The well-known internet memes, seeded from Wikipedia's lists (CC BY-SA) and Wikidata (CC0)++ with a Commons picture only when its licence allows it, each an entry of kind `classic` with its origin, its++ spread, its sources and its attribution.- **The citation is checked again.** A daily job reads every public citation URL; a 404/410 marks it `gone`, a401/403 marks it `hidden`, and an entry whose origin is gone is hidden (revision +1, Search tombstone).- **Meme of the week.** People vote once per ISO week; a second vote changes the choice, never adds a ballot.@@@ -29,11 +40,12 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta| Table | Holds | Migration ||---|---|---|--| `meme_entries` | `mem_` + ULID, unique slug (`/m/<slug>`), title, kind `joke\|catchphrase\|bit\|meme\|saga`, AI summary, status + `status_since`, first/last seen, visibility `public\|unlisted\|hidden`, `revision` (+1 on every change), timestamps | `migrations/0001_initial.sql` |--| `meme_citations` | entry, role `origin\|use\|example`, source service/type/id, stream/VOD/clip/message ids, canonical VOD `offset_seconds`, verbatim `quote_text`, `quote_hash`, canonical `url`, `observed_at`, `source_status` `live\|hidden\|gone` | `0001_initial.sql` |--| `meme_status_events` | every status change: from, to, reason, who, when | `0001_initial.sql` |++| `meme_entries` | `mem_` + ULID, unique slug (`/m/<slug>`), title, kind `joke\|catchphrase\|bit\|meme\|saga\|classic`, AI summary, status + `status_since`, first/last seen, visibility `public\|unlisted\|hidden`, `revision` (+1 on every change), timestamps, the picture and its licence, the attribution, the Wikidata id | `migrations/0001_initial.sql`, `0002_judge_and_encyclopedia.sql` |++| `meme_citations` | entry, role `origin\|use\|example`, source service/type/id, stream/VOD/clip/message ids, canonical VOD `offset_seconds`, verbatim `quote_text`, `quote_hash`, canonical `url`, `observed_at`, `source_status` `live\|hidden\|gone`; the open web is a source too (`wikipedia`, `wikidata`, `commons`) | `0001_initial.sql`, `0002_judge_and_encyclopedia.sql` |++| `meme_judgements` | one row per entry: the score, the reason, the title/summary/origin/spread/examples the AI wrote, whether the text names looks or carries a denied word, the model and the run | `0002_judge_and_encyclopedia.sql` |++| `meme_status_events` | every status change: from, to, reason, who, when; a judge decision adds `decision`, `from_visibility`, `to_visibility` and `score` | `0001_initial.sql`, `0002_judge_and_encyclopedia.sql` || `meme_votes` | (`week` `YYYY-Www`, `subject`) → entry; one vote a week | `0001_initial.sql` |--| `meme_templates` | id, name, `image_url` (openvibe.media or a local data-free placeholder), `box_count`, source, license; three placeholders seeded | `0001_initial.sql` |++| `meme_templates` | id, name, `image_url` (openvibe.media, a local data-free placeholder or a Wikimedia Commons file), `box_count`, source, license, credit, source page, attribution; three placeholders seeded | `0001_initial.sql`, `0002_judge_and_encyclopedia.sql` || `meme_captions` | `mcp_` + ULID, subject, template, `lines` jsonb (1–4 plain-text lines), visibility, created | `0001_initial.sql` || `meme_reports` | `mrp_` + ULID, entry, subject, kind `takedown\|correction`, reason, note, status | `0001_initial.sql` || `meme_outbox` | Today's outbox shape: fixed `evt_` id, topic, JSON payload, attempts, last error, claim/backoff lease — the Search document queue | `0001_initial.sql` |@@@ -43,8 +55,8 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta## Does not own- **Chat, Live and the VODs** — OpenVibe.Chat and OpenVibe.Live. Meme reads their public history, Rundown and-- transcripts. It never reads a private VOD, never uses a service grant in v1, and never cites VOD **5709** or clip-- **483** (the fixed deny list).++ transcripts. It never reads a private VOD, and it never cites VOD **5709** or clip **483** (the fixed deny list).++ The one service grant Meme holds is OpenVibe.AI's `ai.run.create`, used by the judge and the encyclopedia.- **Identity, roles and accounts** — OpenVibe.Network. Meme verifies session tokens against its JWKS; a staffroute asks for `global_mod` or above.- **Search** — OpenVibe.Search. Meme produces `search.index-document@1` documents in its own outbox; the Search@@@ -61,6 +73,15 @@ Environment variable names are in [.env.example](.env.example); no value is comm`OV_OAUTH_REDIRECT_URI`, `OV_SESSION_AUDIENCE`, `MEME_AUDIENCE`.- **OpenVibe.Chat / Live / Video** — the public reads: `OV_CHAT_URL`, `OV_LIVE_URL`, `OV_VIDEO_URL`,`MEME_UPSTREAM_TIMEOUT_MS`.++- **OpenVibe.AI** — the judge and the encyclopedia's writing pass: `OV_AI_INTERNAL_URL` (default++ `http://127.0.0.1:4700`), `MEME_AI_AUDIENCE` (`openvibe.ai`), `MEME_AI_SCOPE` (`ai.run.create`),++ `MEME_AI_WAIT_MS`, `MEME_AI_TIMEOUT_MS`, `MEME_AI_POLL_MS`, `MEME_AI_MAX_TOKENS`. Meme calls `ai.generate` as++ the service `meme` with a Network client-credentials token; Network grants `ai.run.create` for the `ai.generate`++ namespace. Meme never names a provider — AI's routing decides, and the run reports what answered.++- **Wikipedia, Wikidata and Wikimedia Commons** — the encyclopedia: `MEME_WIKIPEDIA_URL`, `MEME_WIKIDATA_URL`,++ `MEME_COMMONS_URL`, `MEME_WIKI_USER_AGENT`, `MEME_WIKI_TIMEOUT_MS`, `MEME_WIKI_MIN_INTERVAL_MS`,++ `MEME_ENCYCLOPEDIA_LIMIT`, `MEME_ENCYCLOPEDIA_LISTS`, `MEME_ENCYCLOPEDIA_CATEGORIES`,++ `MEME_ENCYCLOPEDIA_MIN_SCORE`, `MEME_TEMPLATES_LIMIT`.- **OpenVibe.Events** — account deliveries: `MEME_EVENTS_SECRET`, `MEME_EVENTS_URL`, `MEME_EVENTS_ENDPOINT`.- **PostgreSQL** — `DATABASE_URL` (through PgBouncer) and `DATABASE_DIRECT_URL` (migrations); embedded PGlite in`MEME_PGLITE_DIR` for development.@@@ -78,7 +99,7 @@ also gets per-actor minute/hour limits (`MEME_LIMITS_MINUTE`, `MEME_LIMITS_HOUR`| Route | What it does ||---|---|| `GET /entries?limit=&cursor=&kind=&status=` | public entries, newest first; opaque keyset cursor; `Cache-Control: public, max-age=60` |--| `GET /entries/:slug` | one entry with `origin`, every `citation`, the use `timeline` and `examples`; a hidden one is 404 unless staff |++| `GET /entries/:slug` | one entry with `origin`, every `citation`, the use `timeline` and `examples`, and the judge's `notes` (where it started, how people use it, the examples); a hidden one is 404 unless staff || `GET /entries/:slug/series?days=30` | uses per UTC day, zero-filled, 1–365 days || `GET /trending?limit=` | public entries, `rising` first, then `peaked`, `classic`, `dead`; last use newest first || `GET /week` | this ISO week's candidates (public entries with a live use that week) and their vote counts |@@@ -88,6 +109,8 @@ also gets per-actor minute/hour limits (`MEME_LIMITS_MINUTE`, `MEME_LIMITS_HOUR`| `GET /captions/:id` | one caption; a public read never returns the author's subject || `POST /entries/:slug/report` | signed in; `{ kind: "takedown"\|"correction", reason, note? }`, stored for staff || `GET /reports` | staff: the open reports |++| `GET /staff/candidates` | staff: the hidden candidates with the judge's score, its reason, the uses, the days and the channels |++| `GET /staff/candidates/:slug` | staff: one candidate — its judgement, its citations and its decision history || `POST /entries/:slug/publish` | staff: hidden → public; `409 meme.origin_gone` when the origin is not live || `POST /entries/:slug/hide` | staff: public → hidden; Search tombstone || `PATCH /entries/:slug` | staff: `{ title, kind, status, reason? }`; a status change writes `meme_status_events` |@@@ -96,6 +119,47 @@ Validation refusals are `400 { code: "meme.invalid", message }`; a blocked capti`400 { code: "meme.blocked_words" }`. Other codes: `meme.sign_in_required`, `meme.staff_required`,`meme.cross_site`, `meme.not_found`, `meme.template_not_found`, `meme.origin_gone`, `meme.rate_limited`.++## The judge++++`server/jobs/judge.js` runs every 30 minutes (`MEME_JUDGE_INTERVAL_MS`), off under `MEME_DRILL`; tests call++`runOnce()`.++++For every candidate nobody has judged yet it:++++1. reads the entry's live citations and opens each one up — the chat lines around a chat message, the transcript++ lines around a VOD second (`server/ai/context.js`), because a joke only makes sense with its neighbours;++2. asks OpenVibe.AI to score it from 0 to 1 on whether it is a real meme, catchphrase or inside joke rather than++ everyday words that recur, and to write the entry from that material only;++3. stores the score, the reason and the written entry in `meme_judgements`;++4. applies the publish rule (`server/ai/gate.js`): score ≥ `MEME_JUDGE_MIN_SCORE` (0.75), at least++ `MEME_JUDGE_MIN_USES` (3) uses across at least `MEME_JUDGE_MIN_DAYS` (2) days or `MEME_JUDGE_MIN_CHANNELS` (2)++ channels, nothing denied, nothing that names looks, the entry complete, and a live origin citation;++5. publishes what cleared the bar and leaves the rest hidden, logging both as a status event with the score and++ the reasons.++++The AI's text is never taken at its word: a title that names looks is dropped (the entry keeps the name it had),++every other field is repaired word by word with `noLooks`, and a field that still names looks — or that carries a++word from the deny list — keeps the entry hidden.++++`node scripts/rejudge.js` runs one round over every hidden candidate and prints what was published and why;++`--dry-run` judges and reports without publishing, `--slug <slug>` judges only those.++++## The encyclopedia++++`node scripts/seed-classics.js` seeds the well-known internet memes; `node scripts/seed-templates.js` seeds the++caption templates. Neither starts the server.++++The classics are read from Wikipedia's lists of Internet phenomena and memes (CC BY-SA) and from the categories++behind them, and every entry is finished from Wikidata (CC0): the Q id, the inception date and the picture. A++Commons picture is used only when its licence allows it (public domain, CC0, CC), and each entry keeps its credit,++its licence, its file page and the attribution of the text it was written from. The same AI pass writes the entry++— the title in natural case, the summary, where it started, how it spread, two or three example uses — and the++same publish rule decides what becomes public: an entry the AI scored under `MEME_ENCYCLOPEDIA_MIN_SCORE` (0.75),++or whose text names looks or carries a denied word, stays hidden for staff with its score.++++Nothing is scraped from a site whose terms forbid it: Wikipedia, Wikidata and Wikimedia Commons only, each with++its User-Agent, paced per host.++## The importer`server/importer.js` runs every 30 minutes (`MEME_IMPORTER_INTERVAL_MS`), off under `MEME_DRILL`; tests call@@@ -180,13 +244,18 @@ npm run test:pg # the same through PostgreSQL + PgBouncer (openvibe-sdkOV_ALLOW_SUITE=1 npm testnpm start # PORT 5120, loopback; DATABASE_URL or an embedded PGlite in developmentnode scripts/seed-proof.js # the first proof, hidden, from the live public APIs++node scripts/rejudge.js # one judge round over every hidden candidate; prints what was published++node scripts/seed-classics.js # the encyclopedia, from Wikipedia and Wikidata++node scripts/seed-templates.js # the caption templates, from Wikimedia Commons```--The tests boot the app with a mock Network, a Chat/Live/Video fixture server, and a temporary database: the--migration, the importer's candidate and its canonical redirect, the deny list, `noLooks`, the one-vote-a-week rule,--the caption limits, staff publish/patch/hide, the Search document (validated against++The tests boot the app with a mock Network, an OpenVibe.AI stand-in, a Chat/Live/Video fixture server, and a++temporary database: the migration, the importer's candidate and its canonical redirect, the deny list, `noLooks`,++the one-vote-a-week rule, the caption limits, staff publish/patch/hide, the Search document (validated against`search.index-document@1`), the citation 404 → gone → hidden path, sign-in, the pages and cache headers, and the--account export/deletion over votes, captions and reports.++account export/deletion over votes, captions and reports. The judge, the publish rule, the encyclopedia's++attribution and the templates' licences have their own files: `test/judge.test.js`, `test/encyclopedia.test.js`++and `test/templates.test.js`.## Deploynnew file mode 100644@@@ -0,0 +1,68 @@++-- phase: expand++-- The judge and the encyclopedia. A candidate is judged by OpenVibe.AI from its own citations: a score, a reason and++-- the written entry (title, summary, where it started, how people use it, two or three examples). The judgement is++-- kept beside the entry, never inside it, so the entry stays the same shape it was.++--++-- The encyclopedia adds entries of kind 'classic': the well-known internet memes, cited to Wikipedia (CC BY-SA) and++-- to Wikidata (CC0) with a Commons image only when its licence allows it. A citation from the open web is a new++-- source_service ('web') and a new source_type ('wikipedia' | 'wikidata' | 'commons'); the OpenVibe sources keep++-- their own. Every decision the judge takes is a status event with its score, so the history says why.++--++-- Nothing here is a secret: a citation stores a public URL and the sha256 of the quoted text.++++-- 'classic' joins the kinds: an encyclopedia entry is not an OpenVibe running joke.++ALTER TABLE meme_entries DROP CONSTRAINT meme_entries_kind;++ALTER TABLE meme_entries ADD CONSTRAINT meme_entries_kind CHECK (kind IN ('joke', 'catchphrase', 'bit', 'meme', 'saga', 'classic'));++++-- The open web is a source now: Wikipedia, Wikidata and Wikimedia Commons.++ALTER TABLE meme_citations DROP CONSTRAINT meme_citations_source_service;++ALTER TABLE meme_citations ADD CONSTRAINT meme_citations_source_service CHECK (source_service IN ('live', 'chat', 'community', 'web'));++ALTER TABLE meme_citations DROP CONSTRAINT meme_citations_source_type;++ALTER TABLE meme_citations ADD CONSTRAINT meme_citations_source_type CHECK (source_type IN ('chat_message', 'vod', 'clip', 'rundown_segment', 'ai_paste', 'stream_memory', 'wikipedia', 'wikidata', 'commons'));++++-- An encyclopedia entry keeps its picture and where it came from. The licence is the Commons file's own.++ALTER TABLE meme_entries ADD COLUMN image_url text;++ALTER TABLE meme_entries ADD COLUMN image_credit text;++ALTER TABLE meme_entries ADD COLUMN image_license text;++ALTER TABLE meme_entries ADD COLUMN image_license_url text;++ALTER TABLE meme_entries ADD COLUMN image_source_url text;++ALTER TABLE meme_entries ADD COLUMN attribution text; -- the licences the entry's text is under++ALTER TABLE meme_entries ADD COLUMN external_id text; -- the Wikidata Q id, when there is one++CREATE UNIQUE INDEX meme_entries_external_id ON meme_entries (external_id) WHERE external_id IS NOT NULL;++++-- One judgement per entry: what the AI decided, why, and what it wrote.++CREATE TABLE meme_judgements (++ entry_id text COLLATE "C" PRIMARY KEY REFERENCES meme_entries (id) ON DELETE CASCADE,++ score numeric(4,3) NOT NULL, -- 0 to 1++ reason text NOT NULL, -- why the AI gave that score++ title text, -- the AI's title, in natural case++ summary text, -- the AI's short summary++ origin text, -- where it started++ spread text, -- how people use it, and how it spread++ examples jsonb NOT NULL DEFAULT '[]'::jsonb, -- two or three example uses++ denied boolean NOT NULL DEFAULT false, -- a denied source or a denied word++ looks boolean NOT NULL DEFAULT false, -- the text named looks++ model text,++ provider text,++ run_id text,++ judged_at timestamptz NOT NULL DEFAULT now(),++ judged_by text COLLATE "C" NOT NULL DEFAULT 'service:meme:judge',++ CONSTRAINT meme_judgements_score CHECK (score >= 0 AND score <= 1),++ CONSTRAINT meme_judgements_examples CHECK (jsonb_typeof(examples) = 'array')++);++CREATE INDEX meme_judgements_by_score ON meme_judgements (score DESC);++++-- A decision is a status event: the visibility moved, the score says why.++ALTER TABLE meme_status_events ADD COLUMN decision text; -- publish | keep_hidden | judge++ALTER TABLE meme_status_events ADD COLUMN from_visibility text;++ALTER TABLE meme_status_events ADD COLUMN to_visibility text;++ALTER TABLE meme_status_events ADD COLUMN score numeric(4,3);++ALTER TABLE meme_status_events ADD CONSTRAINT meme_status_events_decision CHECK (decision IS NULL OR decision IN ('publish', 'keep_hidden', 'judge', 'hide', 'seed'));++++-- A template says where its picture came from and under which licence.++ALTER TABLE meme_templates ADD COLUMN credit text;++ALTER TABLE meme_templates ADD COLUMN source_url text;++ALTER TABLE meme_templates ADD COLUMN attribution text;++-- The picture may be an openvibe.media object, a file of this site, or a Wikimedia Commons file under a free licence.++ALTER TABLE meme_templates DROP CONSTRAINT meme_templates_image;++ALTER TABLE meme_templates ADD CONSTRAINT meme_templates_image CHECK (image_url ~ '^(https://openvibe\.media/|/img/|https://upload\.wikimedia\.org/)');@@@ -8,6 +8,9 @@"start": "node server/index.js","dev": "NODE_ENV=development node server/index.js","seed:proof": "node scripts/seed-proof.js",++ "seed:classics": "node scripts/seed-classics.js",++ "seed:templates": "node scripts/seed-templates.js",++ "rejudge": "node scripts/rejudge.js","test": "node test/run.js","test:pg": "MEME_TEST_STORE=pg node test/run.js"},nnew file mode 100644@@@ -0,0 +1,91 @@++#!/usr/bin/env node++'use strict';++++/**++ * The re-judge round: every candidate is judged again from its own citations, and the publish rule decides what++ * becomes public. The script says what it did, entry by entry, with the score and the reasons.++ *++ * node scripts/rejudge.js # every hidden candidate++ * node scripts/rejudge.js --slug cozmo-s-alive --slug wait-wait-wait++ * node scripts/rejudge.js --dry-run # judge and report, publish nothing++ *++ * A candidate that clears the bar (0.75 or more, used at least three times across at least two days or two++ * channels, nothing denied, nothing about looks) is published and the decision is a status event. Everything else++ * stays hidden for staff, with its score on GET /api/v1/staff/candidates.++ */++process.env.NODE_ENV = process.env.NODE_ENV || 'development';++require('dotenv').config();++++const { load } = require('../server/config');++const { openStore } = require('../server/db');++const { createStore } = require('../server/store');++const { createSearchIndex } = require('../server/search');++const { createUpstreams } = require('../server/upstreams');++const { createMemeAi } = require('../server/ai/client');++const { createContext } = require('../server/ai/context');++const { createJudge } = require('../server/ai/judge');++const { createJudgeJob } = require('../server/jobs/judge');++++const arg = (name) => {++ const i = process.argv.indexOf(`--${name}`);++ if (i < 0) return null;++ const next = process.argv[i + 1];++ return next && !next.startsWith('--') ? next : true;++};++++async function main() {++ const config = load();++ const slugs = process.argv.reduce((acc, v, i, all) => (v === '--slug' && all[i + 1] ? [...acc, all[i + 1]] : acc), []);++ const dryRun = process.argv.includes('--dry-run');++ const s = await openStore(config, { log: console });++ try {++ const search = createSearchIndex({ config, s });++ const entries = createStore(s, { search });++ const upstreams = createUpstreams({ config, fetchImpl: globalThis.fetch, log: console });++ const ai = createMemeAi({ config, fetchImpl: globalThis.fetch, log: console });++ const context = createContext({ config, upstreams, log: console });++ const judge = createJudge({ config, ai, context, log: console });++ const job = createJudgeJob({ config, s, entries, judge, context, log: console });++ if (!ai.enabled) throw new Error('OpenVibe.AI is not configured (OV_OAUTH_CLIENT_SECRET and OV_AI_INTERNAL_URL); nothing was judged');++++ const before = await entries.listHidden({ limit: 500 });++ const out = await job.runOnce({ force: !slugs.length, only: slugs.length ? slugs : null, publish: !dryRun });++ const after = await entries.listHidden({ limit: 500 });++++ const published = [];++ const kept = [];++ for (const entry of before) {++ const verdict = await entries.judgement(entry.id);++ const row = after.find((e) => e.id === entry.id);++ const line = {++ slug: entry.slug,++ title: entry.title,++ score: verdict ? verdict.score : null,++ reason: verdict ? verdict.reason : null,++ uses: row ? row.uses : entry.uses,++ days: row ? row.days : 0,++ channels: row ? row.channels : 0,++ visibility: row ? row.visibility : 'published',++ };++ if (!row) published.push(line);++ else kept.push(line);++ }++ console.log(`[rejudge] ${JSON.stringify(out)}`);++ for (const line of published) console.log(`[rejudge] PUBLISHED ${line.slug} — ${line.score} — ${line.reason}`);++ for (const line of kept) console.log(`[rejudge] hidden ${line.slug} — ${line.score} — ${line.uses} uses, ${line.days} days, ${line.channels} channels`);++ console.log(`[rejudge] published ${published.length}, kept hidden ${kept.length}`);++ if (dryRun) console.log('[rejudge] dry run: the entries above were judged; publish them with the job again without --dry-run');++ process.exitCode = out.ok ? 0 : 1;++ } finally {++ await s.close();++ }++}++++if (require.main === module) {++ main().catch((err) => {++ console.error(`[rejudge] ${err && err.message ? err.message : err}`);++ process.exitCode = 1;++ });++}++++module.exports = { main };nnew file mode 100644@@@ -0,0 +1,66 @@++#!/usr/bin/env node++'use strict';++++/**++ * Seeds the encyclopedia: the well-known internet memes from Wikipedia's lists (CC BY-SA) and Wikidata (CC0), each++ * written by the same AI pass the OpenVibe candidates use, each cited to the text it was written from.++ *++ * node scripts/seed-classics.js # write what is missing, publish what clears the bar++ * node scripts/seed-classics.js --limit 20 # only the first twenty names of the list++ * node scripts/seed-classics.js --dry-run # write nothing; say what would be written++ *++ * The script never starts the HTTP server and never starts a job timer. It reads only the official public APIs of++ * Wikipedia, Wikidata and Wikimedia Commons — never a site whose terms forbid it. What is already seeded (a meme++ * with a Wikidata id already in the table) is left alone, so the script can be run again.++ */++process.env.NODE_ENV = process.env.NODE_ENV || 'development';++require('dotenv').config();++++const { load } = require('../server/config');++const { openStore } = require('../server/db');++const { createStore } = require('../server/store');++const { createSearchIndex } = require('../server/search');++const { createMemeAi } = require('../server/ai/client');++const { createContext } = require('../server/ai/context');++const { createJudge } = require('../server/ai/judge');++const { createSources } = require('../server/classics/sources');++const { createClassicsJob } = require('../server/classics/seed');++++const arg = (name) => {++ const i = process.argv.indexOf(`--${name}`);++ if (i < 0) return null;++ const next = process.argv[i + 1];++ return next && !next.startsWith('--') ? next : true;++};++++async function main() {++ const config = load();++ const limit = arg('limit');++ const dryRun = process.argv.includes('--dry-run');++ const s = await openStore(config, { log: console });++ try {++ const search = createSearchIndex({ config, s });++ const entries = createStore(s, { search });++ const upstreams = { chatContext: async () => [], vodContext: async () => [] };++ const ai = createMemeAi({ config, fetchImpl: globalThis.fetch, log: console });++ const context = createContext({ config, upstreams, log: console });++ const judge = createJudge({ config, ai, context, log: console });++ const sources = createSources({ config, fetchImpl: globalThis.fetch, log: console });++ const job = createClassicsJob({ config, s, entries, sources, judge, log: console });++ if (!ai.enabled) throw new Error('OpenVibe.AI is not configured (OV_OAUTH_CLIENT_SECRET and OV_AI_INTERNAL_URL); nothing was written');++ const out = await job.runOnce({ limit: limit ? Number(limit) : null, publish: !dryRun });++ console.log(`[seed-classics] ${JSON.stringify({ ...out, errors: (out.errors || []).slice(0, 10) })}`);++ process.exitCode = out.ok ? 0 : 1;++ } finally {++ await s.close();++ }++}++++if (require.main === module) {++ main().catch((err) => {++ console.error(`[seed-classics] ${err && err.message ? err.message : err}`);++ process.exitCode = 1;++ });++}++++module.exports = { main };nnew file mode 100644@@@ -0,0 +1,55 @@++#!/usr/bin/env node++'use strict';++++/**++ * Seeds the caption templates: the well-known meme templates from Wikimedia Commons, each under a licence that lets++ * us use it (public domain, CC0 or a Creative Commons licence), each with its credit and the file page it came from.++ *++ * node scripts/seed-templates.js # all of them++ * node scripts/seed-templates.js --limit 10 # only the first ten++ *++ * The script never starts the HTTP server. A template whose file cannot be found, or whose licence does not allow++ * reuse, is skipped and listed in the output. The three data-free placeholders of v1 are left as they are.++ */++process.env.NODE_ENV = process.env.NODE_ENV || 'development';++require('dotenv').config();++++const { load } = require('../server/config');++const { openStore } = require('../server/db');++const { createStore } = require('../server/store');++const { createSearchIndex } = require('../server/search');++const { createTemplatesJob } = require('../server/templates/seed');++++const arg = (name) => {++ const i = process.argv.indexOf(`--${name}`);++ if (i < 0) return null;++ const next = process.argv[i + 1];++ return next && !next.startsWith('--') ? next : true;++};++++async function main() {++ const config = load();++ const limit = arg('limit');++ const s = await openStore(config, { log: console });++ try {++ const search = createSearchIndex({ config, s });++ const entries = createStore(s, { search });++ const job = createTemplatesJob({ config, entries, fetchImpl: globalThis.fetch, log: console });++ const out = await job.runOnce({ limit: limit ? Number(limit) : null });++ for (const t of out.templates) console.log(`[seed-templates] ${t.id} · ${t.name} · ${t.license} · ${t.source_url}`);++ for (const m of out.missing) console.log(`[seed-templates] skipped ${m.name}: ${m.reason}`);++ console.log(`[seed-templates] ${JSON.stringify({ seeded: out.seeded, skipped: out.skipped })}`);++ process.exitCode = 0;++ } finally {++ await s.close();++ }++}++++if (require.main === module) {++ main().catch((err) => {++ console.error(`[seed-templates] ${err && err.message ? err.message : err}`);++ process.exitCode = 1;++ });++}++++module.exports = { main };nnew file mode 100644@@@ -0,0 +1,189 @@++'use strict';++++/**++ * OpenVibe.Meme's only model gateway: OpenVibe.AI's `generate` operation, called as the first-party service `meme`++ * with a client-credentials token (audience openvibe.ai, scope ai.run.create). Network grants the token and the++ * `ai.*` namespace lets it run `ai.generate` — the same path Live and Design take.++ *++ * Nothing here knows about memes: the client takes a prompt and a system message and answers the text. The judge++ * (server/ai/judge.js) is what asks a question and reads an answer.++ *++ * A run that never finishes is an error, not a hang: every call carries a deadline. AI's routing is the authority —++ * Meme never names a provider, so a pool's choice is what actually answered, and that is what is recorded.++ */++const { serviceAuth } = require('openvibe-contracts');++++class MemeAiError extends Error {++ constructor(status, code, detail) {++ super(detail || code);++ this.name = 'MemeAiError';++ this.status = status;++ this.code = code;++ this.detail = detail || code;++ }++}++++/** The JSON a model usually meant: a fenced block, a stray sentence around it, or a truncated tail. */++function parseJsonLoose(text) {++ if (!text) return null;++ const raw = String(text).trim();++ const fenced = /```(?:json)?\s*([\s\S]*?)```/i.exec(raw);++ const body = fenced ? fenced[1].trim() : raw;++ try { return JSON.parse(body); } catch { /* not JSON on its own */ }++ const start = body.indexOf('{');++ const end = body.lastIndexOf('}');++ if (start >= 0 && end > start) {++ try { return JSON.parse(body.slice(start, end + 1)); } catch { /* not JSON either */ }++ }++ return null;++}++++function createMemeAi({ config, fetchImpl = globalThis.fetch, log = console }) {++ const ai = config.ai || {};++ const tokenClient = config.oauth.clientSecret++ ? serviceAuth.createTokenClient({++ tokenUrl: `${config.networkInternalUrl || config.networkUrl}/oauth/token`,++ clientId: config.oauth.clientId,++ clientSecret: config.oauth.clientSecret,++ audience: ai.audience,++ scope: ai.scope,++ fetchImpl,++ })++ : null;++ const enabled = Boolean(ai.url && tokenClient);++ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++++ async function call(method, path, body, { retried = false } = {}) {++ if (!enabled) throw new MemeAiError(503, 'meme.ai_not_configured', 'OpenVibe.AI is not configured for OpenVibe.Meme.');++ let headers;++ try {++ headers = { Accept: 'application/json', ...(body ? { 'Content-Type': 'application/json' } : {}), ...(await tokenClient.authHeaders()) };++ } catch (err) {++ throw new MemeAiError(502, 'meme.ai_token_failed', 'OpenVibe.Network did not issue an AI token for Meme.');++ }++ let res;++ try {++ res = await fetchImpl(`${String(ai.url).replace(/\/+$/, '')}${path}`, {++ method,++ headers,++ body: body ? JSON.stringify(body) : undefined,++ signal: AbortSignal.timeout(ai.timeoutMs),++ });++ } catch (err) {++ throw new MemeAiError(502, 'meme.ai_unreachable', err && err.name === 'TimeoutError'++ ? 'OpenVibe.AI did not answer in time.'++ : 'OpenVibe.AI could not be reached.');++ }++ if (res.status === 401 && !retried) {++ tokenClient.invalidate();++ return call(method, path, body, { retried: true });++ }++ const json = await res.json().catch(() => null);++ if (!res.ok) {++ const code = (json && (json.code || json.error)) || `meme.ai_http_${res.status}`;++ const detail = (json && (json.detail || json.message)) || `OpenVibe.AI refused the run (${res.status}).`;++ throw new MemeAiError(res.status >= 500 ? 502 : res.status, String(code).slice(0, 80), String(detail).slice(0, 500));++ }++ return json;++ }++++ /** The run's answer, however the run finished. */++ async function waitFor(run, startedAt) {++ let r = run;++ const deadline = Date.now() + ai.timeoutMs;++ while (r && ['queued', 'running'].includes(r.status)) {++ if (!r.id) throw new MemeAiError(502, 'meme.ai_bad_answer', 'OpenVibe.AI answered with a running run that has no id.');++ if (Date.now() > deadline) throw new MemeAiError(504, 'meme.ai_timeout', `The AI run ${r.id} did not finish in time.`);++ await sleep(ai.pollMs);++ const body = await call('GET', `/api/v1/runs/${encodeURIComponent(r.id)}`);++ r = body && body.run ? body.run : body;++ }++ if (!r) throw new MemeAiError(502, 'meme.ai_bad_answer', 'OpenVibe.AI answered without a run.');++ const done = ['succeeded', 'cached', 'completed', 'ok'].includes(r.status)++ || (r.status == null && r.output != null && !r.error);++ if (!done) {++ const e = r.error || {};++ throw new MemeAiError(502, 'meme.ai_run_failed', `The AI run ${r.id || '?'} ${r.status || 'did not finish'}${e.code ? ` (${e.code}: ${e.detail || ''})` : ''}.`);++ }++ return r;++ }++++ /** The text of a run's output, whatever the workflow wrapped it in. */++ function textOf(output) {++ if (typeof output === 'string') return output;++ if (!output || typeof output !== 'object') return '';++ for (const k of ['text', 'content', 'body']) if (typeof output[k] === 'string') return output[k];++ if (output.message && typeof output.message.content === 'string') return output.message.content;++ if (Array.isArray(output.choices) && output.choices[0] && typeof output.choices[0].message === 'object') {++ return String(output.choices[0].message.content || '');++ }++ return '';++ }++++ /**++ * One run of ai.generate. Answers { text, model, provider, run_id, ms, tokens, cost_usd }.++ * The target names the work, so a run in AI's list says what it was for.++ */++ async function generate({ prompt, system = '', maxTokens = null, temperature = null, target = null, wait = null }) {++ if (!enabled) throw new MemeAiError(503, 'meme.ai_not_configured', 'OpenVibe.AI is not configured for OpenVibe.Meme.');++ const body = {++ prompt: String(prompt || ''),++ ...(system ? { system: String(system) } : {}),++ max_tokens: maxTokens || ai.maxTokens,++ temperature: temperature == null ? ai.temperature : temperature,++ options: { cache: false },++ ...(target ? { target } : {}),++ };++ const startedAt = Date.now();++ const waitMs = Math.min(wait || ai.waitMs, 60_000);++ let created;++ try {++ created = await call('POST', `/api/v1/generate?wait=${waitMs}`, body);++ } catch (err) {++ if (!err || err.status !== 404) throw err;++ // A deployment that exposes the run API but not the convenience operation: the same run, by workflow.++ created = await call('POST', `/api/v1/runs?wait=${waitMs}`, {++ workflow: 'ai.generate',++ input: { prompt: body.prompt, ...(body.system ? { system: body.system } : {}), max_tokens: body.max_tokens, temperature: body.temperature },++ options: body.options,++ ...(target ? { target } : {}),++ });++ }++ const first = created && created.run ? created.run : created;++ const run = await waitFor(first, startedAt);++ const text = textOf(run.output);++ const usage = run.usage || {};++ const prov = run.provenance || {};++ const tokens = usage.tokens && typeof usage.tokens === 'object'++ ? (Number(usage.tokens.input) || 0) + (Number(usage.tokens.output) || 0)++ : (Number(usage.tokens_in) || 0) + (Number(usage.tokens_out) || 0)++ || (Number(usage.input_tokens) || 0) + (Number(usage.output_tokens) || 0)++ || (Number(run.tokens) || 0);++ return {++ text: String(text || ''),++ model: String(prov.model || run.model || (run.output && run.output.model) || 'unknown').slice(0, 120),++ provider: String(prov.provider || run.provider || (run.output && run.output.provider) || 'ai').slice(0, 80),++ run_id: run.id || null,++ ms: Math.max(0, Date.now() - startedAt),++ tokens: Math.max(0, Math.round(tokens)),++ cost_usd: Number(usage.cost_usd != null ? usage.cost_usd : (usage.cost != null ? usage.cost : (run.cost_usd || 0))) || 0,++ };++ }++++ /** One run, asked for JSON, repaired when the model fences it or adds a sentence. */++ async function json({ prompt, system = '', schema = null, temperature = null, target = null }) {++ const ask = schema++ ? `${prompt}\n\nAnswer with one JSON object and nothing else, with exactly these keys: ${Object.keys(schema).join(', ')}.\nThe shape is:\n${JSON.stringify(schema)}`++ : `${prompt}\n\nAnswer with one JSON object and nothing else.`;++ const out = await generate({ prompt: ask, system, temperature: temperature == null ? 0 : temperature, target });++ const parsed = parseJsonLoose(out.text);++ if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {++ throw new MemeAiError(502, 'meme.ai_not_json', 'The AI run did not answer with a JSON object.');++ }++ return { ...out, data: parsed };++ }++++ return { enabled, generate, json, parseJsonLoose, MemeAiError };++}++++module.exports = { createMemeAi, parseJsonLoose, MemeAiError };nnew file mode 100644@@@ -0,0 +1,79 @@++'use strict';++++/**++ * The cited material a judge reads. A citation stores one verbatim line; a joke only makes sense with its++ * neighbours, so before the AI is asked anything, each citation is opened up: the chat lines around a chat message,++ * the transcript lines around a VOD second. Nothing new is created — every line comes from the same public API the++ * citation itself came from, and a citation whose context cannot be read is still cited, just without context.++ *++ * The block that reaches the model is the only place a quote is shown: the verbatim text and the lines around it.++ */++const { parseApiTime } = require('../upstreams');++++const DAY = 86_400_000;++++/** "2026-10-20 10:00:00" → the day it happened, in UTC. */++function utcDay(iso) {++ const s = String(iso || '');++ return s.length >= 10 ? s.slice(0, 10) : '';++}++++function createContext({ config, upstreams, log = console }) {++ const lines = Math.max(1, Number(config.judge && config.judge.contextLines) || 6);++++ /** The lines around one citation. Never throws: a missing context is an empty context. */++ async function around(citation) {++ try {++ if (citation.source_type === 'chat_message' || citation.source_service === 'chat') {++ return await upstreams.chatContext(citation, { lines });++ }++ if (citation.vod_id != null && citation.offset_seconds != null) {++ return await upstreams.vodContext(citation.vod_id, citation.offset_seconds, { lines });++ }++ } catch (err) {++ log.warn(`[OpenVibe.Meme] context for ${citation.source_type}:${citation.source_id}: ${err.message}`);++ }++ return [];++ }++++ /** Every citation with its context, oldest first. */++ async function prepare(citations) {++ const out = [];++ for (const citation of citations || []) {++ out.push({ citation, context: await around(citation) });++ }++ return out;++ }++++ /**++ * The block the judge reads: one numbered use per citation, with the quote and the lines around it.++ * The text is trimmed so a long transcript cannot push the question out of the model's window.++ */++ function render(prepared) {++ const parts = [];++ let n = 0;++ for (const { citation, context } of prepared || []) {++ n += 1;++ const when = utcDay(citation.observed_at);++ const where = citation.source_service === 'live' ? 'OpenVibe.Live'++ : citation.source_service === 'chat' ? 'OpenVibe.Chat'++ : citation.source_service === 'community' ? 'OpenVibe.Community'++ : citation.source_type === 'wikipedia' ? 'Wikipedia'++ : citation.source_type === 'wikidata' ? 'Wikidata'++ : citation.source_type === 'commons' ? 'Wikimedia Commons'++ : citation.source_service;++ const head = `Use ${n} (${where}, ${when || 'unknown date'}): "${String(citation.quote_text || '').slice(0, 600)}"`;++ const linesOut = [];++ for (const line of context || []) {++ if (line.who) linesOut.push(` ${line.who}: ${String(line.text || '').slice(0, 400)}`);++ else linesOut.push(` ${String(line.text || '').slice(0, 400)}`);++ }++ parts.push(linesOut.length ? `${head}\n${linesOut.join('\n')}` : head);++ }++ return parts.join('\n\n');++ }++++ return { prepare, render, around, utcDay };++}++++module.exports = { createContext, utcDay };nnew file mode 100644@@@ -0,0 +1,99 @@++'use strict';++++/**++ * The publish rule. A judged candidate is published only when it clears every bar:++ *++ * - the AI scored it 0.75 or more on "is this a real meme, catchphrase or inside joke";++ * - it was used at least three times, across at least two days or at least two channels;++ * - nothing it cites is on the deny list, and its own text carries no denied word;++ * - its text names nobody's looks;++ * - it is complete: a title, a summary, where it started and how people use it;++ * - it has a live origin citation (an entry without an origin is not an entry).++ *++ * Everything else stays hidden for staff, with its score. The reasons are returned, not just a yes or a no, so the++ * staff list can say why.++ */++const { utcDay } = require('./context');++const { hasLooks } = require('../text/no-looks');++++/** The uses that count: the live origin and use citations. An example is an illustration, not a use. */++function countUses(citations) {++ const live = (citations || []).filter((c) => c.source_status === 'live' && (c.role === 'origin' || c.role === 'use'));++ const days = new Set(live.map((c) => utcDay(c.observed_at)).filter(Boolean)).size;++ const channels = new Set(live.map((c) => c.source_service).filter(Boolean)).size;++ const hasOrigin = live.some((c) => c.role === 'origin');++ return { uses: live.length, days, channels, hasOrigin };++}++++function evaluate({ entry = null, citations = [], judgement = null, gate = {} } = {}) {++ const minScore = Number.isFinite(Number(gate.minScore)) ? Number(gate.minScore) : 0.75;++ const minUses = Math.max(1, Number(gate.minUses) || 3);++ const minDays = Math.max(1, Number(gate.minDays) || 2);++ const minChannels = Math.max(1, Number(gate.minChannels) || 2);++++ const { uses, days, channels, hasOrigin } = countUses(citations);++ const reasons = [];++++ if (!judgement) reasons.push('not judged yet');++ else {++ const score = Number(judgement.score);++ if (!Number.isFinite(score)) reasons.push('no score');++ else if (score < minScore) reasons.push(`scored ${score.toFixed(2)}, under ${minScore.toFixed(2)}`);++ if (judgement.denied) reasons.push('a denied source');++ if (judgement.looks) reasons.push('the text names looks');++ if (!judgement.title) reasons.push('no title');++ if (!judgement.summary) reasons.push('no summary');++ if (!judgement.origin) reasons.push('no origin');++ if (!judgement.spread) reasons.push('no spread');++ if (!Array.isArray(judgement.examples) || judgement.examples.length < 2) reasons.push('fewer than two examples');++ }++ if (entry && entry.title && hasLooks(entry.title)) reasons.push('the title names looks');++ if (uses < minUses) reasons.push(`used ${uses} time${uses === 1 ? '' : 's'}, needs ${minUses}`);++ if (days < minDays && channels < minChannels) {++ reasons.push(`used on ${days} day${days === 1 ? '' : 's'} in ${channels} channel${channels === 1 ? '' : 's'}, needs ${minDays} days or ${minChannels} channels`);++ }++ if (!hasOrigin) reasons.push('no live origin citation');++++ return {++ publish: reasons.length === 0,++ reasons,++ uses,++ days,++ channels,++ score: judgement ? Number(judgement.score) : null,++ minScore,++ };++}++++/**++ * The encyclopedia's rule. A classic has no uses to count: it is published when the AI scored it as a real, widely++ * used meme and its own text is clean (no looks, no denied word, nothing denied in what it cites).++ */++function evaluateClassic({ judgement = null, minScore = 0.75, entry = null } = {}) {++ const reasons = [];++ if (!judgement) reasons.push('not judged yet');++ else {++ const score = Number(judgement.score);++ if (!Number.isFinite(score)) reasons.push('no score');++ else if (score < minScore) reasons.push(`scored ${score.toFixed(2)}, under ${minScore.toFixed(2)}`);++ if (judgement.denied) reasons.push('a denied source');++ if (judgement.looks) reasons.push('the text names looks');++ if (!judgement.title) reasons.push('no title');++ if (!judgement.summary) reasons.push('no summary');++ if (!judgement.origin) reasons.push('no origin');++ if (!judgement.spread) reasons.push('no spread');++ if (!Array.isArray(judgement.examples) || judgement.examples.length < 2) reasons.push('fewer than two examples');++ }++ if (entry && entry.title && hasLooks(entry.title)) reasons.push('the title names looks');++ return {++ publish: reasons.length === 0,++ reasons,++ uses: 0,++ days: 0,++ channels: 0,++ score: judgement ? Number(judgement.score) : null,++ minScore,++ };++}++++module.exports = { evaluate, evaluateClassic, countUses };nnew file mode 100644@@@ -0,0 +1,183 @@++'use strict';++++/**++ * The judge. One AI run per candidate: it scores the candidate from 0 to 1 on whether it is a real meme, catchphrase++ * or inside joke rather than everyday words that happen to recur, and it writes the entry from the same cited++ * material: a title in natural case, a short summary, where it started, how people use it, and two or three example++ * uses.++ *++ * Two rules hold whatever the model says:++ * - the text never names looks (openvibe-sdk/no-looks): a title that names them is not used, and every other++ * field is repaired word by word; an entry whose text named them is never published;++ * - the text never carries a word from the deny list (server/deny-list.js).++ *++ * The score and the reason are stored with the entry, so a staff person sees why a candidate was kept and the++ * history says what the AI decided.++ */++const { noLooks, hasLooks } = require('../text/no-looks');++const { isDenied, hasDeniedWord } = require('../deny-list');++++const JUDGE_SYSTEM = [++ 'You are the editor of OpenVibe.Meme, an open encyclopedia of memes: the running jokes of a streaming network and the memes of the internet.',++ 'You judge one candidate at a time and you use only the cited material in front of you. If the citations do not show it, it is not there.',++ 'You answer with one JSON object and nothing else.',++].join('\n');++++const JUDGE_SCHEMA = Object.freeze({++ score: 'number from 0 to 1',++ reason: 'one or two sentences saying why you gave that score',++ title: 'the name people would actually say, in natural case',++ summary: 'one or two sentences on what it is',++ origin: 'where it started, from the cited uses',++ spread: 'how people use it: how it is deployed and who says it',++ examples: 'two or three short example uses, written the way people would say them',++});++++const SCORE_LINE = [++ 'Score it from 0 to 1:',++ ' 1.00 a clear, shared joke people play off on purpose',++ ' 0.75 the bar: recognisably a catchphrase, bit or meme with a shared meaning',++ ' 0.50 maybe, but it could just be ordinary speech that happens to recur',++ ' 0.00 everyday words; nothing shared, nothing to play off',++].join('\n');++++const RULES = [++ 'Rules:',++ '- Write only what the cited material shows. Never add a fact from outside it, and never invent a person.',++ '- Never describe anyone\'s body, clothing, hair, face or age, and never guess someone\'s gender.',++ '- Keep the title short, in natural case ("Cozmo\'s alive", not "COZMO S ALIVE").',++ '- If the material does not support a field, leave it an empty string rather than guessing.',++].join('\n');++++const maxLen = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);++const clip = (s, n) => String(s == null ? '' : s).trim().slice(0, n);++++/** The score the model gave, kept only when it is a number between 0 and 1. */++function readScore(value) {++ const n = Number(value);++ if (!Number.isFinite(n)) return null;++ if (n < 0) return 0;++ if (n > 1) return 1;++ return n;++}++++/** The examples, as a list of short lines. */++function readExamples(value) {++ const list = Array.isArray(value) ? value : (typeof value === 'string' ? value.split('\n') : []);++ const out = [];++ for (const item of list) {++ const line = maxLen(item, 160);++ if (line) out.push(line);++ if (out.length >= 5) break;++ }++ return out;++}++++/**++ * Clean the AI's own words: no looks, no denied word.++ *++ * A title that names looks is not used at all — it is not repaired word by word, it is dropped, and the entry keeps++ * the name it already had (the phrase it was found under, or the name the source gives it). Every other field is++ * repaired word by word; a field that still names looks after the repair is kept but marked, and an entry with such++ * a field is never published.++ */++function cleanFields({ title, summary, origin, spread, examples }) {++ const t = String(title == null ? '' : title).replace(/\s+/g, ' ').trim();++ const looksTitle = hasLooks(t);++ const fields = {++ title: looksTitle ? '' : maxLen(t, 160),++ summary: maxLen(noLooks(summary), 600),++ origin: maxLen(noLooks(origin), 600),++ spread: maxLen(noLooks(spread), 600),++ examples: (examples || []).map((e) => maxLen(noLooks(e), 200)).filter(Boolean),++ };++ const looks = looksTitle++ || hasLooks(fields.summary) || hasLooks(fields.origin) || hasLooks(fields.spread)++ || fields.examples.some((e) => hasLooks(e));++ const denied = hasDeniedWord(`${fields.title} ${fields.summary} ${fields.origin} ${fields.spread} ${fields.examples.join(' ')}`);++ return { ...fields, looks, denied };++}++++/**++ * Judge one candidate or one classic. `kind` is 'candidate' (an OpenVibe running joke, cited to its own moments)++ * or 'classic' (an internet meme, cited to Wikipedia and Wikidata).++ */++function createJudge({ config, ai, context, log = console }) {++ /**++ * @param {object} subject { title, kind, phrase }++ * @param {Array} prepared the citations with their context, from server/ai/context.js++ */++ async function judge(subject, prepared) {++ if (!ai || !ai.enabled) throw new Error('OpenVibe.AI is not configured for OpenVibe.Meme.');++ const block = context.render(prepared);++ const isClassic = subject.kind === 'classic';++ const what = isClassic++ ? `Write the encyclopedia entry for the internet meme "${subject.title}".`++ : `Judge this candidate phrase from OpenVibe: "${subject.title}".`;++ const ask = isClassic++ ? [++ what,++ '',++ 'Score it from 0 to 1 on how much it is a real, widely used meme rather than a one-off video, a product or a person:',++ ' 1.00 a well-known meme with a shared meaning',++ ' 0.75 the bar: a recognisable, reused meme',++ ' 0.50 maybe, but it is niche or long forgotten',++ ' 0.00 not a meme',++ '',++ 'Then write the entry from the cited sources only: the title in natural case, a short summary, where it started, how it spread and how people use it, and two or three example uses.',++ '',++ 'Cited sources:',++ block,++ '',++ RULES,++ ].join('\n')++ : [++ what,++ '',++ 'Is it a real meme, catchphrase or inside joke — something people play off on purpose, with a shared meaning — or is it just everyday words that happen to recur?',++ '',++ SCORE_LINE,++ '',++ 'Then write the entry from the cited uses only: the title in natural case, a short summary, where it started, how people use it, and two or three example uses.',++ '',++ 'Cited uses:',++ block,++ '',++ RULES,++ ].join('\n');++++ const out = await ai.json({++ prompt: ask,++ system: JUDGE_SYSTEM,++ schema: JUDGE_SCHEMA,++ temperature: 0.2,++ target: { service: 'meme', type: 'judgement', id: `judge_${String(subject.id || subject.title).slice(0, 100)}` },++ });++++ const data = out.data || {};++ const score = readScore(data.score);++ if (score == null) throw new Error('the AI run did not give a score');++ const cleaned = cleanFields({++ title: data.title,++ summary: data.summary,++ origin: data.origin,++ spread: data.spread,++ examples: readExamples(data.examples),++ });++ const deniedCitation = (prepared || []).some(({ citation }) => isDenied(citation));++ return {++ score,++ reason: maxLen(data.reason, 600) || 'The AI gave no reason.',++ ...cleaned,++ denied: cleaned.denied || deniedCitation,++ looks: cleaned.looks,++ model: out.model,++ provider: out.provider,++ run_id: out.run_id,++ };++ }++++ return { judge, cleanFields, readScore, readExamples, enabled: Boolean(ai && ai.enabled), JUDGE_SCHEMA, JUDGE_SYSTEM };++}++++module.exports = { createJudge, cleanFields, readScore, readExamples, JUDGE_SCHEMA, JUDGE_SYSTEM };@@@ -38,6 +38,10 @@ const { createSearchIndex } = require('./search');const { createUpstreams } = require('./upstreams');const { createImporter } = require('./importer');const { createReverifier } = require('./reverify');++const { createMemeAi } = require('./ai/client');++const { createContext } = require('./ai/context');++const { createJudge } = require('./ai/judge');++const { createJudgeJob } = require('./jobs/judge');const { createNetworkSender } = require('openvibe-sdk/account-data');const PUBLIC_DIR = path.join(__dirname, '..', 'public');@@@ -64,6 +68,12 @@ async function createApp(opts = {}) {const upstreams = createUpstreams({ config, fetchImpl, log });const importer = createImporter({ config, s, entries, upstreams, fetchImpl, log });const reverifier = createReverifier({ config, s, entries, upstreams, log });++ // The judge: OpenVibe.AI scores each candidate from its own citations and writes the entry; the publish rule++ // decides what becomes public.++ const ai = createMemeAi({ config, fetchImpl, log });++ const context = createContext({ config, upstreams, log });++ const judge = createJudge({ config, ai, context, log });++ const judgeJob = createJudgeJob({ config, s, entries, judge, context, log });// Account export and deletion (ADR-033). The sender posts to Network's internal routes with this service's own// client-credentials token; a test injects a stand-in through opts.accountSend.@@@ -74,6 +84,7 @@ async function createApp(opts = {}) {const ctx = {config, s, keys, sso, principal, search, entries, upstreams, importer, reverifier, accountData, accountSend, log,++ ai, context, judge, judgeJob,};const app = express();nnew file mode 100644@@@ -0,0 +1,265 @@++'use strict';++++/**++ * The encyclopedia's seed. It reads the well-known internet memes from the open lists (Wikipedia's lists of Internet++ * phenomena and memes, CC BY-SA) and from Wikidata (CC0), writes each one with the same AI pass the OpenVibe++ * candidates use, and keeps every entry's source, picture licence and attribution.++ *++ * One entry per meme:++ * kind 'classic', status 'classic', the AI's title and summary, where it started and how it spread, two or three++ * example uses, and the citations it was written from: the Wikipedia article (the origin), the list's own words++ * about it, the Wikidata facts, and the Commons picture under its own licence.++ *++ * The same publish rule as the candidates: the AI scored it 0.75 or more on "is this a real, widely used meme",++ * the text names no looks and no denied word, and the entry is complete. What does not clear the bar stays hidden++ * for staff, with its score — the list holds videos, products and people too, and not all of them are memes.++ */++const { isExplicit } = require('../text/no-looks');++const { hasDeniedWord } = require('../deny-list');++const { evaluateClassic } = require('../ai/gate');++const { slugify } = require('../store');++++/** The attribution every classic carries: Wikipedia's and Wikidata's licences. */++const ATTRIBUTION = 'Text from Wikipedia (CC BY-SA 4.0); facts from Wikidata (CC0).';++++/** A Wikidata date: "+1997-00-00T00:00:00Z" (a year) or "1997-05-01T00:00:00Z". */++function toIsoDate(value) {++ const s = String(value == null ? '' : value).replace(/^\+/, '');++ const m = /^(\d{4})(?:-(\d{2}))?(?:-(\d{2}))?/.exec(s);++ if (!m) return null;++ const year = Number(m[1]);++ const month = m[2] && m[2] !== '00' ? Number(m[2]) : 1;++ const day = m[3] && m[3] !== '00' ? Number(m[3]) : 1;++ if (year < 1900 || year > 2100) return null;++ const d = new Date(Date.UTC(year, month - 1, day));++ return Number.isNaN(d.getTime()) ? null : d.toISOString();++}++++/** The year a description mentions first, when Wikidata has no date: "…in 2013…" */++function yearFrom(text) {++ const m = /\b(19[5-9]\d|20[0-2]\d)\b/.exec(String(text || ''));++ if (!m) return null;++ return `${m[1]}-01-01T00:00:00.000Z`;++}++++/** A citation for the open web. It is stored like any other: a public URL and the sha256 of the quoted text. */++function webCitation({ source_type, source_id, quote_text, url, observed_at, role = 'use' }) {++ return {++ role,++ source_service: 'web',++ source_type,++ source_id: String(source_id).slice(0, 200),++ quote_text: String(quote_text || '').trim(),++ url: String(url),++ observed_at: observed_at || new Date().toISOString(),++ source_status: 'live',++ };++}++++function createClassicsJob({ config, s, entries, sources, judge, log = console }) {++ const stats = { runs: 0, found: 0, seeded: 0, published: 0, kept: 0, skipped: 0, errors: 0, last_run_at: null, last_error: null };++ const opts = config.encyclopedia || {};++++ /** The lists: Wikipedia's list of Internet phenomena and the category behind it. */++ async function collect() {++ const found = [];++ for (const title of opts.lists || []) {++ try {++ const entriesOfList = await sources.listEntries(title);++ for (const e of entriesOfList) found.push({ ...e, list: title });++ } catch (err) {++ log.warn(`[OpenVibe.Meme] classics: list ${title}: ${err.message}`);++ }++ }++ for (const category of opts.categories || []) {++ try {++ for (const e of await sources.categoryMembers(category)) found.push({ ...e, list: `Category:${category}` });++ } catch (err) {++ log.warn(`[OpenVibe.Meme] classics: category ${category}: ${err.message}`);++ }++ }++ // One entry per name: the first one wins.++ const seen = new Set();++ const out = [];++ for (const e of found) {++ const key = e.title.trim().toLowerCase();++ if (!key || seen.has(key)) continue;++ seen.add(key);++ out.push(e);++ }++ return out.slice(0, Math.max(1, Number(opts.limit) || 400));++ }++++ /**++ * The citations of one classic, and the facts behind them. Returns null when the sources are too thin to write++ * an entry from, or when the text is not something this site publishes.++ */++ async function buildCandidate(item, { qids, facts, leads, images }) {++ const title = item.title;++ const article = item.target && item.target.includes('/') ? null : item.target;++ const resolved = article ? qids.get(article) : null;++ const fact = resolved ? facts.get(resolved.qid) : null;++ const lead = article ? (leads.get(article) || '') : '';++++ // The list's own words, and the article's lead: the two texts the entry is written from.++ const listText = item.description || '';++ if (!listText && !lead) return null;++++ // Nothing explicit and nothing from the deny list ever enters the database, in a quote either.++ if (isExplicit(listText, lead)) return null;++ if (hasDeniedWord(`${title} ${listText} ${lead}`)) return null;++++ const now = new Date(s.now()).toISOString();++ const inception = (fact && toIsoDate(fact.inception)) || yearFrom(listText) || yearFrom(lead) || now;++ const cites = [];++++ if (lead) {++ cites.push(webCitation({++ source_type: 'wikipedia',++ source_id: article || title,++ quote_text: lead.slice(0, 4000),++ url: `https://en.wikipedia.org/wiki/${encodeURIComponent(String(article || title).replace(/ /g, '_'))}`,++ observed_at: inception,++ role: 'origin',++ }));++ }++ if (listText) {++ cites.push(webCitation({++ source_type: 'wikipedia',++ source_id: `${item.list || 'Wikipedia list'}#${title}`,++ quote_text: listText.slice(0, 4000),++ url: `https://en.wikipedia.org/wiki/${encodeURIComponent(String(item.list || 'List_of_Internet_phenomena').replace(/ /g, '_'))}`,++ observed_at: inception,++ role: 'use',++ }));++ }++ if (resolved && fact) {++ const when = fact.inception ? `First recorded: ${fact.inception}.` : '';++ cites.push(webCitation({++ source_type: 'wikidata',++ source_id: resolved.qid,++ quote_text: `Wikidata ${resolved.qid}: ${fact.label || title}${fact.description ? ` — ${fact.description}` : ''}. ${when}`.trim(),++ url: `https://www.wikidata.org/wiki/${resolved.qid}`,++ observed_at: inception,++ role: 'use',++ }));++ }++ let image = null;++ if (fact && fact.image && images.has(fact.image)) {++ image = images.get(fact.image);++ cites.push(webCitation({++ source_type: 'commons',++ source_id: fact.image,++ quote_text: `${image.license}${image.license_url ? ` (${image.license_url})` : ''}. ${image.credit}.`,++ url: image.page_url,++ observed_at: now,++ role: 'example',++ }));++ }++ if (!cites.length) return null;++ return { item, cites, inception, image, qid: resolved ? resolved.qid : null, title, lead, listText };++ }++++ /**++ * Seed the encyclopedia. `limit` caps how many entries this run writes; `publish` is false for a dry run.++ * Returns the counts: found, seeded, published, kept hidden, skipped.++ */++ async function runOnce({ limit = null, publish = true } = {}) {++ const runAt = new Date(s.now()).toISOString();++ const errors = [];++ if (!judge || !judge.enabled) {++ const message = 'OpenVibe.AI is not configured: nothing was written.';++ stats.last_error = message;++ return { ok: false, seeded: 0, published: 0, kept: 0, skipped: 0, error: message };++ }++ const found = await collect();++ stats.found = found.length;++ const max = Math.min(1000, Math.max(1, Number(limit) || opts.limit || 400));++++ // The facts for every name at once: a Wikipedia title → Q id, the entities, the leads, the pictures.++ const articles = [...new Set(found.map((e) => e.target).filter((t) => t && !t.includes('/')))];++ const qids = await sources.resolveQids(articles);++ const facts = await sources.entities([...qids.values()].map((v) => v.qid));++ const leads = await sources.leadExtract(articles);++ const images = await sources.imageInfo([...facts.values()].map((f) => f.image).filter(Boolean));++++ let seeded = 0;++ let published = 0;++ let kept = 0;++ let skipped = 0;++ for (const item of found) {++ if (seeded >= max) break;++ try {++ const built = await buildCandidate(item, { qids, facts, leads, images });++ if (!built) { skipped++; continue; }++ // One entry per meme: a Wikidata id already seeded is left alone.++ if (built.qid) {++ const existing = await s.db.maybe('SELECT id FROM meme_entries WHERE external_id = $1', [built.qid]);++ if (existing) { skipped++; continue; }++ }++ const slug = slugify(built.item.title);++ const existing = await entries.bySlug(slug);++ if (existing) { skipped++; continue; }++++ // The AI writes the entry from the cited text only.++ const prepared = built.cites.map((citation) => ({ citation, context: [] }));++ const verdict = await judge.judge({ id: built.title, title: built.title, kind: 'classic' }, prepared);++ const title = verdict.title || built.title;++ const decision = evaluateClassic({ judgement: verdict, minScore: opts.minScore || 0.75, entry: { title } });++ const summary = verdict.summary || built.listText.slice(0, 300);++ const created = await entries.createEntry({++ slug,++ title,++ kind: 'classic',++ summary,++ status: 'classic',++ first_seen_at: built.inception,++ last_seen_at: new Date(s.now()).toISOString(),++ visibility: publish && decision.publish ? 'public' : 'hidden',++ citations: built.cites,++ image_url: built.image ? built.image.url : null,++ image_credit: built.image ? built.image.credit : null,++ image_license: built.image ? built.image.license : null,++ image_license_url: built.image ? built.image.license_url : null,++ image_source_url: built.image ? built.image.page_url : null,++ attribution: ATTRIBUTION,++ external_id: built.qid,++ });++ await entries.saveJudgement(created.entry.id, verdict);++ await entries.logDecision({++ entryId: created.entry.id,++ decision: decision.publish ? 'publish' : 'keep_hidden',++ fromStatus: 'rising',++ toStatus: 'classic',++ fromVisibility: 'hidden',++ toVisibility: created.entry.visibility,++ score: decision.score,++ reason: decision.publish++ ? `the AI scored it ${Number(decision.score).toFixed(2)}: ${verdict.reason || ''}`.slice(0, 500)++ : `kept hidden: ${decision.reasons.join('; ')}`.slice(0, 500),++ });++ seeded++;++ if (decision.publish) published++;++ else kept++;++ } catch (err) {++ errors.push(`${item.title}: ${err.message}`);++ log.warn(`[OpenVibe.Meme] classics ${item.title} failed: ${err.message}`);++ }++ }++ stats.runs++;++ stats.seeded += seeded;++ stats.published += published;++ stats.kept += kept;++ stats.skipped += skipped;++ stats.errors += errors.length;++ stats.last_run_at = runAt;++ stats.last_error = errors[0] || null;++ const out = { ok: errors.length === 0, found: found.length, seeded, published, kept, skipped, errors };++ log.log(`[OpenVibe.Meme] classics: ${found.length} found, ${seeded} seeded, ${published} published, ${kept} kept hidden, ${skipped} skipped${errors.length ? `, ${errors.length} errors` : ''}`);++ return out;++ }++++ return { runOnce, collect, buildCandidate, stats: () => ({ ...stats }) };++}++++module.exports = { createClassicsJob, toIsoDate, yearFrom, webCitation, ATTRIBUTION };nnew file mode 100644@@@ -0,0 +1,314 @@++'use strict';++++/**++ * The open sources of the encyclopedia: Wikipedia (CC BY-SA), Wikidata (CC0) and Wikimedia Commons (each file under++ * its own licence). Nothing is scraped from a site whose terms forbid it; these three have APIs that say what may++ * be used and how it must be credited.++ *++ * listEntries(title) a Wikipedia list read into entries: the name, the section and the list's own words++ * categoryMembers(cat) the pages of a Wikipedia category (the second list of well-known memes)++ * resolveQids(titles) a Wikipedia title → its Wikidata Q id++ * entities(qids) Wikidata: the inception date, the picture, the label and the description++ * leadExtract(titles) a Wikipedia article's lead, in plain words++ * imageInfo(files) a Commons file: the picture, its licence and its credit++ *++ * Every request names the service (a User-Agent with a contact address) and is paced per host, as the Wikimedia++ * API policy asks.++ */++const { setTimeout: sleep } = require('node:timers/promises');++++class SourceError extends Error {++ constructor(message, { code = 'meme.source', status = 502 } = {}) {++ super(message);++ this.name = 'SourceError';++ this.code = code;++ this.status = status;++ }++}++++/** Templates that carry no words of their own. */++const EMPTY_TEMPLATES = new Set(['snd', 'lang', 'nowrap', 'small', 'sic', 'efn', 'citation', 'cite web', 'cite book',++ 'cite magazine', 'cite news', 'cite journal', 'ill', 'nobold', 'not italic', 'abbr', 'portal', 'anchor', 'vanchor',++ 'clr', 'color', 'center', 'blockquote', 'div col', 'div col end', 'flatlist', 'hlist', 'columns-list', 'ref']);++++const chunk = (list, n) => {++ const out = [];++ for (let i = 0; i < list.length; i += n) out.push(list.slice(i, i + n));++ return out;++};++++/** The words of a wiki text, without the markup: links keep their display text, templates keep their content. */++function plainText(text) {++ let s = String(text == null ? '' : text);++ s = s.replace(/<!--[\s\S]*?-->/g, ' ');++ s = s.replace(/<ref[^>]*\/>/gi, ' ');++ s = s.replace(/<ref[^>]*>[\s\S]*?<\/ref>/gi, ' ');++ s = s.replace(/<[^>]+>/g, ' ');++ // Links: [[Target|Display]] → Display, [[Target]] → Target.++ for (let i = 0; i < 6; i++) {++ const before = s;++ s = s.replace(/\[\[([^\]|]+)\|([^\]]*)\]\]/g, '$2');++ s = s.replace(/\[\[([^\]]*)\]\]/g, '$1');++ if (s === before) break;++ }++ // Templates: an empty one goes, another keeps its content.++ for (let i = 0; i < 8; i++) {++ const before = s;++ s = s.replace(/\{\{([^{}|]*)(?:\|([^{}]*))?\}\}/g, (m, name, arg) => {++ const key = String(name).trim().toLowerCase();++ if (EMPTY_TEMPLATES.has(key)) return ' ';++ return ` ${arg == null ? '' : arg} `;++ });++ if (s === before) break;++ }++ s = s.replace(/'''''|'''|''/g, '');++ s = s.replace(/ /g, ' ').replace(/&/g, '&').replace(/"/g, '"').replace(/'/g, "'").replace(/</g, '<').replace(/>/g, '>');++ return s.replace(/\s+/g, ' ').trim();++}++++/** The name a list entry carries: the link's display text, or its target without the section. */++function entryTitle(raw) {++ const link = /\[\[([^\]|]+)(?:\|([^\]]*))?\]\]/.exec(raw);++ if (link) {++ const display = (link[2] || '').replace(/'''/g, '').trim();++ if (display) return display;++ return String(link[1]).split('#')[0].replace(/_/g, ' ').trim();++ }++ const quoted = /^["“”'](.+?)["“”']/.exec(String(raw).trim());++ if (quoted) return quoted[1].trim();++ return '';++}++++/**++ * One Wikipedia list, read into entries. Each entry is { title, target, section, description } where the++ * description is the list's own words about that entry.++ */++function parseList(wikitext) {++ const out = [];++ let section = '';++ for (const line of String(wikitext || '').split('\n')) {++ const heading = /^(={2,6})\s*(.+?)\s*\1\s*$/.exec(line);++ if (heading) { section = plainText(heading[2]); continue; }++ if (!/^\*+\s*\S/.test(line)) continue;++ if (/^\*\*+/.test(line)) continue; // a sub-bullet is a note, not an entry++ if (/^(See also|References|Notes|External links)$/i.test(section)) continue;++ const body = line.replace(/^\*+\s*/, '');++ const raw = body.split('{{snd}}')[0];++ const title = entryTitle(raw);++ if (!title || title.length > 120) continue;++ const target = (/\[\[([^\]|]+)(?:\|[^\]]*)?\]\]/.exec(raw) || [])[1] || title;++ const description = plainText(body.split('{{snd}}').slice(1).join('{{snd}}') || body.slice(raw.length));++ out.push({ title, target: String(target).replace(/_/g, ' '), section, description });++ }++ return out;++}++++/** True when a licence lets us use the picture: public domain, CC0, or a Creative Commons licence. */++function isFreeLicense(name) {++ const s = String(name || '').trim().toLowerCase();++ if (!s) return false;++ if (/^(cc0|public domain|pd|no restrictions|cc pd)/.test(s)) return true;++ if (/^cc[ -]?(by|by-sa|by-nc|by-nd)[ -]?\d/.test(s)) return true;++ if (s === 'cc by') return true;++ if (s.startsWith('cc by')) return true;++ return false;++}++++/** The Commons metadata, without its HTML. */++function cleanMeta(value) {++ if (!value) return '';++ return plainText(String(value).replace(/<[^>]+>/g, ' ')).slice(0, 300);++}++++function createSources({ config, fetchImpl = globalThis.fetch, log = console }) {++ const opts = config.encyclopedia || {};++ const userAgent = opts.userAgent || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])';++ const timeoutMs = Math.max(1000, Number(opts.timeoutMs) || 20_000);++ const lastCall = new Map();++++ /** One request per host, at least minIntervalMs apart, with a retry on 429 and 5xx. */++ async function call(url, { label = 'Wikipedia', retries = 2 } = {}) {++ const host = new URL(url).host;++ const wait = Math.max(0, (lastCall.get(host) || 0) + (Number(opts.minIntervalMs) || 0) - Date.now());++ if (wait) await sleep(wait);++ lastCall.set(host, Date.now());++ let lastErr = null;++ for (let attempt = 0; attempt <= retries; attempt++) {++ let res;++ try {++ res = await fetchImpl(url, {++ headers: { 'User-Agent': userAgent, Accept: 'application/json' },++ signal: AbortSignal.timeout(timeoutMs),++ redirect: 'follow',++ });++ } catch (err) {++ lastErr = new SourceError(`${label}: ${err && err.message}`, { code: 'meme.source.unreachable' });++ await sleep(500 * (attempt + 1));++ continue;++ }++ if (res.ok) {++ const text = await res.text();++ try { return JSON.parse(text); } catch { throw new SourceError(`${label} answered something that is not JSON`, { code: 'meme.source.body' }); }++ }++ if (res.status === 404) return { notFound: true };++ lastErr = new SourceError(`${label} answered ${res.status}`, { status: res.status >= 500 ? 502 : res.status });++ if (res.status === 429 || res.status >= 500) {++ await sleep(800 * (attempt + 1));++ continue;++ }++ throw lastErr;++ }++ throw lastErr || new SourceError(`${label} did not answer`);++ }++++ const api = (base, params) => `${String(base).replace(/\/+$/, '')}/w/api.php?${new URLSearchParams({ format: 'json', formatversion: '2', ...params })}`;++++ /** One Wikipedia list, read into entries. */++ async function listEntries(title) {++ const data = await call(api(opts.wikipedia, {++ action: 'parse', page: String(title || '').replace(/ /g, '_'), prop: 'wikitext', redirects: '1',++ }), { label: `Wikipedia ${title}` });++ if (!data || data.notFound || !data.parse) return [];++ return parseList(data.parse.wikitext);++ }++++ /** The pages of a Wikipedia category: the second list of well-known memes. */++ async function categoryMembers(category) {++ const out = [];++ let cont = null;++ for (let page = 0; page < 8; page++) {++ const data = await call(api(opts.wikipedia, {++ action: 'query', list: 'categorymembers', cmtitle: `Category:${String(category).replace(/ /g, '_')}`,++ cmlimit: '500', cmnamespace: '0', ...(cont ? { cmcontinue: cont } : {}),++ }), { label: `Wikipedia category ${category}` });++ if (!data || data.notFound) break;++ for (const m of data.query && data.query.categorymembers ? data.query.categorymembers : []) {++ if (m && m.title) out.push({ title: m.title, target: m.title, section: 'Category', description: '' });++ }++ cont = data.continue && data.continue.cmcontinue ? data.continue.cmcontinue : null;++ if (!cont) break;++ }++ return out;++ }++++ /** Wikipedia titles → their Wikidata Q id (redirects followed). */++ async function resolveQids(titles) {++ const wanted = [...new Set((titles || []).map((t) => String(t || '').trim()).filter(Boolean))];++ const out = new Map();++ for (const group of chunk(wanted, 40)) {++ const data = await call(api(opts.wikipedia, {++ action: 'query', redirects: '1', prop: 'pageprops', ppprop: 'wikibase_item', titles: group.join('|'),++ }), { label: 'Wikipedia' });++ if (!data || data.notFound) continue;++ const pages = (data.query && data.query.pages) || [];++ const redirects = (data.query && data.query.redirects) || [];++ const redirectTo = new Map(redirects.map((r) => [String(r.from), String(r.to)]));++ const byTitle = new Map(pages.map((p) => [String(p.title), p]));++ for (const title of group) {++ const landed = redirectTo.get(title) || title;++ const page = byTitle.get(landed) || byTitle.get(title);++ const qid = page && page.pageprops && page.pageprops.wikibase_item;++ if (qid && /^Q[1-9][0-9]{0,11}$/.test(qid)) out.set(title, { qid, title: page.title });++ }++ }++ return out;++ }++++ /** Q ids → the facts the encyclopedia needs: when it started, its picture, its label. */++ async function entities(qids) {++ const wanted = [...new Set((qids || []).filter((q) => /^Q[1-9][0-9]{0,11}$/.test(String(q))))];++ const out = new Map();++ for (const group of chunk(wanted, 40)) {++ const data = await call(api(opts.wikidata, {++ action: 'wbgetentities', ids: group.join('|'), props: 'claims|labels|descriptions|sitelinks', languages: 'en',++ }), { label: 'Wikidata' });++ if (!data || data.notFound) continue;++ const all = data.entities || {};++ for (const qid of group) {++ const e = all[qid];++ if (!e) continue;++ const claims = e.claims || {};++ const first = (prop) => {++ const list = claims[prop];++ if (!Array.isArray(list) || !list.length) return null;++ const v = list[0] && list[0].mainsnak && list[0].mainsnak.datavalue && list[0].mainsnak.datavalue.value;++ return v == null ? null : v;++ };++ const inception = first('P571');++ const image = first('P18');++ out.set(qid, {++ qid,++ label: (e.labels && e.labels.en && e.labels.en.value) || null,++ description: (e.descriptions && e.descriptions.en && e.descriptions.en.value) || null,++ inception: inception && inception.time ? String(inception.time).replace(/^\+/, '') : null,++ image: typeof image === 'string' ? image : null,++ article: (e.sitelinks && e.sitelinks.enwiki && e.sitelinks.enwiki.title) || null,++ });++ }++ }++ return out;++ }++++ /** Wikipedia articles → their lead, in plain words. */++ async function leadExtract(titles) {++ const wanted = [...new Set((titles || []).map((t) => String(t || '').trim()).filter(Boolean))];++ const out = new Map();++ for (const group of chunk(wanted, 20)) {++ const data = await call(api(opts.wikipedia, {++ action: 'query', prop: 'extracts', exintro: '1', explaintext: '1', redirects: '1',++ exlimit: 'max', titles: group.join('|'),++ }), { label: 'Wikipedia' });++ if (!data || data.notFound) continue;++ const pages = (data.query && data.query.pages) || [];++ const redirects = (data.query && data.query.redirects) || [];++ const redirectTo = new Map(redirects.map((r) => [String(r.from), String(r.to)]));++ const byTitle = new Map(pages.map((p) => [String(p.title), p]));++ for (const title of group) {++ const landed = redirectTo.get(title) || title;++ const page = byTitle.get(landed) || byTitle.get(title);++ const extract = page && typeof page.extract === 'string' ? page.extract.trim() : '';++ if (extract) out.set(title, extract.slice(0, 4000));++ }++ }++ return out;++ }++++ /** Commons files → the picture, its licence and its credit. A file under a licence that forbids reuse is dropped. */++ async function imageInfo(files) {++ const wanted = [...new Set((files || []).map((f) => String(f || '').trim()).filter(Boolean))];++ const out = new Map();++ for (const group of chunk(wanted, 40)) {++ const data = await call(api(opts.commons, {++ action: 'query', prop: 'imageinfo', iiprop: 'url|extmetadata', iiurlwidth: '1200',++ titles: group.map((f) => (f.startsWith('File:') ? f : `File:${f}`)).join('|'),++ }), { label: 'Wikimedia Commons' });++ if (!data || data.notFound) continue;++ const pages = (data.query && data.query.pages) || [];++ for (const page of pages) {++ const info = page && page.imageinfo && page.imageinfo[0];++ if (!info) continue;++ const meta = info.extmetadata || {};++ const val = (name) => cleanMeta(meta[name] && meta[name].value);++ const license = val('LicenseShortName') || val('License') || '';++ const licenseUrl = val('LicenseUrl');++ if (!isFreeLicense(license)) continue;++ const url = info.thumburl || info.url;++ if (!url || !/^https:\/\/upload\.wikimedia\.org\//.test(url)) continue;++ out.set(String(page.title).replace(/^File:/, ''), {++ url,++ original_url: info.url || url,++ page_url: `https://commons.wikimedia.org/wiki/${encodeURIComponent(String(page.title).replace(/ /g, '_'))}`,++ license,++ license_url: licenseUrl,++ credit: val('Artist') || val('Credit') || 'Wikimedia Commons',++ usage_terms: val('UsageTerms'),++ });++ }++ }++ return out;++ }++++ return { listEntries, categoryMembers, resolveQids, entities, leadExtract, imageInfo, isFreeLicense, plainText, parseList };++}++++module.exports = { createSources, parseList, plainText, isFreeLicense, entryTitle, SourceError };@@@ -70,6 +70,33 @@ function load(env = process.env) {timeoutMs: Math.max(500, int(env.MEME_UPSTREAM_TIMEOUT_MS, 8000)),},++ // OpenVibe.AI: the judge and the encyclopedia's writing pass. Meme calls ai.generate as the service++ // `meme` with a client-credentials token (audience openvibe.ai, scope ai.run.create); Network grants the++ // token and the ai.* namespace. Meme never names a provider: AI's routing decides, and the run reports++ // what answered.++ ai: {++ url: trim(env.OV_AI_INTERNAL_URL || 'http://127.0.0.1:4700'),++ audience: env.MEME_AI_AUDIENCE || 'openvibe.ai',++ scope: env.MEME_AI_SCOPE || 'ai.run.create',++ waitMs: Math.max(1000, int(env.MEME_AI_WAIT_MS, 55_000)),++ timeoutMs: Math.max(1000, int(env.MEME_AI_TIMEOUT_MS, 120_000)),++ pollMs: Math.max(100, int(env.MEME_AI_POLL_MS, 1500)),++ maxTokens: Math.max(200, int(env.MEME_AI_MAX_TOKENS, 1400)),++ temperature: 0.2,++ },++++ // The judge: it scores a candidate from its own citations and writes the entry. Only what clears the bar is++ // published; everything else stays hidden for staff with its score. Off under MEME_DRILL.++ judge: {++ intervalMs: Math.max(60_000, int(env.MEME_JUDGE_INTERVAL_MS, 30 * 60_000)),++ minScore: Number.isFinite(Number(env.MEME_JUDGE_MIN_SCORE)) ? Number(env.MEME_JUDGE_MIN_SCORE) : 0.75,++ minUses: Math.max(1, int(env.MEME_JUDGE_MIN_USES, 3)),++ minDays: Math.max(1, int(env.MEME_JUDGE_MIN_DAYS, 2)),++ minChannels: Math.max(1, int(env.MEME_JUDGE_MIN_CHANNELS, 2)),++ maxPerRun: Math.min(30, Math.max(1, int(env.MEME_JUDGE_MAX_PER_RUN, 10))),++ contextLines: Math.max(1, int(env.MEME_JUDGE_CONTEXT_LINES, 6)),++ },++// The candidate importer: every 30 minutes, off under MEME_DRILL (tests call runOnce()).importer: {intervalMs: Math.max(60_000, int(env.MEME_IMPORTER_INTERVAL_MS, 30 * 60_000)),@@@ -82,6 +109,32 @@ function load(env = process.env) {// Every citation is re-verified daily, off under MEME_DRILL.reverify: { intervalMs: Math.max(60_000, int(env.MEME_REVERIFY_INTERVAL_MS, 24 * 60 * 60_000)) },++++ // The encyclopedia: the well-known internet memes, cited to Wikipedia (CC BY-SA) and to Wikidata (CC0),++ // written by the same AI pass. It is a one-off seed, not a timer: the job runs from scripts/seed-classics.js.++ encyclopedia: {++ wikipedia: trim(env.MEME_WIKIPEDIA_URL || 'https://en.wikipedia.org'),++ wikidata: trim(env.MEME_WIKIDATA_URL || 'https://www.wikidata.org'),++ commons: trim(env.MEME_COMMONS_URL || 'https://commons.wikimedia.org'),++ userAgent: env.MEME_WIKI_USER_AGENT || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])',++ timeoutMs: Math.max(1000, int(env.MEME_WIKI_TIMEOUT_MS, 20_000)),++ minIntervalMs: Math.max(0, int(env.MEME_WIKI_MIN_INTERVAL_MS, 250)),++ limit: Math.min(1000, Math.max(1, int(env.MEME_ENCYCLOPEDIA_LIMIT, 400))),++ // The lists the seeds are read from (Wikipedia, CC BY-SA): the phenomena list and the category behind it.++ lists: String(env.MEME_ENCYCLOPEDIA_LISTS || 'List_of_Internet_phenomena').split(',').map((x) => x.trim()).filter(Boolean),++ categories: String(env.MEME_ENCYCLOPEDIA_CATEGORIES || 'Internet memes,Memes').split(',').map((x) => x.trim()).filter(Boolean),++ minScore: Number.isFinite(Number(env.MEME_ENCYCLOPEDIA_MIN_SCORE)) ? Number(env.MEME_ENCYCLOPEDIA_MIN_SCORE) : 0.75,++ },++++ // The templates: the well-known public-domain or freely licensed meme templates, from Wikimedia Commons,++ // each with its licence and credit.++ templates: {++ commons: trim(env.MEME_COMMONS_URL || 'https://commons.wikimedia.org'),++ userAgent: env.MEME_WIKI_USER_AGENT || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])',++ timeoutMs: Math.max(1000, int(env.MEME_WIKI_TIMEOUT_MS, 20_000)),++ minIntervalMs: Math.max(0, int(env.MEME_WIKI_MIN_INTERVAL_MS, 250)),++ limit: Math.min(200, Math.max(1, int(env.MEME_TEMPLATES_LIMIT, 40))),++ },};}@@@ -5,9 +5,14 @@* 404 anonymously; clip 483 is Live's AI auto-clip of sexual content (server/ai/explicit.js), and VOD 5709 has no* public row at all. A citation that would carry either id is dropped before it is written, and the seed script and* the importer both call this filter. No title, thumbnail or quote from either ever enters the database.++ *++ * The list also names the words no AI-written entry text may carry. A verbatim quote is never rewritten (that would++ * falsify the citation), but a title, a summary, an origin note and an example are the site's own words: they pass++ * this filter before they are stored, and an entry whose text carries one is never published.*/const DENIED_VOD_IDS = Object.freeze(new Set([5709]));const DENIED_CLIP_IDS = Object.freeze(new Set([483]));++const DENIED_WORDS = Object.freeze(['nigger', 'faggot', 'retard', 'kys', 'rape', 'pedo', 'nazi']);function num(v) {if (v == null || v === '') return null;@@@ -35,4 +40,16 @@ function allowed(uses) {return (Array.isArray(uses) ? uses : []).filter((u) => !isDenied(u));}--module.exports = { DENIED_VOD_IDS, DENIED_CLIP_IDS, isDenied, isDeniedVod, isDeniedClip, allowed };++/** True when a text carries a word the deny list names. The word must stand on its own. */++function hasDeniedWord(text) {++ const s = String(text == null ? '' : text);++ if (!s) return false;++ return DENIED_WORDS.some((w) => new RegExp(`(^|[^a-z0-9])${w}([^a-z0-9]|$)`, 'i').test(s));++}++++/** The text of a field the site's own words carry: clean when it names nothing on the deny list. */++function isCleanText(text) {++ return !hasDeniedWord(text);++}++++module.exports = { DENIED_VOD_IDS, DENIED_CLIP_IDS, DENIED_WORDS, isDenied, isDeniedVod, isDeniedClip, allowed, hasDeniedWord, isCleanText };@@@ -14,6 +14,8 @@* GET /captions/:id one saved caption* POST /entries/:slug/report a takedown/correction report (signed in)* GET /reports open reports (staff)++ * GET /staff/candidates the hidden candidates with the judge's score (staff)++ * GET /staff/candidates/:slug one candidate: its judgement, citations and decisions (staff)* POST /entries/:slug/publish|hide staff* PATCH /entries/:slug staff: title, kind, status*@@@ -116,6 +118,7 @@ function createApi(ctx) {const staff = isStaff(req.principal);if (!row || (row.visibility === 'hidden' && !staff)) return problem(res, 404, 'meme.not_found', 'No such meme.');const citations = await entries.citations(row.id);++ const verdict = await entries.judgement(row.id);if (row.visibility === 'public') res.set('Cache-Control', 'public, max-age=60');else if (row.visibility === 'unlisted') res.set('Cache-Control', 'private, max-age=60');return res.json({@@@ -124,6 +127,9 @@ function createApi(ctx) {citations,timeline: citations.filter((c) => c.role !== 'example'),examples: citations.filter((c) => c.role === 'example'),++ // The judge's own words about the entry: where it started and how people use it.++ ...(verdict && verdict.origin ? { notes: { origin: verdict.origin, spread: verdict.spread, examples: verdict.examples } } : {}),++ ...(staff && verdict ? { judgement: { score: verdict.score, reason: verdict.reason, judged_at: verdict.judged_at, model: verdict.model, provider: verdict.provider } } : {}),});});@@@ -226,6 +232,33 @@ function createApi(ctx) {return res.json({ count: reports.length, reports });});++ // ── The staff list: the candidates that stayed hidden ──++ r.get('/staff/candidates', requireStaff, async (req, res) => {++ const limit = asInt(req.query.limit, 100);++ const kind = typeof req.query.kind === 'string' && req.query.kind ? req.query.kind : null;++ if (kind && !KINDS.includes(kind)) return invalid(res, `kind must be one of ${KINDS.join(', ')}`);++ const list = await entries.listHidden({ limit, kind });++ return res.json({ count: list.length, candidates: list });++ });++++ r.get('/staff/candidates/:slug', requireStaff, async (req, res) => {++ const row = await entries.bySlug(String(req.params.slug || ''));++ if (!row) return problem(res, 404, 'meme.not_found', 'No such meme.');++ const [citations, verdict, history] = await Promise.all([++ entries.citations(row.id),++ entries.judgement(row.id),++ entries.decisions(row.id),++ ]);++ return res.json({++ ...entries.wireEntry(row),++ score: verdict ? verdict.score : null,++ reason: verdict ? verdict.reason : null,++ judgement: verdict,++ citations,++ decisions: history,++ });++ });++return r;}@@@ -21,6 +21,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }() => ctx.keys.client.stop(),() => ctx.importer.stop(),() => ctx.reverifier.stop(),++ () => ctx.judgeJob.stop(),async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },() => ctx.s.close(),...extra,@@@ -50,6 +51,7 @@ async function start() {// Jobs. Each refuses to start under MEME_DRILL; tests call their runOnce().ctx.importer.start();ctx.reverifier.start();++ ctx.judgeJob.start();// The two account topics (export_requested, deleted) at OpenVibe.Events: idempotent; off without the URL, the// delivery secret and the client secret. The consumer itself is mounted in server/app.js.nnew file mode 100644@@@ -0,0 +1,156 @@++'use strict';++++/**++ * The judge's job. It takes the candidates the importer left hidden, asks OpenVibe.AI to judge each one from its++ * own citations, stores the score and the written entry, and then applies the publish rule: what clears the bar is++ * published, everything else stays hidden for staff with its score and the reasons it did not clear.++ *++ * Every decision is a status event, so the history of an entry says what the AI decided and why. The job is safe to++ * run twice: a candidate that already has a judgement is only re-judged when it is asked for (a re-judge round), and++ * publishing an entry that is already public changes nothing.++ *++ * runOnce() judge the candidates nobody has judged yet++ * runOnce({ force: true }) judge every hidden candidate again (the re-judge round)++ * runOnce({ only: [slug] }) judge these entries and nothing else++ */++const { evaluate } = require('../ai/gate');++++function createJudgeJob({ config, s, entries, judge, context, log = console }) {++ const stats = { runs: 0, judged: 0, published: 0, kept: 0, errors: 0, last_run_at: null, last_error: null };++ let timers = [];++++ /** The entry's own citations, oldest first: the judge reads them in the order they happened. */++ async function citationsOf(entry) {++ const cites = await entries.citations(entry.id);++ return cites++ .filter((c) => c.source_status === 'live')++ .sort((a, b) => Date.parse(a.observed_at) - Date.parse(b.observed_at));++ }++++ /**++ * Judge one entry and act on the verdict. Returns { entry, judgement, decision }.++ */++ async function judgeEntry(entry) {++ const citations = await citationsOf(entry);++ const prepared = await context.prepare(citations);++ const verdict = await judge.judge({ id: entry.id, title: entry.title, kind: entry.kind, phrase: entry.title }, prepared);++ const saved = await entries.saveJudgement(entry.id, verdict);++ // The AI's title and summary are the entry's own words; the slug never changes.++ await entries.applyJudgement(entry.id, { title: verdict.title, summary: verdict.summary });++ const fresh = await entries.byId(entry.id);++ const decision = evaluate({++ entry: fresh,++ citations,++ judgement: { ...saved, denied: verdict.denied, looks: verdict.looks },++ gate: config.judge,++ });++ return { entry: fresh, judgement: saved, decision, citations };++ }++++ /** Publish an entry that cleared the bar, and say so in the history. */++ async function publishJudged(entry, decision, verdict) {++ const out = await entries.publish(entry.slug, { changedBy: 'service:meme:judge' });++ if (!out.ok) {++ await entries.logDecision({++ entryId: entry.id, decision: 'keep_hidden', fromStatus: entry.status, toStatus: entry.status,++ fromVisibility: 'hidden', toVisibility: 'hidden', score: decision.score,++ reason: `not published: ${out.code || out.message || 'refused'}`,++ });++ return { published: false, out };++ }++ await entries.logDecision({++ entryId: entry.id, decision: 'publish', fromStatus: entry.status, toStatus: out.entry.status,++ fromVisibility: 'hidden', toVisibility: 'public', score: decision.score,++ reason: `the AI scored it ${Number(decision.score).toFixed(2)}: ${verdict.reason || ''}`.slice(0, 500),++ });++ return { published: true, out };++ }++++ /** Keep an entry hidden, and say why in the history. */++ async function keepHidden(entry, decision, verdict) {++ await entries.logDecision({++ entryId: entry.id, decision: 'keep_hidden', fromStatus: entry.status, toStatus: entry.status,++ fromVisibility: entry.visibility, toVisibility: 'hidden', score: decision.score,++ reason: `kept hidden: ${decision.reasons.join('; ')}${verdict && verdict.reason ? ` — ${verdict.reason}` : ''}`.slice(0, 500),++ });++ return { published: false };++ }++++ async function runOnce({ limit = null, force = false, only = null, publish = true } = {}) {++ const runAt = new Date(s.now()).toISOString();++ if (!judge || !judge.enabled) {++ const message = 'OpenVibe.AI is not configured: nothing was judged.';++ stats.last_error = message;++ log.warn(`[OpenVibe.Meme] judge: ${message}`);++ return { ok: false, judged: 0, published: 0, kept: 0, error: message };++ }++ const max = Math.min(50, Math.max(1, Number(limit) || config.judge.maxPerRun));++ const errors = [];++ let candidates = [];++ if (only && only.length) {++ for (const slug of only) {++ const row = await entries.bySlug(String(slug));++ if (row) candidates.push(row);++ }++ } else if (force) {++ candidates = await entries.listHidden({ limit: max });++ } else {++ candidates = await entries.unjudged({ limit: max });++ }++++ let published = 0;++ let kept = 0;++ for (const candidate of candidates) {++ try {++ const { entry, decision, judgement: verdict } = await judgeEntry(candidate);++ if (decision.publish && publish) {++ const r = await publishJudged(entry, decision, verdict);++ if (r.published) published++;++ else kept++;++ } else if (decision.publish) {++ // A dry run: the judgement is stored, the entry stays where it is.++ await entries.logDecision({++ entryId: entry.id, decision: 'judge', fromStatus: entry.status, toStatus: entry.status,++ fromVisibility: entry.visibility, toVisibility: entry.visibility, score: decision.score,++ reason: `judged ${Number(decision.score).toFixed(2)}: ${verdict.reason || ''}`.slice(0, 500),++ });++ kept++;++ } else {++ await keepHidden(entry, decision, verdict);++ kept++;++ }++ } catch (err) {++ errors.push(`${candidate.slug}: ${err.message}`);++ log.warn(`[OpenVibe.Meme] judge ${candidate.slug} failed: ${err.message}`);++ }++ }++ stats.runs++;++ stats.judged += candidates.length - errors.length;++ stats.published += published;++ stats.kept += kept;++ stats.errors += errors.length;++ stats.last_run_at = runAt;++ stats.last_error = errors[0] || null;++ const out = { ok: errors.length === 0, judged: candidates.length - errors.length, published, kept, errors };++ log.log(`[OpenVibe.Meme] judge: ${candidates.length} judged, ${published} published, ${kept} kept hidden${errors.length ? `, ${errors.length} errors` : ''}`);++ return out;++ }++++ function start() {++ if (config.drill || timers.length) return false;++ const quiet = () => runOnce().catch((err) => log.warn(`[OpenVibe.Meme] judge tick failed: ${err.message}`));++ const kick = setTimeout(quiet, 45_000);++ const tick = setInterval(quiet, config.judge.intervalMs);++ if (kick.unref) kick.unref();++ if (tick.unref) tick.unref();++ timers = [kick, tick];++ return true;++ }++ function stop() {++ for (const t of timers.splice(0)) { clearTimeout(t); clearInterval(t); }++ }++++ return { runOnce, start, stop, judgeEntry, stats: () => ({ ...stats, running: timers.length > 0 }) };++}++++module.exports = { createJudgeJob };nnew file mode 100644@@@ -0,0 +1,94 @@++'use strict';++++/**++ * The one outbound HTTP helper the open-source jobs share: identify every request, pace it per host, keep the++ * answer's fetch time, and turn a failure into a short, non-secret error. The URLs come from config (the official++ * public hosts); no user-chosen URL is ever fetched.++ */++const { setTimeout: sleep } = require('node:timers/promises');++++function hostOf(url) {++ try { return new URL(url).host; } catch { return String(url); }++}++++/** Serialise requests per host, at least minIntervalMs apart (the Wikimedia API policy asks for it). */++function createPacer(minIntervalMs = 0) {++ const last = new Map();++ let chain = Promise.resolve();++ return function pace(url) {++ const host = hostOf(url);++ const run = chain.then(async () => {++ const wait = Math.max(0, (last.get(host) || 0) + minIntervalMs - Date.now());++ if (wait) await sleep(wait);++ last.set(host, Date.now());++ });++ chain = run.catch(() => {});++ return run.then(() => () => {});++ };++}++++class SourceError extends Error {++ constructor(message, { code = 'meme.source', status = 502 } = {}) {++ super(message);++ this.name = 'SourceError';++ this.code = code;++ this.status = status;++ }++}++++/**++ * fetchText(url, { fetchImpl, userAgent, headers, timeoutMs, pace, retries })++ * Returns { text, status, url, fetchedAt }. Retries 429/5xx twice with a short backoff; a 4xx (except 429) is final.++ */++async function fetchText(url, opts = {}) {++ const fetchImpl = opts.fetchImpl || globalThis.fetch;++ const headers = {++ 'User-Agent': opts.userAgent || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])',++ Accept: opts.accept || 'application/json, text/plain;q=0.9, */*;q=0.5',++ ...(opts.headers || {}),++ };++ const retries = Number.isFinite(opts.retries) ? opts.retries : 2;++ let release = null;++ let lastErr = null;++ for (let attempt = 0; attempt <= retries; attempt += 1) {++ if (opts.pace && !release) release = await opts.pace(url);++ try {++ const res = await fetchImpl(url, {++ method: opts.method || 'GET',++ headers,++ body: opts.body,++ signal: AbortSignal.timeout(Math.max(1000, Number(opts.timeoutMs) || 20_000)),++ redirect: 'follow',++ });++ const text = await res.text();++ if (res.ok) {++ if (release) { release(); release = null; }++ return { text, status: res.status, url, fetchedAt: new Date().toISOString() };++ }++ lastErr = new SourceError(`${opts.label || hostOf(url)} answered ${res.status}`, { status: res.status === 404 ? 404 : 502 });++ if (res.status === 429 || res.status >= 500) {++ if (release) { release(); release = null; }++ if (attempt < retries) {++ const retryAfter = Number(opts.headers && opts.headers.get && opts.headers.get('retry-after'));++ await sleep(Number.isFinite(retryAfter) && retryAfter > 0 ? Math.min(10_000, retryAfter * 1000) : 800 * (attempt + 1));++ continue;++ }++ }++ if (release) { release(); release = null; }++ throw lastErr;++ } catch (err) {++ if (release) { release(); release = null; }++ lastErr = err instanceof SourceError ? err : new SourceError(`${opts.label || hostOf(url)}: ${(err && err.message) || 'request failed'}`);++ if (attempt < retries) { await sleep(600 * (attempt + 1)); continue; }++ throw lastErr;++ }++ }++ throw lastErr || new SourceError(`${opts.label || hostOf(url)}: request failed`);++}++++async function fetchJson(url, opts = {}) {++ const out = await fetchText(url, opts);++ try { return { ...out, json: JSON.parse(out.text) }; } catch { throw new SourceError(`${opts.label || hostOf(url)} answered something that is not JSON`); }++}++++module.exports = { createPacer, fetchText, fetchJson, SourceError, hostOf };@@@ -14,7 +14,7 @@ const { cleanSummary } = require('./text/no-looks');const { weekBounds } = require('./week');const SLUG_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;--const KINDS = Object.freeze(['joke', 'catchphrase', 'bit', 'meme', 'saga']);++const KINDS = Object.freeze(['joke', 'catchphrase', 'bit', 'meme', 'saga', 'classic']);const STATUSES = Object.freeze(['rising', 'peaked', 'classic', 'dead']);const VISIBILITIES = Object.freeze(['public', 'unlisted', 'hidden']);const CAPTION_VISIBILITIES = Object.freeze(['public', 'unlisted', 'hidden']);@@@ -43,6 +43,36 @@ function wireEntry(row) {revision: Number(row.revision) || 1,created_at: iso(row.created_at),updated_at: iso(row.updated_at),++ ...(row.image_url ? { image_url: row.image_url } : {}),++ ...(row.image_credit ? { image_credit: row.image_credit } : {}),++ ...(row.image_license ? { image_license: row.image_license } : {}),++ ...(row.image_license_url ? { image_license_url: row.image_license_url } : {}),++ ...(row.image_source_url ? { image_source_url: row.image_source_url } : {}),++ ...(row.attribution ? { attribution: row.attribution } : {}),++ ...(row.external_id ? { external_id: row.external_id } : {}),++ };++}++++function wireJudgement(row) {++ if (!row) return null;++ let examples = row.examples;++ if (typeof examples === 'string') { try { examples = JSON.parse(examples); } catch { examples = []; } }++ return {++ entry_id: row.entry_id,++ score: Number(row.score),++ reason: row.reason,++ title: row.title == null ? null : row.title,++ summary: row.summary == null ? null : row.summary,++ origin: row.origin == null ? null : row.origin,++ spread: row.spread == null ? null : row.spread,++ examples: Array.isArray(examples) ? examples.map(String) : [],++ denied: Boolean(row.denied),++ looks: Boolean(row.looks),++ model: row.model == null ? null : row.model,++ provider: row.provider == null ? null : row.provider,++ run_id: row.run_id == null ? null : row.run_id,++ judged_at: iso(row.judged_at),++ judged_by: row.judged_by,};}@@@ -71,7 +101,13 @@ function wireCitation(row) {function wireTemplate(row) {if (!row) return null;-- return { id: row.id, name: row.name, image_url: row.image_url, box_count: Number(row.box_count), source: row.source, license: row.license, created_at: iso(row.created_at) };++ return {++ id: row.id, name: row.name, image_url: row.image_url, box_count: Number(row.box_count), source: row.source,++ license: row.license, created_at: iso(row.created_at),++ ...(row.credit ? { credit: row.credit } : {}),++ ...(row.source_url ? { source_url: row.source_url } : {}),++ ...(row.attribution ? { attribution: row.attribution } : {}),++ };}function wireCaption(row) {@@@ -275,21 +311,30 @@ function createStore(s, { search }) {}/**-- * Insert one hidden candidate and its citations in the caller's transaction. Used by the importer and the-- * seed script only; it never publishes. Citations that name the deny list are dropped here too.++ * Insert one entry and its citations in the caller's transaction. A candidate is inserted hidden; an++ * encyclopedia entry ('classic') is inserted public. Citations that name the deny list are dropped here too.*/-- async function createCandidate({ slug = null, title, kind = 'meme', summary = null, status = 'rising', first_seen_at, last_seen_at, visibility = 'hidden', citations: cites = [] }, t = null) {++ async function createEntry({++ slug = null, title, kind = 'meme', summary = null, status = 'rising', first_seen_at, last_seen_at,++ visibility = 'hidden', citations: cites = [], image_url = null, image_credit = null, image_license = null,++ image_license_url = null, image_source_url = null, attribution = null, external_id = null,++ }, t = null) {const run = async (tx) => {const id = s.newId('mem');const cleanSlug = await uniqueSlug(slug || slugify(title), tx);const cleanTitle = String(title || '').replace(/\s+/g, ' ').trim().slice(0, 160) || 'Untitled meme';++ const cleanKind = KINDS.includes(kind) ? kind : 'meme';let first = iso(first_seen_at) || s.iso();let last = iso(last_seen_at) || first;if (Date.parse(first) > Date.parse(last)) { const swap = first; first = last; last = swap; }const row = await q(tx).maybe(-- `INSERT INTO meme_entries (id, slug, title, kind, summary, status, status_since, first_seen_at, last_seen_at, visibility, revision, created_at, updated_at)-- VALUES ($1, $2, $3, $4, $5, $6, $7::timestamptz, $8::timestamptz, $9::timestamptz, $10, 1, now(), now()) RETURNING *`,-- [id, cleanSlug, cleanTitle, kind, cleanSummary(cleanTitle, summary), status, iso(last), first, last, visibility],++ `INSERT INTO meme_entries (id, slug, title, kind, summary, status, status_since, first_seen_at, last_seen_at, visibility, revision, created_at, updated_at,++ image_url, image_credit, image_license, image_license_url, image_source_url, attribution, external_id)++ VALUES ($1, $2, $3, $4, $5, $6, $7::timestamptz, $8::timestamptz, $9::timestamptz, $10, 1, now(), now(),++ $11, $12, $13, $14, $15, $16, $17) RETURNING *`,++ [id, cleanSlug, cleanTitle, cleanKind, cleanSummary(cleanTitle, summary), status, iso(last), first, last, visibility,++ image_url || null, image_credit || null, image_license || null, image_license_url || null, image_source_url || null,++ attribution || null, external_id || null],);const added = await addCitations(row.id, cites, tx);const entry = wireEntry(row);@@@ -299,6 +344,11 @@ function createStore(s, { search }) {return t ? await run(t) : await s.tx(run);}++ /** A candidate is an entry that stays hidden until the judge clears it. */++ async function createCandidate(opts, t = null) {++ return await createEntry({ ...opts, visibility: opts.visibility || 'hidden' }, t);++ }++/** Insert citations in the caller's transaction; the sha256 is computed from the verbatim quote. */async function addCitations(entryId, cites = [], t = null) {const run = async (tx) => {@@@ -422,6 +472,21 @@ function createStore(s, { search }) {return wireTemplate(await db.maybe('SELECT * FROM meme_templates WHERE id = $1', [String(id || '')]));}++ /**++ * One template, written from an open source: its picture, where the picture came from, under which licence and++ * who made it. A template that is already there is left alone.++ */++ async function addTemplate({ id, name, image_url, box_count = 2, source, license, credit = null, source_url = null, attribution = null }) {++ const row = await db.maybe(++ `INSERT INTO meme_templates (id, name, image_url, box_count, source, license, credit, source_url, attribution)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)++ ON CONFLICT (id) DO NOTHING RETURNING *`,++ [String(id || ''), String(name || '').slice(0, 80), String(image_url || ''), Math.max(1, Math.min(8, Number(box_count) || 2)),++ String(source || ''), String(license || ''), credit || null, source_url || null, attribution || null],++ );++ return wireTemplate(row || await db.maybe('SELECT * FROM meme_templates WHERE id = $1', [String(id || '')]));++ }++async function createCaption({ subject, templateId, lines, visibility = 'public' }) {if (!CAPTION_VISIBILITIES.includes(visibility)) return refusal(400, 'meme.invalid', `visibility must be one of ${CAPTION_VISIBILITIES.join(', ')}`, 'visibility');const checked = captionLines(lines);@@@ -471,6 +536,137 @@ function createStore(s, { search }) {};}++ // ── The judge ───────────────────────────────────────────++ /** The verdict and the written entry, stored beside the entry (never inside it). */++ async function saveJudgement(entryId, j, t = null) {++ const run = async (tx) => {++ const row = await q(tx).maybe(++ `INSERT INTO meme_judgements (entry_id, score, reason, title, summary, origin, spread, examples, denied, looks, model, provider, run_id, judged_at, judged_by)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10, $11, $12, $13, now(), $14)++ ON CONFLICT (entry_id) DO UPDATE SET++ score = EXCLUDED.score, reason = EXCLUDED.reason, title = EXCLUDED.title, summary = EXCLUDED.summary,++ origin = EXCLUDED.origin, spread = EXCLUDED.spread, examples = EXCLUDED.examples,++ denied = EXCLUDED.denied, looks = EXCLUDED.looks, model = EXCLUDED.model, provider = EXCLUDED.provider,++ run_id = EXCLUDED.run_id, judged_at = now(), judged_by = EXCLUDED.judged_by++ RETURNING *`,++ [entryId, Number(j.score), String(j.reason || ''), j.title || null, j.summary || null, j.origin || null, j.spread || null,++ JSON.stringify(Array.isArray(j.examples) ? j.examples : []), Boolean(j.denied), Boolean(j.looks),++ j.model || null, j.provider || null, j.run_id || null, j.judged_by || 'service:meme:judge'],++ );++ return wireJudgement(row);++ };++ return t ? await run(t) : await s.tx(run);++ }++++ async function judgement(entryId, t = null) {++ return wireJudgement(await q(t).maybe('SELECT * FROM meme_judgements WHERE entry_id = $1', [String(entryId || '')]));++ }++++ /**++ * Take the AI's title and summary for the entry. The title is only used when it is a real title (no looks) and++ * the entry keeps its own slug: a published link never breaks.++ */++ async function applyJudgement(entryId, { title, summary }, { changedBy = 'service:meme:judge' } = {}) {++ return await s.tx(async (t) => {++ const row = await byId(entryId, t);++ if (!row) return null;++ const sets = [];++ const args = [entryId];++ const cleanTitle = String(title == null ? '' : title).replace(/\s+/g, ' ').trim().slice(0, 160);++ if (cleanTitle && cleanTitle !== row.title) { args.push(cleanTitle); sets.push(`title = $${args.length}`); }++ const cleanSummary = summary == null ? null : String(summary).replace(/\s+/g, ' ').trim().slice(0, 600);++ if (cleanSummary && cleanSummary !== row.summary) { args.push(cleanSummary); sets.push(`summary = $${args.length}`); }++ if (!sets.length) return { entry: wireEntry(row), changed: false };++ const updated = await t.maybe(++ `UPDATE meme_entries SET ${sets.join(', ')}, revision = revision + 1, updated_at = now() WHERE id = $1 RETURNING *`,++ args,++ );++ if (updated.visibility === 'public') {++ const all = await liveCitations(entryId, t);++ await search.sync(t, wireEntry(updated), all);++ }++ return { entry: wireEntry(updated), changed: true };++ });++ }++++ /**++ * One decision, one status event: the visibility moved, the score says why. The status itself does not move++ * (a published candidate keeps the status the importer gave it), so from_status and to_status are the same and++ * the decision columns carry the truth.++ */++ async function logDecision({ entryId, decision, fromStatus = null, toStatus = null, fromVisibility = null, toVisibility = null, score = null, reason = null, changedBy = 'service:meme:judge' }) {++ const row = await db.maybe(++ `INSERT INTO meme_status_events (entry_id, from_status, to_status, reason, changed_by, decision, from_visibility, to_visibility, score)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING *`,++ [entryId, fromStatus || null, toStatus || 'rising', reason ? String(reason).slice(0, 500) : null, changedBy, decision,++ fromVisibility || null, toVisibility || null, score == null ? null : Number(score)],++ );++ return row;++ }++++ /** The decision history of one entry, newest first. */++ async function decisions(entryId, { limit = 50 } = {}) {++ return (await db.many(++ `SELECT id, entry_id, from_status, to_status, reason, changed_by, decision, from_visibility, to_visibility, score, changed_at++ FROM meme_status_events WHERE entry_id = $1 ORDER BY changed_at DESC, id DESC LIMIT $2`,++ [String(entryId || ''), Math.min(200, Math.max(1, Number(limit) || 50))],++ )).map((r) => ({++ id: Number(r.id), entry_id: r.entry_id, from_status: r.from_status, to_status: r.to_status,++ reason: r.reason, changed_by: r.changed_by, decision: r.decision,++ from_visibility: r.from_visibility, to_visibility: r.to_visibility,++ score: r.score == null ? null : Number(r.score), changed_at: iso(r.changed_at),++ }));++ }++++ /**++ * The staff list: every entry that is still hidden, with the judge's score and the numbers behind it. Data++ * only — the page is the orchestrator's.++ */++ async function listHidden({ limit = 100, kind = null } = {}) {++ const n = Math.min(500, Math.max(1, Number(limit) || 100));++ const args = [];++ let where = "e.visibility = 'hidden'";++ if (kind) { if (!KINDS.includes(kind)) return []; args.push(kind); where += ` AND e.kind = $${args.length}`; }++ args.push(n);++ const rows = await db.many(++ `SELECT e.*, j.score AS j_score, j.reason AS j_reason, j.title AS j_title, j.judged_at AS j_judged_at,++ j.model AS j_model, j.provider AS j_provider, j.denied AS j_denied, j.looks AS j_looks,++ (SELECT COUNT(*)::int FROM meme_citations c WHERE c.entry_id = e.id AND c.role IN ('origin','use') AND c.source_status = 'live') AS uses,++ (SELECT COUNT(DISTINCT to_char(c.observed_at AT TIME ZONE 'UTC', 'YYYY-MM-DD'))::int FROM meme_citations c WHERE c.entry_id = e.id AND c.role IN ('origin','use') AND c.source_status = 'live') AS days,++ (SELECT COUNT(DISTINCT c.source_service)::int FROM meme_citations c WHERE c.entry_id = e.id AND c.role IN ('origin','use') AND c.source_status = 'live') AS channels++ FROM meme_entries e LEFT JOIN meme_judgements j ON j.entry_id = e.id++ WHERE ${where}++ ORDER BY (j.score IS NULL) ASC, j.score DESC, e.last_seen_at DESC, e.id DESC LIMIT $${args.length}`,++ args,++ );++ return rows.map((r) => ({++ ...wireEntry(r),++ score: r.j_score == null ? null : Number(r.j_score),++ reason: r.j_reason == null ? null : r.j_reason,++ judged_title: r.j_title == null ? null : r.j_title,++ judged_at: iso(r.j_judged_at),++ model: r.j_model == null ? null : r.j_model,++ provider: r.j_provider == null ? null : r.j_provider,++ denied: Boolean(r.j_denied),++ looks: Boolean(r.j_looks),++ uses: Number(r.uses) || 0,++ days: Number(r.days) || 0,++ channels: Number(r.channels) || 0,++ }));++ }++++ /** The hidden entries a judge has not looked at yet (the ones the job takes next). */++ async function unjudged({ limit = 10 } = {}) {++ const n = Math.min(100, Math.max(1, Number(limit) || 10));++ const rows = await db.many(++ `SELECT e.* FROM meme_entries e LEFT JOIN meme_judgements j ON j.entry_id = e.id++ WHERE e.visibility = 'hidden' AND j.entry_id IS NULL++ ORDER BY e.last_seen_at DESC, e.id DESC LIMIT $1`,++ [n],++ );++ return rows.map(wireEntry);++ }++async function createReport({ entryId, subject, kind, reason, note = null }) {const row = await db.maybe("SELECT id FROM meme_entries WHERE id = $1 AND visibility = 'public'", [String(entryId || '')]);if (!row) return refusal(404, 'meme.not_found', 'No such public entry.');@@@ -494,11 +690,12 @@ function createStore(s, { search }) {}return {-- wireEntry, wireCitation, wireTemplate, wireCaption, wireReport,++ wireEntry, wireCitation, wireTemplate, wireCaption, wireReport, wireJudgement,bySlug, byId, citations, liveCitations, list, trending, week, series, uniqueSlug,-- createCandidate, addCitations, enrich, reindex, publish, hide, patch,-- templates, templateById, createCaption, captionById, captionCountForTemplate,++ createCandidate, createEntry, addCitations, enrich, reindex, publish, hide, patch,++ templates, templateById, addTemplate, createCaption, captionById, captionCountForTemplate,castVote, createReport, listOpenReports,++ saveJudgement, judgement, applyJudgement, logDecision, decisions, listHidden, unjudged,KINDS, STATUSES, VISIBILITIES, BLOCKED_WORDS, MAX_LINES, MAX_LINE,};}nnew file mode 100644@@@ -0,0 +1,170 @@++'use strict';++++/**++ * The templates. OpenVibe.Meme's caption generator draws its text over a picture; the pictures are the well-known++ * meme templates, taken from Wikimedia Commons under a licence that lets us use them: public domain, CC0 or a++ * Creative Commons licence. A file under any other licence is left out.++ *++ * Every template keeps its own licence and credit, and the file page it came from, so the picture is attributed++ * where it is shown. The three data-free placeholders of v1 (tpl_classic, tpl_reaction, tpl_triptych) stay: they++ * are the site's own, and they need no attribution.++ *++ * The names are the well-known ones; the file is found in Commons and checked, not assumed. A template whose file++ * cannot be found, or whose licence does not allow reuse, is skipped and reported.++ */++const { createPacer, fetchText, fetchJson } = require('../sources/http');++++/** The well-known templates: the name, the word the file must carry, and how many text boxes it has. */++const WELL_KNOWN = Object.freeze([++ { name: 'Drake Hotline Bling', token: 'drake', box_count: 2 },++ { name: 'Distracted Boyfriend', token: 'distracted', box_count: 1 },++ { name: 'Expanding Brain', token: 'brain', box_count: 4 },++ { name: 'Change My Mind', token: 'change', box_count: 1 },++ { name: 'Is This a Pigeon', token: 'pigeon', box_count: 2 },++ { name: 'Woman Yelling at a Cat', token: 'yelling', box_count: 2 },++ { name: 'Surprised Pikachu', token: 'pikachu', box_count: 2 },++ { name: 'Two Buttons', token: 'button', box_count: 3 },++ { name: 'Boardroom Meeting', token: 'boardroom', box_count: 2 },++ { name: 'Running Away from a Balloon', token: 'balloon', box_count: 2 },++ { name: 'Crying Cat', token: 'cat', box_count: 2 },++ { name: 'Pointing at the Dog', token: 'dog', box_count: 2 },++ { name: 'They Don\'t Know', token: 'know', box_count: 2 },++ { name: 'Trade Offer', token: 'trade', box_count: 2 },++ { name: 'Left Exit 12 Off Ramp', token: 'ramp', box_count: 2 },++ { name: 'Hide the Pain Harold', token: 'harold', box_count: 2 },++ { name: 'Roll Safe', token: 'roll', box_count: 2 },++ { name: 'Disaster Girl', token: 'disaster', box_count: 2 },++ { name: 'Philosoraptor', token: 'raptor', box_count: 2 },++ { name: 'Success Kid', token: 'success', box_count: 2 },++ { name: 'Mocking SpongeBob', token: 'spongebob', box_count: 2 },++ { name: 'This Is Fine', token: 'fine', box_count: 2 },++ { name: 'Waiting Skeleton', token: 'skeleton', box_count: 2 },++ { name: 'X All the Y', token: 'all', box_count: 2 },++ { name: 'Sweating Guy', token: 'sweat', box_count: 2 },++ { name: 'Gru\'s Plan', token: 'gru', box_count: 4 },++ { name: 'Skeleton Jumping', token: 'skeleton', box_count: 1 },++ { name: 'Afraid to Ask Andy', token: 'andy', box_count: 2 },++ { name: 'Me Gusta Mucho', token: 'gusta', box_count: 2 },++ { name: 'Seagull Full of Fries', token: 'seagull', box_count: 2 },++ { name: 'Chad and Virgo', token: 'virgo', box_count: 2 },++ { name: 'Stonks', token: 'stonks', box_count: 2 },++ { name: 'Cheugy', token: 'cheugy', box_count: 2 },++ { name: 'Elon Musk', token: 'musk', box_count: 2 },++ { name: 'No U Mad', token: 'mad', box_count: 2 },++ { name: 'Nutcracker', token: 'nutcracker', box_count: 2 },++ { name: 'Evil Toddler', token: 'toddler', box_count: 2 },++ { name: 'Sweating Buddha', token: 'buddha', box_count: 2 },++ { name: 'Yao Ming Face', token: 'yao', box_count: 2 },++ { name: 'Il Posto di Lavoro', token: 'lavoro', box_count: 2 },++ { name: 'Gru and the Plan', token: 'gru', box_count: 2 },++ { name: 'Duck and Cover', token: 'duck', box_count: 2 },++ { name: 'Scumbag Steve', token: 'steve', box_count: 2 },++ { name: 'Overly Attached Girlfriend', token: 'girlfriend', box_count: 2 },++ { name: 'Inhaling Seagull', token: 'seagull', box_count: 2 },++ { name: 'Annoyed Cat', token: 'cat', box_count: 2 },++]);++++const slug = (name) => String(name || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 60);++++/** The Commons file's licence must allow reuse. */++function isFreeLicense(name) {++ const s = String(name || '').trim().toLowerCase();++ if (!s) return false;++ if (/^(cc0|public domain|pd|no restrictions|cc pd)/.test(s)) return true;++ if (/^cc[ -]?(by|by-sa|by-nc|by-nd)[ -]?\d/.test(s)) return true;++ if (s.startsWith('cc by')) return true;++ return false;++}++++const clean = (v) => String(v == null ? '' : v).replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim().slice(0, 300);++++function createTemplatesJob({ config, entries, fetchImpl = globalThis.fetch, log = console }) {++ const opts = config.templates || {};++ const userAgent = opts.userAgent || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])';++ const timeoutMs = Math.max(1000, Number(opts.timeoutMs) || 20_000);++ const pace = createPacer(Number(opts.minIntervalMs) || 250);++++ const api = (params) => `${String(opts.commons).replace(/\/+$/, '')}/w/api.php?${new URLSearchParams({ format: 'json', formatversion: '2', ...params })}`;++++ /** The Commons files whose name carries the token. */++ async function search(token) {++ const { json } = await fetchJson(api({++ action: 'query', list: 'search', srsearch: `${token} filetype:bitmap`, srnamespace: '6', srlimit: '12',++ }), { fetchImpl, userAgent, timeoutMs, pace, label: 'Commons search' });++ const hits = (json && json.query && json.query.search) || [];++ return hits.map((h) => String(h.title || '')).filter(Boolean);++ }++++ /** The file's picture, licence and credit. A file under a licence that forbids reuse is dropped. */++ async function info(titles) {++ const wanted = [...new Set(titles)].slice(0, 20);++ if (!wanted.length) return new Map();++ const { json } = await fetchJson(api({++ action: 'query', prop: 'imageinfo', iiprop: 'url|extmetadata|mime', iiurlwidth: '1200', titles: wanted.join('|'),++ }), { fetchImpl, userAgent, timeoutMs, pace, label: 'Commons' });++ const out = new Map();++ for (const page of (json && json.query && json.query.pages) || []) {++ const info = page && page.imageinfo && page.imageinfo[0];++ if (!info) continue;++ if (info.mime && !String(info.mime).startsWith('image/')) continue;++ const meta = info.extmetadata || {};++ const val = (name) => clean(meta[name] && meta[name].value);++ const license = val('LicenseShortName') || val('License') || '';++ if (!isFreeLicense(license)) continue;++ const url = info.thumburl || info.url;++ if (!url || !/^https:\/\/upload\.wikimedia\.org\//.test(url)) continue;++ out.set(String(page.title), {++ url,++ page_url: `https://commons.wikimedia.org/wiki/${encodeURIComponent(String(page.title).replace(/ /g, '_'))}`,++ license,++ license_url: val('LicenseUrl'),++ credit: val('Artist') || val('Credit') || 'Wikimedia Commons',++ });++ }++ return out;++ }++++ /**++ * Seed the templates. Returns { seeded, skipped, templates } — the templates carry their licence and credit.++ */++ async function runOnce({ limit = null, publish = true } = {}) {++ const wanted = (limit ? WELL_KNOWN.slice(0, Math.max(1, Number(limit))) : WELL_KNOWN);++ const seeded = [];++ const skipped = [];++ for (const tpl of wanted) {++ try {++ const titles = await search(tpl.token);++ const files = await info(titles);++ let found = null;++ for (const [title, file] of files) {++ if (!title.toLowerCase().includes(tpl.token.toLowerCase())) continue;++ found = file;++ break;++ }++ if (!found) { skipped.push({ name: tpl.name, reason: 'no freely licensed file found' }); continue; }++ const id = `tpl_commons_${slug(tpl.name)}`;++ const row = await entries.addTemplate({++ id,++ name: tpl.name,++ image_url: found.url,++ box_count: tpl.box_count || 2,++ source: 'Wikimedia Commons',++ license: found.license,++ credit: found.credit,++ source_url: found.page_url,++ attribution: `${found.credit} — ${found.license}, via Wikimedia Commons`.slice(0, 400),++ });++ seeded.push(row);++ } catch (err) {++ skipped.push({ name: tpl.name, reason: err.message });++ log.warn(`[OpenVibe.Meme] template ${tpl.name}: ${err.message}`);++ }++ }++ log.log(`[OpenVibe.Meme] templates: ${seeded.length} seeded, ${skipped.length} skipped`);++ return { ok: true, seeded: seeded.length, skipped: skipped.length, templates: seeded, missing: skipped };++ }++++ return { runOnce, search, info, WELL_KNOWN };++}++++module.exports = { createTemplatesJob, WELL_KNOWN, isFreeLicense, slug };@@@ -6,7 +6,7 @@* names looks at all is not repaired word by word — it is replaced by the entry's title, as Meme's plan says.* Verbatim quotes are never passed through this: rewriting a quote would falsify the citation.*/--const { noLooks, hasLooks } = require('openvibe-sdk/no-looks');++const { noLooks, hasLooks, isExplicit } = require('openvibe-sdk/no-looks');/** The summary to store for an AI-written entry summary. */function cleanSummary(title, summary) {@@@ -18,4 +18,4 @@ function cleanSummary(title, summary) {return cleaned || t;}--module.exports = { cleanSummary, noLooks, hasLooks };++module.exports = { cleanSummary, noLooks, hasLooks, isExplicit };@@@ -155,8 +155,52 @@ function createUpstreams({ config, fetchImpl = globalThis.fetch, log = console }return { status: 'gone' };}++ /**++ * The chat lines around a cited message: the lines before and after it, so the judge reads the joke and not++ * one line alone. The same walk-back as chatMessage() finds the message; the page it lands on holds the rest.++ */++ async function chatContext(citation, { lines = 6 } = {}) {++ const at = new Date(citation.observed_at).getTime();++ if (!Number.isFinite(at)) return [];++ const wanted = String(citation.chat_message_id != null ? citation.chat_message_id : citation.source_id);++ let before = chatBefore(at + 60_000);++ for (let page = 0; page < 5; page++) {++ const h = await chatHistory({ before, limit: 500 });++ const idx = h.messages.findIndex((m) => String(m.id) === wanted);++ if (idx >= 0) {++ const half = Math.max(1, Math.ceil(lines / 2));++ return h.messages.slice(Math.max(0, idx - half), idx + half + 1).map((m) => ({++ id: m.id,++ who: m.username ? String(m.username) : null,++ text: String(m.message || ''),++ at: parseApiTime(m.timestamp),++ }));++ }++ const oldest = h.messages.length ? parseApiTime(h.messages[h.messages.length - 1].timestamp) : null;++ if (!oldest || Date.parse(oldest) <= at - 86_400_000) break;++ before = chatBefore(Date.parse(oldest) - 1000);++ }++ return [];++ }++++ /** The transcript lines around a cited second: what was said just before and just after it. */++ async function vodContext(vodId, offsetSeconds, { lines = 6 } = {}) {++ const id = String(vodId == null ? '' : vodId);++ if (!id) return [];++ let data;++ try { data = await transcripts([id]); } catch { return []; }++ const segs = (data[id] && Array.isArray(data[id].segments)) ? data[id].segments : [];++ if (!segs.length) return [];++ const at = Number(offsetSeconds);++ if (!Number.isFinite(at)) return [];++ const before = segs.filter((x) => Number(x.start) <= at).slice(-Math.max(1, Math.floor(lines / 2)));++ const after = segs.filter((x) => Number(x.start) > at).slice(0, Math.max(1, lines - before.length));++ return [...before, ...after].map((x) => ({ start: Number(x.start), text: String(x.text || '') }));++ }++return {chatHistory, chatTimeline, rundownTopics, rundownTopic, vod, transcripts, canonicalVideo, citationUrl, chatMessage,++ chatContext, vodContext,base,};}nnew file mode 100644@@@ -0,0 +1,224 @@++'use strict';++/**++ * The encyclopedia: the well-known internet memes, seeded from the open lists and written by the same AI pass.++ *++ * Every seeded entry is an entry of kind 'classic' with its origin, its spread and its sources; it carries the++ * attribution of the licences its text is under; nothing is published whose text names looks or carries a denied++ * word; and a picture is only used when its licence allows it.++ */++const { boot, check, done } = require('./helpers/boot');++const { createSources } = require('../server/classics/sources');++const { createClassicsJob } = require('../server/classics/seed');++++const LIST_WIKITEXT = `== Films ==++* [[Dog eats husband]]{{snd}}A joke about a woman who cooked her husband and fed him to her dogs.++* [[Success Kid]]{{snd}}A picture of a child raising a barbell, used to describe a small success.++* [[Hidden treasure map]]{{snd}}A map that leads to a hidden treasure.++== Video ==++* [[Nyan cat]]{{snd}}A cat with Pop-Tart wings flying through space with a rainbow trail.++`;++++/** A stand-in for Wikipedia, Wikidata and Wikimedia Commons. */++function wikiFetch(overrides = {}) {++ return async (url) => {++ const u = new URL(url);++ const q = u.searchParams;++ const action = q.get('action');++ const body = (obj) => ({ ok: true, status: 200, text: async () => JSON.stringify(obj), json: async () => obj, headers: new Map() });++++ if (action === 'parse') {++ if (q.get('page') === 'List_of_Internet_phenomena') return body({ parse: { wikitext: LIST_WIKITEXT } });++ return body({ parse: { wikitext: '== Films ==\n* [[Dog eats husband]]{{snd}}A joke about a woman who cooked her husband and fed him to her dogs.\n' } });++ }++ if (action === 'query' && q.get('list') === 'categorymembers') return body({ query: { categorymembers: [] } });++ if (action === 'query' && q.get('prop') === 'pageprops') {++ const titles = String(q.get('titles') || '').split('|');++ return body({++ query: {++ pages: titles.map((title) => ({ title, pageprops: { wikibase_item: overrides.qid || 'Q100' } })),++ },++ });++ }++ if (action === 'query' && q.get('prop') === 'extracts') {++ const titles = String(q.get('titles') || '').split('|');++ return body({++ query: {++ pages: titles.map((title) => ({++ title,++ extract: overrides.extract || 'The joke is a black-humour joke about a woman who cooked her husband and fed him to her dogs, told as a recipe.',++ })),++ },++ });++ }++ if (action === 'query' && q.get('prop') === 'imageinfo') {++ const titles = String(q.get('titles') || '').split('|');++ return body({++ query: {++ pages: titles.map((title) => ({++ title,++ mime: 'image/jpeg',++ imageinfo: [{++ url: 'https://upload.wikimedia.org/wikipedia/commons/a/ab/Dog.jpg',++ thumburl: 'https://upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Dog.jpg/800px-Dog.jpg',++ extmetadata: {++ LicenseShortName: { value: overrides.license || 'CC BY-SA 4.0' },++ LicenseUrl: { value: 'https://creativecommons.org/licenses/by-sa/4.0/' },++ Artist: { value: '<a href="https://commons.wikimedia.org/wiki/User:Someone">Someone</a>' },++ },++ }],++ })),++ },++ });++ }++ if (action === 'wbgetentities') {++ const ids = String(q.get('ids') || '').split('|');++ return body({++ entities: Object.fromEntries(ids.map((id) => [id, {++ id,++ labels: { en: { value: 'Dog eats husband' } },++ descriptions: { en: { value: 'internet meme' } },++ claims: { P571: [{ mainsnak: { datavalue: { value: { time: '+2007-00-00T00:00:00Z' } } } }], P18: [{ mainsnak: { datavalue: { value: 'Dog.jpg' } } }] },++ sitelinks: { enwiki: { title: 'Dog eats husband' } },++ }])),++ });++ }++ return body({});++ };++}++++const answer = (score, extra = {}) => () => JSON.stringify({++ score,++ reason: `the AI scored it ${score}`,++ title: 'Dog eats husband',++ summary: 'A dark joke told as a recipe.',++ origin: 'It comes from a 2007 image macro.',++ spread: 'People tell it as a recipe, with the punchline in the last line.',++ examples: ['recipe: 1 husband, 2 dogs', 'ingredients: one husband'],++ ...extra,++});++++async function seed(t, { answerFn = answer(0.9), fetchImpl = wikiFetch(), config = {} } = {}) {++ const cfg = t.config;++ const sources = createSources({ config: { ...cfg, encyclopedia: { ...cfg.encyclopedia, ...config } }, fetchImpl, log: console });++ const judge = t.ctx.judge;++ const job = createClassicsJob({ config: cfg, s: t.ctx.s, entries: t.ctx.entries, sources, judge, log: console });++ // The stand-in AI answers every prompt with the same verdict.++ t.ai.setAnswer(answerFn);++ return await job.runOnce({ publish: true });++}++++(async () => {++ await check('the encyclopedia is seeded with entries of kind classic, cited and attributed', async () => {++ const t = await boot();++ try {++ const out = await seed(t);++ if (!out.seeded) throw new Error('nothing was seeded');++ const rows = await t.ctx.s.db.many("SELECT * FROM meme_entries WHERE kind = 'classic' ORDER BY slug");++ if (!rows.length) throw new Error('no classic entry was written');++ const row = rows[0];++ if (row.visibility !== 'public') throw new Error(`the entry is ${row.visibility}, not public`);++ if (!row.attribution) throw new Error('the entry carries no attribution');++ if (!/Wikipedia/.test(row.attribution) || !/Wikidata/.test(row.attribution)) throw new Error(`the attribution is ${row.attribution}`);++ if (!row.image_license) throw new Error('the entry carries no licence');++ if (!row.image_credit) throw new Error('the entry carries no credit');++ if (!row.image_url || !/^https:\/\/upload\.wikimedia\.org\//.test(row.image_url)) throw new Error(`the picture is ${row.image_url}`);++ const citations = await t.ctx.entries.citations(row.id);++ if (citations.length < 3) throw new Error(`the entry has ${citations.length} citations`);++ if (!citations.some((c) => c.source_type === 'wikipedia')) throw new Error('no Wikipedia citation');++ if (!citations.some((c) => c.source_type === 'wikidata')) throw new Error('no Wikidata citation');++ if (!citations.some((c) => c.role === 'origin')) throw new Error('no origin citation');++ for (const c of citations) {++ if (!/^https:\/\//.test(c.url)) throw new Error(`a citation has no public URL: ${c.url}`);++ if (!c.quote_text) throw new Error('a citation has no quoted text');++ }++ } finally { await t.close(); }++ });++++ await check('every seeded entry carries its attribution', async () => {++ const t = await boot();++ try {++ await seed(t);++ const rows = await t.ctx.s.db.many("SELECT id, attribution, image_license FROM meme_entries WHERE kind = 'classic'");++ if (!rows.length) throw new Error('nothing was seeded');++ for (const row of rows) {++ if (!row.attribution) throw new Error(`entry ${row.id} has no attribution`);++ if (!row.image_license) throw new Error(`entry ${row.id} has no licence`);++ }++ } finally { await t.close(); }++ });++++ await check('nothing is published whose text names looks', async () => {++ const t = await boot();++ try {++ await seed(t, { answerFn: answer(0.95, { summary: 'A man in a kitchen tells the joke.', title: 'Dog eats husband' }) });++ const rows = await t.ctx.s.db.many("SELECT id, visibility, summary FROM meme_entries WHERE kind = 'classic'");++ for (const row of rows) {++ const { hasLooks } = require('openvibe-sdk/no-looks');++ if (row.visibility === 'public' && hasLooks(row.summary || '')) throw new Error(`entry ${row.id} is public and its summary names looks`);++ }++ } finally { await t.close(); }++ });++++ await check('nothing is published whose text carries a denied word', async () => {++ const t = await boot();++ try {++ await seed(t, { answerFn: answer(0.95, { summary: 'The joke is about a nazi in a kitchen.' }) });++ const rows = await t.ctx.s.db.many("SELECT id, visibility, summary FROM meme_entries WHERE kind = 'classic'");++ for (const row of rows) {++ if (row.visibility === 'public' && /nazi/i.test(row.summary || '')) throw new Error(`entry ${row.id} is public and its summary carries a denied word`);++ }++ } finally { await t.close(); }++ });++++ await check('a classic the AI scored under the bar stays hidden, with its score', async () => {++ const t = await boot();++ try {++ await seed(t, { answerFn: answer(0.4) });++ const rows = await t.ctx.s.db.many("SELECT id, visibility FROM meme_entries WHERE kind = 'classic'");++ if (!rows.length) throw new Error('nothing was seeded');++ for (const row of rows) {++ if (row.visibility !== 'hidden') throw new Error(`entry ${row.id} was published under the bar`);++ }++ const list = await t.ctx.entries.listHidden({ limit: 50 });++ const judged = list.find((e) => e.kind === 'classic');++ if (!judged) throw new Error('the classic is not on the staff list');++ if (Number(judged.score) !== 0.4) throw new Error(`the staff list shows ${judged.score}`);++ } finally { await t.close(); }++ });++++ await check('a picture whose licence forbids reuse is not used', async () => {++ const t = await boot();++ try {++ await seed(t, { fetchImpl: wikiFetch({ license: 'All rights reserved' }) });++ const rows = await t.ctx.s.db.many("SELECT id, image_url FROM meme_entries WHERE kind = 'classic'");++ for (const row of rows) {++ if (row.image_url) throw new Error(`entry ${row.id} uses a picture under a licence that forbids reuse`);++ }++ } finally { await t.close(); }++ });++++ await check('the seeded classics are public in the API', async () => {++ const t = await boot();++ try {++ await seed(t);++ const res = await t.get('/api/v1/entries?limit=50');++ if (res.status !== 200) throw new Error(`the list answered ${res.status}`);++ const classics = res.json().entries.filter((e) => e.kind === 'classic');++ if (!classics.length) throw new Error('no classic entry is public');++ if (!classics[0].attribution) throw new Error('the public entry carries no attribution');++ } finally { await t.close(); }++ });++++ await check('a classic the AI could not write is not seeded at all', async () => {++ const t = await boot();++ try {++ await seed(t, { answerFn: answer(0.9, { title: '', summary: '', origin: '', spread: '', examples: [] }) });++ const rows = await t.ctx.s.db.many("SELECT id, visibility FROM meme_entries WHERE kind = 'classic'");++ for (const row of rows) {++ if (row.visibility === 'public') throw new Error(`entry ${row.id} was published without a written entry`);++ }++ } finally { await t.close(); }++ });++++ done();++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,78 @@++'use strict';++/**++ * An in-process stand-in for OpenVibe.AI. It answers POST /api/v1/generate with one run, exactly the shape the++ * real service returns: { run: { id, status, output: { text }, provenance, usage } }.++ *++ * The answer is a function of the prompt, so a test can say "this candidate scores 0.9" or "this one scores 0.2",++ * and it can read the prompt back to check that the cited material really reached the model. Every call is recorded.++ */++const http = require('http');++++const DEFAULT_ANSWER = JSON.stringify({++ score: 0.9,++ reason: 'People play off this on purpose; it has a shared meaning in the cited uses.',++ title: 'The robot is offline',++ summary: 'A running joke about the robot being offline again.',++ origin: 'It starts in the first cited chat line.',++ spread: 'People say it whenever the robot drops off, as a call back to the stream.',++ examples: ['the robot is offline again', 'still the robot is offline, huh'],++});++++function listen(handler) {++ return new Promise((resolve) => {++ const sockets = new Set();++ const server = http.createServer((req, res) => {++ const chunks = [];++ req.on('data', (c) => chunks.push(c));++ req.on('end', () => {++ const raw = Buffer.concat(chunks).toString('utf8');++ const json = (status, obj) => { res.writeHead(status, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(obj)); };++ Promise.resolve(handler(req, raw, json)).catch((err) => json(500, { error: String(err && err.message) }));++ });++ });++ server.on('connection', (socket) => { sockets.add(socket); socket.on('close', () => sockets.delete(socket)); });++ server.listen(0, '127.0.0.1', () => resolve({++ server,++ url: `http://127.0.0.1:${server.address().port}`,++ close: () => new Promise((r) => { for (const s of sockets) s.destroy(); server.close(r); }),++ }));++ });++}++++async function startAi({ answer } = {}) {++ const state = { calls: [], answer: answer || (() => DEFAULT_ANSWER), runs: 0 };++++ const srv = await listen(async (req, raw, json) => {++ const url = new URL(req.url, 'http://x');++ if (url.pathname === '/api/v1/generate' && req.method === 'POST') {++ const body = JSON.parse(raw || '{}');++ state.calls.push({ prompt: body.prompt, system: body.system, target: body.target });++ state.runs += 1;++ const text = await state.answer(body.prompt, body);++ return json(200, {++ run: {++ id: `run_${state.runs}`,++ status: 'succeeded',++ output: { text: String(text) },++ provenance: { provider: 'mock', model: 'mock-model' },++ usage: { tokens: { input: 100, output: 50 }, cost_usd: 0.001 },++ },++ });++ }++ if (url.pathname.startsWith('/api/v1/runs/')) {++ const id = url.pathname.split('/').pop();++ return json(200, { run: { id, status: 'succeeded', output: { text: DEFAULT_ANSWER } } });++ }++ return json(404, { error: 'not found' });++ });++++ return {++ url: srv.url,++ state,++ calls: state.calls,++ setAnswer(fn) { state.answer = fn; },++ close: srv.close,++ };++}++++module.exports = { startAi, DEFAULT_ANSWER };@@@ -16,6 +16,7 @@ process.env.MEME_LIMITS_HOUR = process.env.MEME_LIMITS_HOUR || '100000';const http = require('http');const { startNetwork } = require('./mocks');const { startUpstreams } = require('./upstreams');++const { startAi } = require('./ai');const captured = [];for (const m of ['log', 'info', 'warn', 'error']) {@@@ -26,6 +27,7 @@ for (const m of ['log', 'info', 'warn', 'error']) {async function boot(opts = {}) {const network = await startNetwork(opts.network || {});const upstreams = await startUpstreams(opts.upstreams || {});++ const ai = await startAi(opts.ai || {});const env = {NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.meme', TRUST_PROXY: '1',OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,@@@ -33,6 +35,7 @@ async function boot(opts = {}) {MEME_DRILL: '1',MEME_EVENTS_URL: network.url,OV_CHAT_URL: upstreams.url, OV_LIVE_URL: upstreams.url, OV_VIDEO_URL: upstreams.url,++ OV_AI_INTERNAL_URL: ai.url,...(opts.env || {}),};for (const [k, v] of Object.entries(opts.env || {})) if (v === null) delete env[k];@@@ -75,17 +78,19 @@ async function boot(opts = {}) {}const t = {-- base, network, upstreams, config, ctx: built.ctx, get, signIn, dbDump,++ base, network, upstreams, ai, config, ctx: built.ctx, get, signIn, dbDump,logs: () => captured.join('\n'),async close() {built.ctx.importer.stop();built.ctx.reverifier.stop();++ built.ctx.judgeJob.stop();await new Promise((r) => server.close(r));built.ctx.keys.client.stop();await built.ctx.telemetry.stop().catch(() => {});await testdb.close();await network.close();await upstreams.close();++ await ai.close();},};return t;nnew file mode 100644@@@ -0,0 +1,237 @@++'use strict';++/**++ * The judge and the publish rule.++ *++ * A candidate is judged from its own citations by OpenVibe.AI (a stand-in here), and only what clears the bar is++ * published: 0.75 or more, used at least three times across at least two days or two channels, nothing denied,++ * nothing about anyone's looks. Everything else stays hidden for staff, with its score, and every decision is a++ * status event.++ */++const { boot, check, done } = require('./helpers/boot');++++const at = (days) => new Date(Date.UTC(2026, 9, 20 + days, 12, 0, 0)).toISOString();++const CHAT_IDS = [1001, 1002, 86960]; // the lines the in-process Chat stand-in holds++++/** A candidate with its uses: each use is a chat line (or a VOD second) on its own day. */++async function candidate(t, { slug, title, uses, channels = ['chat'] }) {++ const cites = uses.map((u, i) => {++ const channel = channels[Math.min(i, channels.length - 1)];++ const live = channel === 'live';++ return {++ role: i === 0 ? 'origin' : 'use',++ source_service: channel,++ source_type: live ? 'vod' : 'chat_message',++ source_id: `${slug}-${i}`,++ chat_message_id: live ? null : CHAT_IDS[i % CHAT_IDS.length],++ vod_id: live ? 9001 : null,++ offset_seconds: live ? 60 + i : null,++ quote_text: `${title} - line ${i + 1}`,++ url: live ? 'https://openvibe.video/v/9001?t=60' : 'https://openvibe.chat/',++ observed_at: u,++ };++ });++ const out = await t.ctx.entries.createCandidate({++ slug, title, kind: 'catchphrase', summary: 'a template summary', status: 'rising',++ first_seen_at: uses[0], last_seen_at: uses[uses.length - 1], visibility: 'hidden', citations: cites,++ });++ return out.entry;++}++++const answer = (score, extra = {}) => (prompt) => JSON.stringify({++ score,++ reason: `the AI scored it ${score}`,++ title: 'Cozmo is alive',++ summary: 'A running joke about the robot being back.',++ origin: 'It starts in the first cited line.',++ spread: 'People say it when the robot comes back.',++ examples: ['the robot is alive', 'cozmo is alive again'],++ ...extra,++});++++(async () => {++ await check('the judge reads the cited material: the quote and the lines around it reach the model', async () => {++ const t2 = await boot();++ try {++ await candidate(t2, { slug: 'robot-offline', title: 'The robot is offline', uses: [at(0), at(2), at(3)] });++ await t2.ctx.judgeJob.runOnce({ force: true });++ const call = t2.ai.calls[0];++ if (!call) throw new Error('the judge did not call the AI');++ if (!/the robot is offline - line 1/i.test(call.prompt)) throw new Error('the prompt did not carry the cited quote');++ if (!/Use 1/.test(call.prompt) || !/Use 3/.test(call.prompt)) throw new Error('the prompt did not carry every use');++ if (!/ada:/.test(call.prompt)) throw new Error('the prompt did not carry the chat lines around the quote');++ } finally { await t2.close(); }++ });++++ await check('the judge stores the score, the reason and the written entry', async () => {++ const t2 = await boot({ ai: { answer: answer(0.93) } });++ try {++ const entry = await candidate(t2, { slug: 'cozmo-s-alive', title: 'Cozmo s alive', uses: [at(0), at(2), at(3)] });++ await t2.ctx.judgeJob.runOnce({ force: true });++ const verdict = await t2.ctx.entries.judgement(entry.id);++ if (!verdict) throw new Error('no judgement was stored');++ if (Number(verdict.score) !== 0.93) throw new Error(`the score is ${verdict.score}`);++ if (!verdict.reason) throw new Error('no reason was stored');++ if (verdict.origin !== 'It starts in the first cited line.') throw new Error('the origin was not stored');++ if (verdict.spread !== 'People say it when the robot comes back.') throw new Error('the spread was not stored');++ if (!Array.isArray(verdict.examples) || verdict.examples.length !== 2) throw new Error('the examples were not stored');++ const after = await t2.ctx.entries.byId(entry.id);++ if (after.title !== 'Cozmo is alive') throw new Error(`the AI title was not used: ${after.title}`);++ } finally { await t2.close(); }++ });++++ await check('a candidate that clears the bar is published', async () => {++ const t2 = await boot({ ai: { answer: answer(0.9) } });++ try {++ const entry = await candidate(t2, { slug: 'yes-please', title: 'Yes please', uses: [at(0), at(2), at(3)] });++ const out = await t2.ctx.judgeJob.runOnce({ force: true });++ if (out.published !== 1) throw new Error(`expected one published entry, got ${out.published}`);++ const after = await t2.ctx.entries.byId(entry.id);++ if (after.visibility !== 'public') throw new Error('the entry is not public');++ } finally { await t2.close(); }++ });++++ await check('the rule: a score under 0.75 keeps the entry hidden', async () => {++ const t2 = await boot({ ai: { answer: answer(0.74) } });++ try {++ const entry = await candidate(t2, { slug: 'weak-phrase', title: 'Yeah thats true', uses: [at(0), at(2), at(3)] });++ const out = await t2.ctx.judgeJob.runOnce({ force: true });++ if (out.published !== 0) throw new Error('a weak entry was published');++ const after = await t2.ctx.entries.byId(entry.id);++ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');++ const list = await t2.ctx.entries.listHidden({ limit: 10 });++ const row = list.find((e) => e.id === entry.id);++ if (!row) throw new Error('the entry is not on the staff list');++ if (Number(row.score) !== 0.74) throw new Error(`the staff list shows ${row.score}`);++ } finally { await t2.close(); }++ });++++ await check('the rule: fewer than three uses keeps the entry hidden', async () => {++ const t2 = await boot({ ai: { answer: answer(0.95) } });++ try {++ const entry = await candidate(t2, { slug: 'twice-only', title: 'Twice only', uses: [at(0), at(2)] });++ const out = await t2.ctx.judgeJob.runOnce({ force: true });++ if (out.published !== 0) throw new Error('a twice-used entry was published');++ const after = await t2.ctx.entries.byId(entry.id);++ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');++ } finally { await t2.close(); }++ });++++ await check('the rule: three uses on one day in one channel keep the entry hidden', async () => {++ const t2 = await boot({ ai: { answer: answer(0.95) } });++ try {++ const entry = await candidate(t2, { slug: 'one-day', title: 'Same day', uses: [at(0), at(0), at(0)] });++ const out = await t2.ctx.judgeJob.runOnce({ force: true });++ if (out.published !== 0) throw new Error('an entry used on one day was published');++ } finally { await t2.close(); }++ });++++ await check('the rule: three uses across two channels clear the bar even on one day', async () => {++ const t2 = await boot({ ai: { answer: answer(0.95) } });++ try {++ const entry = await candidate(t2, { slug: 'two-channels', title: 'Both places', uses: [at(0), at(0), at(0)], channels: ['chat', 'live'] });++ const out = await t2.ctx.judgeJob.runOnce({ force: true });++ if (out.published !== 1) throw new Error('an entry used in two channels was not published');++ const after = await t2.ctx.entries.byId(entry.id);++ if (after.visibility !== 'public') throw new Error('the entry is not public');++ } finally { await t2.close(); }++ });++++ await check('the rule: a denied source is never stored and never published', async () => {++ const t2 = await boot({ ai: { answer: answer(0.99) } });++ try {++ const out = await t2.ctx.entries.createCandidate({++ slug: 'denied-one', title: 'Denied one', kind: 'meme', status: 'rising',++ first_seen_at: at(0), last_seen_at: at(3), visibility: 'hidden',++ citations: [++ { role: 'origin', source_service: 'live', source_type: 'vod', source_id: '5709', vod_id: 5709, offset_seconds: 10, quote_text: 'the robot is offline', url: 'https://openvibe.video/v/5709?t=10', observed_at: at(0) },++ { role: 'use', source_service: 'chat', source_type: 'chat_message', source_id: '1', chat_message_id: 1001, quote_text: 'the robot is offline', url: 'https://openvibe.chat/', observed_at: at(2) },++ { role: 'use', source_service: 'chat', source_type: 'chat_message', source_id: '2', chat_message_id: 1002, quote_text: 'the robot is offline', url: 'https://openvibe.chat/', observed_at: at(3) },++ ],++ });++ const stored = await t2.ctx.entries.citations(out.entry.id);++ if (stored.length !== 2) throw new Error(`the denied citation was stored (${stored.length} citations)`);++ if (stored.some((c) => c.vod_id === 5709)) throw new Error('the denied VOD is in the citations');++ const judged = await t2.ctx.judgeJob.runOnce({ force: true });++ if (judged.published !== 0) throw new Error('an entry citing a denied VOD was published');++ const after = await t2.ctx.entries.byId(out.entry.id);++ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');++ } finally { await t2.close(); }++ });++++ await check('the rule: text that names looks never reaches the public', async () => {++ const t2 = await boot({ ai: { answer: answer(0.99, { title: 'Cozmo in his blue shirt', summary: 'A man in a kitchen says the robot is offline.' }) } });++ try {++ const entry = await candidate(t2, { slug: 'looks-one', title: 'Looks one', uses: [at(0), at(2), at(3)] });++ const out = await t2.ctx.judgeJob.runOnce({ force: true });++ if (out.published !== 0) throw new Error('an entry whose text names looks was published');++ const after = await t2.ctx.entries.byId(entry.id);++ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');++ if (after.title === 'Cozmo in his blue shirt') throw new Error('the title that named looks was used');++ const verdict = await t2.ctx.entries.judgement(entry.id);++ if (!verdict.looks) throw new Error('the judgement does not say the text named looks');++ } finally { await t2.close(); }++ });++++ await check('the rule: text that carries a denied word never reaches the public', async () => {++ const t2 = await boot({ ai: { answer: answer(0.99, { summary: 'The joke is about a nazi in a kitchen.' }) } });++ try {++ const entry = await candidate(t2, { slug: 'denied-word', title: 'Denied word', uses: [at(0), at(2), at(3)] });++ const out = await t2.ctx.judgeJob.runOnce({ force: true });++ if (out.published !== 0) throw new Error('an entry with a denied word was published');++ const verdict = await t2.ctx.entries.judgement(entry.id);++ if (!verdict.denied) throw new Error('the judgement does not say the text is denied');++ } finally { await t2.close(); }++ });++++ await check('every decision is a status event', async () => {++ const t2 = await boot({ ai: { answer: answer(0.9) } });++ try {++ const entry = await candidate(t2, { slug: 'logged-one', title: 'Logged one', uses: [at(0), at(2), at(3)] });++ await t2.ctx.judgeJob.runOnce({ force: true });++ const history = await t2.ctx.entries.decisions(entry.id);++ if (!history.length) throw new Error('no status event was written');++ const decision = history[0];++ if (decision.decision !== 'publish') throw new Error(`the decision is ${decision.decision}`);++ if (decision.from_visibility !== 'hidden' || decision.to_visibility !== 'public') throw new Error('the visibility move was not recorded');++ if (Number(decision.score) !== 0.9) throw new Error(`the event carries the score ${decision.score}`);++ } finally { await t2.close(); }++ });++++ await check('a kept candidate is logged too, with the reasons', async () => {++ const t2 = await boot({ ai: { answer: answer(0.3) } });++ try {++ const entry = await candidate(t2, { slug: 'kept-one', title: 'Kept one', uses: [at(0), at(2), at(3)] });++ await t2.ctx.judgeJob.runOnce({ force: true });++ const history = await t2.ctx.entries.decisions(entry.id);++ if (!history.length) throw new Error('no status event was written');++ if (history[0].decision !== 'keep_hidden') throw new Error(`the decision is ${history[0].decision}`);++ if (!/scored 0.30/.test(history[0].reason || '')) throw new Error(`the reason does not say why: ${history[0].reason}`);++ } finally { await t2.close(); }++ });++++ await check('the staff list shows the hidden candidates with their score', async () => {++ const t2 = await boot({ ai: { answer: answer(0.42) } });++ try {++ const entry = await candidate(t2, { slug: 'staff-one', title: 'Staff one', uses: [at(0), at(2), at(3)] });++ await t2.ctx.judgeJob.runOnce({ force: true });++ const res = await t2.get('/api/v1/staff/candidates', { as: t2.network.addUser('mod', { role: 'global_mod' }) });++ if (res.status !== 200) throw new Error(`the staff list answered ${res.status}`);++ const body = res.json();++ const row = (body.candidates || []).find((c) => c.id === entry.id);++ if (!row) throw new Error('the candidate is not on the staff list');++ if (Number(row.score) !== 0.42) throw new Error(`the list shows the score ${row.score}`);++ if (!row.reason) throw new Error('the list shows no reason');++ if (row.uses !== 3) throw new Error(`the list shows ${row.uses} uses`);++ } finally { await t2.close(); }++ });++++ await check('the staff list is not for anonymous visitors', async () => {++ const t = await boot();++ try {++ const res = await t.get('/api/v1/staff/candidates');++ if (res.status !== 401) throw new Error(`the staff list answered ${res.status} to an anonymous caller`);++ } finally { await t.close(); }++ });++++ done();++})().catch((err) => { console.error(err); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,96 @@++'use strict';++/**++ * The templates: the well-known meme templates from Wikimedia Commons. Each one keeps its licence, its credit and++ * the file page it came from, and a file whose licence forbids reuse is never seeded.++ */++const { boot, check, done } = require('./helpers/boot');++const { createTemplatesJob } = require('../server/templates/seed');++++/** A stand-in for Wikimedia Commons. */++function commonsFetch({ license = 'CC BY-SA 4.0', mime = 'image/jpeg' } = {}) {++ return async (url) => {++ const u = new URL(url);++ const q = u.searchParams;++ const body = (obj) => ({ ok: true, status: 200, text: async () => JSON.stringify(obj), headers: new Map() });++ if (q.get('list') === 'search') {++ return body({ query: { search: [{ title: 'File:Drake Hotline Bling.jpg' }, { title: 'File:Other.jpg' }] } });++ }++ if (q.get('prop') === 'imageinfo') {++ const titles = String(q.get('titles') || '').split('|');++ return body({++ query: {++ pages: titles.map((title) => ({++ title,++ imageinfo: [{++ mime,++ url: 'https://upload.wikimedia.org/wikipedia/commons/a/ab/Drake.jpg',++ thumburl: 'https://upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Drake.jpg/800px-Drake.jpg',++ extmetadata: {++ LicenseShortName: { value: license },++ LicenseUrl: { value: 'https://creativecommons.org/licenses/by-sa/4.0/' },++ Artist: { value: '<a href="https://commons.wikimedia.org/wiki/User:Someone">Someone</a>' },++ },++ }],++ })),++ },++ });++ }++ return body({});++ };++}++++(async () => {++ await check('a template is seeded with its licence, its credit and its source page', async () => {++ const t = await boot();++ try {++ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch(), log: console });++ const out = await job.runOnce({ limit: 1 });++ if (!out.seeded) throw new Error('no template was seeded');++ const rows = await t.ctx.s.db.many('SELECT * FROM meme_templates ORDER BY created_at');++ const row = rows.find((r) => r.id.startsWith('tpl_commons_'));++ if (!row) throw new Error('the seeded template is not in the table');++ if (!row.license) throw new Error('the template carries no licence');++ if (!row.credit) throw new Error('the template carries no credit');++ if (!row.source_url || !/^https:\/\/commons\.wikimedia\.org\//.test(row.source_url)) throw new Error(`the source page is ${row.source_url}`);++ if (!/^https:\/\/upload\.wikimedia\.org\//.test(row.image_url)) throw new Error(`the picture is ${row.image_url}`);++ if (!row.attribution) throw new Error('the template carries no attribution');++ } finally { await t.close(); }++ });++++ await check('the templates are listed with their attribution', async () => {++ const t = await boot();++ try {++ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch(), log: console });++ await job.runOnce({ limit: 1 });++ const res = await t.get('/api/v1/templates');++ if (res.status !== 200) throw new Error(`the templates answered ${res.status}`);++ const body = res.json();++ const seeded = body.templates.find((x) => x.id.startsWith('tpl_commons_'));++ if (!seeded) throw new Error('the seeded template is not listed');++ if (!seeded.license) throw new Error('the listed template carries no licence');++ if (!seeded.credit) throw new Error('the listed template carries no credit');++ } finally { await t.close(); }++ });++++ await check('a file whose licence forbids reuse is never seeded', async () => {++ const t = await boot();++ try {++ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch({ license: 'All rights reserved' }), log: console });++ const out = await job.runOnce({ limit: 3 });++ if (out.seeded) throw new Error(`${out.seeded} template(s) were seeded from a file that forbids reuse`);++ const rows = await t.ctx.s.db.many("SELECT id FROM meme_templates WHERE id LIKE 'tpl_commons_%'");++ if (rows.length) throw new Error('a template was written from a file that forbids reuse');++ } finally { await t.close(); }++ });++++ await check('a file that is not a picture is never seeded', async () => {++ const t = await boot();++ try {++ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch({ mime: 'video/mp4' }), log: console });++ const out = await job.runOnce({ limit: 3 });++ if (out.seeded) throw new Error(`${out.seeded} template(s) were seeded from a file that is not a picture`);++ } finally { await t.close(); }++ });++++ done();++})().catch((err) => { console.error(err); process.exit(1); });
@@ -36,12 +36,47 @@ MEME_EVENTS_URL=
@@ -36,12 +36,47 @@ MEME_EVENTS_URL=
#MEME_EVENTS_ENDPOINT=http://127.0.0.1:5120/internal/events
#MEME_EVENTS_ENDPOINT=http://127.0.0.1:5120/internal/events
#MEME_EVENTS_INTERVAL_MS=5000
#MEME_EVENTS_INTERVAL_MS=5000
-# The public APIs the importer and the citation re-verifier read. No service grant is used in v1.
+# The public APIs the importer and the citation re-verifier read. No service grant is used for them.
OV_CHAT_URL=https://openvibe.chat
OV_CHAT_URL=https://openvibe.chat
OV_LIVE_URL=https://openvibe.live
OV_LIVE_URL=https://openvibe.live
OV_VIDEO_URL=https://openvibe.video
OV_VIDEO_URL=https://openvibe.video
#MEME_UPSTREAM_TIMEOUT_MS=8000
#MEME_UPSTREAM_TIMEOUT_MS=8000
+# OpenVibe.AI: the judge and the encyclopedia's writing pass. Meme calls ai.generate as the service `meme` with a
+# Network client-credentials token (audience openvibe.ai, scope ai.run.create); Network grants the ai.generate
+# namespace. Meme never names a provider — AI's routing decides.
+OV_AI_INTERNAL_URL=http://127.0.0.1:4700
+#MEME_AI_AUDIENCE=openvibe.ai
+#MEME_AI_SCOPE=ai.run.create
+#MEME_AI_WAIT_MS=55000
+#MEME_AI_TIMEOUT_MS=120000
+#MEME_AI_POLL_MS=1500
+#MEME_AI_MAX_TOKENS=1400
+
+# The judge: it scores a candidate from its own citations and writes the entry. Only what clears the bar is
+# published; everything else stays hidden for staff with its score.
+#MEME_JUDGE_INTERVAL_MS=1800000
+#MEME_JUDGE_MIN_SCORE=0.75
+#MEME_JUDGE_MIN_USES=3
+#MEME_JUDGE_MIN_DAYS=2
+#MEME_JUDGE_MIN_CHANNELS=2
+#MEME_JUDGE_MAX_PER_RUN=10
+#MEME_JUDGE_CONTEXT_LINES=6
+
+# The encyclopedia and the templates: Wikipedia (CC BY-SA), Wikidata (CC0) and Wikimedia Commons (each file under
+# its own licence). Read-only public APIs; a site whose terms forbid it is never scraped.
+#MEME_WIKIPEDIA_URL=https://en.wikipedia.org
+#MEME_WIKIDATA_URL=https://www.wikidata.org
+#MEME_COMMONS_URL=https://commons.wikimedia.org
+#MEME_WIKI_USER_AGENT=OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])
+#MEME_WIKI_TIMEOUT_MS=20000
+#MEME_WIKI_MIN_INTERVAL_MS=250
+#MEME_ENCYCLOPEDIA_LIMIT=400
+#MEME_ENCYCLOPEDIA_LISTS=List_of_Internet_phenomena
+#MEME_ENCYCLOPEDIA_CATEGORIES=Internet memes,Memes
+#MEME_ENCYCLOPEDIA_MIN_SCORE=0.75
+#MEME_TEMPLATES_LIMIT=40
+
# The candidate importer: every 30 minutes, hidden candidates only, off under MEME_DRILL.
# The candidate importer: every 30 minutes, hidden candidates only, off under MEME_DRILL.
#MEME_IMPORTER_INTERVAL_MS=1800000
#MEME_IMPORTER_INTERVAL_MS=1800000
#MEME_IMPORTER_MIN_USES=2
#MEME_IMPORTER_MIN_USES=2
@@ -19,6 +19,17 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta
@@ -19,6 +19,17 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta
- **Candidates stay hidden.** `server/importer.js` reads public Chat/Live APIs only, finds a 3–6 word phrase used at
- **Candidates stay hidden.** `server/importer.js` reads public Chat/Live APIs only, finds a 3–6 word phrase used at
least twice on two different days, attaches citations and inserts the entry `hidden`. It never publishes. A staff
least twice on two different days, attaches citations and inserts the entry `hidden`. It never publishes. A staff
person (Network role `global_mod` or above) publishes it with `POST /api/v1/entries/:slug/publish`.
person (Network role `global_mod` or above) publishes it with `POST /api/v1/entries/:slug/publish`.
+- **A judge writes the entry.** `server/ai/judge.js` asks OpenVibe.AI to score each candidate from its own
+ citations — with the chat lines and transcript lines around each quote — and to write the entry from that material
+ only: a title in natural case, a short summary, where it started, how people use it and two or three example uses.
+ The score and the reason are stored with the entry.
+- **Only what clears the bar is published.** Score 0.75 or more, used at least three times across at least two days
+ or two channels, nothing denied, nothing about anyone's looks, the entry complete and a live origin citation.
+ Everything else stays hidden for staff, with its score, on `GET /api/v1/staff/candidates`. Every decision is a
+ status event.
+- **The encyclopedia.** The well-known internet memes, seeded from Wikipedia's lists (CC BY-SA) and Wikidata (CC0)
+ with a Commons picture only when its licence allows it, each an entry of kind `classic` with its origin, its
+ spread, its sources and its attribution.
- **The citation is checked again.** A daily job reads every public citation URL; a 404/410 marks it `gone`, a
- **The citation is checked again.** A daily job reads every public citation URL; a 404/410 marks it `gone`, a
401/403 marks it `hidden`, and an entry whose origin is gone is hidden (revision +1, Search tombstone).
401/403 marks it `hidden`, and an entry whose origin is gone is hidden (revision +1, Search tombstone).
- **Meme of the week.** People vote once per ISO week; a second vote changes the choice, never adds a ballot.
- **Meme of the week.** People vote once per ISO week; a second vote changes the choice, never adds a ballot.
@@ -29,11 +40,12 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta
@@ -29,11 +40,12 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta
| Table | Holds | Migration |
| Table | Holds | Migration |
|---|---|---|
|---|---|---|
-| `meme_entries` | `mem_` + ULID, unique slug (`/m/<slug>`), title, kind `joke\|catchphrase\|bit\|meme\|saga`, AI summary, status + `status_since`, first/last seen, visibility `public\|unlisted\|hidden`, `revision` (+1 on every change), timestamps | `migrations/0001_initial.sql` |
-| `meme_citations` | entry, role `origin\|use\|example`, source service/type/id, stream/VOD/clip/message ids, canonical VOD `offset_seconds`, verbatim `quote_text`, `quote_hash`, canonical `url`, `observed_at`, `source_status` `live\|hidden\|gone` | `0001_initial.sql` |
-| `meme_status_events` | every status change: from, to, reason, who, when | `0001_initial.sql` |
+| `meme_entries` | `mem_` + ULID, unique slug (`/m/<slug>`), title, kind `joke\|catchphrase\|bit\|meme\|saga\|classic`, AI summary, status + `status_since`, first/last seen, visibility `public\|unlisted\|hidden`, `revision` (+1 on every change), timestamps, the picture and its licence, the attribution, the Wikidata id | `migrations/0001_initial.sql`, `0002_judge_and_encyclopedia.sql` |
+| `meme_citations` | entry, role `origin\|use\|example`, source service/type/id, stream/VOD/clip/message ids, canonical VOD `offset_seconds`, verbatim `quote_text`, `quote_hash`, canonical `url`, `observed_at`, `source_status` `live\|hidden\|gone`; the open web is a source too (`wikipedia`, `wikidata`, `commons`) | `0001_initial.sql`, `0002_judge_and_encyclopedia.sql` |
+| `meme_judgements` | one row per entry: the score, the reason, the title/summary/origin/spread/examples the AI wrote, whether the text names looks or carries a denied word, the model and the run | `0002_judge_and_encyclopedia.sql` |
+| `meme_status_events` | every status change: from, to, reason, who, when; a judge decision adds `decision`, `from_visibility`, `to_visibility` and `score` | `0001_initial.sql`, `0002_judge_and_encyclopedia.sql` |
| `meme_votes` | (`week` `YYYY-Www`, `subject`) → entry; one vote a week | `0001_initial.sql` |
| `meme_votes` | (`week` `YYYY-Www`, `subject`) → entry; one vote a week | `0001_initial.sql` |
-| `meme_templates` | id, name, `image_url` (openvibe.media or a local data-free placeholder), `box_count`, source, license; three placeholders seeded | `0001_initial.sql` |
+| `meme_templates` | id, name, `image_url` (openvibe.media, a local data-free placeholder or a Wikimedia Commons file), `box_count`, source, license, credit, source page, attribution; three placeholders seeded | `0001_initial.sql`, `0002_judge_and_encyclopedia.sql` |
| `meme_captions` | `mcp_` + ULID, subject, template, `lines` jsonb (1–4 plain-text lines), visibility, created | `0001_initial.sql` |
| `meme_captions` | `mcp_` + ULID, subject, template, `lines` jsonb (1–4 plain-text lines), visibility, created | `0001_initial.sql` |
| `meme_reports` | `mrp_` + ULID, entry, subject, kind `takedown\|correction`, reason, note, status | `0001_initial.sql` |
| `meme_reports` | `mrp_` + ULID, entry, subject, kind `takedown\|correction`, reason, note, status | `0001_initial.sql` |
| `meme_outbox` | Today's outbox shape: fixed `evt_` id, topic, JSON payload, attempts, last error, claim/backoff lease — the Search document queue | `0001_initial.sql` |
| `meme_outbox` | Today's outbox shape: fixed `evt_` id, topic, JSON payload, attempts, last error, claim/backoff lease — the Search document queue | `0001_initial.sql` |
@@ -43,8 +55,8 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta
@@ -43,8 +55,8 @@ a chat line, a VOD second, a Rundown segment — a timeline of later uses, a sta
## Does not own
## Does not own
- **Chat, Live and the VODs** — OpenVibe.Chat and OpenVibe.Live. Meme reads their public history, Rundown and
- **Chat, Live and the VODs** — OpenVibe.Chat and OpenVibe.Live. Meme reads their public history, Rundown and
- transcripts. It never reads a private VOD, never uses a service grant in v1, and never cites VOD **5709** or clip
- **483** (the fixed deny list).
+ transcripts. It never reads a private VOD, and it never cites VOD **5709** or clip **483** (the fixed deny list).
+ The one service grant Meme holds is OpenVibe.AI's `ai.run.create`, used by the judge and the encyclopedia.
- **Identity, roles and accounts** — OpenVibe.Network. Meme verifies session tokens against its JWKS; a staff
- **Identity, roles and accounts** — OpenVibe.Network. Meme verifies session tokens against its JWKS; a staff
route asks for `global_mod` or above.
route asks for `global_mod` or above.
- **Search** — OpenVibe.Search. Meme produces `search.index-document@1` documents in its own outbox; the Search
- **Search** — OpenVibe.Search. Meme produces `search.index-document@1` documents in its own outbox; the Search
@@ -61,6 +73,15 @@ Environment variable names are in [.env.example](.env.example); no value is comm
@@ -61,6 +73,15 @@ Environment variable names are in [.env.example](.env.example); no value is comm
`OV_OAUTH_REDIRECT_URI`, `OV_SESSION_AUDIENCE`, `MEME_AUDIENCE`.
`OV_OAUTH_REDIRECT_URI`, `OV_SESSION_AUDIENCE`, `MEME_AUDIENCE`.
- **OpenVibe.Chat / Live / Video** — the public reads: `OV_CHAT_URL`, `OV_LIVE_URL`, `OV_VIDEO_URL`,
- **OpenVibe.Chat / Live / Video** — the public reads: `OV_CHAT_URL`, `OV_LIVE_URL`, `OV_VIDEO_URL`,
`MEME_UPSTREAM_TIMEOUT_MS`.
`MEME_UPSTREAM_TIMEOUT_MS`.
+- **OpenVibe.AI** — the judge and the encyclopedia's writing pass: `OV_AI_INTERNAL_URL` (default
+ `http://127.0.0.1:4700`), `MEME_AI_AUDIENCE` (`openvibe.ai`), `MEME_AI_SCOPE` (`ai.run.create`),
+ `MEME_AI_WAIT_MS`, `MEME_AI_TIMEOUT_MS`, `MEME_AI_POLL_MS`, `MEME_AI_MAX_TOKENS`. Meme calls `ai.generate` as
+ the service `meme` with a Network client-credentials token; Network grants `ai.run.create` for the `ai.generate`
+ namespace. Meme never names a provider — AI's routing decides, and the run reports what answered.
+- **Wikipedia, Wikidata and Wikimedia Commons** — the encyclopedia: `MEME_WIKIPEDIA_URL`, `MEME_WIKIDATA_URL`,
+ `MEME_COMMONS_URL`, `MEME_WIKI_USER_AGENT`, `MEME_WIKI_TIMEOUT_MS`, `MEME_WIKI_MIN_INTERVAL_MS`,
+ `MEME_ENCYCLOPEDIA_LIMIT`, `MEME_ENCYCLOPEDIA_LISTS`, `MEME_ENCYCLOPEDIA_CATEGORIES`,
+ `MEME_ENCYCLOPEDIA_MIN_SCORE`, `MEME_TEMPLATES_LIMIT`.
- **OpenVibe.Events** — account deliveries: `MEME_EVENTS_SECRET`, `MEME_EVENTS_URL`, `MEME_EVENTS_ENDPOINT`.
- **OpenVibe.Events** — account deliveries: `MEME_EVENTS_SECRET`, `MEME_EVENTS_URL`, `MEME_EVENTS_ENDPOINT`.
- **PostgreSQL** — `DATABASE_URL` (through PgBouncer) and `DATABASE_DIRECT_URL` (migrations); embedded PGlite in
- **PostgreSQL** — `DATABASE_URL` (through PgBouncer) and `DATABASE_DIRECT_URL` (migrations); embedded PGlite in
`MEME_PGLITE_DIR` for development.
`MEME_PGLITE_DIR` for development.
@@ -78,7 +99,7 @@ also gets per-actor minute/hour limits (`MEME_LIMITS_MINUTE`, `MEME_LIMITS_HOUR`
@@ -78,7 +99,7 @@ also gets per-actor minute/hour limits (`MEME_LIMITS_MINUTE`, `MEME_LIMITS_HOUR`
| Route | What it does |
| Route | What it does |
|---|---|
|---|---|
| `GET /entries?limit=&cursor=&kind=&status=` | public entries, newest first; opaque keyset cursor; `Cache-Control: public, max-age=60` |
| `GET /entries?limit=&cursor=&kind=&status=` | public entries, newest first; opaque keyset cursor; `Cache-Control: public, max-age=60` |
-| `GET /entries/:slug` | one entry with `origin`, every `citation`, the use `timeline` and `examples`; a hidden one is 404 unless staff |
+| `GET /entries/:slug` | one entry with `origin`, every `citation`, the use `timeline` and `examples`, and the judge's `notes` (where it started, how people use it, the examples); a hidden one is 404 unless staff |
| `GET /entries/:slug/series?days=30` | uses per UTC day, zero-filled, 1–365 days |
| `GET /entries/:slug/series?days=30` | uses per UTC day, zero-filled, 1–365 days |
| `GET /trending?limit=` | public entries, `rising` first, then `peaked`, `classic`, `dead`; last use newest first |
| `GET /trending?limit=` | public entries, `rising` first, then `peaked`, `classic`, `dead`; last use newest first |
| `GET /week` | this ISO week's candidates (public entries with a live use that week) and their vote counts |
| `GET /week` | this ISO week's candidates (public entries with a live use that week) and their vote counts |
@@ -88,6 +109,8 @@ also gets per-actor minute/hour limits (`MEME_LIMITS_MINUTE`, `MEME_LIMITS_HOUR`
@@ -88,6 +109,8 @@ also gets per-actor minute/hour limits (`MEME_LIMITS_MINUTE`, `MEME_LIMITS_HOUR`
| `GET /captions/:id` | one caption; a public read never returns the author's subject |
| `GET /captions/:id` | one caption; a public read never returns the author's subject |
| `POST /entries/:slug/report` | signed in; `{ kind: "takedown"\|"correction", reason, note? }`, stored for staff |
| `POST /entries/:slug/report` | signed in; `{ kind: "takedown"\|"correction", reason, note? }`, stored for staff |
| `GET /reports` | staff: the open reports |
| `GET /reports` | staff: the open reports |
+| `GET /staff/candidates` | staff: the hidden candidates with the judge's score, its reason, the uses, the days and the channels |
+| `GET /staff/candidates/:slug` | staff: one candidate — its judgement, its citations and its decision history |
| `POST /entries/:slug/publish` | staff: hidden → public; `409 meme.origin_gone` when the origin is not live |
| `POST /entries/:slug/publish` | staff: hidden → public; `409 meme.origin_gone` when the origin is not live |
| `POST /entries/:slug/hide` | staff: public → hidden; Search tombstone |
| `POST /entries/:slug/hide` | staff: public → hidden; Search tombstone |
| `PATCH /entries/:slug` | staff: `{ title, kind, status, reason? }`; a status change writes `meme_status_events` |
| `PATCH /entries/:slug` | staff: `{ title, kind, status, reason? }`; a status change writes `meme_status_events` |
@@ -96,6 +119,47 @@ Validation refusals are `400 { code: "meme.invalid", message }`; a blocked capti
@@ -96,6 +119,47 @@ Validation refusals are `400 { code: "meme.invalid", message }`; a blocked capti
`400 { code: "meme.blocked_words" }`. Other codes: `meme.sign_in_required`, `meme.staff_required`,
`400 { code: "meme.blocked_words" }`. Other codes: `meme.sign_in_required`, `meme.staff_required`,
`meme.cross_site`, `meme.not_found`, `meme.template_not_found`, `meme.origin_gone`, `meme.rate_limited`.
`meme.cross_site`, `meme.not_found`, `meme.template_not_found`, `meme.origin_gone`, `meme.rate_limited`.
+## The judge
+
+`server/jobs/judge.js` runs every 30 minutes (`MEME_JUDGE_INTERVAL_MS`), off under `MEME_DRILL`; tests call
+`runOnce()`.
+
+For every candidate nobody has judged yet it:
+
+1. reads the entry's live citations and opens each one up — the chat lines around a chat message, the transcript
+ lines around a VOD second (`server/ai/context.js`), because a joke only makes sense with its neighbours;
+2. asks OpenVibe.AI to score it from 0 to 1 on whether it is a real meme, catchphrase or inside joke rather than
+ everyday words that recur, and to write the entry from that material only;
+3. stores the score, the reason and the written entry in `meme_judgements`;
+4. applies the publish rule (`server/ai/gate.js`): score ≥ `MEME_JUDGE_MIN_SCORE` (0.75), at least
+ `MEME_JUDGE_MIN_USES` (3) uses across at least `MEME_JUDGE_MIN_DAYS` (2) days or `MEME_JUDGE_MIN_CHANNELS` (2)
+ channels, nothing denied, nothing that names looks, the entry complete, and a live origin citation;
+5. publishes what cleared the bar and leaves the rest hidden, logging both as a status event with the score and
+ the reasons.
+
+The AI's text is never taken at its word: a title that names looks is dropped (the entry keeps the name it had),
+every other field is repaired word by word with `noLooks`, and a field that still names looks — or that carries a
+word from the deny list — keeps the entry hidden.
+
+`node scripts/rejudge.js` runs one round over every hidden candidate and prints what was published and why;
+`--dry-run` judges and reports without publishing, `--slug <slug>` judges only those.
+
+## The encyclopedia
+
+`node scripts/seed-classics.js` seeds the well-known internet memes; `node scripts/seed-templates.js` seeds the
+caption templates. Neither starts the server.
+
+The classics are read from Wikipedia's lists of Internet phenomena and memes (CC BY-SA) and from the categories
+behind them, and every entry is finished from Wikidata (CC0): the Q id, the inception date and the picture. A
+Commons picture is used only when its licence allows it (public domain, CC0, CC), and each entry keeps its credit,
+its licence, its file page and the attribution of the text it was written from. The same AI pass writes the entry
+— the title in natural case, the summary, where it started, how it spread, two or three example uses — and the
+same publish rule decides what becomes public: an entry the AI scored under `MEME_ENCYCLOPEDIA_MIN_SCORE` (0.75),
+or whose text names looks or carries a denied word, stays hidden for staff with its score.
+
+Nothing is scraped from a site whose terms forbid it: Wikipedia, Wikidata and Wikimedia Commons only, each with
+its User-Agent, paced per host.
+
## The importer
## The importer
`server/importer.js` runs every 30 minutes (`MEME_IMPORTER_INTERVAL_MS`), off under `MEME_DRILL`; tests call
`server/importer.js` runs every 30 minutes (`MEME_IMPORTER_INTERVAL_MS`), off under `MEME_DRILL`; tests call
@@ -180,13 +244,18 @@ npm run test:pg # the same through PostgreSQL + PgBouncer (openvibe-sdk
@@ -180,13 +244,18 @@ npm run test:pg # the same through PostgreSQL + PgBouncer (openvibe-sdk
OV_ALLOW_SUITE=1 npm test
OV_ALLOW_SUITE=1 npm test
npm start # PORT 5120, loopback; DATABASE_URL or an embedded PGlite in development
npm start # PORT 5120, loopback; DATABASE_URL or an embedded PGlite in development
node scripts/seed-proof.js # the first proof, hidden, from the live public APIs
node scripts/seed-proof.js # the first proof, hidden, from the live public APIs
+node scripts/rejudge.js # one judge round over every hidden candidate; prints what was published
+node scripts/seed-classics.js # the encyclopedia, from Wikipedia and Wikidata
+node scripts/seed-templates.js # the caption templates, from Wikimedia Commons
```
```
-The tests boot the app with a mock Network, a Chat/Live/Video fixture server, and a temporary database: the
-migration, the importer's candidate and its canonical redirect, the deny list, `noLooks`, the one-vote-a-week rule,
-the caption limits, staff publish/patch/hide, the Search document (validated against
+The tests boot the app with a mock Network, an OpenVibe.AI stand-in, a Chat/Live/Video fixture server, and a
+temporary database: the migration, the importer's candidate and its canonical redirect, the deny list, `noLooks`,
+the one-vote-a-week rule, the caption limits, staff publish/patch/hide, the Search document (validated against
`search.index-document@1`), the citation 404 → gone → hidden path, sign-in, the pages and cache headers, and the
`search.index-document@1`), the citation 404 → gone → hidden path, sign-in, the pages and cache headers, and the
-account export/deletion over votes, captions and reports.
+account export/deletion over votes, captions and reports. The judge, the publish rule, the encyclopedia's
+attribution and the templates' licences have their own files: `test/judge.test.js`, `test/encyclopedia.test.js`
+and `test/templates.test.js`.
## Deploy
## Deploy
new file mode 100644
new file mode 100644
@@ -0,0 +1,68 @@
@@ -0,0 +1,68 @@
+-- phase: expand
+-- The judge and the encyclopedia. A candidate is judged by OpenVibe.AI from its own citations: a score, a reason and
+-- the written entry (title, summary, where it started, how people use it, two or three examples). The judgement is
+-- kept beside the entry, never inside it, so the entry stays the same shape it was.
+--
+-- The encyclopedia adds entries of kind 'classic': the well-known internet memes, cited to Wikipedia (CC BY-SA) and
+-- to Wikidata (CC0) with a Commons image only when its licence allows it. A citation from the open web is a new
+-- source_service ('web') and a new source_type ('wikipedia' | 'wikidata' | 'commons'); the OpenVibe sources keep
+-- their own. Every decision the judge takes is a status event with its score, so the history says why.
+--
+-- Nothing here is a secret: a citation stores a public URL and the sha256 of the quoted text.
+
+-- 'classic' joins the kinds: an encyclopedia entry is not an OpenVibe running joke.
+ALTER TABLE meme_entries DROP CONSTRAINT meme_entries_kind;
+ALTER TABLE meme_entries ADD CONSTRAINT meme_entries_kind CHECK (kind IN ('joke', 'catchphrase', 'bit', 'meme', 'saga', 'classic'));
+
+-- The open web is a source now: Wikipedia, Wikidata and Wikimedia Commons.
+ALTER TABLE meme_citations DROP CONSTRAINT meme_citations_source_service;
+ALTER TABLE meme_citations ADD CONSTRAINT meme_citations_source_service CHECK (source_service IN ('live', 'chat', 'community', 'web'));
+ALTER TABLE meme_citations DROP CONSTRAINT meme_citations_source_type;
+ALTER TABLE meme_citations ADD CONSTRAINT meme_citations_source_type CHECK (source_type IN ('chat_message', 'vod', 'clip', 'rundown_segment', 'ai_paste', 'stream_memory', 'wikipedia', 'wikidata', 'commons'));
+
+-- An encyclopedia entry keeps its picture and where it came from. The licence is the Commons file's own.
+ALTER TABLE meme_entries ADD COLUMN image_url text;
+ALTER TABLE meme_entries ADD COLUMN image_credit text;
+ALTER TABLE meme_entries ADD COLUMN image_license text;
+ALTER TABLE meme_entries ADD COLUMN image_license_url text;
+ALTER TABLE meme_entries ADD COLUMN image_source_url text;
+ALTER TABLE meme_entries ADD COLUMN attribution text; -- the licences the entry's text is under
+ALTER TABLE meme_entries ADD COLUMN external_id text; -- the Wikidata Q id, when there is one
+CREATE UNIQUE INDEX meme_entries_external_id ON meme_entries (external_id) WHERE external_id IS NOT NULL;
+
+-- One judgement per entry: what the AI decided, why, and what it wrote.
+CREATE TABLE meme_judgements (
+ entry_id text COLLATE "C" PRIMARY KEY REFERENCES meme_entries (id) ON DELETE CASCADE,
+ score numeric(4,3) NOT NULL, -- 0 to 1
+ reason text NOT NULL, -- why the AI gave that score
+ title text, -- the AI's title, in natural case
+ summary text, -- the AI's short summary
+ origin text, -- where it started
+ spread text, -- how people use it, and how it spread
+ examples jsonb NOT NULL DEFAULT '[]'::jsonb, -- two or three example uses
+ denied boolean NOT NULL DEFAULT false, -- a denied source or a denied word
+ looks boolean NOT NULL DEFAULT false, -- the text named looks
+ model text,
+ provider text,
+ run_id text,
+ judged_at timestamptz NOT NULL DEFAULT now(),
+ judged_by text COLLATE "C" NOT NULL DEFAULT 'service:meme:judge',
+ CONSTRAINT meme_judgements_score CHECK (score >= 0 AND score <= 1),
+ CONSTRAINT meme_judgements_examples CHECK (jsonb_typeof(examples) = 'array')
+);
+CREATE INDEX meme_judgements_by_score ON meme_judgements (score DESC);
+
+-- A decision is a status event: the visibility moved, the score says why.
+ALTER TABLE meme_status_events ADD COLUMN decision text; -- publish | keep_hidden | judge
+ALTER TABLE meme_status_events ADD COLUMN from_visibility text;
+ALTER TABLE meme_status_events ADD COLUMN to_visibility text;
+ALTER TABLE meme_status_events ADD COLUMN score numeric(4,3);
+ALTER TABLE meme_status_events ADD CONSTRAINT meme_status_events_decision CHECK (decision IS NULL OR decision IN ('publish', 'keep_hidden', 'judge', 'hide', 'seed'));
+
+-- A template says where its picture came from and under which licence.
+ALTER TABLE meme_templates ADD COLUMN credit text;
+ALTER TABLE meme_templates ADD COLUMN source_url text;
+ALTER TABLE meme_templates ADD COLUMN attribution text;
+-- The picture may be an openvibe.media object, a file of this site, or a Wikimedia Commons file under a free licence.
+ALTER TABLE meme_templates DROP CONSTRAINT meme_templates_image;
+ALTER TABLE meme_templates ADD CONSTRAINT meme_templates_image CHECK (image_url ~ '^(https://openvibe\.media/|/img/|https://upload\.wikimedia\.org/)');
@@ -8,6 +8,9 @@
@@ -8,6 +8,9 @@
"start": "node server/index.js",
"start": "node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"seed:proof": "node scripts/seed-proof.js",
"seed:proof": "node scripts/seed-proof.js",
+ "seed:classics": "node scripts/seed-classics.js",
+ "seed:templates": "node scripts/seed-templates.js",
+ "rejudge": "node scripts/rejudge.js",
"test": "node test/run.js",
"test": "node test/run.js",
"test:pg": "MEME_TEST_STORE=pg node test/run.js"
"test:pg": "MEME_TEST_STORE=pg node test/run.js"
},
},
new file mode 100644
new file mode 100644
@@ -0,0 +1,91 @@
@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * The re-judge round: every candidate is judged again from its own citations, and the publish rule decides what
+ * becomes public. The script says what it did, entry by entry, with the score and the reasons.
+ *
+ * node scripts/rejudge.js # every hidden candidate
+ * node scripts/rejudge.js --slug cozmo-s-alive --slug wait-wait-wait
+ * node scripts/rejudge.js --dry-run # judge and report, publish nothing
+ *
+ * A candidate that clears the bar (0.75 or more, used at least three times across at least two days or two
+ * channels, nothing denied, nothing about looks) is published and the decision is a status event. Everything else
+ * stays hidden for staff, with its score on GET /api/v1/staff/candidates.
+ */
+process.env.NODE_ENV = process.env.NODE_ENV || 'development';
+require('dotenv').config();
+
+const { load } = require('../server/config');
+const { openStore } = require('../server/db');
+const { createStore } = require('../server/store');
+const { createSearchIndex } = require('../server/search');
+const { createUpstreams } = require('../server/upstreams');
+const { createMemeAi } = require('../server/ai/client');
+const { createContext } = require('../server/ai/context');
+const { createJudge } = require('../server/ai/judge');
+const { createJudgeJob } = require('../server/jobs/judge');
+
+const arg = (name) => {
+ const i = process.argv.indexOf(`--${name}`);
+ if (i < 0) return null;
+ const next = process.argv[i + 1];
+ return next && !next.startsWith('--') ? next : true;
+};
+
+async function main() {
+ const config = load();
+ const slugs = process.argv.reduce((acc, v, i, all) => (v === '--slug' && all[i + 1] ? [...acc, all[i + 1]] : acc), []);
+ const dryRun = process.argv.includes('--dry-run');
+ const s = await openStore(config, { log: console });
+ try {
+ const search = createSearchIndex({ config, s });
+ const entries = createStore(s, { search });
+ const upstreams = createUpstreams({ config, fetchImpl: globalThis.fetch, log: console });
+ const ai = createMemeAi({ config, fetchImpl: globalThis.fetch, log: console });
+ const context = createContext({ config, upstreams, log: console });
+ const judge = createJudge({ config, ai, context, log: console });
+ const job = createJudgeJob({ config, s, entries, judge, context, log: console });
+ if (!ai.enabled) throw new Error('OpenVibe.AI is not configured (OV_OAUTH_CLIENT_SECRET and OV_AI_INTERNAL_URL); nothing was judged');
+
+ const before = await entries.listHidden({ limit: 500 });
+ const out = await job.runOnce({ force: !slugs.length, only: slugs.length ? slugs : null, publish: !dryRun });
+ const after = await entries.listHidden({ limit: 500 });
+
+ const published = [];
+ const kept = [];
+ for (const entry of before) {
+ const verdict = await entries.judgement(entry.id);
+ const row = after.find((e) => e.id === entry.id);
+ const line = {
+ slug: entry.slug,
+ title: entry.title,
+ score: verdict ? verdict.score : null,
+ reason: verdict ? verdict.reason : null,
+ uses: row ? row.uses : entry.uses,
+ days: row ? row.days : 0,
+ channels: row ? row.channels : 0,
+ visibility: row ? row.visibility : 'published',
+ };
+ if (!row) published.push(line);
+ else kept.push(line);
+ }
+ console.log(`[rejudge] ${JSON.stringify(out)}`);
+ for (const line of published) console.log(`[rejudge] PUBLISHED ${line.slug} — ${line.score} — ${line.reason}`);
+ for (const line of kept) console.log(`[rejudge] hidden ${line.slug} — ${line.score} — ${line.uses} uses, ${line.days} days, ${line.channels} channels`);
+ console.log(`[rejudge] published ${published.length}, kept hidden ${kept.length}`);
+ if (dryRun) console.log('[rejudge] dry run: the entries above were judged; publish them with the job again without --dry-run');
+ process.exitCode = out.ok ? 0 : 1;
+ } finally {
+ await s.close();
+ }
+}
+
+if (require.main === module) {
+ main().catch((err) => {
+ console.error(`[rejudge] ${err && err.message ? err.message : err}`);
+ process.exitCode = 1;
+ });
+}
+
+module.exports = { main };
new file mode 100644
new file mode 100644
@@ -0,0 +1,66 @@
@@ -0,0 +1,66 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Seeds the encyclopedia: the well-known internet memes from Wikipedia's lists (CC BY-SA) and Wikidata (CC0), each
+ * written by the same AI pass the OpenVibe candidates use, each cited to the text it was written from.
+ *
+ * node scripts/seed-classics.js # write what is missing, publish what clears the bar
+ * node scripts/seed-classics.js --limit 20 # only the first twenty names of the list
+ * node scripts/seed-classics.js --dry-run # write nothing; say what would be written
+ *
+ * The script never starts the HTTP server and never starts a job timer. It reads only the official public APIs of
+ * Wikipedia, Wikidata and Wikimedia Commons — never a site whose terms forbid it. What is already seeded (a meme
+ * with a Wikidata id already in the table) is left alone, so the script can be run again.
+ */
+process.env.NODE_ENV = process.env.NODE_ENV || 'development';
+require('dotenv').config();
+
+const { load } = require('../server/config');
+const { openStore } = require('../server/db');
+const { createStore } = require('../server/store');
+const { createSearchIndex } = require('../server/search');
+const { createMemeAi } = require('../server/ai/client');
+const { createContext } = require('../server/ai/context');
+const { createJudge } = require('../server/ai/judge');
+const { createSources } = require('../server/classics/sources');
+const { createClassicsJob } = require('../server/classics/seed');
+
+const arg = (name) => {
+ const i = process.argv.indexOf(`--${name}`);
+ if (i < 0) return null;
+ const next = process.argv[i + 1];
+ return next && !next.startsWith('--') ? next : true;
+};
+
+async function main() {
+ const config = load();
+ const limit = arg('limit');
+ const dryRun = process.argv.includes('--dry-run');
+ const s = await openStore(config, { log: console });
+ try {
+ const search = createSearchIndex({ config, s });
+ const entries = createStore(s, { search });
+ const upstreams = { chatContext: async () => [], vodContext: async () => [] };
+ const ai = createMemeAi({ config, fetchImpl: globalThis.fetch, log: console });
+ const context = createContext({ config, upstreams, log: console });
+ const judge = createJudge({ config, ai, context, log: console });
+ const sources = createSources({ config, fetchImpl: globalThis.fetch, log: console });
+ const job = createClassicsJob({ config, s, entries, sources, judge, log: console });
+ if (!ai.enabled) throw new Error('OpenVibe.AI is not configured (OV_OAUTH_CLIENT_SECRET and OV_AI_INTERNAL_URL); nothing was written');
+ const out = await job.runOnce({ limit: limit ? Number(limit) : null, publish: !dryRun });
+ console.log(`[seed-classics] ${JSON.stringify({ ...out, errors: (out.errors || []).slice(0, 10) })}`);
+ process.exitCode = out.ok ? 0 : 1;
+ } finally {
+ await s.close();
+ }
+}
+
+if (require.main === module) {
+ main().catch((err) => {
+ console.error(`[seed-classics] ${err && err.message ? err.message : err}`);
+ process.exitCode = 1;
+ });
+}
+
+module.exports = { main };
new file mode 100644
new file mode 100644
@@ -0,0 +1,55 @@
@@ -0,0 +1,55 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Seeds the caption templates: the well-known meme templates from Wikimedia Commons, each under a licence that lets
+ * us use it (public domain, CC0 or a Creative Commons licence), each with its credit and the file page it came from.
+ *
+ * node scripts/seed-templates.js # all of them
+ * node scripts/seed-templates.js --limit 10 # only the first ten
+ *
+ * The script never starts the HTTP server. A template whose file cannot be found, or whose licence does not allow
+ * reuse, is skipped and listed in the output. The three data-free placeholders of v1 are left as they are.
+ */
+process.env.NODE_ENV = process.env.NODE_ENV || 'development';
+require('dotenv').config();
+
+const { load } = require('../server/config');
+const { openStore } = require('../server/db');
+const { createStore } = require('../server/store');
+const { createSearchIndex } = require('../server/search');
+const { createTemplatesJob } = require('../server/templates/seed');
+
+const arg = (name) => {
+ const i = process.argv.indexOf(`--${name}`);
+ if (i < 0) return null;
+ const next = process.argv[i + 1];
+ return next && !next.startsWith('--') ? next : true;
+};
+
+async function main() {
+ const config = load();
+ const limit = arg('limit');
+ const s = await openStore(config, { log: console });
+ try {
+ const search = createSearchIndex({ config, s });
+ const entries = createStore(s, { search });
+ const job = createTemplatesJob({ config, entries, fetchImpl: globalThis.fetch, log: console });
+ const out = await job.runOnce({ limit: limit ? Number(limit) : null });
+ for (const t of out.templates) console.log(`[seed-templates] ${t.id} · ${t.name} · ${t.license} · ${t.source_url}`);
+ for (const m of out.missing) console.log(`[seed-templates] skipped ${m.name}: ${m.reason}`);
+ console.log(`[seed-templates] ${JSON.stringify({ seeded: out.seeded, skipped: out.skipped })}`);
+ process.exitCode = 0;
+ } finally {
+ await s.close();
+ }
+}
+
+if (require.main === module) {
+ main().catch((err) => {
+ console.error(`[seed-templates] ${err && err.message ? err.message : err}`);
+ process.exitCode = 1;
+ });
+}
+
+module.exports = { main };
new file mode 100644
new file mode 100644
@@ -0,0 +1,189 @@
@@ -0,0 +1,189 @@
+'use strict';
+
+/**
+ * OpenVibe.Meme's only model gateway: OpenVibe.AI's `generate` operation, called as the first-party service `meme`
+ * with a client-credentials token (audience openvibe.ai, scope ai.run.create). Network grants the token and the
+ * `ai.*` namespace lets it run `ai.generate` — the same path Live and Design take.
+ *
+ * Nothing here knows about memes: the client takes a prompt and a system message and answers the text. The judge
+ * (server/ai/judge.js) is what asks a question and reads an answer.
+ *
+ * A run that never finishes is an error, not a hang: every call carries a deadline. AI's routing is the authority —
+ * Meme never names a provider, so a pool's choice is what actually answered, and that is what is recorded.
+ */
+const { serviceAuth } = require('openvibe-contracts');
+
+class MemeAiError extends Error {
+ constructor(status, code, detail) {
+ super(detail || code);
+ this.name = 'MemeAiError';
+ this.status = status;
+ this.code = code;
+ this.detail = detail || code;
+ }
+}
+
+/** The JSON a model usually meant: a fenced block, a stray sentence around it, or a truncated tail. */
+function parseJsonLoose(text) {
+ if (!text) return null;
+ const raw = String(text).trim();
+ const fenced = /```(?:json)?\s*([\s\S]*?)```/i.exec(raw);
+ const body = fenced ? fenced[1].trim() : raw;
+ try { return JSON.parse(body); } catch { /* not JSON on its own */ }
+ const start = body.indexOf('{');
+ const end = body.lastIndexOf('}');
+ if (start >= 0 && end > start) {
+ try { return JSON.parse(body.slice(start, end + 1)); } catch { /* not JSON either */ }
+ }
+ return null;
+}
+
+function createMemeAi({ config, fetchImpl = globalThis.fetch, log = console }) {
+ const ai = config.ai || {};
+ const tokenClient = config.oauth.clientSecret
+ ? serviceAuth.createTokenClient({
+ tokenUrl: `${config.networkInternalUrl || config.networkUrl}/oauth/token`,
+ clientId: config.oauth.clientId,
+ clientSecret: config.oauth.clientSecret,
+ audience: ai.audience,
+ scope: ai.scope,
+ fetchImpl,
+ })
+ : null;
+ const enabled = Boolean(ai.url && tokenClient);
+ const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+
+ async function call(method, path, body, { retried = false } = {}) {
+ if (!enabled) throw new MemeAiError(503, 'meme.ai_not_configured', 'OpenVibe.AI is not configured for OpenVibe.Meme.');
+ let headers;
+ try {
+ headers = { Accept: 'application/json', ...(body ? { 'Content-Type': 'application/json' } : {}), ...(await tokenClient.authHeaders()) };
+ } catch (err) {
+ throw new MemeAiError(502, 'meme.ai_token_failed', 'OpenVibe.Network did not issue an AI token for Meme.');
+ }
+ let res;
+ try {
+ res = await fetchImpl(`${String(ai.url).replace(/\/+$/, '')}${path}`, {
+ method,
+ headers,
+ body: body ? JSON.stringify(body) : undefined,
+ signal: AbortSignal.timeout(ai.timeoutMs),
+ });
+ } catch (err) {
+ throw new MemeAiError(502, 'meme.ai_unreachable', err && err.name === 'TimeoutError'
+ ? 'OpenVibe.AI did not answer in time.'
+ : 'OpenVibe.AI could not be reached.');
+ }
+ if (res.status === 401 && !retried) {
+ tokenClient.invalidate();
+ return call(method, path, body, { retried: true });
+ }
+ const json = await res.json().catch(() => null);
+ if (!res.ok) {
+ const code = (json && (json.code || json.error)) || `meme.ai_http_${res.status}`;
+ const detail = (json && (json.detail || json.message)) || `OpenVibe.AI refused the run (${res.status}).`;
+ throw new MemeAiError(res.status >= 500 ? 502 : res.status, String(code).slice(0, 80), String(detail).slice(0, 500));
+ }
+ return json;
+ }
+
+ /** The run's answer, however the run finished. */
+ async function waitFor(run, startedAt) {
+ let r = run;
+ const deadline = Date.now() + ai.timeoutMs;
+ while (r && ['queued', 'running'].includes(r.status)) {
+ if (!r.id) throw new MemeAiError(502, 'meme.ai_bad_answer', 'OpenVibe.AI answered with a running run that has no id.');
+ if (Date.now() > deadline) throw new MemeAiError(504, 'meme.ai_timeout', `The AI run ${r.id} did not finish in time.`);
+ await sleep(ai.pollMs);
+ const body = await call('GET', `/api/v1/runs/${encodeURIComponent(r.id)}`);
+ r = body && body.run ? body.run : body;
+ }
+ if (!r) throw new MemeAiError(502, 'meme.ai_bad_answer', 'OpenVibe.AI answered without a run.');
+ const done = ['succeeded', 'cached', 'completed', 'ok'].includes(r.status)
+ || (r.status == null && r.output != null && !r.error);
+ if (!done) {
+ const e = r.error || {};
+ throw new MemeAiError(502, 'meme.ai_run_failed', `The AI run ${r.id || '?'} ${r.status || 'did not finish'}${e.code ? ` (${e.code}: ${e.detail || ''})` : ''}.`);
+ }
+ return r;
+ }
+
+ /** The text of a run's output, whatever the workflow wrapped it in. */
+ function textOf(output) {
+ if (typeof output === 'string') return output;
+ if (!output || typeof output !== 'object') return '';
+ for (const k of ['text', 'content', 'body']) if (typeof output[k] === 'string') return output[k];
+ if (output.message && typeof output.message.content === 'string') return output.message.content;
+ if (Array.isArray(output.choices) && output.choices[0] && typeof output.choices[0].message === 'object') {
+ return String(output.choices[0].message.content || '');
+ }
+ return '';
+ }
+
+ /**
+ * One run of ai.generate. Answers { text, model, provider, run_id, ms, tokens, cost_usd }.
+ * The target names the work, so a run in AI's list says what it was for.
+ */
+ async function generate({ prompt, system = '', maxTokens = null, temperature = null, target = null, wait = null }) {
+ if (!enabled) throw new MemeAiError(503, 'meme.ai_not_configured', 'OpenVibe.AI is not configured for OpenVibe.Meme.');
+ const body = {
+ prompt: String(prompt || ''),
+ ...(system ? { system: String(system) } : {}),
+ max_tokens: maxTokens || ai.maxTokens,
+ temperature: temperature == null ? ai.temperature : temperature,
+ options: { cache: false },
+ ...(target ? { target } : {}),
+ };
+ const startedAt = Date.now();
+ const waitMs = Math.min(wait || ai.waitMs, 60_000);
+ let created;
+ try {
+ created = await call('POST', `/api/v1/generate?wait=${waitMs}`, body);
+ } catch (err) {
+ if (!err || err.status !== 404) throw err;
+ // A deployment that exposes the run API but not the convenience operation: the same run, by workflow.
+ created = await call('POST', `/api/v1/runs?wait=${waitMs}`, {
+ workflow: 'ai.generate',
+ input: { prompt: body.prompt, ...(body.system ? { system: body.system } : {}), max_tokens: body.max_tokens, temperature: body.temperature },
+ options: body.options,
+ ...(target ? { target } : {}),
+ });
+ }
+ const first = created && created.run ? created.run : created;
+ const run = await waitFor(first, startedAt);
+ const text = textOf(run.output);
+ const usage = run.usage || {};
+ const prov = run.provenance || {};
+ const tokens = usage.tokens && typeof usage.tokens === 'object'
+ ? (Number(usage.tokens.input) || 0) + (Number(usage.tokens.output) || 0)
+ : (Number(usage.tokens_in) || 0) + (Number(usage.tokens_out) || 0)
+ || (Number(usage.input_tokens) || 0) + (Number(usage.output_tokens) || 0)
+ || (Number(run.tokens) || 0);
+ return {
+ text: String(text || ''),
+ model: String(prov.model || run.model || (run.output && run.output.model) || 'unknown').slice(0, 120),
+ provider: String(prov.provider || run.provider || (run.output && run.output.provider) || 'ai').slice(0, 80),
+ run_id: run.id || null,
+ ms: Math.max(0, Date.now() - startedAt),
+ tokens: Math.max(0, Math.round(tokens)),
+ cost_usd: Number(usage.cost_usd != null ? usage.cost_usd : (usage.cost != null ? usage.cost : (run.cost_usd || 0))) || 0,
+ };
+ }
+
+ /** One run, asked for JSON, repaired when the model fences it or adds a sentence. */
+ async function json({ prompt, system = '', schema = null, temperature = null, target = null }) {
+ const ask = schema
+ ? `${prompt}\n\nAnswer with one JSON object and nothing else, with exactly these keys: ${Object.keys(schema).join(', ')}.\nThe shape is:\n${JSON.stringify(schema)}`
+ : `${prompt}\n\nAnswer with one JSON object and nothing else.`;
+ const out = await generate({ prompt: ask, system, temperature: temperature == null ? 0 : temperature, target });
+ const parsed = parseJsonLoose(out.text);
+ if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
+ throw new MemeAiError(502, 'meme.ai_not_json', 'The AI run did not answer with a JSON object.');
+ }
+ return { ...out, data: parsed };
+ }
+
+ return { enabled, generate, json, parseJsonLoose, MemeAiError };
+}
+
+module.exports = { createMemeAi, parseJsonLoose, MemeAiError };
new file mode 100644
new file mode 100644
@@ -0,0 +1,79 @@
@@ -0,0 +1,79 @@
+'use strict';
+
+/**
+ * The cited material a judge reads. A citation stores one verbatim line; a joke only makes sense with its
+ * neighbours, so before the AI is asked anything, each citation is opened up: the chat lines around a chat message,
+ * the transcript lines around a VOD second. Nothing new is created — every line comes from the same public API the
+ * citation itself came from, and a citation whose context cannot be read is still cited, just without context.
+ *
+ * The block that reaches the model is the only place a quote is shown: the verbatim text and the lines around it.
+ */
+const { parseApiTime } = require('../upstreams');
+
+const DAY = 86_400_000;
+
+/** "2026-10-20 10:00:00" → the day it happened, in UTC. */
+function utcDay(iso) {
+ const s = String(iso || '');
+ return s.length >= 10 ? s.slice(0, 10) : '';
+}
+
+function createContext({ config, upstreams, log = console }) {
+ const lines = Math.max(1, Number(config.judge && config.judge.contextLines) || 6);
+
+ /** The lines around one citation. Never throws: a missing context is an empty context. */
+ async function around(citation) {
+ try {
+ if (citation.source_type === 'chat_message' || citation.source_service === 'chat') {
+ return await upstreams.chatContext(citation, { lines });
+ }
+ if (citation.vod_id != null && citation.offset_seconds != null) {
+ return await upstreams.vodContext(citation.vod_id, citation.offset_seconds, { lines });
+ }
+ } catch (err) {
+ log.warn(`[OpenVibe.Meme] context for ${citation.source_type}:${citation.source_id}: ${err.message}`);
+ }
+ return [];
+ }
+
+ /** Every citation with its context, oldest first. */
+ async function prepare(citations) {
+ const out = [];
+ for (const citation of citations || []) {
+ out.push({ citation, context: await around(citation) });
+ }
+ return out;
+ }
+
+ /**
+ * The block the judge reads: one numbered use per citation, with the quote and the lines around it.
+ * The text is trimmed so a long transcript cannot push the question out of the model's window.
+ */
+ function render(prepared) {
+ const parts = [];
+ let n = 0;
+ for (const { citation, context } of prepared || []) {
+ n += 1;
+ const when = utcDay(citation.observed_at);
+ const where = citation.source_service === 'live' ? 'OpenVibe.Live'
+ : citation.source_service === 'chat' ? 'OpenVibe.Chat'
+ : citation.source_service === 'community' ? 'OpenVibe.Community'
+ : citation.source_type === 'wikipedia' ? 'Wikipedia'
+ : citation.source_type === 'wikidata' ? 'Wikidata'
+ : citation.source_type === 'commons' ? 'Wikimedia Commons'
+ : citation.source_service;
+ const head = `Use ${n} (${where}, ${when || 'unknown date'}): "${String(citation.quote_text || '').slice(0, 600)}"`;
+ const linesOut = [];
+ for (const line of context || []) {
+ if (line.who) linesOut.push(` ${line.who}: ${String(line.text || '').slice(0, 400)}`);
+ else linesOut.push(` ${String(line.text || '').slice(0, 400)}`);
+ }
+ parts.push(linesOut.length ? `${head}\n${linesOut.join('\n')}` : head);
+ }
+ return parts.join('\n\n');
+ }
+
+ return { prepare, render, around, utcDay };
+}
+
+module.exports = { createContext, utcDay };
new file mode 100644
new file mode 100644
@@ -0,0 +1,99 @@
@@ -0,0 +1,99 @@
+'use strict';
+
+/**
+ * The publish rule. A judged candidate is published only when it clears every bar:
+ *
+ * - the AI scored it 0.75 or more on "is this a real meme, catchphrase or inside joke";
+ * - it was used at least three times, across at least two days or at least two channels;
+ * - nothing it cites is on the deny list, and its own text carries no denied word;
+ * - its text names nobody's looks;
+ * - it is complete: a title, a summary, where it started and how people use it;
+ * - it has a live origin citation (an entry without an origin is not an entry).
+ *
+ * Everything else stays hidden for staff, with its score. The reasons are returned, not just a yes or a no, so the
+ * staff list can say why.
+ */
+const { utcDay } = require('./context');
+const { hasLooks } = require('../text/no-looks');
+
+/** The uses that count: the live origin and use citations. An example is an illustration, not a use. */
+function countUses(citations) {
+ const live = (citations || []).filter((c) => c.source_status === 'live' && (c.role === 'origin' || c.role === 'use'));
+ const days = new Set(live.map((c) => utcDay(c.observed_at)).filter(Boolean)).size;
+ const channels = new Set(live.map((c) => c.source_service).filter(Boolean)).size;
+ const hasOrigin = live.some((c) => c.role === 'origin');
+ return { uses: live.length, days, channels, hasOrigin };
+}
+
+function evaluate({ entry = null, citations = [], judgement = null, gate = {} } = {}) {
+ const minScore = Number.isFinite(Number(gate.minScore)) ? Number(gate.minScore) : 0.75;
+ const minUses = Math.max(1, Number(gate.minUses) || 3);
+ const minDays = Math.max(1, Number(gate.minDays) || 2);
+ const minChannels = Math.max(1, Number(gate.minChannels) || 2);
+
+ const { uses, days, channels, hasOrigin } = countUses(citations);
+ const reasons = [];
+
+ if (!judgement) reasons.push('not judged yet');
+ else {
+ const score = Number(judgement.score);
+ if (!Number.isFinite(score)) reasons.push('no score');
+ else if (score < minScore) reasons.push(`scored ${score.toFixed(2)}, under ${minScore.toFixed(2)}`);
+ if (judgement.denied) reasons.push('a denied source');
+ if (judgement.looks) reasons.push('the text names looks');
+ if (!judgement.title) reasons.push('no title');
+ if (!judgement.summary) reasons.push('no summary');
+ if (!judgement.origin) reasons.push('no origin');
+ if (!judgement.spread) reasons.push('no spread');
+ if (!Array.isArray(judgement.examples) || judgement.examples.length < 2) reasons.push('fewer than two examples');
+ }
+ if (entry && entry.title && hasLooks(entry.title)) reasons.push('the title names looks');
+ if (uses < minUses) reasons.push(`used ${uses} time${uses === 1 ? '' : 's'}, needs ${minUses}`);
+ if (days < minDays && channels < minChannels) {
+ reasons.push(`used on ${days} day${days === 1 ? '' : 's'} in ${channels} channel${channels === 1 ? '' : 's'}, needs ${minDays} days or ${minChannels} channels`);
+ }
+ if (!hasOrigin) reasons.push('no live origin citation');
+
+ return {
+ publish: reasons.length === 0,
+ reasons,
+ uses,
+ days,
+ channels,
+ score: judgement ? Number(judgement.score) : null,
+ minScore,
+ };
+}
+
+/**
+ * The encyclopedia's rule. A classic has no uses to count: it is published when the AI scored it as a real, widely
+ * used meme and its own text is clean (no looks, no denied word, nothing denied in what it cites).
+ */
+function evaluateClassic({ judgement = null, minScore = 0.75, entry = null } = {}) {
+ const reasons = [];
+ if (!judgement) reasons.push('not judged yet');
+ else {
+ const score = Number(judgement.score);
+ if (!Number.isFinite(score)) reasons.push('no score');
+ else if (score < minScore) reasons.push(`scored ${score.toFixed(2)}, under ${minScore.toFixed(2)}`);
+ if (judgement.denied) reasons.push('a denied source');
+ if (judgement.looks) reasons.push('the text names looks');
+ if (!judgement.title) reasons.push('no title');
+ if (!judgement.summary) reasons.push('no summary');
+ if (!judgement.origin) reasons.push('no origin');
+ if (!judgement.spread) reasons.push('no spread');
+ if (!Array.isArray(judgement.examples) || judgement.examples.length < 2) reasons.push('fewer than two examples');
+ }
+ if (entry && entry.title && hasLooks(entry.title)) reasons.push('the title names looks');
+ return {
+ publish: reasons.length === 0,
+ reasons,
+ uses: 0,
+ days: 0,
+ channels: 0,
+ score: judgement ? Number(judgement.score) : null,
+ minScore,
+ };
+}
+
+module.exports = { evaluate, evaluateClassic, countUses };
new file mode 100644
new file mode 100644
@@ -0,0 +1,183 @@
@@ -0,0 +1,183 @@
+'use strict';
+
+/**
+ * The judge. One AI run per candidate: it scores the candidate from 0 to 1 on whether it is a real meme, catchphrase
+ * or inside joke rather than everyday words that happen to recur, and it writes the entry from the same cited
+ * material: a title in natural case, a short summary, where it started, how people use it, and two or three example
+ * uses.
+ *
+ * Two rules hold whatever the model says:
+ * - the text never names looks (openvibe-sdk/no-looks): a title that names them is not used, and every other
+ * field is repaired word by word; an entry whose text named them is never published;
+ * - the text never carries a word from the deny list (server/deny-list.js).
+ *
+ * The score and the reason are stored with the entry, so a staff person sees why a candidate was kept and the
+ * history says what the AI decided.
+ */
+const { noLooks, hasLooks } = require('../text/no-looks');
+const { isDenied, hasDeniedWord } = require('../deny-list');
+
+const JUDGE_SYSTEM = [
+ 'You are the editor of OpenVibe.Meme, an open encyclopedia of memes: the running jokes of a streaming network and the memes of the internet.',
+ 'You judge one candidate at a time and you use only the cited material in front of you. If the citations do not show it, it is not there.',
+ 'You answer with one JSON object and nothing else.',
+].join('\n');
+
+const JUDGE_SCHEMA = Object.freeze({
+ score: 'number from 0 to 1',
+ reason: 'one or two sentences saying why you gave that score',
+ title: 'the name people would actually say, in natural case',
+ summary: 'one or two sentences on what it is',
+ origin: 'where it started, from the cited uses',
+ spread: 'how people use it: how it is deployed and who says it',
+ examples: 'two or three short example uses, written the way people would say them',
+});
+
+const SCORE_LINE = [
+ 'Score it from 0 to 1:',
+ ' 1.00 a clear, shared joke people play off on purpose',
+ ' 0.75 the bar: recognisably a catchphrase, bit or meme with a shared meaning',
+ ' 0.50 maybe, but it could just be ordinary speech that happens to recur',
+ ' 0.00 everyday words; nothing shared, nothing to play off',
+].join('\n');
+
+const RULES = [
+ 'Rules:',
+ '- Write only what the cited material shows. Never add a fact from outside it, and never invent a person.',
+ '- Never describe anyone\'s body, clothing, hair, face or age, and never guess someone\'s gender.',
+ '- Keep the title short, in natural case ("Cozmo\'s alive", not "COZMO S ALIVE").',
+ '- If the material does not support a field, leave it an empty string rather than guessing.',
+].join('\n');
+
+const maxLen = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);
+const clip = (s, n) => String(s == null ? '' : s).trim().slice(0, n);
+
+/** The score the model gave, kept only when it is a number between 0 and 1. */
+function readScore(value) {
+ const n = Number(value);
+ if (!Number.isFinite(n)) return null;
+ if (n < 0) return 0;
+ if (n > 1) return 1;
+ return n;
+}
+
+/** The examples, as a list of short lines. */
+function readExamples(value) {
+ const list = Array.isArray(value) ? value : (typeof value === 'string' ? value.split('\n') : []);
+ const out = [];
+ for (const item of list) {
+ const line = maxLen(item, 160);
+ if (line) out.push(line);
+ if (out.length >= 5) break;
+ }
+ return out;
+}
+
+/**
+ * Clean the AI's own words: no looks, no denied word.
+ *
+ * A title that names looks is not used at all — it is not repaired word by word, it is dropped, and the entry keeps
+ * the name it already had (the phrase it was found under, or the name the source gives it). Every other field is
+ * repaired word by word; a field that still names looks after the repair is kept but marked, and an entry with such
+ * a field is never published.
+ */
+function cleanFields({ title, summary, origin, spread, examples }) {
+ const t = String(title == null ? '' : title).replace(/\s+/g, ' ').trim();
+ const looksTitle = hasLooks(t);
+ const fields = {
+ title: looksTitle ? '' : maxLen(t, 160),
+ summary: maxLen(noLooks(summary), 600),
+ origin: maxLen(noLooks(origin), 600),
+ spread: maxLen(noLooks(spread), 600),
+ examples: (examples || []).map((e) => maxLen(noLooks(e), 200)).filter(Boolean),
+ };
+ const looks = looksTitle
+ || hasLooks(fields.summary) || hasLooks(fields.origin) || hasLooks(fields.spread)
+ || fields.examples.some((e) => hasLooks(e));
+ const denied = hasDeniedWord(`${fields.title} ${fields.summary} ${fields.origin} ${fields.spread} ${fields.examples.join(' ')}`);
+ return { ...fields, looks, denied };
+}
+
+/**
+ * Judge one candidate or one classic. `kind` is 'candidate' (an OpenVibe running joke, cited to its own moments)
+ * or 'classic' (an internet meme, cited to Wikipedia and Wikidata).
+ */
+function createJudge({ config, ai, context, log = console }) {
+ /**
+ * @param {object} subject { title, kind, phrase }
+ * @param {Array} prepared the citations with their context, from server/ai/context.js
+ */
+ async function judge(subject, prepared) {
+ if (!ai || !ai.enabled) throw new Error('OpenVibe.AI is not configured for OpenVibe.Meme.');
+ const block = context.render(prepared);
+ const isClassic = subject.kind === 'classic';
+ const what = isClassic
+ ? `Write the encyclopedia entry for the internet meme "${subject.title}".`
+ : `Judge this candidate phrase from OpenVibe: "${subject.title}".`;
+ const ask = isClassic
+ ? [
+ what,
+ '',
+ 'Score it from 0 to 1 on how much it is a real, widely used meme rather than a one-off video, a product or a person:',
+ ' 1.00 a well-known meme with a shared meaning',
+ ' 0.75 the bar: a recognisable, reused meme',
+ ' 0.50 maybe, but it is niche or long forgotten',
+ ' 0.00 not a meme',
+ '',
+ 'Then write the entry from the cited sources only: the title in natural case, a short summary, where it started, how it spread and how people use it, and two or three example uses.',
+ '',
+ 'Cited sources:',
+ block,
+ '',
+ RULES,
+ ].join('\n')
+ : [
+ what,
+ '',
+ 'Is it a real meme, catchphrase or inside joke — something people play off on purpose, with a shared meaning — or is it just everyday words that happen to recur?',
+ '',
+ SCORE_LINE,
+ '',
+ 'Then write the entry from the cited uses only: the title in natural case, a short summary, where it started, how people use it, and two or three example uses.',
+ '',
+ 'Cited uses:',
+ block,
+ '',
+ RULES,
+ ].join('\n');
+
+ const out = await ai.json({
+ prompt: ask,
+ system: JUDGE_SYSTEM,
+ schema: JUDGE_SCHEMA,
+ temperature: 0.2,
+ target: { service: 'meme', type: 'judgement', id: `judge_${String(subject.id || subject.title).slice(0, 100)}` },
+ });
+
+ const data = out.data || {};
+ const score = readScore(data.score);
+ if (score == null) throw new Error('the AI run did not give a score');
+ const cleaned = cleanFields({
+ title: data.title,
+ summary: data.summary,
+ origin: data.origin,
+ spread: data.spread,
+ examples: readExamples(data.examples),
+ });
+ const deniedCitation = (prepared || []).some(({ citation }) => isDenied(citation));
+ return {
+ score,
+ reason: maxLen(data.reason, 600) || 'The AI gave no reason.',
+ ...cleaned,
+ denied: cleaned.denied || deniedCitation,
+ looks: cleaned.looks,
+ model: out.model,
+ provider: out.provider,
+ run_id: out.run_id,
+ };
+ }
+
+ return { judge, cleanFields, readScore, readExamples, enabled: Boolean(ai && ai.enabled), JUDGE_SCHEMA, JUDGE_SYSTEM };
+}
+
+module.exports = { createJudge, cleanFields, readScore, readExamples, JUDGE_SCHEMA, JUDGE_SYSTEM };
@@ -38,6 +38,10 @@ const { createSearchIndex } = require('./search');
@@ -38,6 +38,10 @@ const { createSearchIndex } = require('./search');
const { createUpstreams } = require('./upstreams');
const { createUpstreams } = require('./upstreams');
const { createImporter } = require('./importer');
const { createImporter } = require('./importer');
const { createReverifier } = require('./reverify');
const { createReverifier } = require('./reverify');
+const { createMemeAi } = require('./ai/client');
+const { createContext } = require('./ai/context');
+const { createJudge } = require('./ai/judge');
+const { createJudgeJob } = require('./jobs/judge');
const { createNetworkSender } = require('openvibe-sdk/account-data');
const { createNetworkSender } = require('openvibe-sdk/account-data');
const PUBLIC_DIR = path.join(__dirname, '..', 'public');
const PUBLIC_DIR = path.join(__dirname, '..', 'public');
@@ -64,6 +68,12 @@ async function createApp(opts = {}) {
@@ -64,6 +68,12 @@ async function createApp(opts = {}) {
const upstreams = createUpstreams({ config, fetchImpl, log });
const upstreams = createUpstreams({ config, fetchImpl, log });
const importer = createImporter({ config, s, entries, upstreams, fetchImpl, log });
const importer = createImporter({ config, s, entries, upstreams, fetchImpl, log });
const reverifier = createReverifier({ config, s, entries, upstreams, log });
const reverifier = createReverifier({ config, s, entries, upstreams, log });
+ // The judge: OpenVibe.AI scores each candidate from its own citations and writes the entry; the publish rule
+ // decides what becomes public.
+ const ai = createMemeAi({ config, fetchImpl, log });
+ const context = createContext({ config, upstreams, log });
+ const judge = createJudge({ config, ai, context, log });
+ const judgeJob = createJudgeJob({ config, s, entries, judge, context, log });
// Account export and deletion (ADR-033). The sender posts to Network's internal routes with this service's own
// Account export and deletion (ADR-033). The sender posts to Network's internal routes with this service's own
// client-credentials token; a test injects a stand-in through opts.accountSend.
// client-credentials token; a test injects a stand-in through opts.accountSend.
@@ -74,6 +84,7 @@ async function createApp(opts = {}) {
@@ -74,6 +84,7 @@ async function createApp(opts = {}) {
const ctx = {
const ctx = {
config, s, keys, sso, principal, search, entries, upstreams, importer, reverifier, accountData, accountSend, log,
config, s, keys, sso, principal, search, entries, upstreams, importer, reverifier, accountData, accountSend, log,
+ ai, context, judge, judgeJob,
};
};
const app = express();
const app = express();
new file mode 100644
new file mode 100644
@@ -0,0 +1,265 @@
@@ -0,0 +1,265 @@
+'use strict';
+
+/**
+ * The encyclopedia's seed. It reads the well-known internet memes from the open lists (Wikipedia's lists of Internet
+ * phenomena and memes, CC BY-SA) and from Wikidata (CC0), writes each one with the same AI pass the OpenVibe
+ * candidates use, and keeps every entry's source, picture licence and attribution.
+ *
+ * One entry per meme:
+ * kind 'classic', status 'classic', the AI's title and summary, where it started and how it spread, two or three
+ * example uses, and the citations it was written from: the Wikipedia article (the origin), the list's own words
+ * about it, the Wikidata facts, and the Commons picture under its own licence.
+ *
+ * The same publish rule as the candidates: the AI scored it 0.75 or more on "is this a real, widely used meme",
+ * the text names no looks and no denied word, and the entry is complete. What does not clear the bar stays hidden
+ * for staff, with its score — the list holds videos, products and people too, and not all of them are memes.
+ */
+const { isExplicit } = require('../text/no-looks');
+const { hasDeniedWord } = require('../deny-list');
+const { evaluateClassic } = require('../ai/gate');
+const { slugify } = require('../store');
+
+/** The attribution every classic carries: Wikipedia's and Wikidata's licences. */
+const ATTRIBUTION = 'Text from Wikipedia (CC BY-SA 4.0); facts from Wikidata (CC0).';
+
+/** A Wikidata date: "+1997-00-00T00:00:00Z" (a year) or "1997-05-01T00:00:00Z". */
+function toIsoDate(value) {
+ const s = String(value == null ? '' : value).replace(/^\+/, '');
+ const m = /^(\d{4})(?:-(\d{2}))?(?:-(\d{2}))?/.exec(s);
+ if (!m) return null;
+ const year = Number(m[1]);
+ const month = m[2] && m[2] !== '00' ? Number(m[2]) : 1;
+ const day = m[3] && m[3] !== '00' ? Number(m[3]) : 1;
+ if (year < 1900 || year > 2100) return null;
+ const d = new Date(Date.UTC(year, month - 1, day));
+ return Number.isNaN(d.getTime()) ? null : d.toISOString();
+}
+
+/** The year a description mentions first, when Wikidata has no date: "…in 2013…" */
+function yearFrom(text) {
+ const m = /\b(19[5-9]\d|20[0-2]\d)\b/.exec(String(text || ''));
+ if (!m) return null;
+ return `${m[1]}-01-01T00:00:00.000Z`;
+}
+
+/** A citation for the open web. It is stored like any other: a public URL and the sha256 of the quoted text. */
+function webCitation({ source_type, source_id, quote_text, url, observed_at, role = 'use' }) {
+ return {
+ role,
+ source_service: 'web',
+ source_type,
+ source_id: String(source_id).slice(0, 200),
+ quote_text: String(quote_text || '').trim(),
+ url: String(url),
+ observed_at: observed_at || new Date().toISOString(),
+ source_status: 'live',
+ };
+}
+
+function createClassicsJob({ config, s, entries, sources, judge, log = console }) {
+ const stats = { runs: 0, found: 0, seeded: 0, published: 0, kept: 0, skipped: 0, errors: 0, last_run_at: null, last_error: null };
+ const opts = config.encyclopedia || {};
+
+ /** The lists: Wikipedia's list of Internet phenomena and the category behind it. */
+ async function collect() {
+ const found = [];
+ for (const title of opts.lists || []) {
+ try {
+ const entriesOfList = await sources.listEntries(title);
+ for (const e of entriesOfList) found.push({ ...e, list: title });
+ } catch (err) {
+ log.warn(`[OpenVibe.Meme] classics: list ${title}: ${err.message}`);
+ }
+ }
+ for (const category of opts.categories || []) {
+ try {
+ for (const e of await sources.categoryMembers(category)) found.push({ ...e, list: `Category:${category}` });
+ } catch (err) {
+ log.warn(`[OpenVibe.Meme] classics: category ${category}: ${err.message}`);
+ }
+ }
+ // One entry per name: the first one wins.
+ const seen = new Set();
+ const out = [];
+ for (const e of found) {
+ const key = e.title.trim().toLowerCase();
+ if (!key || seen.has(key)) continue;
+ seen.add(key);
+ out.push(e);
+ }
+ return out.slice(0, Math.max(1, Number(opts.limit) || 400));
+ }
+
+ /**
+ * The citations of one classic, and the facts behind them. Returns null when the sources are too thin to write
+ * an entry from, or when the text is not something this site publishes.
+ */
+ async function buildCandidate(item, { qids, facts, leads, images }) {
+ const title = item.title;
+ const article = item.target && item.target.includes('/') ? null : item.target;
+ const resolved = article ? qids.get(article) : null;
+ const fact = resolved ? facts.get(resolved.qid) : null;
+ const lead = article ? (leads.get(article) || '') : '';
+
+ // The list's own words, and the article's lead: the two texts the entry is written from.
+ const listText = item.description || '';
+ if (!listText && !lead) return null;
+
+ // Nothing explicit and nothing from the deny list ever enters the database, in a quote either.
+ if (isExplicit(listText, lead)) return null;
+ if (hasDeniedWord(`${title} ${listText} ${lead}`)) return null;
+
+ const now = new Date(s.now()).toISOString();
+ const inception = (fact && toIsoDate(fact.inception)) || yearFrom(listText) || yearFrom(lead) || now;
+ const cites = [];
+
+ if (lead) {
+ cites.push(webCitation({
+ source_type: 'wikipedia',
+ source_id: article || title,
+ quote_text: lead.slice(0, 4000),
+ url: `https://en.wikipedia.org/wiki/${encodeURIComponent(String(article || title).replace(/ /g, '_'))}`,
+ observed_at: inception,
+ role: 'origin',
+ }));
+ }
+ if (listText) {
+ cites.push(webCitation({
+ source_type: 'wikipedia',
+ source_id: `${item.list || 'Wikipedia list'}#${title}`,
+ quote_text: listText.slice(0, 4000),
+ url: `https://en.wikipedia.org/wiki/${encodeURIComponent(String(item.list || 'List_of_Internet_phenomena').replace(/ /g, '_'))}`,
+ observed_at: inception,
+ role: 'use',
+ }));
+ }
+ if (resolved && fact) {
+ const when = fact.inception ? `First recorded: ${fact.inception}.` : '';
+ cites.push(webCitation({
+ source_type: 'wikidata',
+ source_id: resolved.qid,
+ quote_text: `Wikidata ${resolved.qid}: ${fact.label || title}${fact.description ? ` — ${fact.description}` : ''}. ${when}`.trim(),
+ url: `https://www.wikidata.org/wiki/${resolved.qid}`,
+ observed_at: inception,
+ role: 'use',
+ }));
+ }
+ let image = null;
+ if (fact && fact.image && images.has(fact.image)) {
+ image = images.get(fact.image);
+ cites.push(webCitation({
+ source_type: 'commons',
+ source_id: fact.image,
+ quote_text: `${image.license}${image.license_url ? ` (${image.license_url})` : ''}. ${image.credit}.`,
+ url: image.page_url,
+ observed_at: now,
+ role: 'example',
+ }));
+ }
+ if (!cites.length) return null;
+ return { item, cites, inception, image, qid: resolved ? resolved.qid : null, title, lead, listText };
+ }
+
+ /**
+ * Seed the encyclopedia. `limit` caps how many entries this run writes; `publish` is false for a dry run.
+ * Returns the counts: found, seeded, published, kept hidden, skipped.
+ */
+ async function runOnce({ limit = null, publish = true } = {}) {
+ const runAt = new Date(s.now()).toISOString();
+ const errors = [];
+ if (!judge || !judge.enabled) {
+ const message = 'OpenVibe.AI is not configured: nothing was written.';
+ stats.last_error = message;
+ return { ok: false, seeded: 0, published: 0, kept: 0, skipped: 0, error: message };
+ }
+ const found = await collect();
+ stats.found = found.length;
+ const max = Math.min(1000, Math.max(1, Number(limit) || opts.limit || 400));
+
+ // The facts for every name at once: a Wikipedia title → Q id, the entities, the leads, the pictures.
+ const articles = [...new Set(found.map((e) => e.target).filter((t) => t && !t.includes('/')))];
+ const qids = await sources.resolveQids(articles);
+ const facts = await sources.entities([...qids.values()].map((v) => v.qid));
+ const leads = await sources.leadExtract(articles);
+ const images = await sources.imageInfo([...facts.values()].map((f) => f.image).filter(Boolean));
+
+ let seeded = 0;
+ let published = 0;
+ let kept = 0;
+ let skipped = 0;
+ for (const item of found) {
+ if (seeded >= max) break;
+ try {
+ const built = await buildCandidate(item, { qids, facts, leads, images });
+ if (!built) { skipped++; continue; }
+ // One entry per meme: a Wikidata id already seeded is left alone.
+ if (built.qid) {
+ const existing = await s.db.maybe('SELECT id FROM meme_entries WHERE external_id = $1', [built.qid]);
+ if (existing) { skipped++; continue; }
+ }
+ const slug = slugify(built.item.title);
+ const existing = await entries.bySlug(slug);
+ if (existing) { skipped++; continue; }
+
+ // The AI writes the entry from the cited text only.
+ const prepared = built.cites.map((citation) => ({ citation, context: [] }));
+ const verdict = await judge.judge({ id: built.title, title: built.title, kind: 'classic' }, prepared);
+ const title = verdict.title || built.title;
+ const decision = evaluateClassic({ judgement: verdict, minScore: opts.minScore || 0.75, entry: { title } });
+ const summary = verdict.summary || built.listText.slice(0, 300);
+ const created = await entries.createEntry({
+ slug,
+ title,
+ kind: 'classic',
+ summary,
+ status: 'classic',
+ first_seen_at: built.inception,
+ last_seen_at: new Date(s.now()).toISOString(),
+ visibility: publish && decision.publish ? 'public' : 'hidden',
+ citations: built.cites,
+ image_url: built.image ? built.image.url : null,
+ image_credit: built.image ? built.image.credit : null,
+ image_license: built.image ? built.image.license : null,
+ image_license_url: built.image ? built.image.license_url : null,
+ image_source_url: built.image ? built.image.page_url : null,
+ attribution: ATTRIBUTION,
+ external_id: built.qid,
+ });
+ await entries.saveJudgement(created.entry.id, verdict);
+ await entries.logDecision({
+ entryId: created.entry.id,
+ decision: decision.publish ? 'publish' : 'keep_hidden',
+ fromStatus: 'rising',
+ toStatus: 'classic',
+ fromVisibility: 'hidden',
+ toVisibility: created.entry.visibility,
+ score: decision.score,
+ reason: decision.publish
+ ? `the AI scored it ${Number(decision.score).toFixed(2)}: ${verdict.reason || ''}`.slice(0, 500)
+ : `kept hidden: ${decision.reasons.join('; ')}`.slice(0, 500),
+ });
+ seeded++;
+ if (decision.publish) published++;
+ else kept++;
+ } catch (err) {
+ errors.push(`${item.title}: ${err.message}`);
+ log.warn(`[OpenVibe.Meme] classics ${item.title} failed: ${err.message}`);
+ }
+ }
+ stats.runs++;
+ stats.seeded += seeded;
+ stats.published += published;
+ stats.kept += kept;
+ stats.skipped += skipped;
+ stats.errors += errors.length;
+ stats.last_run_at = runAt;
+ stats.last_error = errors[0] || null;
+ const out = { ok: errors.length === 0, found: found.length, seeded, published, kept, skipped, errors };
+ log.log(`[OpenVibe.Meme] classics: ${found.length} found, ${seeded} seeded, ${published} published, ${kept} kept hidden, ${skipped} skipped${errors.length ? `, ${errors.length} errors` : ''}`);
+ return out;
+ }
+
+ return { runOnce, collect, buildCandidate, stats: () => ({ ...stats }) };
+}
+
+module.exports = { createClassicsJob, toIsoDate, yearFrom, webCitation, ATTRIBUTION };
new file mode 100644
new file mode 100644
@@ -0,0 +1,314 @@
@@ -0,0 +1,314 @@
+'use strict';
+
+/**
+ * The open sources of the encyclopedia: Wikipedia (CC BY-SA), Wikidata (CC0) and Wikimedia Commons (each file under
+ * its own licence). Nothing is scraped from a site whose terms forbid it; these three have APIs that say what may
+ * be used and how it must be credited.
+ *
+ * listEntries(title) a Wikipedia list read into entries: the name, the section and the list's own words
+ * categoryMembers(cat) the pages of a Wikipedia category (the second list of well-known memes)
+ * resolveQids(titles) a Wikipedia title → its Wikidata Q id
+ * entities(qids) Wikidata: the inception date, the picture, the label and the description
+ * leadExtract(titles) a Wikipedia article's lead, in plain words
+ * imageInfo(files) a Commons file: the picture, its licence and its credit
+ *
+ * Every request names the service (a User-Agent with a contact address) and is paced per host, as the Wikimedia
+ * API policy asks.
+ */
+const { setTimeout: sleep } = require('node:timers/promises');
+
+class SourceError extends Error {
+ constructor(message, { code = 'meme.source', status = 502 } = {}) {
+ super(message);
+ this.name = 'SourceError';
+ this.code = code;
+ this.status = status;
+ }
+}
+
+/** Templates that carry no words of their own. */
+const EMPTY_TEMPLATES = new Set(['snd', 'lang', 'nowrap', 'small', 'sic', 'efn', 'citation', 'cite web', 'cite book',
+ 'cite magazine', 'cite news', 'cite journal', 'ill', 'nobold', 'not italic', 'abbr', 'portal', 'anchor', 'vanchor',
+ 'clr', 'color', 'center', 'blockquote', 'div col', 'div col end', 'flatlist', 'hlist', 'columns-list', 'ref']);
+
+const chunk = (list, n) => {
+ const out = [];
+ for (let i = 0; i < list.length; i += n) out.push(list.slice(i, i + n));
+ return out;
+};
+
+/** The words of a wiki text, without the markup: links keep their display text, templates keep their content. */
+function plainText(text) {
+ let s = String(text == null ? '' : text);
+ s = s.replace(/<!--[\s\S]*?-->/g, ' ');
+ s = s.replace(/<ref[^>]*\/>/gi, ' ');
+ s = s.replace(/<ref[^>]*>[\s\S]*?<\/ref>/gi, ' ');
+ s = s.replace(/<[^>]+>/g, ' ');
+ // Links: [[Target|Display]] → Display, [[Target]] → Target.
+ for (let i = 0; i < 6; i++) {
+ const before = s;
+ s = s.replace(/\[\[([^\]|]+)\|([^\]]*)\]\]/g, '$2');
+ s = s.replace(/\[\[([^\]]*)\]\]/g, '$1');
+ if (s === before) break;
+ }
+ // Templates: an empty one goes, another keeps its content.
+ for (let i = 0; i < 8; i++) {
+ const before = s;
+ s = s.replace(/\{\{([^{}|]*)(?:\|([^{}]*))?\}\}/g, (m, name, arg) => {
+ const key = String(name).trim().toLowerCase();
+ if (EMPTY_TEMPLATES.has(key)) return ' ';
+ return ` ${arg == null ? '' : arg} `;
+ });
+ if (s === before) break;
+ }
+ s = s.replace(/'''''|'''|''/g, '');
+ s = s.replace(/ /g, ' ').replace(/&/g, '&').replace(/"/g, '"').replace(/'/g, "'").replace(/</g, '<').replace(/>/g, '>');
+ return s.replace(/\s+/g, ' ').trim();
+}
+
+/** The name a list entry carries: the link's display text, or its target without the section. */
+function entryTitle(raw) {
+ const link = /\[\[([^\]|]+)(?:\|([^\]]*))?\]\]/.exec(raw);
+ if (link) {
+ const display = (link[2] || '').replace(/'''/g, '').trim();
+ if (display) return display;
+ return String(link[1]).split('#')[0].replace(/_/g, ' ').trim();
+ }
+ const quoted = /^["“”'](.+?)["“”']/.exec(String(raw).trim());
+ if (quoted) return quoted[1].trim();
+ return '';
+}
+
+/**
+ * One Wikipedia list, read into entries. Each entry is { title, target, section, description } where the
+ * description is the list's own words about that entry.
+ */
+function parseList(wikitext) {
+ const out = [];
+ let section = '';
+ for (const line of String(wikitext || '').split('\n')) {
+ const heading = /^(={2,6})\s*(.+?)\s*\1\s*$/.exec(line);
+ if (heading) { section = plainText(heading[2]); continue; }
+ if (!/^\*+\s*\S/.test(line)) continue;
+ if (/^\*\*+/.test(line)) continue; // a sub-bullet is a note, not an entry
+ if (/^(See also|References|Notes|External links)$/i.test(section)) continue;
+ const body = line.replace(/^\*+\s*/, '');
+ const raw = body.split('{{snd}}')[0];
+ const title = entryTitle(raw);
+ if (!title || title.length > 120) continue;
+ const target = (/\[\[([^\]|]+)(?:\|[^\]]*)?\]\]/.exec(raw) || [])[1] || title;
+ const description = plainText(body.split('{{snd}}').slice(1).join('{{snd}}') || body.slice(raw.length));
+ out.push({ title, target: String(target).replace(/_/g, ' '), section, description });
+ }
+ return out;
+}
+
+/** True when a licence lets us use the picture: public domain, CC0, or a Creative Commons licence. */
+function isFreeLicense(name) {
+ const s = String(name || '').trim().toLowerCase();
+ if (!s) return false;
+ if (/^(cc0|public domain|pd|no restrictions|cc pd)/.test(s)) return true;
+ if (/^cc[ -]?(by|by-sa|by-nc|by-nd)[ -]?\d/.test(s)) return true;
+ if (s === 'cc by') return true;
+ if (s.startsWith('cc by')) return true;
+ return false;
+}
+
+/** The Commons metadata, without its HTML. */
+function cleanMeta(value) {
+ if (!value) return '';
+ return plainText(String(value).replace(/<[^>]+>/g, ' ')).slice(0, 300);
+}
+
+function createSources({ config, fetchImpl = globalThis.fetch, log = console }) {
+ const opts = config.encyclopedia || {};
+ const userAgent = opts.userAgent || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])';
+ const timeoutMs = Math.max(1000, Number(opts.timeoutMs) || 20_000);
+ const lastCall = new Map();
+
+ /** One request per host, at least minIntervalMs apart, with a retry on 429 and 5xx. */
+ async function call(url, { label = 'Wikipedia', retries = 2 } = {}) {
+ const host = new URL(url).host;
+ const wait = Math.max(0, (lastCall.get(host) || 0) + (Number(opts.minIntervalMs) || 0) - Date.now());
+ if (wait) await sleep(wait);
+ lastCall.set(host, Date.now());
+ let lastErr = null;
+ for (let attempt = 0; attempt <= retries; attempt++) {
+ let res;
+ try {
+ res = await fetchImpl(url, {
+ headers: { 'User-Agent': userAgent, Accept: 'application/json' },
+ signal: AbortSignal.timeout(timeoutMs),
+ redirect: 'follow',
+ });
+ } catch (err) {
+ lastErr = new SourceError(`${label}: ${err && err.message}`, { code: 'meme.source.unreachable' });
+ await sleep(500 * (attempt + 1));
+ continue;
+ }
+ if (res.ok) {
+ const text = await res.text();
+ try { return JSON.parse(text); } catch { throw new SourceError(`${label} answered something that is not JSON`, { code: 'meme.source.body' }); }
+ }
+ if (res.status === 404) return { notFound: true };
+ lastErr = new SourceError(`${label} answered ${res.status}`, { status: res.status >= 500 ? 502 : res.status });
+ if (res.status === 429 || res.status >= 500) {
+ await sleep(800 * (attempt + 1));
+ continue;
+ }
+ throw lastErr;
+ }
+ throw lastErr || new SourceError(`${label} did not answer`);
+ }
+
+ const api = (base, params) => `${String(base).replace(/\/+$/, '')}/w/api.php?${new URLSearchParams({ format: 'json', formatversion: '2', ...params })}`;
+
+ /** One Wikipedia list, read into entries. */
+ async function listEntries(title) {
+ const data = await call(api(opts.wikipedia, {
+ action: 'parse', page: String(title || '').replace(/ /g, '_'), prop: 'wikitext', redirects: '1',
+ }), { label: `Wikipedia ${title}` });
+ if (!data || data.notFound || !data.parse) return [];
+ return parseList(data.parse.wikitext);
+ }
+
+ /** The pages of a Wikipedia category: the second list of well-known memes. */
+ async function categoryMembers(category) {
+ const out = [];
+ let cont = null;
+ for (let page = 0; page < 8; page++) {
+ const data = await call(api(opts.wikipedia, {
+ action: 'query', list: 'categorymembers', cmtitle: `Category:${String(category).replace(/ /g, '_')}`,
+ cmlimit: '500', cmnamespace: '0', ...(cont ? { cmcontinue: cont } : {}),
+ }), { label: `Wikipedia category ${category}` });
+ if (!data || data.notFound) break;
+ for (const m of data.query && data.query.categorymembers ? data.query.categorymembers : []) {
+ if (m && m.title) out.push({ title: m.title, target: m.title, section: 'Category', description: '' });
+ }
+ cont = data.continue && data.continue.cmcontinue ? data.continue.cmcontinue : null;
+ if (!cont) break;
+ }
+ return out;
+ }
+
+ /** Wikipedia titles → their Wikidata Q id (redirects followed). */
+ async function resolveQids(titles) {
+ const wanted = [...new Set((titles || []).map((t) => String(t || '').trim()).filter(Boolean))];
+ const out = new Map();
+ for (const group of chunk(wanted, 40)) {
+ const data = await call(api(opts.wikipedia, {
+ action: 'query', redirects: '1', prop: 'pageprops', ppprop: 'wikibase_item', titles: group.join('|'),
+ }), { label: 'Wikipedia' });
+ if (!data || data.notFound) continue;
+ const pages = (data.query && data.query.pages) || [];
+ const redirects = (data.query && data.query.redirects) || [];
+ const redirectTo = new Map(redirects.map((r) => [String(r.from), String(r.to)]));
+ const byTitle = new Map(pages.map((p) => [String(p.title), p]));
+ for (const title of group) {
+ const landed = redirectTo.get(title) || title;
+ const page = byTitle.get(landed) || byTitle.get(title);
+ const qid = page && page.pageprops && page.pageprops.wikibase_item;
+ if (qid && /^Q[1-9][0-9]{0,11}$/.test(qid)) out.set(title, { qid, title: page.title });
+ }
+ }
+ return out;
+ }
+
+ /** Q ids → the facts the encyclopedia needs: when it started, its picture, its label. */
+ async function entities(qids) {
+ const wanted = [...new Set((qids || []).filter((q) => /^Q[1-9][0-9]{0,11}$/.test(String(q))))];
+ const out = new Map();
+ for (const group of chunk(wanted, 40)) {
+ const data = await call(api(opts.wikidata, {
+ action: 'wbgetentities', ids: group.join('|'), props: 'claims|labels|descriptions|sitelinks', languages: 'en',
+ }), { label: 'Wikidata' });
+ if (!data || data.notFound) continue;
+ const all = data.entities || {};
+ for (const qid of group) {
+ const e = all[qid];
+ if (!e) continue;
+ const claims = e.claims || {};
+ const first = (prop) => {
+ const list = claims[prop];
+ if (!Array.isArray(list) || !list.length) return null;
+ const v = list[0] && list[0].mainsnak && list[0].mainsnak.datavalue && list[0].mainsnak.datavalue.value;
+ return v == null ? null : v;
+ };
+ const inception = first('P571');
+ const image = first('P18');
+ out.set(qid, {
+ qid,
+ label: (e.labels && e.labels.en && e.labels.en.value) || null,
+ description: (e.descriptions && e.descriptions.en && e.descriptions.en.value) || null,
+ inception: inception && inception.time ? String(inception.time).replace(/^\+/, '') : null,
+ image: typeof image === 'string' ? image : null,
+ article: (e.sitelinks && e.sitelinks.enwiki && e.sitelinks.enwiki.title) || null,
+ });
+ }
+ }
+ return out;
+ }
+
+ /** Wikipedia articles → their lead, in plain words. */
+ async function leadExtract(titles) {
+ const wanted = [...new Set((titles || []).map((t) => String(t || '').trim()).filter(Boolean))];
+ const out = new Map();
+ for (const group of chunk(wanted, 20)) {
+ const data = await call(api(opts.wikipedia, {
+ action: 'query', prop: 'extracts', exintro: '1', explaintext: '1', redirects: '1',
+ exlimit: 'max', titles: group.join('|'),
+ }), { label: 'Wikipedia' });
+ if (!data || data.notFound) continue;
+ const pages = (data.query && data.query.pages) || [];
+ const redirects = (data.query && data.query.redirects) || [];
+ const redirectTo = new Map(redirects.map((r) => [String(r.from), String(r.to)]));
+ const byTitle = new Map(pages.map((p) => [String(p.title), p]));
+ for (const title of group) {
+ const landed = redirectTo.get(title) || title;
+ const page = byTitle.get(landed) || byTitle.get(title);
+ const extract = page && typeof page.extract === 'string' ? page.extract.trim() : '';
+ if (extract) out.set(title, extract.slice(0, 4000));
+ }
+ }
+ return out;
+ }
+
+ /** Commons files → the picture, its licence and its credit. A file under a licence that forbids reuse is dropped. */
+ async function imageInfo(files) {
+ const wanted = [...new Set((files || []).map((f) => String(f || '').trim()).filter(Boolean))];
+ const out = new Map();
+ for (const group of chunk(wanted, 40)) {
+ const data = await call(api(opts.commons, {
+ action: 'query', prop: 'imageinfo', iiprop: 'url|extmetadata', iiurlwidth: '1200',
+ titles: group.map((f) => (f.startsWith('File:') ? f : `File:${f}`)).join('|'),
+ }), { label: 'Wikimedia Commons' });
+ if (!data || data.notFound) continue;
+ const pages = (data.query && data.query.pages) || [];
+ for (const page of pages) {
+ const info = page && page.imageinfo && page.imageinfo[0];
+ if (!info) continue;
+ const meta = info.extmetadata || {};
+ const val = (name) => cleanMeta(meta[name] && meta[name].value);
+ const license = val('LicenseShortName') || val('License') || '';
+ const licenseUrl = val('LicenseUrl');
+ if (!isFreeLicense(license)) continue;
+ const url = info.thumburl || info.url;
+ if (!url || !/^https:\/\/upload\.wikimedia\.org\//.test(url)) continue;
+ out.set(String(page.title).replace(/^File:/, ''), {
+ url,
+ original_url: info.url || url,
+ page_url: `https://commons.wikimedia.org/wiki/${encodeURIComponent(String(page.title).replace(/ /g, '_'))}`,
+ license,
+ license_url: licenseUrl,
+ credit: val('Artist') || val('Credit') || 'Wikimedia Commons',
+ usage_terms: val('UsageTerms'),
+ });
+ }
+ }
+ return out;
+ }
+
+ return { listEntries, categoryMembers, resolveQids, entities, leadExtract, imageInfo, isFreeLicense, plainText, parseList };
+}
+
+module.exports = { createSources, parseList, plainText, isFreeLicense, entryTitle, SourceError };
@@ -70,6 +70,33 @@ function load(env = process.env) {
@@ -70,6 +70,33 @@ function load(env = process.env) {
timeoutMs: Math.max(500, int(env.MEME_UPSTREAM_TIMEOUT_MS, 8000)),
timeoutMs: Math.max(500, int(env.MEME_UPSTREAM_TIMEOUT_MS, 8000)),
},
},
+ // OpenVibe.AI: the judge and the encyclopedia's writing pass. Meme calls ai.generate as the service
+ // `meme` with a client-credentials token (audience openvibe.ai, scope ai.run.create); Network grants the
+ // token and the ai.* namespace. Meme never names a provider: AI's routing decides, and the run reports
+ // what answered.
+ ai: {
+ url: trim(env.OV_AI_INTERNAL_URL || 'http://127.0.0.1:4700'),
+ audience: env.MEME_AI_AUDIENCE || 'openvibe.ai',
+ scope: env.MEME_AI_SCOPE || 'ai.run.create',
+ waitMs: Math.max(1000, int(env.MEME_AI_WAIT_MS, 55_000)),
+ timeoutMs: Math.max(1000, int(env.MEME_AI_TIMEOUT_MS, 120_000)),
+ pollMs: Math.max(100, int(env.MEME_AI_POLL_MS, 1500)),
+ maxTokens: Math.max(200, int(env.MEME_AI_MAX_TOKENS, 1400)),
+ temperature: 0.2,
+ },
+
+ // The judge: it scores a candidate from its own citations and writes the entry. Only what clears the bar is
+ // published; everything else stays hidden for staff with its score. Off under MEME_DRILL.
+ judge: {
+ intervalMs: Math.max(60_000, int(env.MEME_JUDGE_INTERVAL_MS, 30 * 60_000)),
+ minScore: Number.isFinite(Number(env.MEME_JUDGE_MIN_SCORE)) ? Number(env.MEME_JUDGE_MIN_SCORE) : 0.75,
+ minUses: Math.max(1, int(env.MEME_JUDGE_MIN_USES, 3)),
+ minDays: Math.max(1, int(env.MEME_JUDGE_MIN_DAYS, 2)),
+ minChannels: Math.max(1, int(env.MEME_JUDGE_MIN_CHANNELS, 2)),
+ maxPerRun: Math.min(30, Math.max(1, int(env.MEME_JUDGE_MAX_PER_RUN, 10))),
+ contextLines: Math.max(1, int(env.MEME_JUDGE_CONTEXT_LINES, 6)),
+ },
+
// The candidate importer: every 30 minutes, off under MEME_DRILL (tests call runOnce()).
// The candidate importer: every 30 minutes, off under MEME_DRILL (tests call runOnce()).
importer: {
importer: {
intervalMs: Math.max(60_000, int(env.MEME_IMPORTER_INTERVAL_MS, 30 * 60_000)),
intervalMs: Math.max(60_000, int(env.MEME_IMPORTER_INTERVAL_MS, 30 * 60_000)),
@@ -82,6 +109,32 @@ function load(env = process.env) {
@@ -82,6 +109,32 @@ function load(env = process.env) {
// Every citation is re-verified daily, off under MEME_DRILL.
// Every citation is re-verified daily, off under MEME_DRILL.
reverify: { intervalMs: Math.max(60_000, int(env.MEME_REVERIFY_INTERVAL_MS, 24 * 60 * 60_000)) },
reverify: { intervalMs: Math.max(60_000, int(env.MEME_REVERIFY_INTERVAL_MS, 24 * 60 * 60_000)) },
+
+ // The encyclopedia: the well-known internet memes, cited to Wikipedia (CC BY-SA) and to Wikidata (CC0),
+ // written by the same AI pass. It is a one-off seed, not a timer: the job runs from scripts/seed-classics.js.
+ encyclopedia: {
+ wikipedia: trim(env.MEME_WIKIPEDIA_URL || 'https://en.wikipedia.org'),
+ wikidata: trim(env.MEME_WIKIDATA_URL || 'https://www.wikidata.org'),
+ commons: trim(env.MEME_COMMONS_URL || 'https://commons.wikimedia.org'),
+ userAgent: env.MEME_WIKI_USER_AGENT || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])',
+ timeoutMs: Math.max(1000, int(env.MEME_WIKI_TIMEOUT_MS, 20_000)),
+ minIntervalMs: Math.max(0, int(env.MEME_WIKI_MIN_INTERVAL_MS, 250)),
+ limit: Math.min(1000, Math.max(1, int(env.MEME_ENCYCLOPEDIA_LIMIT, 400))),
+ // The lists the seeds are read from (Wikipedia, CC BY-SA): the phenomena list and the category behind it.
+ lists: String(env.MEME_ENCYCLOPEDIA_LISTS || 'List_of_Internet_phenomena').split(',').map((x) => x.trim()).filter(Boolean),
+ categories: String(env.MEME_ENCYCLOPEDIA_CATEGORIES || 'Internet memes,Memes').split(',').map((x) => x.trim()).filter(Boolean),
+ minScore: Number.isFinite(Number(env.MEME_ENCYCLOPEDIA_MIN_SCORE)) ? Number(env.MEME_ENCYCLOPEDIA_MIN_SCORE) : 0.75,
+ },
+
+ // The templates: the well-known public-domain or freely licensed meme templates, from Wikimedia Commons,
+ // each with its licence and credit.
+ templates: {
+ commons: trim(env.MEME_COMMONS_URL || 'https://commons.wikimedia.org'),
+ userAgent: env.MEME_WIKI_USER_AGENT || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])',
+ timeoutMs: Math.max(1000, int(env.MEME_WIKI_TIMEOUT_MS, 20_000)),
+ minIntervalMs: Math.max(0, int(env.MEME_WIKI_MIN_INTERVAL_MS, 250)),
+ limit: Math.min(200, Math.max(1, int(env.MEME_TEMPLATES_LIMIT, 40))),
+ },
};
};
}
}
@@ -5,9 +5,14 @@
@@ -5,9 +5,14 @@
* 404 anonymously; clip 483 is Live's AI auto-clip of sexual content (server/ai/explicit.js), and VOD 5709 has no
* 404 anonymously; clip 483 is Live's AI auto-clip of sexual content (server/ai/explicit.js), and VOD 5709 has no
* public row at all. A citation that would carry either id is dropped before it is written, and the seed script and
* public row at all. A citation that would carry either id is dropped before it is written, and the seed script and
* the importer both call this filter. No title, thumbnail or quote from either ever enters the database.
* the importer both call this filter. No title, thumbnail or quote from either ever enters the database.
+ *
+ * The list also names the words no AI-written entry text may carry. A verbatim quote is never rewritten (that would
+ * falsify the citation), but a title, a summary, an origin note and an example are the site's own words: they pass
+ * this filter before they are stored, and an entry whose text carries one is never published.
*/
*/
const DENIED_VOD_IDS = Object.freeze(new Set([5709]));
const DENIED_VOD_IDS = Object.freeze(new Set([5709]));
const DENIED_CLIP_IDS = Object.freeze(new Set([483]));
const DENIED_CLIP_IDS = Object.freeze(new Set([483]));
+const DENIED_WORDS = Object.freeze(['nigger', 'faggot', 'retard', 'kys', 'rape', 'pedo', 'nazi']);
function num(v) {
function num(v) {
if (v == null || v === '') return null;
if (v == null || v === '') return null;
@@ -35,4 +40,16 @@ function allowed(uses) {
@@ -35,4 +40,16 @@ function allowed(uses) {
return (Array.isArray(uses) ? uses : []).filter((u) => !isDenied(u));
return (Array.isArray(uses) ? uses : []).filter((u) => !isDenied(u));
}
}
-module.exports = { DENIED_VOD_IDS, DENIED_CLIP_IDS, isDenied, isDeniedVod, isDeniedClip, allowed };
+/** True when a text carries a word the deny list names. The word must stand on its own. */
+function hasDeniedWord(text) {
+ const s = String(text == null ? '' : text);
+ if (!s) return false;
+ return DENIED_WORDS.some((w) => new RegExp(`(^|[^a-z0-9])${w}([^a-z0-9]|$)`, 'i').test(s));
+}
+
+/** The text of a field the site's own words carry: clean when it names nothing on the deny list. */
+function isCleanText(text) {
+ return !hasDeniedWord(text);
+}
+
+module.exports = { DENIED_VOD_IDS, DENIED_CLIP_IDS, DENIED_WORDS, isDenied, isDeniedVod, isDeniedClip, allowed, hasDeniedWord, isCleanText };
@@ -14,6 +14,8 @@
@@ -14,6 +14,8 @@
* GET /captions/:id one saved caption
* GET /captions/:id one saved caption
* POST /entries/:slug/report a takedown/correction report (signed in)
* POST /entries/:slug/report a takedown/correction report (signed in)
* GET /reports open reports (staff)
* GET /reports open reports (staff)
+ * GET /staff/candidates the hidden candidates with the judge's score (staff)
+ * GET /staff/candidates/:slug one candidate: its judgement, citations and decisions (staff)
* POST /entries/:slug/publish|hide staff
* POST /entries/:slug/publish|hide staff
* PATCH /entries/:slug staff: title, kind, status
* PATCH /entries/:slug staff: title, kind, status
*
*
@@ -116,6 +118,7 @@ function createApi(ctx) {
@@ -116,6 +118,7 @@ function createApi(ctx) {
const staff = isStaff(req.principal);
const staff = isStaff(req.principal);
if (!row || (row.visibility === 'hidden' && !staff)) return problem(res, 404, 'meme.not_found', 'No such meme.');
if (!row || (row.visibility === 'hidden' && !staff)) return problem(res, 404, 'meme.not_found', 'No such meme.');
const citations = await entries.citations(row.id);
const citations = await entries.citations(row.id);
+ const verdict = await entries.judgement(row.id);
if (row.visibility === 'public') res.set('Cache-Control', 'public, max-age=60');
if (row.visibility === 'public') res.set('Cache-Control', 'public, max-age=60');
else if (row.visibility === 'unlisted') res.set('Cache-Control', 'private, max-age=60');
else if (row.visibility === 'unlisted') res.set('Cache-Control', 'private, max-age=60');
return res.json({
return res.json({
@@ -124,6 +127,9 @@ function createApi(ctx) {
@@ -124,6 +127,9 @@ function createApi(ctx) {
citations,
citations,
timeline: citations.filter((c) => c.role !== 'example'),
timeline: citations.filter((c) => c.role !== 'example'),
examples: citations.filter((c) => c.role === 'example'),
examples: citations.filter((c) => c.role === 'example'),
+ // The judge's own words about the entry: where it started and how people use it.
+ ...(verdict && verdict.origin ? { notes: { origin: verdict.origin, spread: verdict.spread, examples: verdict.examples } } : {}),
+ ...(staff && verdict ? { judgement: { score: verdict.score, reason: verdict.reason, judged_at: verdict.judged_at, model: verdict.model, provider: verdict.provider } } : {}),
});
});
});
});
@@ -226,6 +232,33 @@ function createApi(ctx) {
@@ -226,6 +232,33 @@ function createApi(ctx) {
return res.json({ count: reports.length, reports });
return res.json({ count: reports.length, reports });
});
});
+ // ── The staff list: the candidates that stayed hidden ──
+ r.get('/staff/candidates', requireStaff, async (req, res) => {
+ const limit = asInt(req.query.limit, 100);
+ const kind = typeof req.query.kind === 'string' && req.query.kind ? req.query.kind : null;
+ if (kind && !KINDS.includes(kind)) return invalid(res, `kind must be one of ${KINDS.join(', ')}`);
+ const list = await entries.listHidden({ limit, kind });
+ return res.json({ count: list.length, candidates: list });
+ });
+
+ r.get('/staff/candidates/:slug', requireStaff, async (req, res) => {
+ const row = await entries.bySlug(String(req.params.slug || ''));
+ if (!row) return problem(res, 404, 'meme.not_found', 'No such meme.');
+ const [citations, verdict, history] = await Promise.all([
+ entries.citations(row.id),
+ entries.judgement(row.id),
+ entries.decisions(row.id),
+ ]);
+ return res.json({
+ ...entries.wireEntry(row),
+ score: verdict ? verdict.score : null,
+ reason: verdict ? verdict.reason : null,
+ judgement: verdict,
+ citations,
+ decisions: history,
+ });
+ });
+
return r;
return r;
}
}
@@ -21,6 +21,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
@@ -21,6 +21,7 @@ function createLifecycle({ server, ctx, exit, signals, timers = [], extra = [] }
() => ctx.keys.client.stop(),
() => ctx.keys.client.stop(),
() => ctx.importer.stop(),
() => ctx.importer.stop(),
() => ctx.reverifier.stop(),
() => ctx.reverifier.stop(),
+ () => ctx.judgeJob.stop(),
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
async () => { clearInterval(ctx.telemetryTick); if (ctx.retention) await ctx.retention.stop(); if (ctx.telemetry) await ctx.telemetry.stop(); },
() => ctx.s.close(),
() => ctx.s.close(),
...extra,
...extra,
@@ -50,6 +51,7 @@ async function start() {
@@ -50,6 +51,7 @@ async function start() {
// Jobs. Each refuses to start under MEME_DRILL; tests call their runOnce().
// Jobs. Each refuses to start under MEME_DRILL; tests call their runOnce().
ctx.importer.start();
ctx.importer.start();
ctx.reverifier.start();
ctx.reverifier.start();
+ ctx.judgeJob.start();
// The two account topics (export_requested, deleted) at OpenVibe.Events: idempotent; off without the URL, the
// The two account topics (export_requested, deleted) at OpenVibe.Events: idempotent; off without the URL, the
// delivery secret and the client secret. The consumer itself is mounted in server/app.js.
// delivery secret and the client secret. The consumer itself is mounted in server/app.js.
new file mode 100644
new file mode 100644
@@ -0,0 +1,156 @@
@@ -0,0 +1,156 @@
+'use strict';
+
+/**
+ * The judge's job. It takes the candidates the importer left hidden, asks OpenVibe.AI to judge each one from its
+ * own citations, stores the score and the written entry, and then applies the publish rule: what clears the bar is
+ * published, everything else stays hidden for staff with its score and the reasons it did not clear.
+ *
+ * Every decision is a status event, so the history of an entry says what the AI decided and why. The job is safe to
+ * run twice: a candidate that already has a judgement is only re-judged when it is asked for (a re-judge round), and
+ * publishing an entry that is already public changes nothing.
+ *
+ * runOnce() judge the candidates nobody has judged yet
+ * runOnce({ force: true }) judge every hidden candidate again (the re-judge round)
+ * runOnce({ only: [slug] }) judge these entries and nothing else
+ */
+const { evaluate } = require('../ai/gate');
+
+function createJudgeJob({ config, s, entries, judge, context, log = console }) {
+ const stats = { runs: 0, judged: 0, published: 0, kept: 0, errors: 0, last_run_at: null, last_error: null };
+ let timers = [];
+
+ /** The entry's own citations, oldest first: the judge reads them in the order they happened. */
+ async function citationsOf(entry) {
+ const cites = await entries.citations(entry.id);
+ return cites
+ .filter((c) => c.source_status === 'live')
+ .sort((a, b) => Date.parse(a.observed_at) - Date.parse(b.observed_at));
+ }
+
+ /**
+ * Judge one entry and act on the verdict. Returns { entry, judgement, decision }.
+ */
+ async function judgeEntry(entry) {
+ const citations = await citationsOf(entry);
+ const prepared = await context.prepare(citations);
+ const verdict = await judge.judge({ id: entry.id, title: entry.title, kind: entry.kind, phrase: entry.title }, prepared);
+ const saved = await entries.saveJudgement(entry.id, verdict);
+ // The AI's title and summary are the entry's own words; the slug never changes.
+ await entries.applyJudgement(entry.id, { title: verdict.title, summary: verdict.summary });
+ const fresh = await entries.byId(entry.id);
+ const decision = evaluate({
+ entry: fresh,
+ citations,
+ judgement: { ...saved, denied: verdict.denied, looks: verdict.looks },
+ gate: config.judge,
+ });
+ return { entry: fresh, judgement: saved, decision, citations };
+ }
+
+ /** Publish an entry that cleared the bar, and say so in the history. */
+ async function publishJudged(entry, decision, verdict) {
+ const out = await entries.publish(entry.slug, { changedBy: 'service:meme:judge' });
+ if (!out.ok) {
+ await entries.logDecision({
+ entryId: entry.id, decision: 'keep_hidden', fromStatus: entry.status, toStatus: entry.status,
+ fromVisibility: 'hidden', toVisibility: 'hidden', score: decision.score,
+ reason: `not published: ${out.code || out.message || 'refused'}`,
+ });
+ return { published: false, out };
+ }
+ await entries.logDecision({
+ entryId: entry.id, decision: 'publish', fromStatus: entry.status, toStatus: out.entry.status,
+ fromVisibility: 'hidden', toVisibility: 'public', score: decision.score,
+ reason: `the AI scored it ${Number(decision.score).toFixed(2)}: ${verdict.reason || ''}`.slice(0, 500),
+ });
+ return { published: true, out };
+ }
+
+ /** Keep an entry hidden, and say why in the history. */
+ async function keepHidden(entry, decision, verdict) {
+ await entries.logDecision({
+ entryId: entry.id, decision: 'keep_hidden', fromStatus: entry.status, toStatus: entry.status,
+ fromVisibility: entry.visibility, toVisibility: 'hidden', score: decision.score,
+ reason: `kept hidden: ${decision.reasons.join('; ')}${verdict && verdict.reason ? ` — ${verdict.reason}` : ''}`.slice(0, 500),
+ });
+ return { published: false };
+ }
+
+ async function runOnce({ limit = null, force = false, only = null, publish = true } = {}) {
+ const runAt = new Date(s.now()).toISOString();
+ if (!judge || !judge.enabled) {
+ const message = 'OpenVibe.AI is not configured: nothing was judged.';
+ stats.last_error = message;
+ log.warn(`[OpenVibe.Meme] judge: ${message}`);
+ return { ok: false, judged: 0, published: 0, kept: 0, error: message };
+ }
+ const max = Math.min(50, Math.max(1, Number(limit) || config.judge.maxPerRun));
+ const errors = [];
+ let candidates = [];
+ if (only && only.length) {
+ for (const slug of only) {
+ const row = await entries.bySlug(String(slug));
+ if (row) candidates.push(row);
+ }
+ } else if (force) {
+ candidates = await entries.listHidden({ limit: max });
+ } else {
+ candidates = await entries.unjudged({ limit: max });
+ }
+
+ let published = 0;
+ let kept = 0;
+ for (const candidate of candidates) {
+ try {
+ const { entry, decision, judgement: verdict } = await judgeEntry(candidate);
+ if (decision.publish && publish) {
+ const r = await publishJudged(entry, decision, verdict);
+ if (r.published) published++;
+ else kept++;
+ } else if (decision.publish) {
+ // A dry run: the judgement is stored, the entry stays where it is.
+ await entries.logDecision({
+ entryId: entry.id, decision: 'judge', fromStatus: entry.status, toStatus: entry.status,
+ fromVisibility: entry.visibility, toVisibility: entry.visibility, score: decision.score,
+ reason: `judged ${Number(decision.score).toFixed(2)}: ${verdict.reason || ''}`.slice(0, 500),
+ });
+ kept++;
+ } else {
+ await keepHidden(entry, decision, verdict);
+ kept++;
+ }
+ } catch (err) {
+ errors.push(`${candidate.slug}: ${err.message}`);
+ log.warn(`[OpenVibe.Meme] judge ${candidate.slug} failed: ${err.message}`);
+ }
+ }
+ stats.runs++;
+ stats.judged += candidates.length - errors.length;
+ stats.published += published;
+ stats.kept += kept;
+ stats.errors += errors.length;
+ stats.last_run_at = runAt;
+ stats.last_error = errors[0] || null;
+ const out = { ok: errors.length === 0, judged: candidates.length - errors.length, published, kept, errors };
+ log.log(`[OpenVibe.Meme] judge: ${candidates.length} judged, ${published} published, ${kept} kept hidden${errors.length ? `, ${errors.length} errors` : ''}`);
+ return out;
+ }
+
+ function start() {
+ if (config.drill || timers.length) return false;
+ const quiet = () => runOnce().catch((err) => log.warn(`[OpenVibe.Meme] judge tick failed: ${err.message}`));
+ const kick = setTimeout(quiet, 45_000);
+ const tick = setInterval(quiet, config.judge.intervalMs);
+ if (kick.unref) kick.unref();
+ if (tick.unref) tick.unref();
+ timers = [kick, tick];
+ return true;
+ }
+ function stop() {
+ for (const t of timers.splice(0)) { clearTimeout(t); clearInterval(t); }
+ }
+
+ return { runOnce, start, stop, judgeEntry, stats: () => ({ ...stats, running: timers.length > 0 }) };
+}
+
+module.exports = { createJudgeJob };
new file mode 100644
new file mode 100644
@@ -0,0 +1,94 @@
@@ -0,0 +1,94 @@
+'use strict';
+
+/**
+ * The one outbound HTTP helper the open-source jobs share: identify every request, pace it per host, keep the
+ * answer's fetch time, and turn a failure into a short, non-secret error. The URLs come from config (the official
+ * public hosts); no user-chosen URL is ever fetched.
+ */
+const { setTimeout: sleep } = require('node:timers/promises');
+
+function hostOf(url) {
+ try { return new URL(url).host; } catch { return String(url); }
+}
+
+/** Serialise requests per host, at least minIntervalMs apart (the Wikimedia API policy asks for it). */
+function createPacer(minIntervalMs = 0) {
+ const last = new Map();
+ let chain = Promise.resolve();
+ return function pace(url) {
+ const host = hostOf(url);
+ const run = chain.then(async () => {
+ const wait = Math.max(0, (last.get(host) || 0) + minIntervalMs - Date.now());
+ if (wait) await sleep(wait);
+ last.set(host, Date.now());
+ });
+ chain = run.catch(() => {});
+ return run.then(() => () => {});
+ };
+}
+
+class SourceError extends Error {
+ constructor(message, { code = 'meme.source', status = 502 } = {}) {
+ super(message);
+ this.name = 'SourceError';
+ this.code = code;
+ this.status = status;
+ }
+}
+
+/**
+ * fetchText(url, { fetchImpl, userAgent, headers, timeoutMs, pace, retries })
+ * Returns { text, status, url, fetchedAt }. Retries 429/5xx twice with a short backoff; a 4xx (except 429) is final.
+ */
+async function fetchText(url, opts = {}) {
+ const fetchImpl = opts.fetchImpl || globalThis.fetch;
+ const headers = {
+ 'User-Agent': opts.userAgent || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])',
+ Accept: opts.accept || 'application/json, text/plain;q=0.9, */*;q=0.5',
+ ...(opts.headers || {}),
+ };
+ const retries = Number.isFinite(opts.retries) ? opts.retries : 2;
+ let release = null;
+ let lastErr = null;
+ for (let attempt = 0; attempt <= retries; attempt += 1) {
+ if (opts.pace && !release) release = await opts.pace(url);
+ try {
+ const res = await fetchImpl(url, {
+ method: opts.method || 'GET',
+ headers,
+ body: opts.body,
+ signal: AbortSignal.timeout(Math.max(1000, Number(opts.timeoutMs) || 20_000)),
+ redirect: 'follow',
+ });
+ const text = await res.text();
+ if (res.ok) {
+ if (release) { release(); release = null; }
+ return { text, status: res.status, url, fetchedAt: new Date().toISOString() };
+ }
+ lastErr = new SourceError(`${opts.label || hostOf(url)} answered ${res.status}`, { status: res.status === 404 ? 404 : 502 });
+ if (res.status === 429 || res.status >= 500) {
+ if (release) { release(); release = null; }
+ if (attempt < retries) {
+ const retryAfter = Number(opts.headers && opts.headers.get && opts.headers.get('retry-after'));
+ await sleep(Number.isFinite(retryAfter) && retryAfter > 0 ? Math.min(10_000, retryAfter * 1000) : 800 * (attempt + 1));
+ continue;
+ }
+ }
+ if (release) { release(); release = null; }
+ throw lastErr;
+ } catch (err) {
+ if (release) { release(); release = null; }
+ lastErr = err instanceof SourceError ? err : new SourceError(`${opts.label || hostOf(url)}: ${(err && err.message) || 'request failed'}`);
+ if (attempt < retries) { await sleep(600 * (attempt + 1)); continue; }
+ throw lastErr;
+ }
+ }
+ throw lastErr || new SourceError(`${opts.label || hostOf(url)}: request failed`);
+}
+
+async function fetchJson(url, opts = {}) {
+ const out = await fetchText(url, opts);
+ try { return { ...out, json: JSON.parse(out.text) }; } catch { throw new SourceError(`${opts.label || hostOf(url)} answered something that is not JSON`); }
+}
+
+module.exports = { createPacer, fetchText, fetchJson, SourceError, hostOf };
@@ -14,7 +14,7 @@ const { cleanSummary } = require('./text/no-looks');
@@ -14,7 +14,7 @@ const { cleanSummary } = require('./text/no-looks');
const { weekBounds } = require('./week');
const { weekBounds } = require('./week');
const SLUG_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
const SLUG_RE = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
-const KINDS = Object.freeze(['joke', 'catchphrase', 'bit', 'meme', 'saga']);
+const KINDS = Object.freeze(['joke', 'catchphrase', 'bit', 'meme', 'saga', 'classic']);
const STATUSES = Object.freeze(['rising', 'peaked', 'classic', 'dead']);
const STATUSES = Object.freeze(['rising', 'peaked', 'classic', 'dead']);
const VISIBILITIES = Object.freeze(['public', 'unlisted', 'hidden']);
const VISIBILITIES = Object.freeze(['public', 'unlisted', 'hidden']);
const CAPTION_VISIBILITIES = Object.freeze(['public', 'unlisted', 'hidden']);
const CAPTION_VISIBILITIES = Object.freeze(['public', 'unlisted', 'hidden']);
@@ -43,6 +43,36 @@ function wireEntry(row) {
@@ -43,6 +43,36 @@ function wireEntry(row) {
revision: Number(row.revision) || 1,
revision: Number(row.revision) || 1,
created_at: iso(row.created_at),
created_at: iso(row.created_at),
updated_at: iso(row.updated_at),
updated_at: iso(row.updated_at),
+ ...(row.image_url ? { image_url: row.image_url } : {}),
+ ...(row.image_credit ? { image_credit: row.image_credit } : {}),
+ ...(row.image_license ? { image_license: row.image_license } : {}),
+ ...(row.image_license_url ? { image_license_url: row.image_license_url } : {}),
+ ...(row.image_source_url ? { image_source_url: row.image_source_url } : {}),
+ ...(row.attribution ? { attribution: row.attribution } : {}),
+ ...(row.external_id ? { external_id: row.external_id } : {}),
+ };
+}
+
+function wireJudgement(row) {
+ if (!row) return null;
+ let examples = row.examples;
+ if (typeof examples === 'string') { try { examples = JSON.parse(examples); } catch { examples = []; } }
+ return {
+ entry_id: row.entry_id,
+ score: Number(row.score),
+ reason: row.reason,
+ title: row.title == null ? null : row.title,
+ summary: row.summary == null ? null : row.summary,
+ origin: row.origin == null ? null : row.origin,
+ spread: row.spread == null ? null : row.spread,
+ examples: Array.isArray(examples) ? examples.map(String) : [],
+ denied: Boolean(row.denied),
+ looks: Boolean(row.looks),
+ model: row.model == null ? null : row.model,
+ provider: row.provider == null ? null : row.provider,
+ run_id: row.run_id == null ? null : row.run_id,
+ judged_at: iso(row.judged_at),
+ judged_by: row.judged_by,
};
};
}
}
@@ -71,7 +101,13 @@ function wireCitation(row) {
@@ -71,7 +101,13 @@ function wireCitation(row) {
function wireTemplate(row) {
function wireTemplate(row) {
if (!row) return null;
if (!row) return null;
- return { id: row.id, name: row.name, image_url: row.image_url, box_count: Number(row.box_count), source: row.source, license: row.license, created_at: iso(row.created_at) };
+ return {
+ id: row.id, name: row.name, image_url: row.image_url, box_count: Number(row.box_count), source: row.source,
+ license: row.license, created_at: iso(row.created_at),
+ ...(row.credit ? { credit: row.credit } : {}),
+ ...(row.source_url ? { source_url: row.source_url } : {}),
+ ...(row.attribution ? { attribution: row.attribution } : {}),
+ };
}
}
function wireCaption(row) {
function wireCaption(row) {
@@ -275,21 +311,30 @@ function createStore(s, { search }) {
@@ -275,21 +311,30 @@ function createStore(s, { search }) {
}
}
/**
/**
- * Insert one hidden candidate and its citations in the caller's transaction. Used by the importer and the
- * seed script only; it never publishes. Citations that name the deny list are dropped here too.
+ * Insert one entry and its citations in the caller's transaction. A candidate is inserted hidden; an
+ * encyclopedia entry ('classic') is inserted public. Citations that name the deny list are dropped here too.
*/
*/
- async function createCandidate({ slug = null, title, kind = 'meme', summary = null, status = 'rising', first_seen_at, last_seen_at, visibility = 'hidden', citations: cites = [] }, t = null) {
+ async function createEntry({
+ slug = null, title, kind = 'meme', summary = null, status = 'rising', first_seen_at, last_seen_at,
+ visibility = 'hidden', citations: cites = [], image_url = null, image_credit = null, image_license = null,
+ image_license_url = null, image_source_url = null, attribution = null, external_id = null,
+ }, t = null) {
const run = async (tx) => {
const run = async (tx) => {
const id = s.newId('mem');
const id = s.newId('mem');
const cleanSlug = await uniqueSlug(slug || slugify(title), tx);
const cleanSlug = await uniqueSlug(slug || slugify(title), tx);
const cleanTitle = String(title || '').replace(/\s+/g, ' ').trim().slice(0, 160) || 'Untitled meme';
const cleanTitle = String(title || '').replace(/\s+/g, ' ').trim().slice(0, 160) || 'Untitled meme';
+ const cleanKind = KINDS.includes(kind) ? kind : 'meme';
let first = iso(first_seen_at) || s.iso();
let first = iso(first_seen_at) || s.iso();
let last = iso(last_seen_at) || first;
let last = iso(last_seen_at) || first;
if (Date.parse(first) > Date.parse(last)) { const swap = first; first = last; last = swap; }
if (Date.parse(first) > Date.parse(last)) { const swap = first; first = last; last = swap; }
const row = await q(tx).maybe(
const row = await q(tx).maybe(
- `INSERT INTO meme_entries (id, slug, title, kind, summary, status, status_since, first_seen_at, last_seen_at, visibility, revision, created_at, updated_at)
- VALUES ($1, $2, $3, $4, $5, $6, $7::timestamptz, $8::timestamptz, $9::timestamptz, $10, 1, now(), now()) RETURNING *`,
- [id, cleanSlug, cleanTitle, kind, cleanSummary(cleanTitle, summary), status, iso(last), first, last, visibility],
+ `INSERT INTO meme_entries (id, slug, title, kind, summary, status, status_since, first_seen_at, last_seen_at, visibility, revision, created_at, updated_at,
+ image_url, image_credit, image_license, image_license_url, image_source_url, attribution, external_id)
+ VALUES ($1, $2, $3, $4, $5, $6, $7::timestamptz, $8::timestamptz, $9::timestamptz, $10, 1, now(), now(),
+ $11, $12, $13, $14, $15, $16, $17) RETURNING *`,
+ [id, cleanSlug, cleanTitle, cleanKind, cleanSummary(cleanTitle, summary), status, iso(last), first, last, visibility,
+ image_url || null, image_credit || null, image_license || null, image_license_url || null, image_source_url || null,
+ attribution || null, external_id || null],
);
);
const added = await addCitations(row.id, cites, tx);
const added = await addCitations(row.id, cites, tx);
const entry = wireEntry(row);
const entry = wireEntry(row);
@@ -299,6 +344,11 @@ function createStore(s, { search }) {
@@ -299,6 +344,11 @@ function createStore(s, { search }) {
return t ? await run(t) : await s.tx(run);
return t ? await run(t) : await s.tx(run);
}
}
+ /** A candidate is an entry that stays hidden until the judge clears it. */
+ async function createCandidate(opts, t = null) {
+ return await createEntry({ ...opts, visibility: opts.visibility || 'hidden' }, t);
+ }
+
/** Insert citations in the caller's transaction; the sha256 is computed from the verbatim quote. */
/** Insert citations in the caller's transaction; the sha256 is computed from the verbatim quote. */
async function addCitations(entryId, cites = [], t = null) {
async function addCitations(entryId, cites = [], t = null) {
const run = async (tx) => {
const run = async (tx) => {
@@ -422,6 +472,21 @@ function createStore(s, { search }) {
@@ -422,6 +472,21 @@ function createStore(s, { search }) {
return wireTemplate(await db.maybe('SELECT * FROM meme_templates WHERE id = $1', [String(id || '')]));
return wireTemplate(await db.maybe('SELECT * FROM meme_templates WHERE id = $1', [String(id || '')]));
}
}
+ /**
+ * One template, written from an open source: its picture, where the picture came from, under which licence and
+ * who made it. A template that is already there is left alone.
+ */
+ async function addTemplate({ id, name, image_url, box_count = 2, source, license, credit = null, source_url = null, attribution = null }) {
+ const row = await db.maybe(
+ `INSERT INTO meme_templates (id, name, image_url, box_count, source, license, credit, source_url, attribution)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
+ ON CONFLICT (id) DO NOTHING RETURNING *`,
+ [String(id || ''), String(name || '').slice(0, 80), String(image_url || ''), Math.max(1, Math.min(8, Number(box_count) || 2)),
+ String(source || ''), String(license || ''), credit || null, source_url || null, attribution || null],
+ );
+ return wireTemplate(row || await db.maybe('SELECT * FROM meme_templates WHERE id = $1', [String(id || '')]));
+ }
+
async function createCaption({ subject, templateId, lines, visibility = 'public' }) {
async function createCaption({ subject, templateId, lines, visibility = 'public' }) {
if (!CAPTION_VISIBILITIES.includes(visibility)) return refusal(400, 'meme.invalid', `visibility must be one of ${CAPTION_VISIBILITIES.join(', ')}`, 'visibility');
if (!CAPTION_VISIBILITIES.includes(visibility)) return refusal(400, 'meme.invalid', `visibility must be one of ${CAPTION_VISIBILITIES.join(', ')}`, 'visibility');
const checked = captionLines(lines);
const checked = captionLines(lines);
@@ -471,6 +536,137 @@ function createStore(s, { search }) {
@@ -471,6 +536,137 @@ function createStore(s, { search }) {
};
};
}
}
+ // ── The judge ───────────────────────────────────────────
+ /** The verdict and the written entry, stored beside the entry (never inside it). */
+ async function saveJudgement(entryId, j, t = null) {
+ const run = async (tx) => {
+ const row = await q(tx).maybe(
+ `INSERT INTO meme_judgements (entry_id, score, reason, title, summary, origin, spread, examples, denied, looks, model, provider, run_id, judged_at, judged_by)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, $10, $11, $12, $13, now(), $14)
+ ON CONFLICT (entry_id) DO UPDATE SET
+ score = EXCLUDED.score, reason = EXCLUDED.reason, title = EXCLUDED.title, summary = EXCLUDED.summary,
+ origin = EXCLUDED.origin, spread = EXCLUDED.spread, examples = EXCLUDED.examples,
+ denied = EXCLUDED.denied, looks = EXCLUDED.looks, model = EXCLUDED.model, provider = EXCLUDED.provider,
+ run_id = EXCLUDED.run_id, judged_at = now(), judged_by = EXCLUDED.judged_by
+ RETURNING *`,
+ [entryId, Number(j.score), String(j.reason || ''), j.title || null, j.summary || null, j.origin || null, j.spread || null,
+ JSON.stringify(Array.isArray(j.examples) ? j.examples : []), Boolean(j.denied), Boolean(j.looks),
+ j.model || null, j.provider || null, j.run_id || null, j.judged_by || 'service:meme:judge'],
+ );
+ return wireJudgement(row);
+ };
+ return t ? await run(t) : await s.tx(run);
+ }
+
+ async function judgement(entryId, t = null) {
+ return wireJudgement(await q(t).maybe('SELECT * FROM meme_judgements WHERE entry_id = $1', [String(entryId || '')]));
+ }
+
+ /**
+ * Take the AI's title and summary for the entry. The title is only used when it is a real title (no looks) and
+ * the entry keeps its own slug: a published link never breaks.
+ */
+ async function applyJudgement(entryId, { title, summary }, { changedBy = 'service:meme:judge' } = {}) {
+ return await s.tx(async (t) => {
+ const row = await byId(entryId, t);
+ if (!row) return null;
+ const sets = [];
+ const args = [entryId];
+ const cleanTitle = String(title == null ? '' : title).replace(/\s+/g, ' ').trim().slice(0, 160);
+ if (cleanTitle && cleanTitle !== row.title) { args.push(cleanTitle); sets.push(`title = $${args.length}`); }
+ const cleanSummary = summary == null ? null : String(summary).replace(/\s+/g, ' ').trim().slice(0, 600);
+ if (cleanSummary && cleanSummary !== row.summary) { args.push(cleanSummary); sets.push(`summary = $${args.length}`); }
+ if (!sets.length) return { entry: wireEntry(row), changed: false };
+ const updated = await t.maybe(
+ `UPDATE meme_entries SET ${sets.join(', ')}, revision = revision + 1, updated_at = now() WHERE id = $1 RETURNING *`,
+ args,
+ );
+ if (updated.visibility === 'public') {
+ const all = await liveCitations(entryId, t);
+ await search.sync(t, wireEntry(updated), all);
+ }
+ return { entry: wireEntry(updated), changed: true };
+ });
+ }
+
+ /**
+ * One decision, one status event: the visibility moved, the score says why. The status itself does not move
+ * (a published candidate keeps the status the importer gave it), so from_status and to_status are the same and
+ * the decision columns carry the truth.
+ */
+ async function logDecision({ entryId, decision, fromStatus = null, toStatus = null, fromVisibility = null, toVisibility = null, score = null, reason = null, changedBy = 'service:meme:judge' }) {
+ const row = await db.maybe(
+ `INSERT INTO meme_status_events (entry_id, from_status, to_status, reason, changed_by, decision, from_visibility, to_visibility, score)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING *`,
+ [entryId, fromStatus || null, toStatus || 'rising', reason ? String(reason).slice(0, 500) : null, changedBy, decision,
+ fromVisibility || null, toVisibility || null, score == null ? null : Number(score)],
+ );
+ return row;
+ }
+
+ /** The decision history of one entry, newest first. */
+ async function decisions(entryId, { limit = 50 } = {}) {
+ return (await db.many(
+ `SELECT id, entry_id, from_status, to_status, reason, changed_by, decision, from_visibility, to_visibility, score, changed_at
+ FROM meme_status_events WHERE entry_id = $1 ORDER BY changed_at DESC, id DESC LIMIT $2`,
+ [String(entryId || ''), Math.min(200, Math.max(1, Number(limit) || 50))],
+ )).map((r) => ({
+ id: Number(r.id), entry_id: r.entry_id, from_status: r.from_status, to_status: r.to_status,
+ reason: r.reason, changed_by: r.changed_by, decision: r.decision,
+ from_visibility: r.from_visibility, to_visibility: r.to_visibility,
+ score: r.score == null ? null : Number(r.score), changed_at: iso(r.changed_at),
+ }));
+ }
+
+ /**
+ * The staff list: every entry that is still hidden, with the judge's score and the numbers behind it. Data
+ * only — the page is the orchestrator's.
+ */
+ async function listHidden({ limit = 100, kind = null } = {}) {
+ const n = Math.min(500, Math.max(1, Number(limit) || 100));
+ const args = [];
+ let where = "e.visibility = 'hidden'";
+ if (kind) { if (!KINDS.includes(kind)) return []; args.push(kind); where += ` AND e.kind = $${args.length}`; }
+ args.push(n);
+ const rows = await db.many(
+ `SELECT e.*, j.score AS j_score, j.reason AS j_reason, j.title AS j_title, j.judged_at AS j_judged_at,
+ j.model AS j_model, j.provider AS j_provider, j.denied AS j_denied, j.looks AS j_looks,
+ (SELECT COUNT(*)::int FROM meme_citations c WHERE c.entry_id = e.id AND c.role IN ('origin','use') AND c.source_status = 'live') AS uses,
+ (SELECT COUNT(DISTINCT to_char(c.observed_at AT TIME ZONE 'UTC', 'YYYY-MM-DD'))::int FROM meme_citations c WHERE c.entry_id = e.id AND c.role IN ('origin','use') AND c.source_status = 'live') AS days,
+ (SELECT COUNT(DISTINCT c.source_service)::int FROM meme_citations c WHERE c.entry_id = e.id AND c.role IN ('origin','use') AND c.source_status = 'live') AS channels
+ FROM meme_entries e LEFT JOIN meme_judgements j ON j.entry_id = e.id
+ WHERE ${where}
+ ORDER BY (j.score IS NULL) ASC, j.score DESC, e.last_seen_at DESC, e.id DESC LIMIT $${args.length}`,
+ args,
+ );
+ return rows.map((r) => ({
+ ...wireEntry(r),
+ score: r.j_score == null ? null : Number(r.j_score),
+ reason: r.j_reason == null ? null : r.j_reason,
+ judged_title: r.j_title == null ? null : r.j_title,
+ judged_at: iso(r.j_judged_at),
+ model: r.j_model == null ? null : r.j_model,
+ provider: r.j_provider == null ? null : r.j_provider,
+ denied: Boolean(r.j_denied),
+ looks: Boolean(r.j_looks),
+ uses: Number(r.uses) || 0,
+ days: Number(r.days) || 0,
+ channels: Number(r.channels) || 0,
+ }));
+ }
+
+ /** The hidden entries a judge has not looked at yet (the ones the job takes next). */
+ async function unjudged({ limit = 10 } = {}) {
+ const n = Math.min(100, Math.max(1, Number(limit) || 10));
+ const rows = await db.many(
+ `SELECT e.* FROM meme_entries e LEFT JOIN meme_judgements j ON j.entry_id = e.id
+ WHERE e.visibility = 'hidden' AND j.entry_id IS NULL
+ ORDER BY e.last_seen_at DESC, e.id DESC LIMIT $1`,
+ [n],
+ );
+ return rows.map(wireEntry);
+ }
+
async function createReport({ entryId, subject, kind, reason, note = null }) {
async function createReport({ entryId, subject, kind, reason, note = null }) {
const row = await db.maybe("SELECT id FROM meme_entries WHERE id = $1 AND visibility = 'public'", [String(entryId || '')]);
const row = await db.maybe("SELECT id FROM meme_entries WHERE id = $1 AND visibility = 'public'", [String(entryId || '')]);
if (!row) return refusal(404, 'meme.not_found', 'No such public entry.');
if (!row) return refusal(404, 'meme.not_found', 'No such public entry.');
@@ -494,11 +690,12 @@ function createStore(s, { search }) {
@@ -494,11 +690,12 @@ function createStore(s, { search }) {
}
}
return {
return {
- wireEntry, wireCitation, wireTemplate, wireCaption, wireReport,
+ wireEntry, wireCitation, wireTemplate, wireCaption, wireReport, wireJudgement,
bySlug, byId, citations, liveCitations, list, trending, week, series, uniqueSlug,
bySlug, byId, citations, liveCitations, list, trending, week, series, uniqueSlug,
- createCandidate, addCitations, enrich, reindex, publish, hide, patch,
- templates, templateById, createCaption, captionById, captionCountForTemplate,
+ createCandidate, createEntry, addCitations, enrich, reindex, publish, hide, patch,
+ templates, templateById, addTemplate, createCaption, captionById, captionCountForTemplate,
castVote, createReport, listOpenReports,
castVote, createReport, listOpenReports,
+ saveJudgement, judgement, applyJudgement, logDecision, decisions, listHidden, unjudged,
KINDS, STATUSES, VISIBILITIES, BLOCKED_WORDS, MAX_LINES, MAX_LINE,
KINDS, STATUSES, VISIBILITIES, BLOCKED_WORDS, MAX_LINES, MAX_LINE,
};
};
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,170 @@
@@ -0,0 +1,170 @@
+'use strict';
+
+/**
+ * The templates. OpenVibe.Meme's caption generator draws its text over a picture; the pictures are the well-known
+ * meme templates, taken from Wikimedia Commons under a licence that lets us use them: public domain, CC0 or a
+ * Creative Commons licence. A file under any other licence is left out.
+ *
+ * Every template keeps its own licence and credit, and the file page it came from, so the picture is attributed
+ * where it is shown. The three data-free placeholders of v1 (tpl_classic, tpl_reaction, tpl_triptych) stay: they
+ * are the site's own, and they need no attribution.
+ *
+ * The names are the well-known ones; the file is found in Commons and checked, not assumed. A template whose file
+ * cannot be found, or whose licence does not allow reuse, is skipped and reported.
+ */
+const { createPacer, fetchText, fetchJson } = require('../sources/http');
+
+/** The well-known templates: the name, the word the file must carry, and how many text boxes it has. */
+const WELL_KNOWN = Object.freeze([
+ { name: 'Drake Hotline Bling', token: 'drake', box_count: 2 },
+ { name: 'Distracted Boyfriend', token: 'distracted', box_count: 1 },
+ { name: 'Expanding Brain', token: 'brain', box_count: 4 },
+ { name: 'Change My Mind', token: 'change', box_count: 1 },
+ { name: 'Is This a Pigeon', token: 'pigeon', box_count: 2 },
+ { name: 'Woman Yelling at a Cat', token: 'yelling', box_count: 2 },
+ { name: 'Surprised Pikachu', token: 'pikachu', box_count: 2 },
+ { name: 'Two Buttons', token: 'button', box_count: 3 },
+ { name: 'Boardroom Meeting', token: 'boardroom', box_count: 2 },
+ { name: 'Running Away from a Balloon', token: 'balloon', box_count: 2 },
+ { name: 'Crying Cat', token: 'cat', box_count: 2 },
+ { name: 'Pointing at the Dog', token: 'dog', box_count: 2 },
+ { name: 'They Don\'t Know', token: 'know', box_count: 2 },
+ { name: 'Trade Offer', token: 'trade', box_count: 2 },
+ { name: 'Left Exit 12 Off Ramp', token: 'ramp', box_count: 2 },
+ { name: 'Hide the Pain Harold', token: 'harold', box_count: 2 },
+ { name: 'Roll Safe', token: 'roll', box_count: 2 },
+ { name: 'Disaster Girl', token: 'disaster', box_count: 2 },
+ { name: 'Philosoraptor', token: 'raptor', box_count: 2 },
+ { name: 'Success Kid', token: 'success', box_count: 2 },
+ { name: 'Mocking SpongeBob', token: 'spongebob', box_count: 2 },
+ { name: 'This Is Fine', token: 'fine', box_count: 2 },
+ { name: 'Waiting Skeleton', token: 'skeleton', box_count: 2 },
+ { name: 'X All the Y', token: 'all', box_count: 2 },
+ { name: 'Sweating Guy', token: 'sweat', box_count: 2 },
+ { name: 'Gru\'s Plan', token: 'gru', box_count: 4 },
+ { name: 'Skeleton Jumping', token: 'skeleton', box_count: 1 },
+ { name: 'Afraid to Ask Andy', token: 'andy', box_count: 2 },
+ { name: 'Me Gusta Mucho', token: 'gusta', box_count: 2 },
+ { name: 'Seagull Full of Fries', token: 'seagull', box_count: 2 },
+ { name: 'Chad and Virgo', token: 'virgo', box_count: 2 },
+ { name: 'Stonks', token: 'stonks', box_count: 2 },
+ { name: 'Cheugy', token: 'cheugy', box_count: 2 },
+ { name: 'Elon Musk', token: 'musk', box_count: 2 },
+ { name: 'No U Mad', token: 'mad', box_count: 2 },
+ { name: 'Nutcracker', token: 'nutcracker', box_count: 2 },
+ { name: 'Evil Toddler', token: 'toddler', box_count: 2 },
+ { name: 'Sweating Buddha', token: 'buddha', box_count: 2 },
+ { name: 'Yao Ming Face', token: 'yao', box_count: 2 },
+ { name: 'Il Posto di Lavoro', token: 'lavoro', box_count: 2 },
+ { name: 'Gru and the Plan', token: 'gru', box_count: 2 },
+ { name: 'Duck and Cover', token: 'duck', box_count: 2 },
+ { name: 'Scumbag Steve', token: 'steve', box_count: 2 },
+ { name: 'Overly Attached Girlfriend', token: 'girlfriend', box_count: 2 },
+ { name: 'Inhaling Seagull', token: 'seagull', box_count: 2 },
+ { name: 'Annoyed Cat', token: 'cat', box_count: 2 },
+]);
+
+const slug = (name) => String(name || '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 60);
+
+/** The Commons file's licence must allow reuse. */
+function isFreeLicense(name) {
+ const s = String(name || '').trim().toLowerCase();
+ if (!s) return false;
+ if (/^(cc0|public domain|pd|no restrictions|cc pd)/.test(s)) return true;
+ if (/^cc[ -]?(by|by-sa|by-nc|by-nd)[ -]?\d/.test(s)) return true;
+ if (s.startsWith('cc by')) return true;
+ return false;
+}
+
+const clean = (v) => String(v == null ? '' : v).replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim().slice(0, 300);
+
+function createTemplatesJob({ config, entries, fetchImpl = globalThis.fetch, log = console }) {
+ const opts = config.templates || {};
+ const userAgent = opts.userAgent || 'OpenVibe.Meme/0.2 (+https://openvibe.meme; [email protected])';
+ const timeoutMs = Math.max(1000, Number(opts.timeoutMs) || 20_000);
+ const pace = createPacer(Number(opts.minIntervalMs) || 250);
+
+ const api = (params) => `${String(opts.commons).replace(/\/+$/, '')}/w/api.php?${new URLSearchParams({ format: 'json', formatversion: '2', ...params })}`;
+
+ /** The Commons files whose name carries the token. */
+ async function search(token) {
+ const { json } = await fetchJson(api({
+ action: 'query', list: 'search', srsearch: `${token} filetype:bitmap`, srnamespace: '6', srlimit: '12',
+ }), { fetchImpl, userAgent, timeoutMs, pace, label: 'Commons search' });
+ const hits = (json && json.query && json.query.search) || [];
+ return hits.map((h) => String(h.title || '')).filter(Boolean);
+ }
+
+ /** The file's picture, licence and credit. A file under a licence that forbids reuse is dropped. */
+ async function info(titles) {
+ const wanted = [...new Set(titles)].slice(0, 20);
+ if (!wanted.length) return new Map();
+ const { json } = await fetchJson(api({
+ action: 'query', prop: 'imageinfo', iiprop: 'url|extmetadata|mime', iiurlwidth: '1200', titles: wanted.join('|'),
+ }), { fetchImpl, userAgent, timeoutMs, pace, label: 'Commons' });
+ const out = new Map();
+ for (const page of (json && json.query && json.query.pages) || []) {
+ const info = page && page.imageinfo && page.imageinfo[0];
+ if (!info) continue;
+ if (info.mime && !String(info.mime).startsWith('image/')) continue;
+ const meta = info.extmetadata || {};
+ const val = (name) => clean(meta[name] && meta[name].value);
+ const license = val('LicenseShortName') || val('License') || '';
+ if (!isFreeLicense(license)) continue;
+ const url = info.thumburl || info.url;
+ if (!url || !/^https:\/\/upload\.wikimedia\.org\//.test(url)) continue;
+ out.set(String(page.title), {
+ url,
+ page_url: `https://commons.wikimedia.org/wiki/${encodeURIComponent(String(page.title).replace(/ /g, '_'))}`,
+ license,
+ license_url: val('LicenseUrl'),
+ credit: val('Artist') || val('Credit') || 'Wikimedia Commons',
+ });
+ }
+ return out;
+ }
+
+ /**
+ * Seed the templates. Returns { seeded, skipped, templates } — the templates carry their licence and credit.
+ */
+ async function runOnce({ limit = null, publish = true } = {}) {
+ const wanted = (limit ? WELL_KNOWN.slice(0, Math.max(1, Number(limit))) : WELL_KNOWN);
+ const seeded = [];
+ const skipped = [];
+ for (const tpl of wanted) {
+ try {
+ const titles = await search(tpl.token);
+ const files = await info(titles);
+ let found = null;
+ for (const [title, file] of files) {
+ if (!title.toLowerCase().includes(tpl.token.toLowerCase())) continue;
+ found = file;
+ break;
+ }
+ if (!found) { skipped.push({ name: tpl.name, reason: 'no freely licensed file found' }); continue; }
+ const id = `tpl_commons_${slug(tpl.name)}`;
+ const row = await entries.addTemplate({
+ id,
+ name: tpl.name,
+ image_url: found.url,
+ box_count: tpl.box_count || 2,
+ source: 'Wikimedia Commons',
+ license: found.license,
+ credit: found.credit,
+ source_url: found.page_url,
+ attribution: `${found.credit} — ${found.license}, via Wikimedia Commons`.slice(0, 400),
+ });
+ seeded.push(row);
+ } catch (err) {
+ skipped.push({ name: tpl.name, reason: err.message });
+ log.warn(`[OpenVibe.Meme] template ${tpl.name}: ${err.message}`);
+ }
+ }
+ log.log(`[OpenVibe.Meme] templates: ${seeded.length} seeded, ${skipped.length} skipped`);
+ return { ok: true, seeded: seeded.length, skipped: skipped.length, templates: seeded, missing: skipped };
+ }
+
+ return { runOnce, search, info, WELL_KNOWN };
+}
+
+module.exports = { createTemplatesJob, WELL_KNOWN, isFreeLicense, slug };
@@ -6,7 +6,7 @@
@@ -6,7 +6,7 @@
* names looks at all is not repaired word by word — it is replaced by the entry's title, as Meme's plan says.
* names looks at all is not repaired word by word — it is replaced by the entry's title, as Meme's plan says.
* Verbatim quotes are never passed through this: rewriting a quote would falsify the citation.
* Verbatim quotes are never passed through this: rewriting a quote would falsify the citation.
*/
*/
-const { noLooks, hasLooks } = require('openvibe-sdk/no-looks');
+const { noLooks, hasLooks, isExplicit } = require('openvibe-sdk/no-looks');
/** The summary to store for an AI-written entry summary. */
/** The summary to store for an AI-written entry summary. */
function cleanSummary(title, summary) {
function cleanSummary(title, summary) {
@@ -18,4 +18,4 @@ function cleanSummary(title, summary) {
@@ -18,4 +18,4 @@ function cleanSummary(title, summary) {
return cleaned || t;
return cleaned || t;
}
}
-module.exports = { cleanSummary, noLooks, hasLooks };
+module.exports = { cleanSummary, noLooks, hasLooks, isExplicit };
@@ -155,8 +155,52 @@ function createUpstreams({ config, fetchImpl = globalThis.fetch, log = console }
@@ -155,8 +155,52 @@ function createUpstreams({ config, fetchImpl = globalThis.fetch, log = console }
return { status: 'gone' };
return { status: 'gone' };
}
}
+ /**
+ * The chat lines around a cited message: the lines before and after it, so the judge reads the joke and not
+ * one line alone. The same walk-back as chatMessage() finds the message; the page it lands on holds the rest.
+ */
+ async function chatContext(citation, { lines = 6 } = {}) {
+ const at = new Date(citation.observed_at).getTime();
+ if (!Number.isFinite(at)) return [];
+ const wanted = String(citation.chat_message_id != null ? citation.chat_message_id : citation.source_id);
+ let before = chatBefore(at + 60_000);
+ for (let page = 0; page < 5; page++) {
+ const h = await chatHistory({ before, limit: 500 });
+ const idx = h.messages.findIndex((m) => String(m.id) === wanted);
+ if (idx >= 0) {
+ const half = Math.max(1, Math.ceil(lines / 2));
+ return h.messages.slice(Math.max(0, idx - half), idx + half + 1).map((m) => ({
+ id: m.id,
+ who: m.username ? String(m.username) : null,
+ text: String(m.message || ''),
+ at: parseApiTime(m.timestamp),
+ }));
+ }
+ const oldest = h.messages.length ? parseApiTime(h.messages[h.messages.length - 1].timestamp) : null;
+ if (!oldest || Date.parse(oldest) <= at - 86_400_000) break;
+ before = chatBefore(Date.parse(oldest) - 1000);
+ }
+ return [];
+ }
+
+ /** The transcript lines around a cited second: what was said just before and just after it. */
+ async function vodContext(vodId, offsetSeconds, { lines = 6 } = {}) {
+ const id = String(vodId == null ? '' : vodId);
+ if (!id) return [];
+ let data;
+ try { data = await transcripts([id]); } catch { return []; }
+ const segs = (data[id] && Array.isArray(data[id].segments)) ? data[id].segments : [];
+ if (!segs.length) return [];
+ const at = Number(offsetSeconds);
+ if (!Number.isFinite(at)) return [];
+ const before = segs.filter((x) => Number(x.start) <= at).slice(-Math.max(1, Math.floor(lines / 2)));
+ const after = segs.filter((x) => Number(x.start) > at).slice(0, Math.max(1, lines - before.length));
+ return [...before, ...after].map((x) => ({ start: Number(x.start), text: String(x.text || '') }));
+ }
+
return {
return {
chatHistory, chatTimeline, rundownTopics, rundownTopic, vod, transcripts, canonicalVideo, citationUrl, chatMessage,
chatHistory, chatTimeline, rundownTopics, rundownTopic, vod, transcripts, canonicalVideo, citationUrl, chatMessage,
+ chatContext, vodContext,
base,
base,
};
};
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,224 @@
@@ -0,0 +1,224 @@
+'use strict';
+/**
+ * The encyclopedia: the well-known internet memes, seeded from the open lists and written by the same AI pass.
+ *
+ * Every seeded entry is an entry of kind 'classic' with its origin, its spread and its sources; it carries the
+ * attribution of the licences its text is under; nothing is published whose text names looks or carries a denied
+ * word; and a picture is only used when its licence allows it.
+ */
+const { boot, check, done } = require('./helpers/boot');
+const { createSources } = require('../server/classics/sources');
+const { createClassicsJob } = require('../server/classics/seed');
+
+const LIST_WIKITEXT = `== Films ==
+* [[Dog eats husband]]{{snd}}A joke about a woman who cooked her husband and fed him to her dogs.
+* [[Success Kid]]{{snd}}A picture of a child raising a barbell, used to describe a small success.
+* [[Hidden treasure map]]{{snd}}A map that leads to a hidden treasure.
+== Video ==
+* [[Nyan cat]]{{snd}}A cat with Pop-Tart wings flying through space with a rainbow trail.
+`;
+
+/** A stand-in for Wikipedia, Wikidata and Wikimedia Commons. */
+function wikiFetch(overrides = {}) {
+ return async (url) => {
+ const u = new URL(url);
+ const q = u.searchParams;
+ const action = q.get('action');
+ const body = (obj) => ({ ok: true, status: 200, text: async () => JSON.stringify(obj), json: async () => obj, headers: new Map() });
+
+ if (action === 'parse') {
+ if (q.get('page') === 'List_of_Internet_phenomena') return body({ parse: { wikitext: LIST_WIKITEXT } });
+ return body({ parse: { wikitext: '== Films ==\n* [[Dog eats husband]]{{snd}}A joke about a woman who cooked her husband and fed him to her dogs.\n' } });
+ }
+ if (action === 'query' && q.get('list') === 'categorymembers') return body({ query: { categorymembers: [] } });
+ if (action === 'query' && q.get('prop') === 'pageprops') {
+ const titles = String(q.get('titles') || '').split('|');
+ return body({
+ query: {
+ pages: titles.map((title) => ({ title, pageprops: { wikibase_item: overrides.qid || 'Q100' } })),
+ },
+ });
+ }
+ if (action === 'query' && q.get('prop') === 'extracts') {
+ const titles = String(q.get('titles') || '').split('|');
+ return body({
+ query: {
+ pages: titles.map((title) => ({
+ title,
+ extract: overrides.extract || 'The joke is a black-humour joke about a woman who cooked her husband and fed him to her dogs, told as a recipe.',
+ })),
+ },
+ });
+ }
+ if (action === 'query' && q.get('prop') === 'imageinfo') {
+ const titles = String(q.get('titles') || '').split('|');
+ return body({
+ query: {
+ pages: titles.map((title) => ({
+ title,
+ mime: 'image/jpeg',
+ imageinfo: [{
+ url: 'https://upload.wikimedia.org/wikipedia/commons/a/ab/Dog.jpg',
+ thumburl: 'https://upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Dog.jpg/800px-Dog.jpg',
+ extmetadata: {
+ LicenseShortName: { value: overrides.license || 'CC BY-SA 4.0' },
+ LicenseUrl: { value: 'https://creativecommons.org/licenses/by-sa/4.0/' },
+ Artist: { value: '<a href="https://commons.wikimedia.org/wiki/User:Someone">Someone</a>' },
+ },
+ }],
+ })),
+ },
+ });
+ }
+ if (action === 'wbgetentities') {
+ const ids = String(q.get('ids') || '').split('|');
+ return body({
+ entities: Object.fromEntries(ids.map((id) => [id, {
+ id,
+ labels: { en: { value: 'Dog eats husband' } },
+ descriptions: { en: { value: 'internet meme' } },
+ claims: { P571: [{ mainsnak: { datavalue: { value: { time: '+2007-00-00T00:00:00Z' } } } }], P18: [{ mainsnak: { datavalue: { value: 'Dog.jpg' } } }] },
+ sitelinks: { enwiki: { title: 'Dog eats husband' } },
+ }])),
+ });
+ }
+ return body({});
+ };
+}
+
+const answer = (score, extra = {}) => () => JSON.stringify({
+ score,
+ reason: `the AI scored it ${score}`,
+ title: 'Dog eats husband',
+ summary: 'A dark joke told as a recipe.',
+ origin: 'It comes from a 2007 image macro.',
+ spread: 'People tell it as a recipe, with the punchline in the last line.',
+ examples: ['recipe: 1 husband, 2 dogs', 'ingredients: one husband'],
+ ...extra,
+});
+
+async function seed(t, { answerFn = answer(0.9), fetchImpl = wikiFetch(), config = {} } = {}) {
+ const cfg = t.config;
+ const sources = createSources({ config: { ...cfg, encyclopedia: { ...cfg.encyclopedia, ...config } }, fetchImpl, log: console });
+ const judge = t.ctx.judge;
+ const job = createClassicsJob({ config: cfg, s: t.ctx.s, entries: t.ctx.entries, sources, judge, log: console });
+ // The stand-in AI answers every prompt with the same verdict.
+ t.ai.setAnswer(answerFn);
+ return await job.runOnce({ publish: true });
+}
+
+(async () => {
+ await check('the encyclopedia is seeded with entries of kind classic, cited and attributed', async () => {
+ const t = await boot();
+ try {
+ const out = await seed(t);
+ if (!out.seeded) throw new Error('nothing was seeded');
+ const rows = await t.ctx.s.db.many("SELECT * FROM meme_entries WHERE kind = 'classic' ORDER BY slug");
+ if (!rows.length) throw new Error('no classic entry was written');
+ const row = rows[0];
+ if (row.visibility !== 'public') throw new Error(`the entry is ${row.visibility}, not public`);
+ if (!row.attribution) throw new Error('the entry carries no attribution');
+ if (!/Wikipedia/.test(row.attribution) || !/Wikidata/.test(row.attribution)) throw new Error(`the attribution is ${row.attribution}`);
+ if (!row.image_license) throw new Error('the entry carries no licence');
+ if (!row.image_credit) throw new Error('the entry carries no credit');
+ if (!row.image_url || !/^https:\/\/upload\.wikimedia\.org\//.test(row.image_url)) throw new Error(`the picture is ${row.image_url}`);
+ const citations = await t.ctx.entries.citations(row.id);
+ if (citations.length < 3) throw new Error(`the entry has ${citations.length} citations`);
+ if (!citations.some((c) => c.source_type === 'wikipedia')) throw new Error('no Wikipedia citation');
+ if (!citations.some((c) => c.source_type === 'wikidata')) throw new Error('no Wikidata citation');
+ if (!citations.some((c) => c.role === 'origin')) throw new Error('no origin citation');
+ for (const c of citations) {
+ if (!/^https:\/\//.test(c.url)) throw new Error(`a citation has no public URL: ${c.url}`);
+ if (!c.quote_text) throw new Error('a citation has no quoted text');
+ }
+ } finally { await t.close(); }
+ });
+
+ await check('every seeded entry carries its attribution', async () => {
+ const t = await boot();
+ try {
+ await seed(t);
+ const rows = await t.ctx.s.db.many("SELECT id, attribution, image_license FROM meme_entries WHERE kind = 'classic'");
+ if (!rows.length) throw new Error('nothing was seeded');
+ for (const row of rows) {
+ if (!row.attribution) throw new Error(`entry ${row.id} has no attribution`);
+ if (!row.image_license) throw new Error(`entry ${row.id} has no licence`);
+ }
+ } finally { await t.close(); }
+ });
+
+ await check('nothing is published whose text names looks', async () => {
+ const t = await boot();
+ try {
+ await seed(t, { answerFn: answer(0.95, { summary: 'A man in a kitchen tells the joke.', title: 'Dog eats husband' }) });
+ const rows = await t.ctx.s.db.many("SELECT id, visibility, summary FROM meme_entries WHERE kind = 'classic'");
+ for (const row of rows) {
+ const { hasLooks } = require('openvibe-sdk/no-looks');
+ if (row.visibility === 'public' && hasLooks(row.summary || '')) throw new Error(`entry ${row.id} is public and its summary names looks`);
+ }
+ } finally { await t.close(); }
+ });
+
+ await check('nothing is published whose text carries a denied word', async () => {
+ const t = await boot();
+ try {
+ await seed(t, { answerFn: answer(0.95, { summary: 'The joke is about a nazi in a kitchen.' }) });
+ const rows = await t.ctx.s.db.many("SELECT id, visibility, summary FROM meme_entries WHERE kind = 'classic'");
+ for (const row of rows) {
+ if (row.visibility === 'public' && /nazi/i.test(row.summary || '')) throw new Error(`entry ${row.id} is public and its summary carries a denied word`);
+ }
+ } finally { await t.close(); }
+ });
+
+ await check('a classic the AI scored under the bar stays hidden, with its score', async () => {
+ const t = await boot();
+ try {
+ await seed(t, { answerFn: answer(0.4) });
+ const rows = await t.ctx.s.db.many("SELECT id, visibility FROM meme_entries WHERE kind = 'classic'");
+ if (!rows.length) throw new Error('nothing was seeded');
+ for (const row of rows) {
+ if (row.visibility !== 'hidden') throw new Error(`entry ${row.id} was published under the bar`);
+ }
+ const list = await t.ctx.entries.listHidden({ limit: 50 });
+ const judged = list.find((e) => e.kind === 'classic');
+ if (!judged) throw new Error('the classic is not on the staff list');
+ if (Number(judged.score) !== 0.4) throw new Error(`the staff list shows ${judged.score}`);
+ } finally { await t.close(); }
+ });
+
+ await check('a picture whose licence forbids reuse is not used', async () => {
+ const t = await boot();
+ try {
+ await seed(t, { fetchImpl: wikiFetch({ license: 'All rights reserved' }) });
+ const rows = await t.ctx.s.db.many("SELECT id, image_url FROM meme_entries WHERE kind = 'classic'");
+ for (const row of rows) {
+ if (row.image_url) throw new Error(`entry ${row.id} uses a picture under a licence that forbids reuse`);
+ }
+ } finally { await t.close(); }
+ });
+
+ await check('the seeded classics are public in the API', async () => {
+ const t = await boot();
+ try {
+ await seed(t);
+ const res = await t.get('/api/v1/entries?limit=50');
+ if (res.status !== 200) throw new Error(`the list answered ${res.status}`);
+ const classics = res.json().entries.filter((e) => e.kind === 'classic');
+ if (!classics.length) throw new Error('no classic entry is public');
+ if (!classics[0].attribution) throw new Error('the public entry carries no attribution');
+ } finally { await t.close(); }
+ });
+
+ await check('a classic the AI could not write is not seeded at all', async () => {
+ const t = await boot();
+ try {
+ await seed(t, { answerFn: answer(0.9, { title: '', summary: '', origin: '', spread: '', examples: [] }) });
+ const rows = await t.ctx.s.db.many("SELECT id, visibility FROM meme_entries WHERE kind = 'classic'");
+ for (const row of rows) {
+ if (row.visibility === 'public') throw new Error(`entry ${row.id} was published without a written entry`);
+ }
+ } finally { await t.close(); }
+ });
+
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,78 @@
@@ -0,0 +1,78 @@
+'use strict';
+/**
+ * An in-process stand-in for OpenVibe.AI. It answers POST /api/v1/generate with one run, exactly the shape the
+ * real service returns: { run: { id, status, output: { text }, provenance, usage } }.
+ *
+ * The answer is a function of the prompt, so a test can say "this candidate scores 0.9" or "this one scores 0.2",
+ * and it can read the prompt back to check that the cited material really reached the model. Every call is recorded.
+ */
+const http = require('http');
+
+const DEFAULT_ANSWER = JSON.stringify({
+ score: 0.9,
+ reason: 'People play off this on purpose; it has a shared meaning in the cited uses.',
+ title: 'The robot is offline',
+ summary: 'A running joke about the robot being offline again.',
+ origin: 'It starts in the first cited chat line.',
+ spread: 'People say it whenever the robot drops off, as a call back to the stream.',
+ examples: ['the robot is offline again', 'still the robot is offline, huh'],
+});
+
+function listen(handler) {
+ return new Promise((resolve) => {
+ const sockets = new Set();
+ const server = http.createServer((req, res) => {
+ const chunks = [];
+ req.on('data', (c) => chunks.push(c));
+ req.on('end', () => {
+ const raw = Buffer.concat(chunks).toString('utf8');
+ const json = (status, obj) => { res.writeHead(status, { 'Content-Type': 'application/json' }); res.end(JSON.stringify(obj)); };
+ Promise.resolve(handler(req, raw, json)).catch((err) => json(500, { error: String(err && err.message) }));
+ });
+ });
+ server.on('connection', (socket) => { sockets.add(socket); socket.on('close', () => sockets.delete(socket)); });
+ server.listen(0, '127.0.0.1', () => resolve({
+ server,
+ url: `http://127.0.0.1:${server.address().port}`,
+ close: () => new Promise((r) => { for (const s of sockets) s.destroy(); server.close(r); }),
+ }));
+ });
+}
+
+async function startAi({ answer } = {}) {
+ const state = { calls: [], answer: answer || (() => DEFAULT_ANSWER), runs: 0 };
+
+ const srv = await listen(async (req, raw, json) => {
+ const url = new URL(req.url, 'http://x');
+ if (url.pathname === '/api/v1/generate' && req.method === 'POST') {
+ const body = JSON.parse(raw || '{}');
+ state.calls.push({ prompt: body.prompt, system: body.system, target: body.target });
+ state.runs += 1;
+ const text = await state.answer(body.prompt, body);
+ return json(200, {
+ run: {
+ id: `run_${state.runs}`,
+ status: 'succeeded',
+ output: { text: String(text) },
+ provenance: { provider: 'mock', model: 'mock-model' },
+ usage: { tokens: { input: 100, output: 50 }, cost_usd: 0.001 },
+ },
+ });
+ }
+ if (url.pathname.startsWith('/api/v1/runs/')) {
+ const id = url.pathname.split('/').pop();
+ return json(200, { run: { id, status: 'succeeded', output: { text: DEFAULT_ANSWER } } });
+ }
+ return json(404, { error: 'not found' });
+ });
+
+ return {
+ url: srv.url,
+ state,
+ calls: state.calls,
+ setAnswer(fn) { state.answer = fn; },
+ close: srv.close,
+ };
+}
+
+module.exports = { startAi, DEFAULT_ANSWER };
@@ -16,6 +16,7 @@ process.env.MEME_LIMITS_HOUR = process.env.MEME_LIMITS_HOUR || '100000';
@@ -16,6 +16,7 @@ process.env.MEME_LIMITS_HOUR = process.env.MEME_LIMITS_HOUR || '100000';
const http = require('http');
const http = require('http');
const { startNetwork } = require('./mocks');
const { startNetwork } = require('./mocks');
const { startUpstreams } = require('./upstreams');
const { startUpstreams } = require('./upstreams');
+const { startAi } = require('./ai');
const captured = [];
const captured = [];
for (const m of ['log', 'info', 'warn', 'error']) {
for (const m of ['log', 'info', 'warn', 'error']) {
@@ -26,6 +27,7 @@ for (const m of ['log', 'info', 'warn', 'error']) {
@@ -26,6 +27,7 @@ for (const m of ['log', 'info', 'warn', 'error']) {
async function boot(opts = {}) {
async function boot(opts = {}) {
const network = await startNetwork(opts.network || {});
const network = await startNetwork(opts.network || {});
const upstreams = await startUpstreams(opts.upstreams || {});
const upstreams = await startUpstreams(opts.upstreams || {});
+ const ai = await startAi(opts.ai || {});
const env = {
const env = {
NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.meme', TRUST_PROXY: '1',
NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.meme', TRUST_PROXY: '1',
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
@@ -33,6 +35,7 @@ async function boot(opts = {}) {
@@ -33,6 +35,7 @@ async function boot(opts = {}) {
MEME_DRILL: '1',
MEME_DRILL: '1',
MEME_EVENTS_URL: network.url,
MEME_EVENTS_URL: network.url,
OV_CHAT_URL: upstreams.url, OV_LIVE_URL: upstreams.url, OV_VIDEO_URL: upstreams.url,
OV_CHAT_URL: upstreams.url, OV_LIVE_URL: upstreams.url, OV_VIDEO_URL: upstreams.url,
+ OV_AI_INTERNAL_URL: ai.url,
...(opts.env || {}),
...(opts.env || {}),
};
};
for (const [k, v] of Object.entries(opts.env || {})) if (v === null) delete env[k];
for (const [k, v] of Object.entries(opts.env || {})) if (v === null) delete env[k];
@@ -75,17 +78,19 @@ async function boot(opts = {}) {
@@ -75,17 +78,19 @@ async function boot(opts = {}) {
}
}
const t = {
const t = {
- base, network, upstreams, config, ctx: built.ctx, get, signIn, dbDump,
+ base, network, upstreams, ai, config, ctx: built.ctx, get, signIn, dbDump,
logs: () => captured.join('\n'),
logs: () => captured.join('\n'),
async close() {
async close() {
built.ctx.importer.stop();
built.ctx.importer.stop();
built.ctx.reverifier.stop();
built.ctx.reverifier.stop();
+ built.ctx.judgeJob.stop();
await new Promise((r) => server.close(r));
await new Promise((r) => server.close(r));
built.ctx.keys.client.stop();
built.ctx.keys.client.stop();
await built.ctx.telemetry.stop().catch(() => {});
await built.ctx.telemetry.stop().catch(() => {});
await testdb.close();
await testdb.close();
await network.close();
await network.close();
await upstreams.close();
await upstreams.close();
+ await ai.close();
},
},
};
};
return t;
return t;
new file mode 100644
new file mode 100644
@@ -0,0 +1,237 @@
@@ -0,0 +1,237 @@
+'use strict';
+/**
+ * The judge and the publish rule.
+ *
+ * A candidate is judged from its own citations by OpenVibe.AI (a stand-in here), and only what clears the bar is
+ * published: 0.75 or more, used at least three times across at least two days or two channels, nothing denied,
+ * nothing about anyone's looks. Everything else stays hidden for staff, with its score, and every decision is a
+ * status event.
+ */
+const { boot, check, done } = require('./helpers/boot');
+
+const at = (days) => new Date(Date.UTC(2026, 9, 20 + days, 12, 0, 0)).toISOString();
+const CHAT_IDS = [1001, 1002, 86960]; // the lines the in-process Chat stand-in holds
+
+/** A candidate with its uses: each use is a chat line (or a VOD second) on its own day. */
+async function candidate(t, { slug, title, uses, channels = ['chat'] }) {
+ const cites = uses.map((u, i) => {
+ const channel = channels[Math.min(i, channels.length - 1)];
+ const live = channel === 'live';
+ return {
+ role: i === 0 ? 'origin' : 'use',
+ source_service: channel,
+ source_type: live ? 'vod' : 'chat_message',
+ source_id: `${slug}-${i}`,
+ chat_message_id: live ? null : CHAT_IDS[i % CHAT_IDS.length],
+ vod_id: live ? 9001 : null,
+ offset_seconds: live ? 60 + i : null,
+ quote_text: `${title} - line ${i + 1}`,
+ url: live ? 'https://openvibe.video/v/9001?t=60' : 'https://openvibe.chat/',
+ observed_at: u,
+ };
+ });
+ const out = await t.ctx.entries.createCandidate({
+ slug, title, kind: 'catchphrase', summary: 'a template summary', status: 'rising',
+ first_seen_at: uses[0], last_seen_at: uses[uses.length - 1], visibility: 'hidden', citations: cites,
+ });
+ return out.entry;
+}
+
+const answer = (score, extra = {}) => (prompt) => JSON.stringify({
+ score,
+ reason: `the AI scored it ${score}`,
+ title: 'Cozmo is alive',
+ summary: 'A running joke about the robot being back.',
+ origin: 'It starts in the first cited line.',
+ spread: 'People say it when the robot comes back.',
+ examples: ['the robot is alive', 'cozmo is alive again'],
+ ...extra,
+});
+
+(async () => {
+ await check('the judge reads the cited material: the quote and the lines around it reach the model', async () => {
+ const t2 = await boot();
+ try {
+ await candidate(t2, { slug: 'robot-offline', title: 'The robot is offline', uses: [at(0), at(2), at(3)] });
+ await t2.ctx.judgeJob.runOnce({ force: true });
+ const call = t2.ai.calls[0];
+ if (!call) throw new Error('the judge did not call the AI');
+ if (!/the robot is offline - line 1/i.test(call.prompt)) throw new Error('the prompt did not carry the cited quote');
+ if (!/Use 1/.test(call.prompt) || !/Use 3/.test(call.prompt)) throw new Error('the prompt did not carry every use');
+ if (!/ada:/.test(call.prompt)) throw new Error('the prompt did not carry the chat lines around the quote');
+ } finally { await t2.close(); }
+ });
+
+ await check('the judge stores the score, the reason and the written entry', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.93) } });
+ try {
+ const entry = await candidate(t2, { slug: 'cozmo-s-alive', title: 'Cozmo s alive', uses: [at(0), at(2), at(3)] });
+ await t2.ctx.judgeJob.runOnce({ force: true });
+ const verdict = await t2.ctx.entries.judgement(entry.id);
+ if (!verdict) throw new Error('no judgement was stored');
+ if (Number(verdict.score) !== 0.93) throw new Error(`the score is ${verdict.score}`);
+ if (!verdict.reason) throw new Error('no reason was stored');
+ if (verdict.origin !== 'It starts in the first cited line.') throw new Error('the origin was not stored');
+ if (verdict.spread !== 'People say it when the robot comes back.') throw new Error('the spread was not stored');
+ if (!Array.isArray(verdict.examples) || verdict.examples.length !== 2) throw new Error('the examples were not stored');
+ const after = await t2.ctx.entries.byId(entry.id);
+ if (after.title !== 'Cozmo is alive') throw new Error(`the AI title was not used: ${after.title}`);
+ } finally { await t2.close(); }
+ });
+
+ await check('a candidate that clears the bar is published', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.9) } });
+ try {
+ const entry = await candidate(t2, { slug: 'yes-please', title: 'Yes please', uses: [at(0), at(2), at(3)] });
+ const out = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (out.published !== 1) throw new Error(`expected one published entry, got ${out.published}`);
+ const after = await t2.ctx.entries.byId(entry.id);
+ if (after.visibility !== 'public') throw new Error('the entry is not public');
+ } finally { await t2.close(); }
+ });
+
+ await check('the rule: a score under 0.75 keeps the entry hidden', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.74) } });
+ try {
+ const entry = await candidate(t2, { slug: 'weak-phrase', title: 'Yeah thats true', uses: [at(0), at(2), at(3)] });
+ const out = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (out.published !== 0) throw new Error('a weak entry was published');
+ const after = await t2.ctx.entries.byId(entry.id);
+ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');
+ const list = await t2.ctx.entries.listHidden({ limit: 10 });
+ const row = list.find((e) => e.id === entry.id);
+ if (!row) throw new Error('the entry is not on the staff list');
+ if (Number(row.score) !== 0.74) throw new Error(`the staff list shows ${row.score}`);
+ } finally { await t2.close(); }
+ });
+
+ await check('the rule: fewer than three uses keeps the entry hidden', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.95) } });
+ try {
+ const entry = await candidate(t2, { slug: 'twice-only', title: 'Twice only', uses: [at(0), at(2)] });
+ const out = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (out.published !== 0) throw new Error('a twice-used entry was published');
+ const after = await t2.ctx.entries.byId(entry.id);
+ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');
+ } finally { await t2.close(); }
+ });
+
+ await check('the rule: three uses on one day in one channel keep the entry hidden', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.95) } });
+ try {
+ const entry = await candidate(t2, { slug: 'one-day', title: 'Same day', uses: [at(0), at(0), at(0)] });
+ const out = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (out.published !== 0) throw new Error('an entry used on one day was published');
+ } finally { await t2.close(); }
+ });
+
+ await check('the rule: three uses across two channels clear the bar even on one day', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.95) } });
+ try {
+ const entry = await candidate(t2, { slug: 'two-channels', title: 'Both places', uses: [at(0), at(0), at(0)], channels: ['chat', 'live'] });
+ const out = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (out.published !== 1) throw new Error('an entry used in two channels was not published');
+ const after = await t2.ctx.entries.byId(entry.id);
+ if (after.visibility !== 'public') throw new Error('the entry is not public');
+ } finally { await t2.close(); }
+ });
+
+ await check('the rule: a denied source is never stored and never published', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.99) } });
+ try {
+ const out = await t2.ctx.entries.createCandidate({
+ slug: 'denied-one', title: 'Denied one', kind: 'meme', status: 'rising',
+ first_seen_at: at(0), last_seen_at: at(3), visibility: 'hidden',
+ citations: [
+ { role: 'origin', source_service: 'live', source_type: 'vod', source_id: '5709', vod_id: 5709, offset_seconds: 10, quote_text: 'the robot is offline', url: 'https://openvibe.video/v/5709?t=10', observed_at: at(0) },
+ { role: 'use', source_service: 'chat', source_type: 'chat_message', source_id: '1', chat_message_id: 1001, quote_text: 'the robot is offline', url: 'https://openvibe.chat/', observed_at: at(2) },
+ { role: 'use', source_service: 'chat', source_type: 'chat_message', source_id: '2', chat_message_id: 1002, quote_text: 'the robot is offline', url: 'https://openvibe.chat/', observed_at: at(3) },
+ ],
+ });
+ const stored = await t2.ctx.entries.citations(out.entry.id);
+ if (stored.length !== 2) throw new Error(`the denied citation was stored (${stored.length} citations)`);
+ if (stored.some((c) => c.vod_id === 5709)) throw new Error('the denied VOD is in the citations');
+ const judged = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (judged.published !== 0) throw new Error('an entry citing a denied VOD was published');
+ const after = await t2.ctx.entries.byId(out.entry.id);
+ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');
+ } finally { await t2.close(); }
+ });
+
+ await check('the rule: text that names looks never reaches the public', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.99, { title: 'Cozmo in his blue shirt', summary: 'A man in a kitchen says the robot is offline.' }) } });
+ try {
+ const entry = await candidate(t2, { slug: 'looks-one', title: 'Looks one', uses: [at(0), at(2), at(3)] });
+ const out = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (out.published !== 0) throw new Error('an entry whose text names looks was published');
+ const after = await t2.ctx.entries.byId(entry.id);
+ if (after.visibility !== 'hidden') throw new Error('the entry is not hidden');
+ if (after.title === 'Cozmo in his blue shirt') throw new Error('the title that named looks was used');
+ const verdict = await t2.ctx.entries.judgement(entry.id);
+ if (!verdict.looks) throw new Error('the judgement does not say the text named looks');
+ } finally { await t2.close(); }
+ });
+
+ await check('the rule: text that carries a denied word never reaches the public', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.99, { summary: 'The joke is about a nazi in a kitchen.' }) } });
+ try {
+ const entry = await candidate(t2, { slug: 'denied-word', title: 'Denied word', uses: [at(0), at(2), at(3)] });
+ const out = await t2.ctx.judgeJob.runOnce({ force: true });
+ if (out.published !== 0) throw new Error('an entry with a denied word was published');
+ const verdict = await t2.ctx.entries.judgement(entry.id);
+ if (!verdict.denied) throw new Error('the judgement does not say the text is denied');
+ } finally { await t2.close(); }
+ });
+
+ await check('every decision is a status event', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.9) } });
+ try {
+ const entry = await candidate(t2, { slug: 'logged-one', title: 'Logged one', uses: [at(0), at(2), at(3)] });
+ await t2.ctx.judgeJob.runOnce({ force: true });
+ const history = await t2.ctx.entries.decisions(entry.id);
+ if (!history.length) throw new Error('no status event was written');
+ const decision = history[0];
+ if (decision.decision !== 'publish') throw new Error(`the decision is ${decision.decision}`);
+ if (decision.from_visibility !== 'hidden' || decision.to_visibility !== 'public') throw new Error('the visibility move was not recorded');
+ if (Number(decision.score) !== 0.9) throw new Error(`the event carries the score ${decision.score}`);
+ } finally { await t2.close(); }
+ });
+
+ await check('a kept candidate is logged too, with the reasons', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.3) } });
+ try {
+ const entry = await candidate(t2, { slug: 'kept-one', title: 'Kept one', uses: [at(0), at(2), at(3)] });
+ await t2.ctx.judgeJob.runOnce({ force: true });
+ const history = await t2.ctx.entries.decisions(entry.id);
+ if (!history.length) throw new Error('no status event was written');
+ if (history[0].decision !== 'keep_hidden') throw new Error(`the decision is ${history[0].decision}`);
+ if (!/scored 0.30/.test(history[0].reason || '')) throw new Error(`the reason does not say why: ${history[0].reason}`);
+ } finally { await t2.close(); }
+ });
+
+ await check('the staff list shows the hidden candidates with their score', async () => {
+ const t2 = await boot({ ai: { answer: answer(0.42) } });
+ try {
+ const entry = await candidate(t2, { slug: 'staff-one', title: 'Staff one', uses: [at(0), at(2), at(3)] });
+ await t2.ctx.judgeJob.runOnce({ force: true });
+ const res = await t2.get('/api/v1/staff/candidates', { as: t2.network.addUser('mod', { role: 'global_mod' }) });
+ if (res.status !== 200) throw new Error(`the staff list answered ${res.status}`);
+ const body = res.json();
+ const row = (body.candidates || []).find((c) => c.id === entry.id);
+ if (!row) throw new Error('the candidate is not on the staff list');
+ if (Number(row.score) !== 0.42) throw new Error(`the list shows the score ${row.score}`);
+ if (!row.reason) throw new Error('the list shows no reason');
+ if (row.uses !== 3) throw new Error(`the list shows ${row.uses} uses`);
+ } finally { await t2.close(); }
+ });
+
+ await check('the staff list is not for anonymous visitors', async () => {
+ const t = await boot();
+ try {
+ const res = await t.get('/api/v1/staff/candidates');
+ if (res.status !== 401) throw new Error(`the staff list answered ${res.status} to an anonymous caller`);
+ } finally { await t.close(); }
+ });
+
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,96 @@
@@ -0,0 +1,96 @@
+'use strict';
+/**
+ * The templates: the well-known meme templates from Wikimedia Commons. Each one keeps its licence, its credit and
+ * the file page it came from, and a file whose licence forbids reuse is never seeded.
+ */
+const { boot, check, done } = require('./helpers/boot');
+const { createTemplatesJob } = require('../server/templates/seed');
+
+/** A stand-in for Wikimedia Commons. */
+function commonsFetch({ license = 'CC BY-SA 4.0', mime = 'image/jpeg' } = {}) {
+ return async (url) => {
+ const u = new URL(url);
+ const q = u.searchParams;
+ const body = (obj) => ({ ok: true, status: 200, text: async () => JSON.stringify(obj), headers: new Map() });
+ if (q.get('list') === 'search') {
+ return body({ query: { search: [{ title: 'File:Drake Hotline Bling.jpg' }, { title: 'File:Other.jpg' }] } });
+ }
+ if (q.get('prop') === 'imageinfo') {
+ const titles = String(q.get('titles') || '').split('|');
+ return body({
+ query: {
+ pages: titles.map((title) => ({
+ title,
+ imageinfo: [{
+ mime,
+ url: 'https://upload.wikimedia.org/wikipedia/commons/a/ab/Drake.jpg',
+ thumburl: 'https://upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Drake.jpg/800px-Drake.jpg',
+ extmetadata: {
+ LicenseShortName: { value: license },
+ LicenseUrl: { value: 'https://creativecommons.org/licenses/by-sa/4.0/' },
+ Artist: { value: '<a href="https://commons.wikimedia.org/wiki/User:Someone">Someone</a>' },
+ },
+ }],
+ })),
+ },
+ });
+ }
+ return body({});
+ };
+}
+
+(async () => {
+ await check('a template is seeded with its licence, its credit and its source page', async () => {
+ const t = await boot();
+ try {
+ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch(), log: console });
+ const out = await job.runOnce({ limit: 1 });
+ if (!out.seeded) throw new Error('no template was seeded');
+ const rows = await t.ctx.s.db.many('SELECT * FROM meme_templates ORDER BY created_at');
+ const row = rows.find((r) => r.id.startsWith('tpl_commons_'));
+ if (!row) throw new Error('the seeded template is not in the table');
+ if (!row.license) throw new Error('the template carries no licence');
+ if (!row.credit) throw new Error('the template carries no credit');
+ if (!row.source_url || !/^https:\/\/commons\.wikimedia\.org\//.test(row.source_url)) throw new Error(`the source page is ${row.source_url}`);
+ if (!/^https:\/\/upload\.wikimedia\.org\//.test(row.image_url)) throw new Error(`the picture is ${row.image_url}`);
+ if (!row.attribution) throw new Error('the template carries no attribution');
+ } finally { await t.close(); }
+ });
+
+ await check('the templates are listed with their attribution', async () => {
+ const t = await boot();
+ try {
+ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch(), log: console });
+ await job.runOnce({ limit: 1 });
+ const res = await t.get('/api/v1/templates');
+ if (res.status !== 200) throw new Error(`the templates answered ${res.status}`);
+ const body = res.json();
+ const seeded = body.templates.find((x) => x.id.startsWith('tpl_commons_'));
+ if (!seeded) throw new Error('the seeded template is not listed');
+ if (!seeded.license) throw new Error('the listed template carries no licence');
+ if (!seeded.credit) throw new Error('the listed template carries no credit');
+ } finally { await t.close(); }
+ });
+
+ await check('a file whose licence forbids reuse is never seeded', async () => {
+ const t = await boot();
+ try {
+ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch({ license: 'All rights reserved' }), log: console });
+ const out = await job.runOnce({ limit: 3 });
+ if (out.seeded) throw new Error(`${out.seeded} template(s) were seeded from a file that forbids reuse`);
+ const rows = await t.ctx.s.db.many("SELECT id FROM meme_templates WHERE id LIKE 'tpl_commons_%'");
+ if (rows.length) throw new Error('a template was written from a file that forbids reuse');
+ } finally { await t.close(); }
+ });
+
+ await check('a file that is not a picture is never seeded', async () => {
+ const t = await boot();
+ try {
+ const job = createTemplatesJob({ config: t.config, entries: t.ctx.entries, fetchImpl: commonsFetch({ mime: 'video/mp4' }), log: console });
+ const out = await job.runOnce({ limit: 3 });
+ if (out.seeded) throw new Error(`${out.seeded} template(s) were seeded from a file that is not a picture`);
+ } finally { await t.close(); }
+ });
+
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });