openvibers/openvibe.sources Commits
Sources: GitHub repositories (code), star history and the code API (#284)
Changes
@@@ -85,8 +85,10 @@ VALKEY_PREFIX=#SOURCES_MAX_ITEMS_PER_FETCH=12000# Data-source keys (D11 owner settings): the admin panel's sources.data_sources namespace wins; this--# is the fallback when no value was entered there.++# is the fallback when no value was entered there. GITHUB_TOKEN is the last fallback (a read-only token:++# 30 search requests a minute instead of 10; the name only, never the value).#SOURCES_GITHUB_TOKEN=++#GITHUB_TOKEN=# Source credentials: a registry entry names the variable (auth.env), never the value. Names# must start with SOURCES_CRED_. Example (none of the seeds needs one):@@@ -57,10 +57,12 @@ start and flushes on stop.## The registryA source records: `key`, `name`, `type` (`rss|atom|sitemap|jsonld|api|bulk|crawl|manual`), `category`--(`news|blog|reviews|deals|coupons|trade|hosting|crowdfunding|games|housing|food|jobs`, read from `sources.source@1`; housing, food and jobs since contracts 0.210.0), `endpoints` (for `api`: a field mapping, no code per++(`news|blog|reviews|deals|coupons|trade|hosting|crowdfunding|games|housing|food|jobs|code`, read from `sources.source@1`; housing, food and jobs since contracts 0.210.0, code since 0.411.1), `endpoints` (for `api`: a field mapping, no code perprovider; for `bulk`: one dataset file with a format, a byte cap and a row limit), `auth` (`none|header|bearer|query` with the **name** of an environment variable —names must start with `SOURCES_CRED_`, so an entry can never point at this service's own--secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`),++secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`) — the GitHub read uses++`GITHUB_TOKEN` (or `SOURCES_GITHUB_TOKEN`, or the owner's D11 setting) by name when set, and is anonymous within++GitHub's 10-a-minute search limit otherwise —`poll_interval_sec`, `stale_after_sec`, `max_items`, `enabled`, `review_required`,`sensitivity`, `default_indexability`, `search_visibility` and `retire_after_sec` (since contracts 0.211.0).@@@ -108,7 +110,7 @@ succeeded. Health status: `healthy | stale | failing | never_fetched | disabled| `rss`, `atom` (`feed@3`) | RSS 2.0, RSS 1.0/RDF, Atom 1.0 | guid/id identity, else link; text-only summary capped at 1000 chars; a deal feed's own offer elements (DealNews' `dealnews:price` with its currency, `dealnews:retailer`, `dealnews:expires`) become `fields.price`, `currency`, `seller` and `valid_until` exactly as stated (the item stays an article; nothing is read out of the headline); its `dealnews:category` fills `fields.categories` when the item has no `<category>`, and `dealnews:staffPick` is `fields.staff_pick` (true/false, absent when not stated) || `sitemap` (`sitemap@1`) | `<urlset>` (+ Google News extension), `<sitemapindex>` | index children are listed, not followed || `jsonld` (`jsonld@2`) | schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) | prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error`; `list_price` from a ListPrice/StrikethroughPrice/MSRP/SRP `priceSpecification`, as stated |--| `api` (`api@1`) | JSON, XML, TSV or CSV official files via a registry mapping | `items_path`, `items_shape` (`array` default, `map` for records keyed by id), `fields`, `extra`, `where`/`filter`; epoch fields only where mapped as such; `resolver: "hn-whoishiring"` walks the Hacker News thread's JSON documents instead |++| `api` (`api@1`) | JSON, XML, TSV or CSV official files via a registry mapping | `items_path`, `items_shape` (`array` default, `map` for records keyed by id), `fields`, `extra`, `where`/`filter`; epoch fields only where mapped as such; `resolver: "hn-whoishiring"` walks the Hacker News thread's JSON documents instead; `resolver: "github"` walks GitHub's repository search plan (server/github.js) || `bulk` (`bulk@1`) | one dataset file, streamed (CSV, JSONL or TSV) | `max_bytes` is the size cap, `max_rows` the per-run row limit; the stream stops at the limit, so a multi-GB file is never held in memory. `resolver: "off-delta"` reads a text index and only the gzipped JSONL delta parts it lists, keeping the last finished part and the current part's row offset in `source_state` (Open Food Facts) || `crawl` (`jsonld@2`, `page@1`) | a whole site from its seeds: its sitemaps and, if asked, its links (see below) | only pages that publish JSON-LD are kept, unless `require_structured: false` keeps a `page` item from the page's own search metadata || `manual` (`manual@1`) | items entered through the API with an evidence URL | no fetcher |@@@ -261,7 +263,7 @@ the endpoint's last good parse saw. At most 500 items are retired a run; the res[`sources.item@1`](docs/contracts-proposal/).**Retention.** Items do not grow forever: `server/retention.js` declares, in one place, a window per category (news 30--days, blog 90, reviews 180, deals 14, coupons 90, trade 365, hosting 30, crowdfunding 90, games 14, food 365, jobs 14)++days, blog 90, reviews 180, deals 14, coupons 90, trade 365, hosting 30, crowdfunding 90, games 14, food 365, jobs 14, code 30)and one `keepLatest` rule for housing, where HUD's yearly Fair Market Rents must keep the latest value per source key.An item's age is `retrieved_at` while it is live and `removed_at` once removed, so a consumer following the change feedstill sees a removal for the full window; `first_seen_at` is never the age, since a source that keeps listing a job or@@@ -283,7 +285,7 @@ service calls them with its service token and `X-OV-Subject: usr_…`.| Capability | Routes ||---|---|| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health`; `GET /api/v1/merchant/sources[/:id]` |--| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |++| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1`; `GET /api/v1/code/repos?list=trending\|week\|month\|hot\|new\|top&lang=&topic=&limit=`, `GET /api/v1/code/search?q=&lang=&sort=stars\|updated\|best&page=` || `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}`; `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` |`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —@@@ -367,7 +369,7 @@ for the N-1 rollback window.## Seeds--[seeds/sources.json](seeds/sources.json) holds the 67 registered sources, each with its robots, terms and licence notes (the production rows are the record; the terms notes match them as of the 2026-10-23 review).++[seeds/sources.json](seeds/sources.json) holds the 69 registered sources, each with its robots, terms and licence notes (the production rows are the record; the terms notes match them as of the 2026-10-23 review).`npm run seed` creates the ones that do not exist yet, **all disabled**, and never changes an existing one; a personenables a source after reading its terms (`PATCH /api/v1/sources/:key {"enabled": true}`), and the fetcher re-checksrobots.txt on every run. Production on 2026-10-22 (the food row checked 2026-11-06):@@@ -384,6 +386,7 @@ robots.txt on every run. Production on 2026-10-22 (the food row checked 2026-11-| crowdfunding | `opencollective-accounts` (api) | 1 of 1 || games | `steam-top-sellers`, `steam-new-releases`, `steam-specials` (api: the store's featured lists, in their order) | 3 of 3 || food | `open-food-facts`, `openfda-food-enforcement`, `bls-average-prices`, `usda-ers-fruit-vegetable-prices` (api); `themealdb` | 4 of 5: `themealdb` is off (it needs a key, which is the owner's call); `usda-ers-fruit-vegetable-prices` was enabled on 2026-11-06 after its terms review, and its 155 prices are recorded in `test/fixtures/` |++| code | `github` (api, `resolver: "github"`: REST search only, star history per day) | seeded disabled: enable after the terms review |Candidates rejected on 2026-09-22: CheapShark's API (`/api/1.0/` is disallowed in itsrobots.txt) and EDGAR's `getcurrent` Atom feed (`/cgi-bin` disallowed). The seeded feeds were@@@ -9,7 +9,7 @@"properties": {"id": { "type": "string", "pattern": "^itm_[0-9A-HJKMNP-TV-Z]{26}$" },"source_key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },-- "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade", "hosting", "crowdfunding", "games", "housing", "food", "jobs"] },++ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade", "hosting", "crowdfunding", "games", "housing", "food", "jobs", "code"] },"kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$", "description": "article | url | sitemap | product | offer | review | review_signal | record | coupon …" },"identity": { "type": "string", "minLength": 1, "maxLength": 2048, "description": "The source's own id (guid, @id, API id) or the canonical URL." },"canonical_url": { "type": ["string", "null"], "format": "uri" },@@@ -56,7 +56,8 @@"games","housing","food",-- "jobs"++ "jobs",++ "code"]},"homepage_url": {@@@ -166,9 +167,10 @@"resolver": {"enum": ["hn-whoishiring",-- "off-delta"++ "off-delta",++ "github"],-- "description": "An api endpoint whose records are walked from several JSON URLs instead of one items_path, or a bulk endpoint whose rows come from the gzipped JSONL parts a text index lists. hn-whoishiring reads user/whoishiring.json, the latest \"Who is hiring\" story's top-level kids, and each item/<id>.json; it stores one job per comment, a short excerpt with emails and phones removed, and links the comment (Sources, 2026-10-31). off-delta reads index.txt and then only the delta parts not finished yet, resuming a part at its stored row offset (Open Food Facts; Sources, 2026-10-31). The endpoint URL is a .json URL for hn-whoishiring and the index's .txt URL for off-delta."++ "description": "An api endpoint whose records are walked from several JSON URLs instead of one items_path, or a bulk endpoint whose rows come from the gzipped JSONL parts a text index lists. hn-whoishiring reads user/whoishiring.json, the latest \"Who is hiring\" story's top-level kids, and each item/<id>.json; it stores one job per comment, a short excerpt with emails and phones removed, and links the comment (Sources, 2026-10-31). off-delta reads index.txt and then only the delta parts not finished yet, resuming a part at its stored row offset (Open Food Facts; Sources, 2026-10-31). The endpoint URL is a .json URL for hn-whoishiring and the index's .txt URL for off-delta. github walks a share of a search plan on api.github.com/search/repositories per run (new repositories today, this week and this month by stars, recently pushed, all-time top, and twelve languages), keeps a star count per repository per day, and returns repository items; its endpoint URL is an api.github.com URL and never a github.com page (Sources, 2026-11-12)."},"item_kind": {"type": "string",nnew file mode 100644@@@ -0,0 +1,14 @@++-- phase: expand++-- A GitHub repository's star count per day (server/adapters/github.js, server/github.js). The code API computes++-- "stars gained over 1 day / 7 days" from these snapshots, so a repository's age in the tracked set is visible:++-- a repository first seen yesterday has a 1-day gain and a null 7-day gain, never a fabricated zero.++-- One row per source, repository and UTC day; a later sighting the same day replaces the count.++CREATE TABLE IF NOT EXISTS github_stars (++ source_key text COLLATE "C" NOT NULL,++ full_name text COLLATE "C" NOT NULL,++ day text COLLATE "C" NOT NULL,++ stars integer NOT NULL CHECK (stars >= 0),++ captured_at bigint NOT NULL,++ PRIMARY KEY (source_key, full_name, day)++);++CREATE INDEX IF NOT EXISTS idx_github_stars_day ON github_stars (source_key, day);@@@ -12,7 +12,7 @@"dotenv": "^16.4.7","express": "^4.21.2","fast-xml-parser": "^4.5.3",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.410.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.411.1","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.150.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.214.0","pg": "^8.23.0"@@@ -1125,9 +1125,9 @@}},"node_modules/openvibe-contracts": {-- "version": "0.410.0",-- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.410.0",-- "integrity": "sha512-rmmq2mebXim2fv+sip2YQ0PrrZLxGUhEfe1w+1EFfGdoVvLI5tp0/eavY2sW6/VU+rzs7cQMC/A0NZD5th5wrA==",++ "version": "0.411.1",++ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.411.1",++ "integrity": "sha512-WuIkcAdEaGwsz6Wsyt1Ear2hGLm4vr+NSait0MEVtJBiinwGBU7AgBTGFnUVCMRg4V40EfJZSYXXk7g4buduCA==","license": "AGPL-3.0-only","dependencies": {"ajv": "^8.17.1",@@@ -30,7 +30,7 @@"dotenv": "^16.4.7","express": "^4.21.2","fast-xml-parser": "^4.5.3",-- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.410.0",++ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.411.1","openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.150.0","openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.214.0","pg": "^8.23.0"@@@ -2604,6 +2604,34 @@"review_required": true,"sensitivity": "none","default_indexability": "noindex"++ },++ {++ "key": "github",++ "name": "GitHub: trending, top and new repositories",++ "type": "api",++ "category": "code",++ "homepage_url": "https://github.com/",++ "endpoints": [++ {++ "url": "https://api.github.com/search/repositories?q=stars:%3E10000&sort=stars&order=desc&per_page=100",++ "format": "json",++ "resolver": "github",++ "item_kind": "repository",++ "max_bytes": 4194304,++ "timeout_ms": 30000++ }++ ],++ "robots_note": "api.github.com/robots.txt checked 2026-11-12 from openvibe-ovh: HTTP 404, so RFC 9309 leaves the REST API unrestricted (4xx means no rules). No request goes to github.com or its /trending page; only api.github.com/search/repositories. The fetcher re-checks robots.txt on every run and on every redirect hop.",++ "terms_note": "GitHub's REST API is documented for automated use (https://docs.github.com/rest/search/search#search-repositories) and the Acceptable Use Policies allow reading public repository metadata through the API while forbidding scraping the website or circumventing rate limits (https://docs.github.com/site-policy/acceptable-use-policies/github-acceptable-use-policies). This source reads only the search API, passes no personal data, stores only what a public repository states about itself (full name, URL, description, language, topics, stars, forks, open issues, licence, dates, owner avatar, homepage), links every item to GitHub, and never copies repository code or README text. It honours the search rate-limit headers and waits out a refusal. Re-verify before enabling.",++ "license_note": "Repository metadata is read under GitHub's API terms; each repository's own licence is shown only as GitHub reports it (SPDX id or name), never assumed. The repository, its code and its text remain the owner's.",++ "min_interval_ms": 6500,++ "poll_interval_sec": 120,++ "stale_after_sec": 21600,++ "max_items": 3000,++ "enabled": false,++ "review_required": true,++ "sensitivity": "none",++ "default_indexability": "noindex"}]}nnew file mode 100644@@@ -0,0 +1,174 @@++'use strict';++/**++ * GitHub repositories for Sources, read from the REST search API only (api.github.com/search/repositories),++ * never from github.com/trending. One run walks a share of the query plan:++ *++ * - repositories created today, this week and this month, by stars;++ * - the most-starred repositories pushed in the last 30 days;++ * - the all-time most-starred;++ * - the same recent list per language, for the twelve languages with the most repositories on GitHub.++ *++ * Search is limited to 10 requests a minute without a token and 30 with one, so the cursor lives in++ * source_state and a run asks for at most its budget of queries; the remaining ones come on the next run.++ * The scheduler decides when that is (never an interval loop inside a fetch).++ *++ * A star count is kept per repository per day (github_stars). The API computes stars gained over 1 and 7++ * days from those snapshots: a repository first seen yesterday has no 7-day figure and says null, never 0.++ *++ * The mapping keeps what the repository states: full name, URL, description, language, topics, stars,++ * forks, open issues, licence, created, pushed, owner avatar and homepage. A field GitHub does not state++ * is null.++ */++const { toText, toIsoDate, canonicalUrl, toId } = require('./util');++++const PARSER_VERSION = 'github@1';++const STATE_NAME = 'github.cursor';++const API = 'https://api.github.com/search/repositories';++const PER_PAGE = 100;++++// The twelve languages GitHub's own search facets put at the top of public repositories. A list, not a++// count: the plan is stable and reviewable, and a language added later starts on the next run.++const LANGUAGES = Object.freeze(['JavaScript', 'TypeScript', 'Python', 'Go', 'Rust', 'Java', 'C++', 'C#', 'PHP', 'Ruby', 'Swift', 'Kotlin']);++++const isoDay = (ms) => new Date(ms).toISOString().slice(0, 10);++const daysAgo = (ms, n) => isoDay(ms - n * 86400000);++++/** The queries, in a fixed order: base lists first, then one recent list per language. */++function plan(now = Date.now()) {++ const today = isoDay(now);++ const week = daysAgo(now, 7);++ const month = daysAgo(now, 30);++ const pushed = daysAgo(now, 30);++ const base = [++ { key: 'trending', q: `created:>${today}`, sort: 'stars', order: 'desc' },++ { key: 'week', q: `created:>${week}`, sort: 'stars', order: 'desc' },++ { key: 'month', q: `created:>${month}`, sort: 'stars', order: 'desc' },++ { key: 'pushed', q: `pushed:>${pushed} stars:>100`, sort: 'stars', order: 'desc' },++ { key: 'top', q: 'stars:>10000', sort: 'stars', order: 'desc' },++ { key: 'new', q: `created:>${week}`, sort: 'updated', order: 'desc' },++ ];++ const languages = LANGUAGES.map((language) => ({++ key: `lang:${language}`,++ q: `language:"${language}" created:>${month}`,++ sort: 'stars', order: 'desc',++ }));++ return [...base, ...languages];++}++++function queryUrl(query) {++ const u = new URL(API);++ u.searchParams.set('q', query.q);++ if (query.sort) u.searchParams.set('sort', query.sort);++ if (query.order) u.searchParams.set('order', query.order);++ u.searchParams.set('per_page', String(PER_PAGE));++ return u.toString();++}++++/** The public item shape: what Exchange and other products read. A number GitHub did not state is null. */++function publicRepository(item, gains = {}) {++ const f = item.fields || {};++ return {++ full_name: item.identity,++ url: item.canonical_url,++ description: item.summary,++ language: f.language == null ? null : f.language,++ topics: Array.isArray(f.topics) ? f.topics : [],++ stars: f.stars == null ? null : f.stars,++ stars_gained_1d: gains.days_1 == null ? null : gains.days_1,++ stars_gained_7d: gains.days_7 == null ? null : gains.days_7,++ forks: f.forks == null ? null : f.forks,++ open_issues: f.open_issues == null ? null : f.open_issues,++ license: f.license == null ? null : f.license,++ created: f.created == null ? null : f.created,++ pushed: f.pushed == null ? null : f.pushed,++ owner_avatar_url: f.owner_avatar_url == null ? null : f.owner_avatar_url,++ homepage: f.homepage == null ? null : f.homepage,++ source: 'github',++ };++}++++function repoItem(repo, summaryMax = 1000) {++ const full = toId(repo && repo.full_name, 200);++ if (!full) return null;++ const owner = repo.owner && typeof repo.owner === 'object' ? repo.owner : {};++ const license = repo.license && typeof repo.license === 'object' ? repo.license : null;++ const stars = Number.isFinite(Number(repo.stargazers_count)) ? Math.max(0, Math.trunc(Number(repo.stargazers_count))) : null;++ const fields = {++ stars,++ forks: Number.isFinite(Number(repo.forks_count)) ? Math.max(0, Math.trunc(Number(repo.forks_count))) : null,++ open_issues: Number.isFinite(Number(repo.open_issues_count)) ? Math.max(0, Math.trunc(Number(repo.open_issues_count))) : null,++ language: toText(repo.language, 60),++ topics: Array.isArray(repo.topics) ? repo.topics.map((t) => toText(t, 50)).filter(Boolean).slice(0, 20) : [],++ // SPDX id when GitHub has one; NOASSERTION means GitHub read a licence file but could not name it,++ // so the repository's own string is kept instead of inventing an id.++ license: license ? (license.spdx_id && license.spdx_id !== 'NOASSERTION' ? toText(license.spdx_id, 80) : toText(license.name, 120)) : null,++ created: toIsoDate(repo.created_at),++ pushed: toIsoDate(repo.pushed_at),++ owner_avatar_url: canonicalUrl(owner.avatar_url),++ homepage: canonicalUrl(repo.homepage),++ archived: repo.archived === true,++ fork: repo.fork === true,++ };++ return {++ identity: full,++ kind: 'repository',++ canonical_url: canonicalUrl(repo.html_url) || `https://github.com/${full}`,++ title: full,++ summary: toText(repo.description, summaryMax),++ authors: owner.login ? [toText(owner.login, 200)].filter(Boolean) : [],++ published_at: fields.created,++ updated_at: fields.pushed,++ fields,++ };++}++++/** Parse one search answer. A body without an items array is a parse error, never an empty success. */++function parseSearch(body) {++ if (!body || typeof body !== 'object' || !Array.isArray(body.items)) {++ throw Object.assign(new Error('GitHub search did not return an items array'), { code: 'parse_error' });++ }++ return { items: body.items, total: Number.isFinite(Number(body.total_count)) ? Number(body.total_count) : null, incomplete: body.incomplete_results === true };++}++++/**++ * Keep one star count per repository per day. Called inside the run transaction; a repository seen twice in++ * one run keeps the last count, which is the same answer GitHub gave later.++ */++async function recordStars(db, sourceKey, list, at) {++ if (!Array.isArray(list) || !list.length) return 0;++ const day = isoDay(at);++ const stmt = db.prepare(`INSERT INTO github_stars (source_key, full_name, day, stars, captured_at) VALUES (@source_key, @full_name, @day, @stars, @captured_at)++ ON CONFLICT (source_key, full_name, day) DO UPDATE SET stars = excluded.stars, captured_at = excluded.captured_at`);++ let n = 0;++ for (const row of list) {++ if (!row || !row.full_name || !Number.isFinite(row.stars)) continue;++ await stmt.run({ source_key: sourceKey, full_name: row.full_name, day, stars: row.stars, captured_at: at });++ n++;++ }++ return n;++}++++/** Stars gained over 1 and 7 days from daily snapshots; a window with no snapshot is null. */++function gainsFrom(history, todayMs) {++ const today = isoDay(todayMs);++ const yesterday = isoDay(todayMs - 86400000);++ const weekAgo = isoDay(todayMs - 7 * 86400000);++ const earlier = (limit) => {++ let best = null;++ for (const h of history) if (h.day <= limit && (best == null || h.day > best.day)) best = h;++ return best;++ };++ const latest = earlier(today);++ const d1 = earlier(yesterday);++ const d7 = earlier(weekAgo);++ const stars = latest ? latest.stars : null;++ return {++ stars,++ days_1: stars != null && d1 ? stars - d1.stars : null,++ days_7: stars != null && d7 ? stars - d7.stars : null,++ };++}++++module.exports = {++ PARSER_VERSION, STATE_NAME, API, LANGUAGES, plan, queryUrl, repoItem, parseSearch, publicRepository, recordStars, gainsFrom, isoDay,++};@@@ -10,6 +10,7 @@ const jsonld = require('./jsonld');const api = require('./api');const bulk = require('./bulk');const hn = require('./hn');++const github = require('./github');const ADAPTERS = {rss: { parse: feed.parse, version: feed.PARSER_VERSION, accept: 'application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1' },@@@ -24,7 +25,12 @@ const ADAPTERS = {};// A resolver reads several URLs of one API and returns the same { items, skipped, meta } an adapter's parse does.--const RESOLVERS = { 'hn-whoishiring': { resolve: hn.resolve, version: hn.PARSER_VERSION } };++const RESOLVERS = {++ 'hn-whoishiring': { resolve: hn.resolve, version: hn.PARSER_VERSION },++ // GitHub is walked by the client in server/github.js (a share of the search plan per run, rate-limit++ // aware); the resolver entry names its parser version for every item it produces.++ github: { resolve: null, version: github.PARSER_VERSION },++};const MANUAL_VERSION = 'manual@1';nnew file mode 100644@@@ -0,0 +1,148 @@++'use strict';++/**++ * Code API (service tokens): GitHub repositories kept by the `github` source.++ *++ * GET /api/v1/code/repos?list=trending|week|month|hot|new|top&lang=&topic=&limit=++ * The cached rankings. `hot` is stars gained in the last 24 hours and 7 days from github_stars,++ * never a total. A list with no reading for a repository leaves its gain null.++ * GET /api/v1/code/search?q=&lang=&sort=stars|updated|best&page=++ * A live GitHub repository search through the same client as the scheduled fetch, cached ten++ * minutes, with GitHub's own query syntax passed through.++ *++ * Both need sources.item.read. The public API is Exchange's; it reads these over loopback.++ */++const express = require('express');++const { http } = require('openvibe-contracts');++const { CAPS } = require('../auth');++const github = require('../adapters/github');++++const LISTS = Object.freeze(['trending', 'week', 'month', 'hot', 'new', 'top']);++const SORTS = Object.freeze(['stars', 'updated', 'best']);++const DAY = 86400000;++++const text = (v, max) => (v == null ? null : String(v).trim().slice(0, max) || null);++const number = (v) => (Number.isFinite(Number(v)) ? Number(v) : null);++++function rowItem(row) {++ let fields = {};++ try { fields = JSON.parse(row.fields) || {}; } catch { fields = {}; }++ return {++ identity: row.identity, canonical_url: row.canonical_url, summary: row.summary,++ title: row.title, fields,++ };++}++++async function historyFor(db, names, now) {++ if (!names.length) return new Map();++ const since = github.isoDay(now - 8 * DAY);++ const rows = (await db.query('SELECT full_name, day, stars FROM github_stars WHERE day >= $1 AND full_name = ANY($2)', [since, names])).rows;++ const byName = new Map();++ for (const r of rows) {++ if (!byName.has(r.full_name)) byName.set(r.full_name, []);++ byName.get(r.full_name).push({ day: r.day, stars: Number(r.stars) });++ }++ return byName;++}++++function withGains(rows, history, now) {++ return rows.map((row) => {++ const item = rowItem(row);++ const gains = github.gainsFrom(history.get(item.identity) || [], now);++ return { item, gains, repo: github.publicRepository(item, gains) };++ });++}++++function sortFor(list) {++ return (a, b) => {++ if (list === 'hot') {++ const a1 = a.gains.days_1, b1 = b.gains.days_1;++ if (a1 != null && b1 != null && a1 !== b1) return b1 - a1;++ if (a1 != null && b1 == null) return -1;++ if (a1 == null && b1 != null) return 1;++ const a7 = a.gains.days_7, b7 = b.gains.days_7;++ if (a7 != null && b7 != null && a7 !== b7) return b7 - a7;++ }++ if (list === 'new') {++ const ac = Date.parse(a.repo.created || 0) || 0;++ const bc = Date.parse(b.repo.created || 0) || 0;++ if (ac !== bc) return bc - ac;++ }++ const as = number(a.repo.stars), bs = number(b.repo.stars);++ if (as != null && bs != null && as !== bs) return bs - as;++ if (as != null && bs == null) return -1;++ if (as == null && bs != null) return 1;++ return String(a.repo.full_name).localeCompare(String(b.repo.full_name));++ };++}++++function createCodeRouter({ db, registry, auth, limits, github: client }) {++ const router = express.Router();++ const read = [auth.requireCap(CAPS.items), limits.reads('sources.code')];++++ router.get('/api/v1/code/repos', read, async (req, res) => {++ const list = LISTS.includes(String(req.query.list || 'trending')) ? String(req.query.list || 'trending') : null;++ if (!list) return http.sendProblem(res, 422, 'sources.bad_query', { detail: `list must be one of ${LISTS.join('|')}` });++ const lang = text(req.query.lang, 60);++ const topic = text(req.query.topic, 50);++ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 30, 1), 100);++ const t = Date.now();++ const dayStart = Date.parse(github.isoDay(t));++ // Ordering by stars happens in JS: the tracked set is bounded by the source's max_items (3,000) and++ // the gains need the snapshots anyway. A filter narrows it before the sort.++ const rows = (await db.query("SELECT * FROM items WHERE category = 'code' AND removed_at IS NULL ORDER BY retrieved_at DESC LIMIT 5000")).rows;++ const filtered = rows.filter((row) => {++ const item = rowItem(row);++ const f = item.fields || {};++ if (lang && String(f.language || '').toLowerCase() !== lang.toLowerCase()) return false;++ if (topic && !(Array.isArray(f.topics) ? f.topics : []).some((x) => String(x).toLowerCase() === topic.toLowerCase())) return false;++ const created = Date.parse(f.created || '');++ if (list === 'trending' && (!Number.isFinite(created) || created < dayStart)) return false;++ if (list === 'week' && (!Number.isFinite(created) || created < t - 7 * DAY)) return false;++ if (list === 'month' && (!Number.isFinite(created) || created < t - 30 * DAY)) return false;++ if (list === 'new' && (!Number.isFinite(created) || created < t - 7 * DAY)) return false;++ return true;++ });++ const history = await historyFor(db, filtered.map((r) => r.identity), t);++ const ordered = withGains(filtered, history, t).sort(sortFor(list)).slice(0, limit).map((x) => x.repo);++ const source = await registry.get('github');++ const health = source ? registry.health(source) : null;++ res.set('Cache-Control', 'public, max-age=60');++ res.json({++ generated_at: new Date(t).toISOString(),++ list, lang, topic, limit, count: ordered.length,++ source: health ? { key: 'github', status: health.status, stale: health.stale, last_success_at: health.last_success_at } : null,++ items: ordered,++ });++ });++++ router.get('/api/v1/code/search', read, async (req, res) => {++ const q = text(req.query.q, 300);++ if (!q) return http.sendProblem(res, 422, 'sources.bad_query', { detail: 'q is required' });++ const lang = text(req.query.lang, 60);++ const sort = SORTS.includes(String(req.query.sort || 'best')) ? String(req.query.sort || 'best') : 'best';++ const page = Math.min(Math.max(parseInt(req.query.page, 10) || 1, 1), 10);++ let out;++ try {++ out = await client.search({ q, lang, sort, page });++ } catch (err) {++ if (err.code === 'rate_limited') {++ res.set('Retry-After', String(Math.max(1, Math.ceil((err.retryAfterMs || 60000) / 1000))));++ return http.sendProblem(res, 429, 'sources.rate_limited', { detail: 'GitHub asked us to wait before another search', ctx: req.ov });++ }++ if (err.code === 'bad_query') return http.sendProblem(res, 422, 'sources.bad_query', { detail: err.message, ctx: req.ov });++ return http.sendProblem(res, 502, 'sources.github_unavailable', { detail: 'GitHub search did not answer', ctx: req.ov });++ }++ const names = out.items.map((i) => i.full_name);++ const history = await historyFor(db, names, Date.now());++ const items = out.items.map((repo) => {++ const gains = github.gainsFrom(history.get(repo.full_name) || [], Date.now());++ return { ...repo, stars_gained_1d: gains.days_1, stars_gained_7d: gains.days_7 };++ });++ res.set('Cache-Control', 'public, max-age=600');++ res.json({ ...out, items, count: items.length });++ });++++ return router;++}++++module.exports = { createCodeRouter, LISTS, SORTS };@@@ -9,6 +9,7 @@ const { createRelease } = require('openvibe-shared/release');const { createSourcesReadiness, registerSourcesGauges } = require('./observability');const { sourcesRouter } = require('./api/sources');const { itemsRouter } = require('./api/items');++const { createCodeRouter } = require('./api/code');const { merchantRouter } = require('./api/merchant');const { merchantPages } = require('./merchant-pages');const { createActorLimits } = require('./api/actor-limits');@@@ -25,6 +26,7 @@ const TEXT_INDEX = ['','GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)','GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)',++ 'GET /api/v1/code/repos?list=&lang=&topic=&limit=, /api/v1/code/search?q=&lang=&sort=&page= (sources.item.read)','POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)','GET/POST /api/v1/merchant/sources[/:id[/verify|/withdraw]] (sources.source.read/manage, or a merchant session)','GET /api/health, /api/ready, /release.json',@@@ -114,6 +116,7 @@ function createApp({ config, db, registry, items, ingest, scheduler, auth, outboconst limits = createActorLimits({ config, now: limitsNow || (() => Date.now()), registry: metrics.registry, log });app.use(sourcesRouter({ db, registry, ingest, auth, limits }));app.use(itemsRouter({ db, registry, items, auth, outbox, now, limits }));++ app.use(createCodeRouter({ db, registry, auth, limits, github: ingest.github }));// GET /auth/health: does the Network accept this site's OAuth client? 503 only when it refuses it. Sources has// no site-wide sign-in; the merchant flow is the one OAuth client, so it is the one this checks.nnew file mode 100644@@@ -0,0 +1,210 @@++'use strict';++/**++ * The GitHub client shared by ingestion and the code search API. One process has one client, so the++ * token, the request spacing and the last rate-limit headers are the same for a scheduled fetch and a++ * person's search.++ *++ * The token is read by name at request time (server/settings.js: the owner's setting, then++ * SOURCES_GITHUB_TOKEN, then GITHUB_TOKEN). It is never stored, logged, returned or put in a URL.++ * Without one, GitHub's search limit is 10 requests a minute; with one it is 30. The client keeps the++ * last x-ratelimit-remaining/reset, waits out a 403/429 the API sends, and lets a batch end early: a++ * run that already has items records them and says when the next request may go.++ *++ * Nothing here fetches github.com pages. The only host is api.github.com, and every request goes through++ * the same robots, spacing and address guard as every other Sources fetch.++ */++const crypto = require('crypto');++const github = require('./adapters/github');++const settings = require('./settings');++const { decodeBody, ParseError } = require('./adapters/util');++++const HOST = 'api.github.com';++const SEARCH_TTL_MS = 10 * 60 * 1000;++const MAX_CACHE = 200;++++class RateLimitError extends Error {++ constructor(retryAfterMs, detail) {++ super(detail || 'GitHub asked us to wait');++ this.code = 'rate_limited';++ this.retryAfterMs = Math.max(0, Number(retryAfterMs) || 0);++ }++}++++const int = (v) => { const n = Number(v); return Number.isFinite(n) ? n : null; };++const sleep = (ms) => new Promise((r) => setTimeout(r, ms));++++function retryAfterMs(value, now) {++ if (value == null) return null;++ const s = String(value).trim();++ if (/^\d{1,7}$/.test(s)) return Number(s) * 1000;++ const t = Date.parse(s);++ return Number.isFinite(t) ? Math.max(0, t - now) : null;++}++++function createGithub({ config, fetcher, robots, spacer, now = () => Date.now(), log = console, capBytes = 4 * 1024 * 1024 }) {++ const limiter = { remaining: null, resetAt: 0, at: 0 };++ const cache = new Map(); // search URL → { at, value }++ const inflight = new Map();++++ const token = () => {++ try { return settings.githubToken() || config.secrets.GITHUB_TOKEN || ''; } catch { return config.secrets.GITHUB_TOKEN || ''; }++ };++++ function headers() {++ const h = {++ Accept: 'application/vnd.github+json',++ 'X-GitHub-Api-Version': '2022-11-28',++ 'User-Agent': config.fetch.userAgent,++ };++ const t = token();++ if (t) h.Authorization = `Bearer ${t}`;++ return h;++ }++++ /** One API GET: robots, spacing, the token and the rate-limit ledger. Returns { body, status, bytes, headers }. */++ async function request(url, { beforeRequest = null, minIntervalMs = null } = {}) {++ // A used-up search budget is a wait, not a stream of refused requests. GitHub resets the search++ // limit within a minute; resetAt comes from its own headers.++ if (limiter.remaining === 0 && limiter.resetAt > now()) {++ throw new RateLimitError(limiter.resetAt - now(), 'the GitHub search limit is used up');++ }++ const verdict = await robots.check(url);++ if (!verdict.allowed) {++ if (verdict.refusedCode) throw Object.assign(new Error(`${HOST} is not fetchable (${verdict.refusedCode})`), { code: verdict.refusedCode });++ throw Object.assign(new Error(verdict.reason || 'robots.txt disallows this URL'), { code: 'robots_denied' });++ }++ await spacer.space(HOST, minIntervalMs == null ? (token() ? 2000 : 6500) : minIntervalMs);++ if (beforeRequest) await beforeRequest();++ limiter.at = now();++ const res = await fetcher.fetchUrl(url, {++ headers: headers(),++ capBytes,++ sensitiveHeaders: ['Authorization'],++ beforeHop: async (next) => {++ const v = await robots.check(next.toString());++ return v.allowed ? null : { refuse: v.reason };++ },++ });++ const remaining = int(res.headers['x-ratelimit-remaining']);++ const reset = int(res.headers['x-ratelimit-reset']);++ if (remaining != null) limiter.remaining = remaining;++ if (reset != null) limiter.resetAt = reset * 1000;++ if (res.status === 429 || (res.status === 403 && (remaining === 0 || retryAfterMs(res.headers['retry-after'], now()) != null))) {++ const wait = retryAfterMs(res.headers['retry-after'], now())++ ?? (limiter.resetAt > now() ? limiter.resetAt - now() : 60_000);++ throw new RateLimitError(wait, `GitHub answered ${res.status}`);++ }++ if (res.status === 401) throw Object.assign(new Error('GitHub refused the token (401)'), { code: 'http_error', status: 401 });++ if (res.status === 403) throw Object.assign(new Error('GitHub answered 403 (the token may lack access)'), { code: 'http_error', status: 403 });++ if (res.status !== 200 && res.status !== 203) throw Object.assign(new Error(`GitHub answered HTTP ${res.status}`), { code: 'http_error', status: res.status });++ let body;++ try { body = JSON.parse(decodeBody(res.body, res.headers['content-type'])); } catch {++ throw new ParseError('GitHub answered non-JSON');++ }++ return { body, status: res.status, bytes: res.bytes || 0, headers: res.headers };++ }++++ /**++ * One ingestion run's share of the query plan. Starts at the stored cursor, asks at most `budget`++ * questions, and returns the cursor for the next run. A rate limit after some answers keeps those++ * answers and reports when to come back; a rate limit before any is thrown so the run is recorded++ * as rate_limited, never as an empty success.++ */++ async function fetchQueries({ source, cursor = {}, budget = 8, beforeRequest = null } = {}) {++ const list = github.plan(now());++ const start = Number.isInteger(cursor.next) ? ((cursor.next % list.length) + list.length) % list.length : 0;++ const items = new Map();++ const stars = new Map();++ let next = start;++ let queries = 0;++ let lastStatus = null;++ let bytes = 0;++ const hash = crypto.createHash('sha256');++ let notBefore = 0;++ let limited = null;++ for (let i = 0; i < budget; i++) {++ const index = (start + i) % list.length;++ const query = list[index];++ let out;++ try {++ out = await request(github.queryUrl(query), { beforeRequest, minIntervalMs: source.min_interval_ms });++ } catch (err) {++ if (err.code === 'rate_limited') {++ notBefore = Math.max(notBefore, now() + err.retryAfterMs);++ limited = err;++ break;++ }++ throw err;++ }++ lastStatus = out.status;++ bytes += out.bytes;++ hash.update(Buffer.from(JSON.stringify(out.body)));++ const parsed = github.parseSearch(out.body);++ for (const raw of parsed.items) {++ const item = github.repoItem(raw, config.items.summaryMax);++ if (!item) continue;++ items.set(item.identity, item);++ const starsNow = item.fields.stars;++ if (starsNow != null) stars.set(item.identity, starsNow);++ }++ queries++;++ next = (index + 1) % list.length;++ }++ if (!queries && limited) throw limited;++ return {++ items: [...items.values()],++ stars: [...stars.entries()].map(([full_name, n]) => ({ full_name, stars: n })),++ next, queries, lastStatus, bytes,++ rawBodyHash: hash.digest('hex'),++ notBefore,++ limited: Boolean(limited),++ };++ }++++ /**++ * A live repository search for the public API: GitHub's own query syntax passes through, with an++ * optional language and sort. Cached for ten minutes; a request that GitHub rate-limits answers++ * the last copy for up to an hour, marked stale, instead of inventing an empty result.++ */++ async function search({ q, lang = null, sort = 'best', page = 1 } = {}) {++ const text = String(q || '').trim().slice(0, 300);++ if (!text) throw Object.assign(new Error('a search needs q'), { code: 'bad_query' });++ const language = lang ? String(lang).trim().slice(0, 60) : '';++ const order = ['stars', 'updated', 'best'].includes(sort) ? sort : 'best';++ const p = Math.min(Math.max(parseInt(page, 10) || 1, 1), 10);++ const u = new URL(github.API);++ u.searchParams.set('q', language ? `${text} language:"${language.replace(/"/g, '')}"` : text);++ if (order !== 'best') { u.searchParams.set('sort', order); u.searchParams.set('order', 'desc'); }++ u.searchParams.set('per_page', '30');++ u.searchParams.set('page', String(p));++ const url = u.toString();++ const hit = cache.get(url);++ if (hit && now() - hit.at < SEARCH_TTL_MS) return { ...hit.value, cached: true, stale: false };++ if (inflight.has(url)) return await inflight.get(url);++ const job = (async () => {++ try {++ const out = await request(url, { minIntervalMs: token() ? 2000 : 6500 });++ const parsed = github.parseSearch(out.body);++ const items = parsed.items.map((r) => github.repoItem(r, config.items.summaryMax)).filter(Boolean)++ .map((i) => github.publicRepository(i));++ const value = { items, total: parsed.total, incomplete: parsed.incomplete, page: p, sort: order, q: text, lang: language || null };++ cache.set(url, { at: now(), value });++ if (cache.size > MAX_CACHE) cache.delete(cache.keys().next().value);++ return { ...value, cached: false, stale: false };++ } catch (err) {++ if (err.code === 'rate_limited' && hit && now() - hit.at < 60 * 60 * 1000) {++ return { ...hit.value, cached: true, stale: true, retry_after_ms: err.retryAfterMs };++ }++ throw err;++ } finally {++ inflight.delete(url);++ }++ })();++ inflight.set(url, job);++ return await job;++ }++++ return { request, fetchQueries, search, rateLimit: () => ({ ...limiter }), hasToken: () => Boolean(token()) };++}++++module.exports = { createGithub, RateLimitError, retryAfterMs, SEARCH_TTL_MS };@@@ -29,6 +29,8 @@ const crypto = require('crypto');const { ids } = require('openvibe-contracts');const { ADAPTERS, RESOLVERS } = require('./adapters');const offDelta = require('./adapters/off-delta');++const githubAdapter = require('./adapters/github');++const { createGithub } = require('./github');const extract = require('./adapters/extract');const { createCrawler } = require('./crawl');const { sha256, decodeBody, ParseError } = require('./adapters/util');@@@ -46,6 +48,7 @@ function retryAfterMs(value, now) {function createIngest({ db, config, registry, items, robots, fetcher, spacer, emitEvent, now = () => Date.now(), log = console, outbox = null }) {const inflight = new Set();const { space, setCrawlDelay } = spacer;++ const github = createGithub({ config, fetcher, robots, spacer, now, log });let stopping = false;const st = {@@@ -416,6 +419,54 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, emreturn { run, failed: false, success: true, recorded: true };}++ /**++ * A GitHub source: one share of the search plan per run through server/github.js. The cursor is kept++ * in source_state, so a run with no token (10 searches a minute) never tries the whole plan at once.++ * Star counts go to github_stars in the same transaction as the items and the run row.++ */++ async function fetchGithubEndpoint(source, endpoint, trigger) {++ const startedAt = now();++ const run = baseRun(source, endpoint.url, trigger, startedAt);++ let cursor = {};++ try {++ const saved = await st.stateGet.get(source.key, githubAdapter.STATE_NAME);++ if (saved && saved.value) cursor = JSON.parse(saved.value);++ } catch { cursor = {}; }++ const budget = github.hasToken() ? 20 : 8;++ let out;++ try {++ out = await github.fetchQueries({++ source, cursor, budget,++ beforeRequest: async () => { await st.requested.run(now(), source.key); },++ });++ } catch (err) {++ const state = err.code === 'rate_limited' ? 'rate_limited'++ : err.code === 'robots_denied' ? 'robots_denied'++ : err.code === 'parse_error' ? 'parse_error'++ : err.code === 'timeout' ? 'timeout' : 'http_error';++ Object.assign(run, { state, error_code: err.code || 'network', detail: String(err.message).slice(0, 500), finished_at: now() });++ return { run, failed: true, notBefore: err.code === 'rate_limited' ? now() + (err.retryAfterMs || 60000) : null };++ }++ Object.assign(run, {++ http_status: out.lastStatus, bytes: out.bytes, finished_at: now(),++ raw_body_hash: out.rawBodyHash, parser_version: githubAdapter.PARSER_VERSION,++ });++ await db.tx(async () => {++ const counts = await items.ingest(source, out.items, {++ runId: run.id, rawBodyHash: run.raw_body_hash, parserVersion: githubAdapter.PARSER_VERSION, retrievedAt: run.finished_at,++ });++ Object.assign(run, {++ state: 'ok', items_seen: out.items.length, items_created: counts.created, items_updated: counts.updated,++ items_unchanged: counts.unchanged, items_skipped: counts.skipped,++ });++ await githubAdapter.recordStars(db, source.key, out.stars, run.finished_at);++ await st.insertRun.run(run);++ await st.stateSet.run({ k: source.key, n: githubAdapter.STATE_NAME, v: JSON.stringify({ next: out.next, queries: out.queries, at: run.finished_at }), at: run.finished_at });++ await st.saveValidators.run({ k: source.key, u: endpoint.url, etag: null, lm: null, status: out.lastStatus, at: run.finished_at });++ });++ return { run, failed: false, success: true, recorded: true, notBefore: out.notBefore };++ }++/*** A bulk source with `resolver: "off-delta"`: read a text index, then only the delta parts not finished yet.* The full export is one gzip stream and cannot be resumed, so the first run starts at the oldest part in the@@@ -595,9 +646,11 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em? endpoint.resolver === 'off-delta'? await fetchOffDeltaEndpoint(source, endpoint, trigger): await fetchBulkEndpoint(source, endpoint, trigger)-- : endpoint.resolver-- ? await fetchResolverEndpoint(source, endpoint, trigger)-- : await fetchEndpoint(source, endpoint, trigger);++ : endpoint.resolver === 'github'++ ? await fetchGithubEndpoint(source, endpoint, trigger)++ : endpoint.resolver++ ? await fetchResolverEndpoint(source, endpoint, trigger)++ : await fetchEndpoint(source, endpoint, trigger);} catch (err) {// Anything unexpected is still a recorded failure, never a silent gap.const r = baseRun(source, endpoint.url, trigger, t);@@@ -700,7 +753,7 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em/** Shutting down: a crawl ends after the page in flight instead of working through its budget. */function stop() { stopping = true; }-- return { run, crawlStatus, stop, inflight: () => [...inflight] };++ return { run, crawlStatus, stop, inflight: () => [...inflight], github };}module.exports = { createIngest, FAILURE_STATES, retryAfterMs };@@@ -112,13 +112,17 @@ function validateSource(input, { maxItemsCap = 12000 } = {}) {// (only as their defaults) is what lets PATCH { enabled: true } validate a stored resolver.if (!['url', 'format', 'resolver', 'item_kind', 'max_bytes', 'timeout_ms', 'items_shape', 'items_path', 'fields', 'extra'].includes(k)) throw new RegistryError(`endpoints[${i}].${k} is not a resolver field`);}-- if (e.resolver !== 'hn-whoishiring') throw new RegistryError(`endpoints[${i}].resolver must be hn-whoishiring`);++ if (!['hn-whoishiring', 'github'].includes(e.resolver)) throw new RegistryError(`endpoints[${i}].resolver must be hn-whoishiring or github`);if (e.format !== undefined && e.format !== 'json') throw new RegistryError(`endpoints[${i}].format must be json for a resolver`);if (e.items_shape !== undefined && e.items_shape !== 'array') throw new RegistryError(`endpoints[${i}].items_shape must be array for a resolver`);if (e.items_path !== undefined && e.items_path !== '') throw new RegistryError(`endpoints[${i}].items_path must be "" for a resolver`);if (e.fields !== undefined && (!e.fields || typeof e.fields !== 'object' || Array.isArray(e.fields) || Object.keys(e.fields).length)) throw new RegistryError(`endpoints[${i}].fields must be empty for a resolver`);if (e.extra !== undefined && (!e.extra || typeof e.extra !== 'object' || Array.isArray(e.extra) || Object.keys(e.extra).length)) throw new RegistryError(`endpoints[${i}].extra must be empty for a resolver`);-- if (!/\.json(\?|$)/.test(out.url)) throw new RegistryError(`endpoints[${i}].resolver needs a .json endpoint`);++ // GitHub's search answer is one of several queries the resolver plans itself; the endpoint is the++ // API origin, not a fixed .json document. The host is pinned so the token can only go to GitHub.++ if (e.resolver === 'github') {++ if (new URL(out.url).origin !== 'https://api.github.com') throw new RegistryError(`endpoints[${i}].resolver github must be on https://api.github.com`);++ } else if (!/\.json(\?|$)/.test(out.url)) throw new RegistryError(`endpoints[${i}].resolver needs a .json endpoint`);if (e.item_kind !== undefined && !/^[a-z][a-z0-9_]{1,39}$/.test(e.item_kind)) throw new RegistryError(`endpoints[${i}].item_kind is malformed`);if (e.max_bytes !== undefined && (!Number.isInteger(e.max_bytes) || e.max_bytes < 1024 || e.max_bytes > 32 * 1024 * 1024)) throw new RegistryError(`endpoints[${i}].max_bytes must be an integer from 1024 to 33554432`);if (e.timeout_ms !== undefined && (!Number.isInteger(e.timeout_ms) || e.timeout_ms < 1000 || e.timeout_ms > 120000)) throw new RegistryError(`endpoints[${i}].timeout_ms must be an integer from 1000 to 120000`);@@@ -41,6 +41,8 @@* year for a rebuild and the Open Food Facts bulk snapshot, which is regenerated.* - jobs 14 days: Arbeitnow and We Work Remotely retire what the board stopped showing after 7* days; Work copies each listing, so a fortnight keeps every tombstone it may pull.++ * - code 30 days: the GitHub rankings are a current list; a month keeps a repository through a failed++ * pull, and the star history is what says whether it is still hot.** Sources has no openvibe-shared/config namespace, so the rules stay in code (the brief's* alternative); a day count this consequential should change in review, not in a database row.@@@ -74,6 +76,7 @@ const RULES = Object.freeze({housing: { keepLatest: true, reason: 'HUD rents change yearly: the latest value per area is what Rent shows' },food: { days: 365, reason: 'Food keeps the recall history it shows; a year covers a rebuild' },jobs: { days: 14, reason: 'boards retire unseen jobs after 7 days and Work pulls hourly' },++ code: { days: 30, reason: 'GitHub rankings are current; a month keeps the tracked set through a failed pull' },});// The rules are the contract's categories in review; the runtime deliberately does not throw when@@@ -61,7 +61,7 @@ function createSettings({ db, env = process.env, log = console, fetchImpl = glob}/** The read-only GitHub token, read when a GitHub read is made. */-- function githubToken() { return valueOf('github_read_only_token', 'SOURCES_GITHUB_TOKEN', ''); }++ function githubToken() { return valueOf('github_read_only_token', 'SOURCES_GITHUB_TOKEN', '') || valueOf('github_read_only_token', 'GITHUB_TOKEN', ''); }async function test(key) {if (key !== 'github_read_only_token') return { ok: false, detail: 'That setting has no test.' };@@@ -124,7 +124,7 @@ function createSettings({ db, env = process.env, log = console, fetchImpl = globfunction activate(settings) { ACTIVE.settings = settings; return settings; }function githubToken() {if (ACTIVE.settings) return ACTIVE.settings.githubToken();-- return process.env.SOURCES_GITHUB_TOKEN || '';++ return process.env.SOURCES_GITHUB_TOKEN || process.env.GITHUB_TOKEN || '';}module.exports = { SERVICE, NAMESPACE, SCHEMA, SECRETS, createSettings, activate, githubToken };@@@ -1,7 +1,8 @@'use strict';/*** The categories a source may have are sources.source@1's, read from openvibe-contracts: housing, food and jobs since-- * contracts 0.210.0 (plan 2026-10-22, track F1: Rent and Homes, Food and Work move their own fetches into Sources).++ * contracts 0.210.0 (plan 2026-10-22, track F1: Rent and Homes, Food and Work move their own fetches into Sources),++ * and code since 0.411.1 (GitHub repository rankings for Exchange).* The registry accepts every one of them and refuses anything else, and the items API filters by them.*/const assert = require('assert');@@@ -10,15 +11,15 @@ const { suite, sourceDef } = require('./helpers');const t = suite('categories');--t('the categories are the contract\'s, housing, food and jobs included', () => {++t('the categories are the contract\'s, housing, food, jobs and code included', () => {const contract = require('openvibe-contracts/contracts/sources/source.v1.json').properties.category.enum;assert.deepStrictEqual([...CATEGORIES], contract);-- for (const c of ['news', 'hosting', 'games', 'housing', 'food', 'jobs']) assert.ok(CATEGORIES.includes(c), c);++ for (const c of ['news', 'hosting', 'games', 'housing', 'food', 'jobs', 'code']) assert.ok(CATEGORIES.includes(c), c);assert.ok(Object.isFrozen(CATEGORIES));});t('the registry accepts a source in each new category and refuses one outside the contract', () => {-- for (const category of ['housing', 'food', 'jobs']) {++ for (const category of ['housing', 'food', 'jobs', 'code']) {assert.strictEqual(validateSource(sourceDef({ key: `example-${category}`, category, endpoints: ['https://feeds.example.org/feed.xml'] })).category, category);}assert.throws(() => validateSource(sourceDef({ key: 'example-weather', category: 'weather', endpoints: ['https://feeds.example.org/feed.xml'] })), /category must be one of .*housing\|food\|jobs/);nnew file mode 100644@@@ -0,0 +1,100 @@++'use strict';++/**++ * /api/v1/code: the repository rankings and the live search. Ingestion is driven by a stubbed GitHub++ * client (no network); the history is written through the real run transaction, then older days are++ * inserted so "hot" is computed from stars gained, not from totals.++ */++const assert = require('assert');++const { boot, request, serviceToken, suite } = require('./helpers');++const github = require('../server/adapters/github');++++const t = suite('code');++let svc;++const ITEMS = serviceToken('exchange', ['sources.item.read']);++const SOURCES = serviceToken('exchange', ['sources.source.read']);++const NOW = Date.now();++const DAY = 86400000;++++function repo(full, { stars, language, created, topics = ['ai'] }) {++ return {++ full_name: full, html_url: `https://github.com/${full}`, description: `${full} description`,++ stargazers_count: stars, forks_count: 12, open_issues_count: 3, language, topics,++ license: { spdx_id: 'MIT', name: 'MIT License' }, created_at: created, pushed_at: new Date(NOW - 3600000).toISOString(),++ owner: { login: full.split('/')[0], avatar_url: `https://avatars.githubusercontent.com/u/1?v=4` },++ homepage: `https://${full.split('/')[0]}.example`, archived: false, fork: false,++ };++}++++const A = repo('acme/hot', { stars: 500, language: 'Rust', created: new Date(NOW - 3600000).toISOString() });++const B = repo('beta/steady', { stars: 100, language: 'Python', created: new Date(NOW - 2 * DAY).toISOString(), topics: ['tools'] });++++t('boot and one GitHub run', async () => {++ svc = await boot();++ const seed = require('../seeds/sources.json').sources.find((s) => s.key === 'github');++ await svc.registry.create(seed, 'test');++ await svc.db.prepare("UPDATE sources SET enabled = 1, min_interval_ms = 0, last_request_at = NULL WHERE key = 'github'").run();++ svc.ingest.github.hasToken = () => false;++ svc.ingest.github.fetchQueries = async () => ({++ items: [github.repoItem(A, 1000), github.repoItem(B, 1000)],++ stars: [{ full_name: 'acme/hot', stars: 500 }, { full_name: 'beta/steady', stars: 100 }],++ next: 2, queries: 2, lastStatus: 200, bytes: 100, rawBodyHash: 'a'.repeat(64), notBefore: 0, limited: false,++ });++ const out = await svc.ingest.run('github');++ assert.strictEqual(out.runs[0].state, 'ok');++ assert.strictEqual(out.runs[0].items_created, 2);++ const stars = await svc.db.prepare('SELECT full_name, day, stars FROM github_stars ORDER BY full_name').all();++ assert.deepStrictEqual(stars.map((s) => [s.full_name, s.stars]), [['acme/hot', 500], ['beta/steady', 100]]);++ // Older days complete the history: +100/+5 in 1 day and +200/+10 in 7.++ const put = svc.db.prepare('INSERT INTO github_stars (source_key, full_name, day, stars, captured_at) VALUES (?, ?, ?, ?, ?)');++ await put.run('github', 'acme/hot', github.isoDay(NOW - DAY), 400, NOW - DAY);++ await put.run('github', 'acme/hot', github.isoDay(NOW - 7 * DAY), 300, NOW - 7 * DAY);++ await put.run('github', 'beta/steady', github.isoDay(NOW - DAY), 95, NOW - DAY);++ await put.run('github', 'beta/steady', github.isoDay(NOW - 7 * DAY), 90, NOW - 7 * DAY);++});++++t('the rankings need sources.item.read, carry the public item shape and rank hot by stars gained', async () => {++ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/repos')).status, 401);++ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/repos', { token: SOURCES })).status, 403);++ const week = await request(svc.base, 'GET', '/api/v1/code/repos?list=week&limit=10', { token: ITEMS });++ assert.strictEqual(week.status, 200);++ assert.strictEqual(week.body.list, 'week');++ assert.strictEqual(week.body.items.length, 2);++ const one = week.body.items.find((x) => x.full_name === 'acme/hot');++ for (const key of ['full_name', 'url', 'description', 'language', 'topics', 'stars', 'stars_gained_1d', 'stars_gained_7d', 'forks', 'open_issues', 'license', 'created', 'pushed', 'owner_avatar_url', 'homepage']) assert.ok(key in one, key);++ assert.strictEqual(one.stars, 500);++ assert.strictEqual(one.stars_gained_1d, 100);++ assert.strictEqual(one.stars_gained_7d, 200);++ assert.strictEqual(one.license, 'MIT');++ const hot = await request(svc.base, 'GET', '/api/v1/code/repos?list=hot', { token: ITEMS });++ assert.deepStrictEqual(hot.body.items.map((x) => x.full_name), ['acme/hot', 'beta/steady']);++ assert.deepStrictEqual([hot.body.items[0].stars_gained_1d, hot.body.items[0].stars_gained_7d], [100, 200]);++ const rust = await request(svc.base, 'GET', '/api/v1/code/repos?list=month&lang=Rust', { token: ITEMS });++ assert.deepStrictEqual(rust.body.items.map((x) => x.full_name), ['acme/hot']);++ const tools = await request(svc.base, 'GET', '/api/v1/code/repos?list=top&topic=tools', { token: ITEMS });++ assert.deepStrictEqual(tools.body.items.map((x) => x.full_name), ['beta/steady']);++ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/repos?list=weather', { token: ITEMS })).status, 422);++});++++t('the live search passes q and language through, fills the gains, and answers 429 with Retry-After', async () => {++ const seen = [];++ svc.ingest.github.search = async (args) => {++ seen.push(args);++ return { items: [github.publicRepository(github.repoItem(A, 1000))], total: 1, incomplete: false, page: 1, sort: 'stars', q: args.q, lang: args.lang, cached: false, stale: false };++ };++ const r = await request(svc.base, 'GET', '/api/v1/code/search?q=stars:>500%20topic:ai&lang=Rust&sort=stars&page=1', { token: ITEMS });++ assert.strictEqual(r.status, 200);++ assert.deepStrictEqual(seen[0], { q: 'stars:>500 topic:ai', lang: 'Rust', sort: 'stars', page: 1 });++ assert.strictEqual(r.body.items[0].full_name, 'acme/hot');++ assert.strictEqual(r.body.items[0].stars_gained_1d, 100);++ assert.match(r.headers.get('cache-control'), /max-age=600/);++ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/search?q=', { token: ITEMS })).status, 422);++ svc.ingest.github.search = async () => { throw Object.assign(new Error('slow down'), { code: 'rate_limited', retryAfterMs: 30000 }); };++ const limited = await request(svc.base, 'GET', '/api/v1/code/search?q=ai', { token: ITEMS });++ assert.strictEqual(limited.status, 429);++ assert.ok(Number(limited.headers.get('retry-after')) >= 30);++ assert.strictEqual(limited.body.code, 'sources.rate_limited');++});++++t('stop', async () => { await svc.stop(); });++++t.run();nnew file mode 100644@@@ -0,0 +1,233 @@++{++ "total_count": 5757,++ "incomplete_results": false,++ "items": [++ {++ "id": 132750724,++ "node_id": "MDEwOlJlcG9zaXRvcnkxMzI3NTA3MjQ=",++ "name": "build-your-own-x",++ "full_name": "codecrafters-io/build-your-own-x",++ "private": false,++ "owner": {++ "login": "codecrafters-io",++ "id": 58904235,++ "node_id": "MDEyOk9yZ2FuaXphdGlvbjU4OTA0MjM1",++ "avatar_url": "https://avatars.githubusercontent.com/u/58904235?v=4",++ "gravatar_id": "",++ "url": "https://api.github.com/users/codecrafters-io",++ "html_url": "https://github.com/codecrafters-io",++ "followers_url": "https://api.github.com/users/codecrafters-io/followers",++ "following_url": "https://api.github.com/users/codecrafters-io/following{/other_user}",++ "gists_url": "https://api.github.com/users/codecrafters-io/gists{/gist_id}",++ "starred_url": "https://api.github.com/users/codecrafters-io/starred{/owner}{/repo}",++ "subscriptions_url": "https://api.github.com/users/codecrafters-io/subscriptions",++ "organizations_url": "https://api.github.com/users/codecrafters-io/orgs",++ "repos_url": "https://api.github.com/users/codecrafters-io/repos",++ "events_url": "https://api.github.com/users/codecrafters-io/events{/privacy}",++ "received_events_url": "https://api.github.com/users/codecrafters-io/received_events",++ "type": "Organization",++ "user_view_type": "public",++ "site_admin": false++ },++ "html_url": "https://github.com/codecrafters-io/build-your-own-x",++ "description": "Master programming by recreating your favorite technologies from scratch.",++ "fork": false,++ "url": "https://api.github.com/repos/codecrafters-io/build-your-own-x",++ "forks_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/forks",++ "keys_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/keys{/key_id}",++ "collaborators_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/collaborators{/collaborator}",++ "teams_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/teams",++ "hooks_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/hooks",++ "issue_events_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/issues/events{/number}",++ "events_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/events",++ "assignees_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/assignees{/user}",++ "branches_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/branches{/branch}",++ "tags_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/tags",++ "blobs_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/blobs{/sha}",++ "git_tags_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/tags{/sha}",++ "git_refs_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/refs{/sha}",++ "trees_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/trees{/sha}",++ "statuses_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/statuses/{sha}",++ "languages_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/languages",++ "stargazers_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/stargazers",++ "contributors_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/contributors",++ "subscribers_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/subscribers",++ "subscription_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/subscription",++ "commits_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/commits{/sha}",++ "git_commits_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/commits{/sha}",++ "comments_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/comments{/number}",++ "issue_comment_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/issues/comments{/number}",++ "contents_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/contents/{+path}",++ "compare_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/compare/{base}...{head}",++ "merges_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/merges",++ "archive_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/{archive_format}{/ref}",++ "downloads_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/downloads",++ "issues_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/issues{/number}",++ "pulls_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/pulls{/number}",++ "milestones_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/milestones{/number}",++ "notifications_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/notifications{?since,all,participating}",++ "labels_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/labels{/name}",++ "releases_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/releases{/id}",++ "deployments_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/deployments",++ "created_at": "2018-05-09T12:03:18Z",++ "updated_at": "2026-11-12T07:55:05Z",++ "pushed_at": "2026-11-08T16:38:15Z",++ "git_url": "git://github.com/codecrafters-io/build-your-own-x.git",++ "ssh_url": "[email protected]:codecrafters-io/build-your-own-x.git",++ "clone_url": "https://github.com/codecrafters-io/build-your-own-x.git",++ "svn_url": "https://github.com/codecrafters-io/build-your-own-x",++ "homepage": "https://codecrafters.io",++ "size": 1089,++ "stargazers_count": 561961,++ "watchers_count": 561961,++ "language": "Markdown",++ "has_issues": true,++ "has_projects": false,++ "has_downloads": false,++ "has_wiki": false,++ "has_pages": false,++ "has_discussions": false,++ "forks_count": 52587,++ "mirror_url": null,++ "archived": false,++ "disabled": false,++ "open_issues_count": 701,++ "license": null,++ "allow_forking": true,++ "is_template": false,++ "web_commit_signoff_required": false,++ "has_pull_requests": true,++ "pull_request_creation_policy": "all",++ "topics": [++ "awesome-list",++ "free",++ "programming",++ "tutorial-code",++ "tutorial-exercises",++ "tutorials"++ ],++ "visibility": "public",++ "forks": 52587,++ "open_issues": 701,++ "watchers": 561961,++ "default_branch": "master",++ "score": 1.0++ },++ {++ "id": 21737465,++ "node_id": "MDEwOlJlcG9zaXRvcnkyMTczNzQ2NQ==",++ "name": "awesome",++ "full_name": "sindresorhus/awesome",++ "private": false,++ "owner": {++ "login": "sindresorhus",++ "id": 170270,++ "node_id": "MDQ6VXNlcjE3MDI3MA==",++ "avatar_url": "https://avatars.githubusercontent.com/u/170270?v=4",++ "gravatar_id": "",++ "url": "https://api.github.com/users/sindresorhus",++ "html_url": "https://github.com/sindresorhus",++ "followers_url": "https://api.github.com/users/sindresorhus/followers",++ "following_url": "https://api.github.com/users/sindresorhus/following{/other_user}",++ "gists_url": "https://api.github.com/users/sindresorhus/gists{/gist_id}",++ "starred_url": "https://api.github.com/users/sindresorhus/starred{/owner}{/repo}",++ "subscriptions_url": "https://api.github.com/users/sindresorhus/subscriptions",++ "organizations_url": "https://api.github.com/users/sindresorhus/orgs",++ "repos_url": "https://api.github.com/users/sindresorhus/repos",++ "events_url": "https://api.github.com/users/sindresorhus/events{/privacy}",++ "received_events_url": "https://api.github.com/users/sindresorhus/received_events",++ "type": "User",++ "user_view_type": "public",++ "site_admin": false++ },++ "html_url": "https://github.com/sindresorhus/awesome",++ "description": "😎 Awesome lists about all kinds of interesting topics [NOTE: Pull requests are temporarily disabled until I have a chance to catch up with the existing ones]",++ "fork": false,++ "url": "https://api.github.com/repos/sindresorhus/awesome",++ "forks_url": "https://api.github.com/repos/sindresorhus/awesome/forks",++ "keys_url": "https://api.github.com/repos/sindresorhus/awesome/keys{/key_id}",++ "collaborators_url": "https://api.github.com/repos/sindresorhus/awesome/collaborators{/collaborator}",++ "teams_url": "https://api.github.com/repos/sindresorhus/awesome/teams",++ "hooks_url": "https://api.github.com/repos/sindresorhus/awesome/hooks",++ "issue_events_url": "https://api.github.com/repos/sindresorhus/awesome/issues/events{/number}",++ "events_url": "https://api.github.com/repos/sindresorhus/awesome/events",++ "assignees_url": "https://api.github.com/repos/sindresorhus/awesome/assignees{/user}",++ "branches_url": "https://api.github.com/repos/sindresorhus/awesome/branches{/branch}",++ "tags_url": "https://api.github.com/repos/sindresorhus/awesome/tags",++ "blobs_url": "https://api.github.com/repos/sindresorhus/awesome/git/blobs{/sha}",++ "git_tags_url": "https://api.github.com/repos/sindresorhus/awesome/git/tags{/sha}",++ "git_refs_url": "https://api.github.com/repos/sindresorhus/awesome/git/refs{/sha}",++ "trees_url": "https://api.github.com/repos/sindresorhus/awesome/git/trees{/sha}",++ "statuses_url": "https://api.github.com/repos/sindresorhus/awesome/statuses/{sha}",++ "languages_url": "https://api.github.com/repos/sindresorhus/awesome/languages",++ "stargazers_url": "https://api.github.com/repos/sindresorhus/awesome/stargazers",++ "contributors_url": "https://api.github.com/repos/sindresorhus/awesome/contributors",++ "subscribers_url": "https://api.github.com/repos/sindresorhus/awesome/subscribers",++ "subscription_url": "https://api.github.com/repos/sindresorhus/awesome/subscription",++ "commits_url": "https://api.github.com/repos/sindresorhus/awesome/commits{/sha}",++ "git_commits_url": "https://api.github.com/repos/sindresorhus/awesome/git/commits{/sha}",++ "comments_url": "https://api.github.com/repos/sindresorhus/awesome/comments{/number}",++ "issue_comment_url": "https://api.github.com/repos/sindresorhus/awesome/issues/comments{/number}",++ "contents_url": "https://api.github.com/repos/sindresorhus/awesome/contents/{+path}",++ "compare_url": "https://api.github.com/repos/sindresorhus/awesome/compare/{base}...{head}",++ "merges_url": "https://api.github.com/repos/sindresorhus/awesome/merges",++ "archive_url": "https://api.github.com/repos/sindresorhus/awesome/{archive_format}{/ref}",++ "downloads_url": "https://api.github.com/repos/sindresorhus/awesome/downloads",++ "issues_url": "https://api.github.com/repos/sindresorhus/awesome/issues{/number}",++ "pulls_url": "https://api.github.com/repos/sindresorhus/awesome/pulls{/number}",++ "milestones_url": "https://api.github.com/repos/sindresorhus/awesome/milestones{/number}",++ "notifications_url": "https://api.github.com/repos/sindresorhus/awesome/notifications{?since,all,participating}",++ "labels_url": "https://api.github.com/repos/sindresorhus/awesome/labels{/name}",++ "releases_url": "https://api.github.com/repos/sindresorhus/awesome/releases{/id}",++ "deployments_url": "https://api.github.com/repos/sindresorhus/awesome/deployments",++ "created_at": "2014-07-11T13:42:37Z",++ "updated_at": "2026-11-12T07:56:15Z",++ "pushed_at": "2026-09-02T02:10:47Z",++ "git_url": "git://github.com/sindresorhus/awesome.git",++ "ssh_url": "[email protected]:sindresorhus/awesome.git",++ "clone_url": "https://github.com/sindresorhus/awesome.git",++ "svn_url": "https://github.com/sindresorhus/awesome",++ "homepage": "",++ "size": 1566,++ "stargazers_count": 535531,++ "watchers_count": 535531,++ "language": null,++ "has_issues": true,++ "has_projects": false,++ "has_downloads": false,++ "has_wiki": false,++ "has_pages": true,++ "has_discussions": false,++ "forks_count": 38323,++ "mirror_url": null,++ "archived": false,++ "disabled": false,++ "open_issues_count": 103,++ "license": {++ "key": "cc0-1.0",++ "name": "Creative Commons Zero v1.0 Universal",++ "spdx_id": "CC0-1.0",++ "url": "https://api.github.com/licenses/cc0-1.0",++ "node_id": "MDc6TGljZW5zZTY="++ },++ "allow_forking": true,++ "is_template": false,++ "web_commit_signoff_required": false,++ "has_pull_requests": true,++ "pull_request_creation_policy": "collaborators_only",++ "topics": [++ "awesome",++ "awesome-list",++ "lists",++ "resources",++ "unicorns"++ ],++ "visibility": "public",++ "forks": 38323,++ "open_issues": 103,++ "watchers": 535531,++ "default_branch": "main",++ "score": 1.0++ }++ ]++}nnew file mode 100644@@@ -0,0 +1,143 @@++'use strict';++/**++ * GitHub: the query plan, the recorded search answer mapped to items, the star-gain arithmetic from daily++ * snapshots, and the client's cursor, cache and rate-limit behaviour. The fetcher is a stub returning a++ * recorded api.github.com answer, so nothing here touches the network.++ */++const assert = require('assert');++const fs = require('fs');++const path = require('path');++const { suite } = require('./helpers');++const github = require('../server/adapters/github');++const { createGithub } = require('../server/github');++++const t = suite('github');++const RECORDED = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'github', 'search-top.json'), 'utf8'));++const CONFIG = {++ fetch: { userAgent: 'OpenVibeBot/test' },++ items: { summaryMax: 1000 },++ secrets: {},++};++const silent = { log() {}, warn() {}, error() {} };++const allowedRobots = { check: async () => ({ allowed: true, reason: null, crawlDelaySec: 0 }) };++const noSpace = { space: async () => {}, setCrawlDelay() {} };++++function fakeFetcher(handler) {++ return { fetchUrl: async (url, opts) => handler(url, opts) };++}++++t('the plan is today, this week, this month, recently pushed, all-time top, new, and twelve languages', () => {++ const plan = github.plan(Date.parse('2026-11-12T12:00:00Z'));++ assert.deepStrictEqual(plan.slice(0, 6).map((q) => q.key), ['trending', 'week', 'month', 'pushed', 'top', 'new']);++ assert.strictEqual(plan[0].q, 'created:>2026-11-12');++ assert.strictEqual(plan[1].q, 'created:>2026-11-05');++ assert.strictEqual(plan[2].q, 'created:>2026-10-13');++ assert.strictEqual(plan[3].q, 'pushed:>2026-10-13 stars:>100');++ assert.strictEqual(plan.length, 6 + 12);++ assert.deepStrictEqual(github.LANGUAGES.slice(0, 3), ['JavaScript', 'TypeScript', 'Python']);++ assert.ok(plan.some((q) => q.q.includes('language:"Rust"')));++});++++t('a recorded search answer becomes items with every stated field and null for the ones GitHub left out', () => {++ const items = RECORDED.items.map((r) => github.repoItem(r, 1000)).filter(Boolean);++ assert.strictEqual(items.length, 2);++ const first = items[0];++ assert.strictEqual(first.identity, 'codecrafters-io/build-your-own-x');++ assert.strictEqual(first.canonical_url, 'https://github.com/codecrafters-io/build-your-own-x');++ assert.strictEqual(first.fields.stars, 561961);++ assert.strictEqual(first.fields.language, 'Markdown');++ assert.ok(first.fields.owner_avatar_url.startsWith('https://avatars.githubusercontent.com/'));++ assert.ok(Array.isArray(first.fields.topics));++ assert.strictEqual(first.fields.license, null, 'no licence stated is null, never guessed');++ assert.strictEqual(first.fields.created, new Date(RECORDED.items[0].created_at).toISOString());++ const second = items[1];++ assert.ok(second.fields.license, 'a stated licence is kept');++ assert.ok(second.fields.homepage === null || second.fields.homepage.startsWith('https://'));++ const pub = github.publicRepository(first, { days_1: 12, days_7: null });++ assert.strictEqual(pub.stars_gained_1d, 12);++ assert.strictEqual(pub.stars_gained_7d, null);++ assert.strictEqual(pub.source, 'github');++});++++t('a body without an items array is a parse error, never an empty success', () => {++ assert.throws(() => github.parseSearch({ total_count: 0 }), /items array/);++ assert.throws(() => github.parseSearch(null), /items array/);++});++++t('stars gained come from daily snapshots; a window with no snapshot is null, never zero', () => {++ const now = Date.parse('2026-11-12T12:00:00Z');++ const history = [++ { day: '2026-11-04', stars: 100 },++ { day: '2026-11-10', stars: 180 },++ { day: '2026-11-11', stars: 195 },++ { day: '2026-11-12', stars: 210 },++ ];++ const g = github.gainsFrom(history, now);++ assert.deepStrictEqual([g.stars, g.days_1, g.days_7], [210, 15, 110]);++ const young = github.gainsFrom([{ day: '2026-11-12', stars: 14 }], now);++ assert.deepStrictEqual([young.stars, young.days_1, young.days_7], [14, null, null]);++ assert.deepStrictEqual(github.gainsFrom([], now), { stars: null, days_1: null, days_7: null });++});++++t('one run asks its budget of queries from the stored cursor, keeps the stars and returns the next cursor', async () => {++ const seen = [];++ const client = createGithub({++ config: CONFIG, robots: allowedRobots, spacer: noSpace, now: () => Date.parse('2026-11-12T12:00:00Z'), log: silent,++ fetcher: fakeFetcher(async (url) => {++ seen.push(url);++ return { status: 200, body: Buffer.from(JSON.stringify(RECORDED)), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '9', 'x-ratelimit-reset': '1794471413' }, bytes: 100, finalUrl: url };++ }),++ });++ const out = await client.fetchQueries({ source: { key: 'github', min_interval_ms: 0 }, cursor: {}, budget: 2 });++ assert.strictEqual(seen.length, 2);++ assert.match(seen[0], /created%3A%3E2026-11-12|created%3A%3E2026/);++ assert.strictEqual(out.queries, 2);++ assert.strictEqual(out.items.length, 2);++ assert.deepStrictEqual(out.stars.map((s) => s.full_name).sort(), ['codecrafters-io/build-your-own-x', 'sindresorhus/awesome']);++ assert.strictEqual(out.next, 2);++ const next = await client.fetchQueries({ source: { key: 'github', min_interval_ms: 0 }, cursor: { next: out.next }, budget: 1 });++ assert.strictEqual(next.queries, 1);++ assert.strictEqual(next.next, 3);++});++++t('a rate-limited answer backs off: the reset time is honoured and a partial batch keeps what it has', async () => {++ let calls = 0;++ const reset = Math.floor(Date.parse('2026-11-12T12:01:00Z') / 1000);++ const client = createGithub({++ config: CONFIG, robots: allowedRobots, spacer: noSpace, now: () => Date.parse('2026-11-12T12:00:00Z'), log: silent,++ fetcher: fakeFetcher(async (url) => {++ calls++;++ if (calls === 1) return { status: 200, body: Buffer.from(JSON.stringify(RECORDED)), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '1', 'x-ratelimit-reset': String(reset) }, bytes: 100, finalUrl: url };++ return { status: 403, body: Buffer.from('{"message":"API rate limit exceeded"}'), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': String(reset) }, bytes: 40, finalUrl: url };++ }),++ });++ const out = await client.fetchQueries({ source: { key: 'github', min_interval_ms: 0 }, budget: 3 });++ assert.strictEqual(out.queries, 1, 'the batch stopped at the refusal');++ assert.strictEqual(out.limited, true);++ assert.strictEqual(out.items.length, 2);++ assert.ok(out.notBefore >= Date.parse('2026-11-12T12:00:59Z'), 'the next run waits for GitHub reset');++ assert.ok(!JSON.stringify(out).includes('Authorization'), 'the token never reaches a result');++});++++t('a live search is cached for ten minutes and passes GitHub syntax through unchanged', async () => {++ let calls = 0;++ let seenUrl = '';++ const client = createGithub({++ config: CONFIG, robots: allowedRobots, spacer: noSpace, now: () => Date.parse('2026-11-12T12:00:00Z'), log: silent,++ fetcher: fakeFetcher(async (url) => {++ calls++;++ seenUrl = url;++ assert.match(decodeURIComponent(url).replace(/\+/g, ' '), /stars:>500 topic:ai language:"Rust"/);++ return { status: 200, body: Buffer.from(JSON.stringify(RECORDED)), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '8', 'x-ratelimit-reset': '1794471413' }, bytes: 100, finalUrl: url };++ }),++ });++ const first = await client.search({ q: 'stars:>500 topic:ai', lang: 'Rust', sort: 'stars' });++ const second = await client.search({ q: 'stars:>500 topic:ai', lang: 'Rust', sort: 'stars' });++ assert.strictEqual(calls, 1, 'the second read is the cache');++ assert.strictEqual(second.cached, true);++ assert.strictEqual(first.items.length, 2);++ assert.strictEqual(first.page, 1);++ assert.match(decodeURIComponent(seenUrl), /sort=stars/);++});++++t.run();
@@ -85,8 +85,10 @@ VALKEY_PREFIX=
@@ -85,8 +85,10 @@ VALKEY_PREFIX=
#SOURCES_MAX_ITEMS_PER_FETCH=12000
#SOURCES_MAX_ITEMS_PER_FETCH=12000
# Data-source keys (D11 owner settings): the admin panel's sources.data_sources namespace wins; this
# Data-source keys (D11 owner settings): the admin panel's sources.data_sources namespace wins; this
-# is the fallback when no value was entered there.
+# is the fallback when no value was entered there. GITHUB_TOKEN is the last fallback (a read-only token:
+# 30 search requests a minute instead of 10; the name only, never the value).
#SOURCES_GITHUB_TOKEN=
#SOURCES_GITHUB_TOKEN=
+#GITHUB_TOKEN=
# Source credentials: a registry entry names the variable (auth.env), never the value. Names
# Source credentials: a registry entry names the variable (auth.env), never the value. Names
# must start with SOURCES_CRED_. Example (none of the seeds needs one):
# must start with SOURCES_CRED_. Example (none of the seeds needs one):
@@ -57,10 +57,12 @@ start and flushes on stop.
@@ -57,10 +57,12 @@ start and flushes on stop.
## The registry
## The registry
A source records: `key`, `name`, `type` (`rss|atom|sitemap|jsonld|api|bulk|crawl|manual`), `category`
A source records: `key`, `name`, `type` (`rss|atom|sitemap|jsonld|api|bulk|crawl|manual`), `category`
-(`news|blog|reviews|deals|coupons|trade|hosting|crowdfunding|games|housing|food|jobs`, read from `sources.source@1`; housing, food and jobs since contracts 0.210.0), `endpoints` (for `api`: a field mapping, no code per
+(`news|blog|reviews|deals|coupons|trade|hosting|crowdfunding|games|housing|food|jobs|code`, read from `sources.source@1`; housing, food and jobs since contracts 0.210.0, code since 0.411.1), `endpoints` (for `api`: a field mapping, no code per
provider; for `bulk`: one dataset file with a format, a byte cap and a row limit), `auth` (`none|header|bearer|query` with the **name** of an environment variable —
provider; for `bulk`: one dataset file with a format, a byte cap and a row limit), `auth` (`none|header|bearer|query` with the **name** of an environment variable —
names must start with `SOURCES_CRED_`, so an entry can never point at this service's own
names must start with `SOURCES_CRED_`, so an entry can never point at this service's own
-secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`),
+secrets), `robots_note`, `terms_note`, `license_note`, the rate limit (`min_interval_ms`) — the GitHub read uses
+`GITHUB_TOKEN` (or `SOURCES_GITHUB_TOKEN`, or the owner's D11 setting) by name when set, and is anonymous within
+GitHub's 10-a-minute search limit otherwise —
`poll_interval_sec`, `stale_after_sec`, `max_items`, `enabled`, `review_required`,
`poll_interval_sec`, `stale_after_sec`, `max_items`, `enabled`, `review_required`,
`sensitivity`, `default_indexability`, `search_visibility` and `retire_after_sec` (since contracts 0.211.0).
`sensitivity`, `default_indexability`, `search_visibility` and `retire_after_sec` (since contracts 0.211.0).
@@ -108,7 +110,7 @@ succeeded. Health status: `healthy | stale | failing | never_fetched | disabled
@@ -108,7 +110,7 @@ succeeded. Health status: `healthy | stale | failing | never_fetched | disabled
| `rss`, `atom` (`feed@3`) | RSS 2.0, RSS 1.0/RDF, Atom 1.0 | guid/id identity, else link; text-only summary capped at 1000 chars; a deal feed's own offer elements (DealNews' `dealnews:price` with its currency, `dealnews:retailer`, `dealnews:expires`) become `fields.price`, `currency`, `seller` and `valid_until` exactly as stated (the item stays an article; nothing is read out of the headline); its `dealnews:category` fills `fields.categories` when the item has no `<category>`, and `dealnews:staffPick` is `fields.staff_pick` (true/false, absent when not stated) |
| `rss`, `atom` (`feed@3`) | RSS 2.0, RSS 1.0/RDF, Atom 1.0 | guid/id identity, else link; text-only summary capped at 1000 chars; a deal feed's own offer elements (DealNews' `dealnews:price` with its currency, `dealnews:retailer`, `dealnews:expires`) become `fields.price`, `currency`, `seller` and `valid_until` exactly as stated (the item stays an article; nothing is read out of the headline); its `dealnews:category` fills `fields.categories` when the item has no `<category>`, and `dealnews:staffPick` is `fields.staff_pick` (true/false, absent when not stated) |
| `sitemap` (`sitemap@1`) | `<urlset>` (+ Google News extension), `<sitemapindex>` | index children are listed, not followed |
| `sitemap` (`sitemap@1`) | `<urlset>` (+ Google News extension), `<sitemapindex>` | index children are listed, not followed |
| `jsonld` (`jsonld@2`) | schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) | prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error`; `list_price` from a ListPrice/StrikethroughPrice/MSRP/SRP `priceSpecification`, as stated |
| `jsonld` (`jsonld@2`) | schema.org Product, Offer/AggregateOffer, Review, NewsArticle, BlogPosting (+ Article) | prices, ratings and availability exactly as the page states; missing = null; embedded review text not kept; a page without JSON-LD is a `parse_error`; `list_price` from a ListPrice/StrikethroughPrice/MSRP/SRP `priceSpecification`, as stated |
-| `api` (`api@1`) | JSON, XML, TSV or CSV official files via a registry mapping | `items_path`, `items_shape` (`array` default, `map` for records keyed by id), `fields`, `extra`, `where`/`filter`; epoch fields only where mapped as such; `resolver: "hn-whoishiring"` walks the Hacker News thread's JSON documents instead |
+| `api` (`api@1`) | JSON, XML, TSV or CSV official files via a registry mapping | `items_path`, `items_shape` (`array` default, `map` for records keyed by id), `fields`, `extra`, `where`/`filter`; epoch fields only where mapped as such; `resolver: "hn-whoishiring"` walks the Hacker News thread's JSON documents instead; `resolver: "github"` walks GitHub's repository search plan (server/github.js) |
| `bulk` (`bulk@1`) | one dataset file, streamed (CSV, JSONL or TSV) | `max_bytes` is the size cap, `max_rows` the per-run row limit; the stream stops at the limit, so a multi-GB file is never held in memory. `resolver: "off-delta"` reads a text index and only the gzipped JSONL delta parts it lists, keeping the last finished part and the current part's row offset in `source_state` (Open Food Facts) |
| `bulk` (`bulk@1`) | one dataset file, streamed (CSV, JSONL or TSV) | `max_bytes` is the size cap, `max_rows` the per-run row limit; the stream stops at the limit, so a multi-GB file is never held in memory. `resolver: "off-delta"` reads a text index and only the gzipped JSONL delta parts it lists, keeping the last finished part and the current part's row offset in `source_state` (Open Food Facts) |
| `crawl` (`jsonld@2`, `page@1`) | a whole site from its seeds: its sitemaps and, if asked, its links (see below) | only pages that publish JSON-LD are kept, unless `require_structured: false` keeps a `page` item from the page's own search metadata |
| `crawl` (`jsonld@2`, `page@1`) | a whole site from its seeds: its sitemaps and, if asked, its links (see below) | only pages that publish JSON-LD are kept, unless `require_structured: false` keeps a `page` item from the page's own search metadata |
| `manual` (`manual@1`) | items entered through the API with an evidence URL | no fetcher |
| `manual` (`manual@1`) | items entered through the API with an evidence URL | no fetcher |
@@ -261,7 +263,7 @@ the endpoint's last good parse saw. At most 500 items are retired a run; the res
@@ -261,7 +263,7 @@ the endpoint's last good parse saw. At most 500 items are retired a run; the res
[`sources.item@1`](docs/contracts-proposal/).
[`sources.item@1`](docs/contracts-proposal/).
**Retention.** Items do not grow forever: `server/retention.js` declares, in one place, a window per category (news 30
**Retention.** Items do not grow forever: `server/retention.js` declares, in one place, a window per category (news 30
-days, blog 90, reviews 180, deals 14, coupons 90, trade 365, hosting 30, crowdfunding 90, games 14, food 365, jobs 14)
+days, blog 90, reviews 180, deals 14, coupons 90, trade 365, hosting 30, crowdfunding 90, games 14, food 365, jobs 14, code 30)
and one `keepLatest` rule for housing, where HUD's yearly Fair Market Rents must keep the latest value per source key.
and one `keepLatest` rule for housing, where HUD's yearly Fair Market Rents must keep the latest value per source key.
An item's age is `retrieved_at` while it is live and `removed_at` once removed, so a consumer following the change feed
An item's age is `retrieved_at` while it is live and `removed_at` once removed, so a consumer following the change feed
still sees a removal for the full window; `first_seen_at` is never the age, since a source that keeps listing a job or
still sees a removal for the full window; `first_seen_at` is never the age, since a source that keeps listing a job or
@@ -283,7 +285,7 @@ service calls them with its service token and `X-OV-Subject: usr_…`.
@@ -283,7 +285,7 @@ service calls them with its service token and `X-OV-Subject: usr_…`.
| Capability | Routes |
| Capability | Routes |
|---|---|
|---|---|
| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health`; `GET /api/v1/merchant/sources[/:id]` |
| `sources.source.read` | `GET /api/v1/sources`, `/api/v1/sources/:key`, `/api/v1/sources/:key/runs?before=`, `/api/v1/runs?state=failed&after=`, `/api/v1/health`; `GET /api/v1/merchant/sources[/:id]` |
-| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1` |
+| `sources.item.read` | `GET /api/v1/items?source=&category=&after=<change_seq>&include_removed=1`, `GET /api/v1/items/:id?revisions=1`; `GET /api/v1/code/repos?list=trending\|week\|month\|hot\|new\|top&lang=&topic=&limit=`, `GET /api/v1/code/search?q=&lang=&sort=stars\|updated\|best&page=` |
| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}`; `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` |
| `sources.source.manage` | `POST /api/v1/sources`, `PATCH`/`DELETE /api/v1/sources/:key`, `POST /api/v1/sources/:key/fetch`, `POST /api/v1/sources/:key/items` (manual), `DELETE /api/v1/items/:id` `{reason}`; `POST /api/v1/merchant/sources`, `/:id/verify`, `/:id/withdraw` |
`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —
`GET /api/v1/items` pages in change order (creations, revisions and — with `include_removed=1` —
@@ -367,7 +369,7 @@ for the N-1 rollback window.
@@ -367,7 +369,7 @@ for the N-1 rollback window.
## Seeds
## Seeds
-[seeds/sources.json](seeds/sources.json) holds the 67 registered sources, each with its robots, terms and licence notes (the production rows are the record; the terms notes match them as of the 2026-10-23 review).
+[seeds/sources.json](seeds/sources.json) holds the 69 registered sources, each with its robots, terms and licence notes (the production rows are the record; the terms notes match them as of the 2026-10-23 review).
`npm run seed` creates the ones that do not exist yet, **all disabled**, and never changes an existing one; a person
`npm run seed` creates the ones that do not exist yet, **all disabled**, and never changes an existing one; a person
enables a source after reading its terms (`PATCH /api/v1/sources/:key {"enabled": true}`), and the fetcher re-checks
enables a source after reading its terms (`PATCH /api/v1/sources/:key {"enabled": true}`), and the fetcher re-checks
robots.txt on every run. Production on 2026-10-22 (the food row checked 2026-11-06):
robots.txt on every run. Production on 2026-10-22 (the food row checked 2026-11-06):
@@ -384,6 +386,7 @@ robots.txt on every run. Production on 2026-10-22 (the food row checked 2026-11-
@@ -384,6 +386,7 @@ robots.txt on every run. Production on 2026-10-22 (the food row checked 2026-11-
| crowdfunding | `opencollective-accounts` (api) | 1 of 1 |
| crowdfunding | `opencollective-accounts` (api) | 1 of 1 |
| games | `steam-top-sellers`, `steam-new-releases`, `steam-specials` (api: the store's featured lists, in their order) | 3 of 3 |
| games | `steam-top-sellers`, `steam-new-releases`, `steam-specials` (api: the store's featured lists, in their order) | 3 of 3 |
| food | `open-food-facts`, `openfda-food-enforcement`, `bls-average-prices`, `usda-ers-fruit-vegetable-prices` (api); `themealdb` | 4 of 5: `themealdb` is off (it needs a key, which is the owner's call); `usda-ers-fruit-vegetable-prices` was enabled on 2026-11-06 after its terms review, and its 155 prices are recorded in `test/fixtures/` |
| food | `open-food-facts`, `openfda-food-enforcement`, `bls-average-prices`, `usda-ers-fruit-vegetable-prices` (api); `themealdb` | 4 of 5: `themealdb` is off (it needs a key, which is the owner's call); `usda-ers-fruit-vegetable-prices` was enabled on 2026-11-06 after its terms review, and its 155 prices are recorded in `test/fixtures/` |
+| code | `github` (api, `resolver: "github"`: REST search only, star history per day) | seeded disabled: enable after the terms review |
Candidates rejected on 2026-09-22: CheapShark's API (`/api/1.0/` is disallowed in its
Candidates rejected on 2026-09-22: CheapShark's API (`/api/1.0/` is disallowed in its
robots.txt) and EDGAR's `getcurrent` Atom feed (`/cgi-bin` disallowed). The seeded feeds were
robots.txt) and EDGAR's `getcurrent` Atom feed (`/cgi-bin` disallowed). The seeded feeds were
@@ -9,7 +9,7 @@
@@ -9,7 +9,7 @@
"properties": {
"properties": {
"id": { "type": "string", "pattern": "^itm_[0-9A-HJKMNP-TV-Z]{26}$" },
"id": { "type": "string", "pattern": "^itm_[0-9A-HJKMNP-TV-Z]{26}$" },
"source_key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },
"source_key": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },
- "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade", "hosting", "crowdfunding", "games", "housing", "food", "jobs"] },
+ "category": { "enum": ["news", "blog", "reviews", "deals", "coupons", "trade", "hosting", "crowdfunding", "games", "housing", "food", "jobs", "code"] },
"kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$", "description": "article | url | sitemap | product | offer | review | review_signal | record | coupon …" },
"kind": { "type": "string", "pattern": "^[a-z][a-z0-9_]{1,39}$", "description": "article | url | sitemap | product | offer | review | review_signal | record | coupon …" },
"identity": { "type": "string", "minLength": 1, "maxLength": 2048, "description": "The source's own id (guid, @id, API id) or the canonical URL." },
"identity": { "type": "string", "minLength": 1, "maxLength": 2048, "description": "The source's own id (guid, @id, API id) or the canonical URL." },
"canonical_url": { "type": ["string", "null"], "format": "uri" },
"canonical_url": { "type": ["string", "null"], "format": "uri" },
@@ -56,7 +56,8 @@
@@ -56,7 +56,8 @@
"games",
"games",
"housing",
"housing",
"food",
"food",
- "jobs"
+ "jobs",
+ "code"
]
]
},
},
"homepage_url": {
"homepage_url": {
@@ -166,9 +167,10 @@
@@ -166,9 +167,10 @@
"resolver": {
"resolver": {
"enum": [
"enum": [
"hn-whoishiring",
"hn-whoishiring",
- "off-delta"
+ "off-delta",
+ "github"
],
],
- "description": "An api endpoint whose records are walked from several JSON URLs instead of one items_path, or a bulk endpoint whose rows come from the gzipped JSONL parts a text index lists. hn-whoishiring reads user/whoishiring.json, the latest \"Who is hiring\" story's top-level kids, and each item/<id>.json; it stores one job per comment, a short excerpt with emails and phones removed, and links the comment (Sources, 2026-10-31). off-delta reads index.txt and then only the delta parts not finished yet, resuming a part at its stored row offset (Open Food Facts; Sources, 2026-10-31). The endpoint URL is a .json URL for hn-whoishiring and the index's .txt URL for off-delta."
+ "description": "An api endpoint whose records are walked from several JSON URLs instead of one items_path, or a bulk endpoint whose rows come from the gzipped JSONL parts a text index lists. hn-whoishiring reads user/whoishiring.json, the latest \"Who is hiring\" story's top-level kids, and each item/<id>.json; it stores one job per comment, a short excerpt with emails and phones removed, and links the comment (Sources, 2026-10-31). off-delta reads index.txt and then only the delta parts not finished yet, resuming a part at its stored row offset (Open Food Facts; Sources, 2026-10-31). The endpoint URL is a .json URL for hn-whoishiring and the index's .txt URL for off-delta. github walks a share of a search plan on api.github.com/search/repositories per run (new repositories today, this week and this month by stars, recently pushed, all-time top, and twelve languages), keeps a star count per repository per day, and returns repository items; its endpoint URL is an api.github.com URL and never a github.com page (Sources, 2026-11-12)."
},
},
"item_kind": {
"item_kind": {
"type": "string",
"type": "string",
new file mode 100644
new file mode 100644
@@ -0,0 +1,14 @@
@@ -0,0 +1,14 @@
+-- phase: expand
+-- A GitHub repository's star count per day (server/adapters/github.js, server/github.js). The code API computes
+-- "stars gained over 1 day / 7 days" from these snapshots, so a repository's age in the tracked set is visible:
+-- a repository first seen yesterday has a 1-day gain and a null 7-day gain, never a fabricated zero.
+-- One row per source, repository and UTC day; a later sighting the same day replaces the count.
+CREATE TABLE IF NOT EXISTS github_stars (
+ source_key text COLLATE "C" NOT NULL,
+ full_name text COLLATE "C" NOT NULL,
+ day text COLLATE "C" NOT NULL,
+ stars integer NOT NULL CHECK (stars >= 0),
+ captured_at bigint NOT NULL,
+ PRIMARY KEY (source_key, full_name, day)
+);
+CREATE INDEX IF NOT EXISTS idx_github_stars_day ON github_stars (source_key, day);
@@ -12,7 +12,7 @@
@@ -12,7 +12,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"fast-xml-parser": "^4.5.3",
"fast-xml-parser": "^4.5.3",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.410.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.411.1",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.150.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.150.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.214.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.214.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -1125,9 +1125,9 @@
@@ -1125,9 +1125,9 @@
}
}
},
},
"node_modules/openvibe-contracts": {
"node_modules/openvibe-contracts": {
- "version": "0.410.0",
- "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.410.0",
- "integrity": "sha512-rmmq2mebXim2fv+sip2YQ0PrrZLxGUhEfe1w+1EFfGdoVvLI5tp0/eavY2sW6/VU+rzs7cQMC/A0NZD5th5wrA==",
+ "version": "0.411.1",
+ "resolved": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.411.1",
+ "integrity": "sha512-WuIkcAdEaGwsz6Wsyt1Ear2hGLm4vr+NSait0MEVtJBiinwGBU7AgBTGFnUVCMRg4V40EfJZSYXXk7g4buduCA==",
"license": "AGPL-3.0-only",
"license": "AGPL-3.0-only",
"dependencies": {
"dependencies": {
"ajv": "^8.17.1",
"ajv": "^8.17.1",
@@ -30,7 +30,7 @@
@@ -30,7 +30,7 @@
"dotenv": "^16.4.7",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"express": "^4.21.2",
"fast-xml-parser": "^4.5.3",
"fast-xml-parser": "^4.5.3",
- "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.410.0",
+ "openvibe-contracts": "https://codeload.github.com/OpenVibers/OpenVibe.Contracts/tar.gz/refs/tags/v0.411.1",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.150.0",
"openvibe-sdk": "https://codeload.github.com/OpenVibers/OpenVibe.SDK/tar.gz/refs/tags/v0.150.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.214.0",
"openvibe-shared": "https://codeload.github.com/OpenVibers/OpenVibe.Shared/tar.gz/refs/tags/v4.214.0",
"pg": "^8.23.0"
"pg": "^8.23.0"
@@ -2604,6 +2604,34 @@
@@ -2604,6 +2604,34 @@
"review_required": true,
"review_required": true,
"sensitivity": "none",
"sensitivity": "none",
"default_indexability": "noindex"
"default_indexability": "noindex"
+ },
+ {
+ "key": "github",
+ "name": "GitHub: trending, top and new repositories",
+ "type": "api",
+ "category": "code",
+ "homepage_url": "https://github.com/",
+ "endpoints": [
+ {
+ "url": "https://api.github.com/search/repositories?q=stars:%3E10000&sort=stars&order=desc&per_page=100",
+ "format": "json",
+ "resolver": "github",
+ "item_kind": "repository",
+ "max_bytes": 4194304,
+ "timeout_ms": 30000
+ }
+ ],
+ "robots_note": "api.github.com/robots.txt checked 2026-11-12 from openvibe-ovh: HTTP 404, so RFC 9309 leaves the REST API unrestricted (4xx means no rules). No request goes to github.com or its /trending page; only api.github.com/search/repositories. The fetcher re-checks robots.txt on every run and on every redirect hop.",
+ "terms_note": "GitHub's REST API is documented for automated use (https://docs.github.com/rest/search/search#search-repositories) and the Acceptable Use Policies allow reading public repository metadata through the API while forbidding scraping the website or circumventing rate limits (https://docs.github.com/site-policy/acceptable-use-policies/github-acceptable-use-policies). This source reads only the search API, passes no personal data, stores only what a public repository states about itself (full name, URL, description, language, topics, stars, forks, open issues, licence, dates, owner avatar, homepage), links every item to GitHub, and never copies repository code or README text. It honours the search rate-limit headers and waits out a refusal. Re-verify before enabling.",
+ "license_note": "Repository metadata is read under GitHub's API terms; each repository's own licence is shown only as GitHub reports it (SPDX id or name), never assumed. The repository, its code and its text remain the owner's.",
+ "min_interval_ms": 6500,
+ "poll_interval_sec": 120,
+ "stale_after_sec": 21600,
+ "max_items": 3000,
+ "enabled": false,
+ "review_required": true,
+ "sensitivity": "none",
+ "default_indexability": "noindex"
}
}
]
]
}
}
new file mode 100644
new file mode 100644
@@ -0,0 +1,174 @@
@@ -0,0 +1,174 @@
+'use strict';
+/**
+ * GitHub repositories for Sources, read from the REST search API only (api.github.com/search/repositories),
+ * never from github.com/trending. One run walks a share of the query plan:
+ *
+ * - repositories created today, this week and this month, by stars;
+ * - the most-starred repositories pushed in the last 30 days;
+ * - the all-time most-starred;
+ * - the same recent list per language, for the twelve languages with the most repositories on GitHub.
+ *
+ * Search is limited to 10 requests a minute without a token and 30 with one, so the cursor lives in
+ * source_state and a run asks for at most its budget of queries; the remaining ones come on the next run.
+ * The scheduler decides when that is (never an interval loop inside a fetch).
+ *
+ * A star count is kept per repository per day (github_stars). The API computes stars gained over 1 and 7
+ * days from those snapshots: a repository first seen yesterday has no 7-day figure and says null, never 0.
+ *
+ * The mapping keeps what the repository states: full name, URL, description, language, topics, stars,
+ * forks, open issues, licence, created, pushed, owner avatar and homepage. A field GitHub does not state
+ * is null.
+ */
+const { toText, toIsoDate, canonicalUrl, toId } = require('./util');
+
+const PARSER_VERSION = 'github@1';
+const STATE_NAME = 'github.cursor';
+const API = 'https://api.github.com/search/repositories';
+const PER_PAGE = 100;
+
+// The twelve languages GitHub's own search facets put at the top of public repositories. A list, not a
+// count: the plan is stable and reviewable, and a language added later starts on the next run.
+const LANGUAGES = Object.freeze(['JavaScript', 'TypeScript', 'Python', 'Go', 'Rust', 'Java', 'C++', 'C#', 'PHP', 'Ruby', 'Swift', 'Kotlin']);
+
+const isoDay = (ms) => new Date(ms).toISOString().slice(0, 10);
+const daysAgo = (ms, n) => isoDay(ms - n * 86400000);
+
+/** The queries, in a fixed order: base lists first, then one recent list per language. */
+function plan(now = Date.now()) {
+ const today = isoDay(now);
+ const week = daysAgo(now, 7);
+ const month = daysAgo(now, 30);
+ const pushed = daysAgo(now, 30);
+ const base = [
+ { key: 'trending', q: `created:>${today}`, sort: 'stars', order: 'desc' },
+ { key: 'week', q: `created:>${week}`, sort: 'stars', order: 'desc' },
+ { key: 'month', q: `created:>${month}`, sort: 'stars', order: 'desc' },
+ { key: 'pushed', q: `pushed:>${pushed} stars:>100`, sort: 'stars', order: 'desc' },
+ { key: 'top', q: 'stars:>10000', sort: 'stars', order: 'desc' },
+ { key: 'new', q: `created:>${week}`, sort: 'updated', order: 'desc' },
+ ];
+ const languages = LANGUAGES.map((language) => ({
+ key: `lang:${language}`,
+ q: `language:"${language}" created:>${month}`,
+ sort: 'stars', order: 'desc',
+ }));
+ return [...base, ...languages];
+}
+
+function queryUrl(query) {
+ const u = new URL(API);
+ u.searchParams.set('q', query.q);
+ if (query.sort) u.searchParams.set('sort', query.sort);
+ if (query.order) u.searchParams.set('order', query.order);
+ u.searchParams.set('per_page', String(PER_PAGE));
+ return u.toString();
+}
+
+/** The public item shape: what Exchange and other products read. A number GitHub did not state is null. */
+function publicRepository(item, gains = {}) {
+ const f = item.fields || {};
+ return {
+ full_name: item.identity,
+ url: item.canonical_url,
+ description: item.summary,
+ language: f.language == null ? null : f.language,
+ topics: Array.isArray(f.topics) ? f.topics : [],
+ stars: f.stars == null ? null : f.stars,
+ stars_gained_1d: gains.days_1 == null ? null : gains.days_1,
+ stars_gained_7d: gains.days_7 == null ? null : gains.days_7,
+ forks: f.forks == null ? null : f.forks,
+ open_issues: f.open_issues == null ? null : f.open_issues,
+ license: f.license == null ? null : f.license,
+ created: f.created == null ? null : f.created,
+ pushed: f.pushed == null ? null : f.pushed,
+ owner_avatar_url: f.owner_avatar_url == null ? null : f.owner_avatar_url,
+ homepage: f.homepage == null ? null : f.homepage,
+ source: 'github',
+ };
+}
+
+function repoItem(repo, summaryMax = 1000) {
+ const full = toId(repo && repo.full_name, 200);
+ if (!full) return null;
+ const owner = repo.owner && typeof repo.owner === 'object' ? repo.owner : {};
+ const license = repo.license && typeof repo.license === 'object' ? repo.license : null;
+ const stars = Number.isFinite(Number(repo.stargazers_count)) ? Math.max(0, Math.trunc(Number(repo.stargazers_count))) : null;
+ const fields = {
+ stars,
+ forks: Number.isFinite(Number(repo.forks_count)) ? Math.max(0, Math.trunc(Number(repo.forks_count))) : null,
+ open_issues: Number.isFinite(Number(repo.open_issues_count)) ? Math.max(0, Math.trunc(Number(repo.open_issues_count))) : null,
+ language: toText(repo.language, 60),
+ topics: Array.isArray(repo.topics) ? repo.topics.map((t) => toText(t, 50)).filter(Boolean).slice(0, 20) : [],
+ // SPDX id when GitHub has one; NOASSERTION means GitHub read a licence file but could not name it,
+ // so the repository's own string is kept instead of inventing an id.
+ license: license ? (license.spdx_id && license.spdx_id !== 'NOASSERTION' ? toText(license.spdx_id, 80) : toText(license.name, 120)) : null,
+ created: toIsoDate(repo.created_at),
+ pushed: toIsoDate(repo.pushed_at),
+ owner_avatar_url: canonicalUrl(owner.avatar_url),
+ homepage: canonicalUrl(repo.homepage),
+ archived: repo.archived === true,
+ fork: repo.fork === true,
+ };
+ return {
+ identity: full,
+ kind: 'repository',
+ canonical_url: canonicalUrl(repo.html_url) || `https://github.com/${full}`,
+ title: full,
+ summary: toText(repo.description, summaryMax),
+ authors: owner.login ? [toText(owner.login, 200)].filter(Boolean) : [],
+ published_at: fields.created,
+ updated_at: fields.pushed,
+ fields,
+ };
+}
+
+/** Parse one search answer. A body without an items array is a parse error, never an empty success. */
+function parseSearch(body) {
+ if (!body || typeof body !== 'object' || !Array.isArray(body.items)) {
+ throw Object.assign(new Error('GitHub search did not return an items array'), { code: 'parse_error' });
+ }
+ return { items: body.items, total: Number.isFinite(Number(body.total_count)) ? Number(body.total_count) : null, incomplete: body.incomplete_results === true };
+}
+
+/**
+ * Keep one star count per repository per day. Called inside the run transaction; a repository seen twice in
+ * one run keeps the last count, which is the same answer GitHub gave later.
+ */
+async function recordStars(db, sourceKey, list, at) {
+ if (!Array.isArray(list) || !list.length) return 0;
+ const day = isoDay(at);
+ const stmt = db.prepare(`INSERT INTO github_stars (source_key, full_name, day, stars, captured_at) VALUES (@source_key, @full_name, @day, @stars, @captured_at)
+ ON CONFLICT (source_key, full_name, day) DO UPDATE SET stars = excluded.stars, captured_at = excluded.captured_at`);
+ let n = 0;
+ for (const row of list) {
+ if (!row || !row.full_name || !Number.isFinite(row.stars)) continue;
+ await stmt.run({ source_key: sourceKey, full_name: row.full_name, day, stars: row.stars, captured_at: at });
+ n++;
+ }
+ return n;
+}
+
+/** Stars gained over 1 and 7 days from daily snapshots; a window with no snapshot is null. */
+function gainsFrom(history, todayMs) {
+ const today = isoDay(todayMs);
+ const yesterday = isoDay(todayMs - 86400000);
+ const weekAgo = isoDay(todayMs - 7 * 86400000);
+ const earlier = (limit) => {
+ let best = null;
+ for (const h of history) if (h.day <= limit && (best == null || h.day > best.day)) best = h;
+ return best;
+ };
+ const latest = earlier(today);
+ const d1 = earlier(yesterday);
+ const d7 = earlier(weekAgo);
+ const stars = latest ? latest.stars : null;
+ return {
+ stars,
+ days_1: stars != null && d1 ? stars - d1.stars : null,
+ days_7: stars != null && d7 ? stars - d7.stars : null,
+ };
+}
+
+module.exports = {
+ PARSER_VERSION, STATE_NAME, API, LANGUAGES, plan, queryUrl, repoItem, parseSearch, publicRepository, recordStars, gainsFrom, isoDay,
+};
@@ -10,6 +10,7 @@ const jsonld = require('./jsonld');
@@ -10,6 +10,7 @@ const jsonld = require('./jsonld');
const api = require('./api');
const api = require('./api');
const bulk = require('./bulk');
const bulk = require('./bulk');
const hn = require('./hn');
const hn = require('./hn');
+const github = require('./github');
const ADAPTERS = {
const ADAPTERS = {
rss: { parse: feed.parse, version: feed.PARSER_VERSION, accept: 'application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1' },
rss: { parse: feed.parse, version: feed.PARSER_VERSION, accept: 'application/rss+xml, application/atom+xml, application/xml;q=0.9, text/xml;q=0.8, */*;q=0.1' },
@@ -24,7 +25,12 @@ const ADAPTERS = {
@@ -24,7 +25,12 @@ const ADAPTERS = {
};
};
// A resolver reads several URLs of one API and returns the same { items, skipped, meta } an adapter's parse does.
// A resolver reads several URLs of one API and returns the same { items, skipped, meta } an adapter's parse does.
-const RESOLVERS = { 'hn-whoishiring': { resolve: hn.resolve, version: hn.PARSER_VERSION } };
+const RESOLVERS = {
+ 'hn-whoishiring': { resolve: hn.resolve, version: hn.PARSER_VERSION },
+ // GitHub is walked by the client in server/github.js (a share of the search plan per run, rate-limit
+ // aware); the resolver entry names its parser version for every item it produces.
+ github: { resolve: null, version: github.PARSER_VERSION },
+};
const MANUAL_VERSION = 'manual@1';
const MANUAL_VERSION = 'manual@1';
new file mode 100644
new file mode 100644
@@ -0,0 +1,148 @@
@@ -0,0 +1,148 @@
+'use strict';
+/**
+ * Code API (service tokens): GitHub repositories kept by the `github` source.
+ *
+ * GET /api/v1/code/repos?list=trending|week|month|hot|new|top&lang=&topic=&limit=
+ * The cached rankings. `hot` is stars gained in the last 24 hours and 7 days from github_stars,
+ * never a total. A list with no reading for a repository leaves its gain null.
+ * GET /api/v1/code/search?q=&lang=&sort=stars|updated|best&page=
+ * A live GitHub repository search through the same client as the scheduled fetch, cached ten
+ * minutes, with GitHub's own query syntax passed through.
+ *
+ * Both need sources.item.read. The public API is Exchange's; it reads these over loopback.
+ */
+const express = require('express');
+const { http } = require('openvibe-contracts');
+const { CAPS } = require('../auth');
+const github = require('../adapters/github');
+
+const LISTS = Object.freeze(['trending', 'week', 'month', 'hot', 'new', 'top']);
+const SORTS = Object.freeze(['stars', 'updated', 'best']);
+const DAY = 86400000;
+
+const text = (v, max) => (v == null ? null : String(v).trim().slice(0, max) || null);
+const number = (v) => (Number.isFinite(Number(v)) ? Number(v) : null);
+
+function rowItem(row) {
+ let fields = {};
+ try { fields = JSON.parse(row.fields) || {}; } catch { fields = {}; }
+ return {
+ identity: row.identity, canonical_url: row.canonical_url, summary: row.summary,
+ title: row.title, fields,
+ };
+}
+
+async function historyFor(db, names, now) {
+ if (!names.length) return new Map();
+ const since = github.isoDay(now - 8 * DAY);
+ const rows = (await db.query('SELECT full_name, day, stars FROM github_stars WHERE day >= $1 AND full_name = ANY($2)', [since, names])).rows;
+ const byName = new Map();
+ for (const r of rows) {
+ if (!byName.has(r.full_name)) byName.set(r.full_name, []);
+ byName.get(r.full_name).push({ day: r.day, stars: Number(r.stars) });
+ }
+ return byName;
+}
+
+function withGains(rows, history, now) {
+ return rows.map((row) => {
+ const item = rowItem(row);
+ const gains = github.gainsFrom(history.get(item.identity) || [], now);
+ return { item, gains, repo: github.publicRepository(item, gains) };
+ });
+}
+
+function sortFor(list) {
+ return (a, b) => {
+ if (list === 'hot') {
+ const a1 = a.gains.days_1, b1 = b.gains.days_1;
+ if (a1 != null && b1 != null && a1 !== b1) return b1 - a1;
+ if (a1 != null && b1 == null) return -1;
+ if (a1 == null && b1 != null) return 1;
+ const a7 = a.gains.days_7, b7 = b.gains.days_7;
+ if (a7 != null && b7 != null && a7 !== b7) return b7 - a7;
+ }
+ if (list === 'new') {
+ const ac = Date.parse(a.repo.created || 0) || 0;
+ const bc = Date.parse(b.repo.created || 0) || 0;
+ if (ac !== bc) return bc - ac;
+ }
+ const as = number(a.repo.stars), bs = number(b.repo.stars);
+ if (as != null && bs != null && as !== bs) return bs - as;
+ if (as != null && bs == null) return -1;
+ if (as == null && bs != null) return 1;
+ return String(a.repo.full_name).localeCompare(String(b.repo.full_name));
+ };
+}
+
+function createCodeRouter({ db, registry, auth, limits, github: client }) {
+ const router = express.Router();
+ const read = [auth.requireCap(CAPS.items), limits.reads('sources.code')];
+
+ router.get('/api/v1/code/repos', read, async (req, res) => {
+ const list = LISTS.includes(String(req.query.list || 'trending')) ? String(req.query.list || 'trending') : null;
+ if (!list) return http.sendProblem(res, 422, 'sources.bad_query', { detail: `list must be one of ${LISTS.join('|')}` });
+ const lang = text(req.query.lang, 60);
+ const topic = text(req.query.topic, 50);
+ const limit = Math.min(Math.max(parseInt(req.query.limit, 10) || 30, 1), 100);
+ const t = Date.now();
+ const dayStart = Date.parse(github.isoDay(t));
+ // Ordering by stars happens in JS: the tracked set is bounded by the source's max_items (3,000) and
+ // the gains need the snapshots anyway. A filter narrows it before the sort.
+ const rows = (await db.query("SELECT * FROM items WHERE category = 'code' AND removed_at IS NULL ORDER BY retrieved_at DESC LIMIT 5000")).rows;
+ const filtered = rows.filter((row) => {
+ const item = rowItem(row);
+ const f = item.fields || {};
+ if (lang && String(f.language || '').toLowerCase() !== lang.toLowerCase()) return false;
+ if (topic && !(Array.isArray(f.topics) ? f.topics : []).some((x) => String(x).toLowerCase() === topic.toLowerCase())) return false;
+ const created = Date.parse(f.created || '');
+ if (list === 'trending' && (!Number.isFinite(created) || created < dayStart)) return false;
+ if (list === 'week' && (!Number.isFinite(created) || created < t - 7 * DAY)) return false;
+ if (list === 'month' && (!Number.isFinite(created) || created < t - 30 * DAY)) return false;
+ if (list === 'new' && (!Number.isFinite(created) || created < t - 7 * DAY)) return false;
+ return true;
+ });
+ const history = await historyFor(db, filtered.map((r) => r.identity), t);
+ const ordered = withGains(filtered, history, t).sort(sortFor(list)).slice(0, limit).map((x) => x.repo);
+ const source = await registry.get('github');
+ const health = source ? registry.health(source) : null;
+ res.set('Cache-Control', 'public, max-age=60');
+ res.json({
+ generated_at: new Date(t).toISOString(),
+ list, lang, topic, limit, count: ordered.length,
+ source: health ? { key: 'github', status: health.status, stale: health.stale, last_success_at: health.last_success_at } : null,
+ items: ordered,
+ });
+ });
+
+ router.get('/api/v1/code/search', read, async (req, res) => {
+ const q = text(req.query.q, 300);
+ if (!q) return http.sendProblem(res, 422, 'sources.bad_query', { detail: 'q is required' });
+ const lang = text(req.query.lang, 60);
+ const sort = SORTS.includes(String(req.query.sort || 'best')) ? String(req.query.sort || 'best') : 'best';
+ const page = Math.min(Math.max(parseInt(req.query.page, 10) || 1, 1), 10);
+ let out;
+ try {
+ out = await client.search({ q, lang, sort, page });
+ } catch (err) {
+ if (err.code === 'rate_limited') {
+ res.set('Retry-After', String(Math.max(1, Math.ceil((err.retryAfterMs || 60000) / 1000))));
+ return http.sendProblem(res, 429, 'sources.rate_limited', { detail: 'GitHub asked us to wait before another search', ctx: req.ov });
+ }
+ if (err.code === 'bad_query') return http.sendProblem(res, 422, 'sources.bad_query', { detail: err.message, ctx: req.ov });
+ return http.sendProblem(res, 502, 'sources.github_unavailable', { detail: 'GitHub search did not answer', ctx: req.ov });
+ }
+ const names = out.items.map((i) => i.full_name);
+ const history = await historyFor(db, names, Date.now());
+ const items = out.items.map((repo) => {
+ const gains = github.gainsFrom(history.get(repo.full_name) || [], Date.now());
+ return { ...repo, stars_gained_1d: gains.days_1, stars_gained_7d: gains.days_7 };
+ });
+ res.set('Cache-Control', 'public, max-age=600');
+ res.json({ ...out, items, count: items.length });
+ });
+
+ return router;
+}
+
+module.exports = { createCodeRouter, LISTS, SORTS };
@@ -9,6 +9,7 @@ const { createRelease } = require('openvibe-shared/release');
@@ -9,6 +9,7 @@ const { createRelease } = require('openvibe-shared/release');
const { createSourcesReadiness, registerSourcesGauges } = require('./observability');
const { createSourcesReadiness, registerSourcesGauges } = require('./observability');
const { sourcesRouter } = require('./api/sources');
const { sourcesRouter } = require('./api/sources');
const { itemsRouter } = require('./api/items');
const { itemsRouter } = require('./api/items');
+const { createCodeRouter } = require('./api/code');
const { merchantRouter } = require('./api/merchant');
const { merchantRouter } = require('./api/merchant');
const { merchantPages } = require('./merchant-pages');
const { merchantPages } = require('./merchant-pages');
const { createActorLimits } = require('./api/actor-limits');
const { createActorLimits } = require('./api/actor-limits');
@@ -25,6 +26,7 @@ const TEXT_INDEX = [
@@ -25,6 +26,7 @@ const TEXT_INDEX = [
'',
'',
'GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)',
'GET /api/v1/sources, /api/v1/sources/:key, /api/v1/sources/:key/runs, /api/v1/runs, /api/v1/health (sources.source.read)',
'GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)',
'GET /api/v1/items?source=&category=&after=, /api/v1/items/:id (sources.item.read)',
+ 'GET /api/v1/code/repos?list=&lang=&topic=&limit=, /api/v1/code/search?q=&lang=&sort=&page= (sources.item.read)',
'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',
'POST/PATCH/DELETE /api/v1/sources[/:key], POST /api/v1/sources/:key/fetch, /items, DELETE /api/v1/items/:id (sources.source.manage)',
'GET/POST /api/v1/merchant/sources[/:id[/verify|/withdraw]] (sources.source.read/manage, or a merchant session)',
'GET/POST /api/v1/merchant/sources[/:id[/verify|/withdraw]] (sources.source.read/manage, or a merchant session)',
'GET /api/health, /api/ready, /release.json',
'GET /api/health, /api/ready, /release.json',
@@ -114,6 +116,7 @@ function createApp({ config, db, registry, items, ingest, scheduler, auth, outbo
@@ -114,6 +116,7 @@ function createApp({ config, db, registry, items, ingest, scheduler, auth, outbo
const limits = createActorLimits({ config, now: limitsNow || (() => Date.now()), registry: metrics.registry, log });
const limits = createActorLimits({ config, now: limitsNow || (() => Date.now()), registry: metrics.registry, log });
app.use(sourcesRouter({ db, registry, ingest, auth, limits }));
app.use(sourcesRouter({ db, registry, ingest, auth, limits }));
app.use(itemsRouter({ db, registry, items, auth, outbox, now, limits }));
app.use(itemsRouter({ db, registry, items, auth, outbox, now, limits }));
+ app.use(createCodeRouter({ db, registry, auth, limits, github: ingest.github }));
// GET /auth/health: does the Network accept this site's OAuth client? 503 only when it refuses it. Sources has
// GET /auth/health: does the Network accept this site's OAuth client? 503 only when it refuses it. Sources has
// no site-wide sign-in; the merchant flow is the one OAuth client, so it is the one this checks.
// no site-wide sign-in; the merchant flow is the one OAuth client, so it is the one this checks.
new file mode 100644
new file mode 100644
@@ -0,0 +1,210 @@
@@ -0,0 +1,210 @@
+'use strict';
+/**
+ * The GitHub client shared by ingestion and the code search API. One process has one client, so the
+ * token, the request spacing and the last rate-limit headers are the same for a scheduled fetch and a
+ * person's search.
+ *
+ * The token is read by name at request time (server/settings.js: the owner's setting, then
+ * SOURCES_GITHUB_TOKEN, then GITHUB_TOKEN). It is never stored, logged, returned or put in a URL.
+ * Without one, GitHub's search limit is 10 requests a minute; with one it is 30. The client keeps the
+ * last x-ratelimit-remaining/reset, waits out a 403/429 the API sends, and lets a batch end early: a
+ * run that already has items records them and says when the next request may go.
+ *
+ * Nothing here fetches github.com pages. The only host is api.github.com, and every request goes through
+ * the same robots, spacing and address guard as every other Sources fetch.
+ */
+const crypto = require('crypto');
+const github = require('./adapters/github');
+const settings = require('./settings');
+const { decodeBody, ParseError } = require('./adapters/util');
+
+const HOST = 'api.github.com';
+const SEARCH_TTL_MS = 10 * 60 * 1000;
+const MAX_CACHE = 200;
+
+class RateLimitError extends Error {
+ constructor(retryAfterMs, detail) {
+ super(detail || 'GitHub asked us to wait');
+ this.code = 'rate_limited';
+ this.retryAfterMs = Math.max(0, Number(retryAfterMs) || 0);
+ }
+}
+
+const int = (v) => { const n = Number(v); return Number.isFinite(n) ? n : null; };
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+
+function retryAfterMs(value, now) {
+ if (value == null) return null;
+ const s = String(value).trim();
+ if (/^\d{1,7}$/.test(s)) return Number(s) * 1000;
+ const t = Date.parse(s);
+ return Number.isFinite(t) ? Math.max(0, t - now) : null;
+}
+
+function createGithub({ config, fetcher, robots, spacer, now = () => Date.now(), log = console, capBytes = 4 * 1024 * 1024 }) {
+ const limiter = { remaining: null, resetAt: 0, at: 0 };
+ const cache = new Map(); // search URL → { at, value }
+ const inflight = new Map();
+
+ const token = () => {
+ try { return settings.githubToken() || config.secrets.GITHUB_TOKEN || ''; } catch { return config.secrets.GITHUB_TOKEN || ''; }
+ };
+
+ function headers() {
+ const h = {
+ Accept: 'application/vnd.github+json',
+ 'X-GitHub-Api-Version': '2022-11-28',
+ 'User-Agent': config.fetch.userAgent,
+ };
+ const t = token();
+ if (t) h.Authorization = `Bearer ${t}`;
+ return h;
+ }
+
+ /** One API GET: robots, spacing, the token and the rate-limit ledger. Returns { body, status, bytes, headers }. */
+ async function request(url, { beforeRequest = null, minIntervalMs = null } = {}) {
+ // A used-up search budget is a wait, not a stream of refused requests. GitHub resets the search
+ // limit within a minute; resetAt comes from its own headers.
+ if (limiter.remaining === 0 && limiter.resetAt > now()) {
+ throw new RateLimitError(limiter.resetAt - now(), 'the GitHub search limit is used up');
+ }
+ const verdict = await robots.check(url);
+ if (!verdict.allowed) {
+ if (verdict.refusedCode) throw Object.assign(new Error(`${HOST} is not fetchable (${verdict.refusedCode})`), { code: verdict.refusedCode });
+ throw Object.assign(new Error(verdict.reason || 'robots.txt disallows this URL'), { code: 'robots_denied' });
+ }
+ await spacer.space(HOST, minIntervalMs == null ? (token() ? 2000 : 6500) : minIntervalMs);
+ if (beforeRequest) await beforeRequest();
+ limiter.at = now();
+ const res = await fetcher.fetchUrl(url, {
+ headers: headers(),
+ capBytes,
+ sensitiveHeaders: ['Authorization'],
+ beforeHop: async (next) => {
+ const v = await robots.check(next.toString());
+ return v.allowed ? null : { refuse: v.reason };
+ },
+ });
+ const remaining = int(res.headers['x-ratelimit-remaining']);
+ const reset = int(res.headers['x-ratelimit-reset']);
+ if (remaining != null) limiter.remaining = remaining;
+ if (reset != null) limiter.resetAt = reset * 1000;
+ if (res.status === 429 || (res.status === 403 && (remaining === 0 || retryAfterMs(res.headers['retry-after'], now()) != null))) {
+ const wait = retryAfterMs(res.headers['retry-after'], now())
+ ?? (limiter.resetAt > now() ? limiter.resetAt - now() : 60_000);
+ throw new RateLimitError(wait, `GitHub answered ${res.status}`);
+ }
+ if (res.status === 401) throw Object.assign(new Error('GitHub refused the token (401)'), { code: 'http_error', status: 401 });
+ if (res.status === 403) throw Object.assign(new Error('GitHub answered 403 (the token may lack access)'), { code: 'http_error', status: 403 });
+ if (res.status !== 200 && res.status !== 203) throw Object.assign(new Error(`GitHub answered HTTP ${res.status}`), { code: 'http_error', status: res.status });
+ let body;
+ try { body = JSON.parse(decodeBody(res.body, res.headers['content-type'])); } catch {
+ throw new ParseError('GitHub answered non-JSON');
+ }
+ return { body, status: res.status, bytes: res.bytes || 0, headers: res.headers };
+ }
+
+ /**
+ * One ingestion run's share of the query plan. Starts at the stored cursor, asks at most `budget`
+ * questions, and returns the cursor for the next run. A rate limit after some answers keeps those
+ * answers and reports when to come back; a rate limit before any is thrown so the run is recorded
+ * as rate_limited, never as an empty success.
+ */
+ async function fetchQueries({ source, cursor = {}, budget = 8, beforeRequest = null } = {}) {
+ const list = github.plan(now());
+ const start = Number.isInteger(cursor.next) ? ((cursor.next % list.length) + list.length) % list.length : 0;
+ const items = new Map();
+ const stars = new Map();
+ let next = start;
+ let queries = 0;
+ let lastStatus = null;
+ let bytes = 0;
+ const hash = crypto.createHash('sha256');
+ let notBefore = 0;
+ let limited = null;
+ for (let i = 0; i < budget; i++) {
+ const index = (start + i) % list.length;
+ const query = list[index];
+ let out;
+ try {
+ out = await request(github.queryUrl(query), { beforeRequest, minIntervalMs: source.min_interval_ms });
+ } catch (err) {
+ if (err.code === 'rate_limited') {
+ notBefore = Math.max(notBefore, now() + err.retryAfterMs);
+ limited = err;
+ break;
+ }
+ throw err;
+ }
+ lastStatus = out.status;
+ bytes += out.bytes;
+ hash.update(Buffer.from(JSON.stringify(out.body)));
+ const parsed = github.parseSearch(out.body);
+ for (const raw of parsed.items) {
+ const item = github.repoItem(raw, config.items.summaryMax);
+ if (!item) continue;
+ items.set(item.identity, item);
+ const starsNow = item.fields.stars;
+ if (starsNow != null) stars.set(item.identity, starsNow);
+ }
+ queries++;
+ next = (index + 1) % list.length;
+ }
+ if (!queries && limited) throw limited;
+ return {
+ items: [...items.values()],
+ stars: [...stars.entries()].map(([full_name, n]) => ({ full_name, stars: n })),
+ next, queries, lastStatus, bytes,
+ rawBodyHash: hash.digest('hex'),
+ notBefore,
+ limited: Boolean(limited),
+ };
+ }
+
+ /**
+ * A live repository search for the public API: GitHub's own query syntax passes through, with an
+ * optional language and sort. Cached for ten minutes; a request that GitHub rate-limits answers
+ * the last copy for up to an hour, marked stale, instead of inventing an empty result.
+ */
+ async function search({ q, lang = null, sort = 'best', page = 1 } = {}) {
+ const text = String(q || '').trim().slice(0, 300);
+ if (!text) throw Object.assign(new Error('a search needs q'), { code: 'bad_query' });
+ const language = lang ? String(lang).trim().slice(0, 60) : '';
+ const order = ['stars', 'updated', 'best'].includes(sort) ? sort : 'best';
+ const p = Math.min(Math.max(parseInt(page, 10) || 1, 1), 10);
+ const u = new URL(github.API);
+ u.searchParams.set('q', language ? `${text} language:"${language.replace(/"/g, '')}"` : text);
+ if (order !== 'best') { u.searchParams.set('sort', order); u.searchParams.set('order', 'desc'); }
+ u.searchParams.set('per_page', '30');
+ u.searchParams.set('page', String(p));
+ const url = u.toString();
+ const hit = cache.get(url);
+ if (hit && now() - hit.at < SEARCH_TTL_MS) return { ...hit.value, cached: true, stale: false };
+ if (inflight.has(url)) return await inflight.get(url);
+ const job = (async () => {
+ try {
+ const out = await request(url, { minIntervalMs: token() ? 2000 : 6500 });
+ const parsed = github.parseSearch(out.body);
+ const items = parsed.items.map((r) => github.repoItem(r, config.items.summaryMax)).filter(Boolean)
+ .map((i) => github.publicRepository(i));
+ const value = { items, total: parsed.total, incomplete: parsed.incomplete, page: p, sort: order, q: text, lang: language || null };
+ cache.set(url, { at: now(), value });
+ if (cache.size > MAX_CACHE) cache.delete(cache.keys().next().value);
+ return { ...value, cached: false, stale: false };
+ } catch (err) {
+ if (err.code === 'rate_limited' && hit && now() - hit.at < 60 * 60 * 1000) {
+ return { ...hit.value, cached: true, stale: true, retry_after_ms: err.retryAfterMs };
+ }
+ throw err;
+ } finally {
+ inflight.delete(url);
+ }
+ })();
+ inflight.set(url, job);
+ return await job;
+ }
+
+ return { request, fetchQueries, search, rateLimit: () => ({ ...limiter }), hasToken: () => Boolean(token()) };
+}
+
+module.exports = { createGithub, RateLimitError, retryAfterMs, SEARCH_TTL_MS };
@@ -29,6 +29,8 @@ const crypto = require('crypto');
@@ -29,6 +29,8 @@ const crypto = require('crypto');
const { ids } = require('openvibe-contracts');
const { ids } = require('openvibe-contracts');
const { ADAPTERS, RESOLVERS } = require('./adapters');
const { ADAPTERS, RESOLVERS } = require('./adapters');
const offDelta = require('./adapters/off-delta');
const offDelta = require('./adapters/off-delta');
+const githubAdapter = require('./adapters/github');
+const { createGithub } = require('./github');
const extract = require('./adapters/extract');
const extract = require('./adapters/extract');
const { createCrawler } = require('./crawl');
const { createCrawler } = require('./crawl');
const { sha256, decodeBody, ParseError } = require('./adapters/util');
const { sha256, decodeBody, ParseError } = require('./adapters/util');
@@ -46,6 +48,7 @@ function retryAfterMs(value, now) {
@@ -46,6 +48,7 @@ function retryAfterMs(value, now) {
function createIngest({ db, config, registry, items, robots, fetcher, spacer, emitEvent, now = () => Date.now(), log = console, outbox = null }) {
function createIngest({ db, config, registry, items, robots, fetcher, spacer, emitEvent, now = () => Date.now(), log = console, outbox = null }) {
const inflight = new Set();
const inflight = new Set();
const { space, setCrawlDelay } = spacer;
const { space, setCrawlDelay } = spacer;
+ const github = createGithub({ config, fetcher, robots, spacer, now, log });
let stopping = false;
let stopping = false;
const st = {
const st = {
@@ -416,6 +419,54 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em
@@ -416,6 +419,54 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em
return { run, failed: false, success: true, recorded: true };
return { run, failed: false, success: true, recorded: true };
}
}
+ /**
+ * A GitHub source: one share of the search plan per run through server/github.js. The cursor is kept
+ * in source_state, so a run with no token (10 searches a minute) never tries the whole plan at once.
+ * Star counts go to github_stars in the same transaction as the items and the run row.
+ */
+ async function fetchGithubEndpoint(source, endpoint, trigger) {
+ const startedAt = now();
+ const run = baseRun(source, endpoint.url, trigger, startedAt);
+ let cursor = {};
+ try {
+ const saved = await st.stateGet.get(source.key, githubAdapter.STATE_NAME);
+ if (saved && saved.value) cursor = JSON.parse(saved.value);
+ } catch { cursor = {}; }
+ const budget = github.hasToken() ? 20 : 8;
+ let out;
+ try {
+ out = await github.fetchQueries({
+ source, cursor, budget,
+ beforeRequest: async () => { await st.requested.run(now(), source.key); },
+ });
+ } catch (err) {
+ const state = err.code === 'rate_limited' ? 'rate_limited'
+ : err.code === 'robots_denied' ? 'robots_denied'
+ : err.code === 'parse_error' ? 'parse_error'
+ : err.code === 'timeout' ? 'timeout' : 'http_error';
+ Object.assign(run, { state, error_code: err.code || 'network', detail: String(err.message).slice(0, 500), finished_at: now() });
+ return { run, failed: true, notBefore: err.code === 'rate_limited' ? now() + (err.retryAfterMs || 60000) : null };
+ }
+ Object.assign(run, {
+ http_status: out.lastStatus, bytes: out.bytes, finished_at: now(),
+ raw_body_hash: out.rawBodyHash, parser_version: githubAdapter.PARSER_VERSION,
+ });
+ await db.tx(async () => {
+ const counts = await items.ingest(source, out.items, {
+ runId: run.id, rawBodyHash: run.raw_body_hash, parserVersion: githubAdapter.PARSER_VERSION, retrievedAt: run.finished_at,
+ });
+ Object.assign(run, {
+ state: 'ok', items_seen: out.items.length, items_created: counts.created, items_updated: counts.updated,
+ items_unchanged: counts.unchanged, items_skipped: counts.skipped,
+ });
+ await githubAdapter.recordStars(db, source.key, out.stars, run.finished_at);
+ await st.insertRun.run(run);
+ await st.stateSet.run({ k: source.key, n: githubAdapter.STATE_NAME, v: JSON.stringify({ next: out.next, queries: out.queries, at: run.finished_at }), at: run.finished_at });
+ await st.saveValidators.run({ k: source.key, u: endpoint.url, etag: null, lm: null, status: out.lastStatus, at: run.finished_at });
+ });
+ return { run, failed: false, success: true, recorded: true, notBefore: out.notBefore };
+ }
+
/**
/**
* A bulk source with `resolver: "off-delta"`: read a text index, then only the delta parts not finished yet.
* A bulk source with `resolver: "off-delta"`: read a text index, then only the delta parts not finished yet.
* The full export is one gzip stream and cannot be resumed, so the first run starts at the oldest part in the
* The full export is one gzip stream and cannot be resumed, so the first run starts at the oldest part in the
@@ -595,9 +646,11 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em
@@ -595,9 +646,11 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em
? endpoint.resolver === 'off-delta'
? endpoint.resolver === 'off-delta'
? await fetchOffDeltaEndpoint(source, endpoint, trigger)
? await fetchOffDeltaEndpoint(source, endpoint, trigger)
: await fetchBulkEndpoint(source, endpoint, trigger)
: await fetchBulkEndpoint(source, endpoint, trigger)
- : endpoint.resolver
- ? await fetchResolverEndpoint(source, endpoint, trigger)
- : await fetchEndpoint(source, endpoint, trigger);
+ : endpoint.resolver === 'github'
+ ? await fetchGithubEndpoint(source, endpoint, trigger)
+ : endpoint.resolver
+ ? await fetchResolverEndpoint(source, endpoint, trigger)
+ : await fetchEndpoint(source, endpoint, trigger);
} catch (err) {
} catch (err) {
// Anything unexpected is still a recorded failure, never a silent gap.
// Anything unexpected is still a recorded failure, never a silent gap.
const r = baseRun(source, endpoint.url, trigger, t);
const r = baseRun(source, endpoint.url, trigger, t);
@@ -700,7 +753,7 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em
@@ -700,7 +753,7 @@ function createIngest({ db, config, registry, items, robots, fetcher, spacer, em
/** Shutting down: a crawl ends after the page in flight instead of working through its budget. */
/** Shutting down: a crawl ends after the page in flight instead of working through its budget. */
function stop() { stopping = true; }
function stop() { stopping = true; }
- return { run, crawlStatus, stop, inflight: () => [...inflight] };
+ return { run, crawlStatus, stop, inflight: () => [...inflight], github };
}
}
module.exports = { createIngest, FAILURE_STATES, retryAfterMs };
module.exports = { createIngest, FAILURE_STATES, retryAfterMs };
@@ -112,13 +112,17 @@ function validateSource(input, { maxItemsCap = 12000 } = {}) {
@@ -112,13 +112,17 @@ function validateSource(input, { maxItemsCap = 12000 } = {}) {
// (only as their defaults) is what lets PATCH { enabled: true } validate a stored resolver.
// (only as their defaults) is what lets PATCH { enabled: true } validate a stored resolver.
if (!['url', 'format', 'resolver', 'item_kind', 'max_bytes', 'timeout_ms', 'items_shape', 'items_path', 'fields', 'extra'].includes(k)) throw new RegistryError(`endpoints[${i}].${k} is not a resolver field`);
if (!['url', 'format', 'resolver', 'item_kind', 'max_bytes', 'timeout_ms', 'items_shape', 'items_path', 'fields', 'extra'].includes(k)) throw new RegistryError(`endpoints[${i}].${k} is not a resolver field`);
}
}
- if (e.resolver !== 'hn-whoishiring') throw new RegistryError(`endpoints[${i}].resolver must be hn-whoishiring`);
+ if (!['hn-whoishiring', 'github'].includes(e.resolver)) throw new RegistryError(`endpoints[${i}].resolver must be hn-whoishiring or github`);
if (e.format !== undefined && e.format !== 'json') throw new RegistryError(`endpoints[${i}].format must be json for a resolver`);
if (e.format !== undefined && e.format !== 'json') throw new RegistryError(`endpoints[${i}].format must be json for a resolver`);
if (e.items_shape !== undefined && e.items_shape !== 'array') throw new RegistryError(`endpoints[${i}].items_shape must be array for a resolver`);
if (e.items_shape !== undefined && e.items_shape !== 'array') throw new RegistryError(`endpoints[${i}].items_shape must be array for a resolver`);
if (e.items_path !== undefined && e.items_path !== '') throw new RegistryError(`endpoints[${i}].items_path must be "" for a resolver`);
if (e.items_path !== undefined && e.items_path !== '') throw new RegistryError(`endpoints[${i}].items_path must be "" for a resolver`);
if (e.fields !== undefined && (!e.fields || typeof e.fields !== 'object' || Array.isArray(e.fields) || Object.keys(e.fields).length)) throw new RegistryError(`endpoints[${i}].fields must be empty for a resolver`);
if (e.fields !== undefined && (!e.fields || typeof e.fields !== 'object' || Array.isArray(e.fields) || Object.keys(e.fields).length)) throw new RegistryError(`endpoints[${i}].fields must be empty for a resolver`);
if (e.extra !== undefined && (!e.extra || typeof e.extra !== 'object' || Array.isArray(e.extra) || Object.keys(e.extra).length)) throw new RegistryError(`endpoints[${i}].extra must be empty for a resolver`);
if (e.extra !== undefined && (!e.extra || typeof e.extra !== 'object' || Array.isArray(e.extra) || Object.keys(e.extra).length)) throw new RegistryError(`endpoints[${i}].extra must be empty for a resolver`);
- if (!/\.json(\?|$)/.test(out.url)) throw new RegistryError(`endpoints[${i}].resolver needs a .json endpoint`);
+ // GitHub's search answer is one of several queries the resolver plans itself; the endpoint is the
+ // API origin, not a fixed .json document. The host is pinned so the token can only go to GitHub.
+ if (e.resolver === 'github') {
+ if (new URL(out.url).origin !== 'https://api.github.com') throw new RegistryError(`endpoints[${i}].resolver github must be on https://api.github.com`);
+ } else if (!/\.json(\?|$)/.test(out.url)) throw new RegistryError(`endpoints[${i}].resolver needs a .json endpoint`);
if (e.item_kind !== undefined && !/^[a-z][a-z0-9_]{1,39}$/.test(e.item_kind)) throw new RegistryError(`endpoints[${i}].item_kind is malformed`);
if (e.item_kind !== undefined && !/^[a-z][a-z0-9_]{1,39}$/.test(e.item_kind)) throw new RegistryError(`endpoints[${i}].item_kind is malformed`);
if (e.max_bytes !== undefined && (!Number.isInteger(e.max_bytes) || e.max_bytes < 1024 || e.max_bytes > 32 * 1024 * 1024)) throw new RegistryError(`endpoints[${i}].max_bytes must be an integer from 1024 to 33554432`);
if (e.max_bytes !== undefined && (!Number.isInteger(e.max_bytes) || e.max_bytes < 1024 || e.max_bytes > 32 * 1024 * 1024)) throw new RegistryError(`endpoints[${i}].max_bytes must be an integer from 1024 to 33554432`);
if (e.timeout_ms !== undefined && (!Number.isInteger(e.timeout_ms) || e.timeout_ms < 1000 || e.timeout_ms > 120000)) throw new RegistryError(`endpoints[${i}].timeout_ms must be an integer from 1000 to 120000`);
if (e.timeout_ms !== undefined && (!Number.isInteger(e.timeout_ms) || e.timeout_ms < 1000 || e.timeout_ms > 120000)) throw new RegistryError(`endpoints[${i}].timeout_ms must be an integer from 1000 to 120000`);
@@ -41,6 +41,8 @@
@@ -41,6 +41,8 @@
* year for a rebuild and the Open Food Facts bulk snapshot, which is regenerated.
* year for a rebuild and the Open Food Facts bulk snapshot, which is regenerated.
* - jobs 14 days: Arbeitnow and We Work Remotely retire what the board stopped showing after 7
* - jobs 14 days: Arbeitnow and We Work Remotely retire what the board stopped showing after 7
* days; Work copies each listing, so a fortnight keeps every tombstone it may pull.
* days; Work copies each listing, so a fortnight keeps every tombstone it may pull.
+ * - code 30 days: the GitHub rankings are a current list; a month keeps a repository through a failed
+ * pull, and the star history is what says whether it is still hot.
*
*
* Sources has no openvibe-shared/config namespace, so the rules stay in code (the brief's
* Sources has no openvibe-shared/config namespace, so the rules stay in code (the brief's
* alternative); a day count this consequential should change in review, not in a database row.
* alternative); a day count this consequential should change in review, not in a database row.
@@ -74,6 +76,7 @@ const RULES = Object.freeze({
@@ -74,6 +76,7 @@ const RULES = Object.freeze({
housing: { keepLatest: true, reason: 'HUD rents change yearly: the latest value per area is what Rent shows' },
housing: { keepLatest: true, reason: 'HUD rents change yearly: the latest value per area is what Rent shows' },
food: { days: 365, reason: 'Food keeps the recall history it shows; a year covers a rebuild' },
food: { days: 365, reason: 'Food keeps the recall history it shows; a year covers a rebuild' },
jobs: { days: 14, reason: 'boards retire unseen jobs after 7 days and Work pulls hourly' },
jobs: { days: 14, reason: 'boards retire unseen jobs after 7 days and Work pulls hourly' },
+ code: { days: 30, reason: 'GitHub rankings are current; a month keeps the tracked set through a failed pull' },
});
});
// The rules are the contract's categories in review; the runtime deliberately does not throw when
// The rules are the contract's categories in review; the runtime deliberately does not throw when
@@ -61,7 +61,7 @@ function createSettings({ db, env = process.env, log = console, fetchImpl = glob
@@ -61,7 +61,7 @@ function createSettings({ db, env = process.env, log = console, fetchImpl = glob
}
}
/** The read-only GitHub token, read when a GitHub read is made. */
/** The read-only GitHub token, read when a GitHub read is made. */
- function githubToken() { return valueOf('github_read_only_token', 'SOURCES_GITHUB_TOKEN', ''); }
+ function githubToken() { return valueOf('github_read_only_token', 'SOURCES_GITHUB_TOKEN', '') || valueOf('github_read_only_token', 'GITHUB_TOKEN', ''); }
async function test(key) {
async function test(key) {
if (key !== 'github_read_only_token') return { ok: false, detail: 'That setting has no test.' };
if (key !== 'github_read_only_token') return { ok: false, detail: 'That setting has no test.' };
@@ -124,7 +124,7 @@ function createSettings({ db, env = process.env, log = console, fetchImpl = glob
@@ -124,7 +124,7 @@ function createSettings({ db, env = process.env, log = console, fetchImpl = glob
function activate(settings) { ACTIVE.settings = settings; return settings; }
function activate(settings) { ACTIVE.settings = settings; return settings; }
function githubToken() {
function githubToken() {
if (ACTIVE.settings) return ACTIVE.settings.githubToken();
if (ACTIVE.settings) return ACTIVE.settings.githubToken();
- return process.env.SOURCES_GITHUB_TOKEN || '';
+ return process.env.SOURCES_GITHUB_TOKEN || process.env.GITHUB_TOKEN || '';
}
}
module.exports = { SERVICE, NAMESPACE, SCHEMA, SECRETS, createSettings, activate, githubToken };
module.exports = { SERVICE, NAMESPACE, SCHEMA, SECRETS, createSettings, activate, githubToken };
@@ -1,7 +1,8 @@
@@ -1,7 +1,8 @@
'use strict';
'use strict';
/**
/**
* The categories a source may have are sources.source@1's, read from openvibe-contracts: housing, food and jobs since
* The categories a source may have are sources.source@1's, read from openvibe-contracts: housing, food and jobs since
- * contracts 0.210.0 (plan 2026-10-22, track F1: Rent and Homes, Food and Work move their own fetches into Sources).
+ * contracts 0.210.0 (plan 2026-10-22, track F1: Rent and Homes, Food and Work move their own fetches into Sources),
+ * and code since 0.411.1 (GitHub repository rankings for Exchange).
* The registry accepts every one of them and refuses anything else, and the items API filters by them.
* The registry accepts every one of them and refuses anything else, and the items API filters by them.
*/
*/
const assert = require('assert');
const assert = require('assert');
@@ -10,15 +11,15 @@ const { suite, sourceDef } = require('./helpers');
@@ -10,15 +11,15 @@ const { suite, sourceDef } = require('./helpers');
const t = suite('categories');
const t = suite('categories');
-t('the categories are the contract\'s, housing, food and jobs included', () => {
+t('the categories are the contract\'s, housing, food, jobs and code included', () => {
const contract = require('openvibe-contracts/contracts/sources/source.v1.json').properties.category.enum;
const contract = require('openvibe-contracts/contracts/sources/source.v1.json').properties.category.enum;
assert.deepStrictEqual([...CATEGORIES], contract);
assert.deepStrictEqual([...CATEGORIES], contract);
- for (const c of ['news', 'hosting', 'games', 'housing', 'food', 'jobs']) assert.ok(CATEGORIES.includes(c), c);
+ for (const c of ['news', 'hosting', 'games', 'housing', 'food', 'jobs', 'code']) assert.ok(CATEGORIES.includes(c), c);
assert.ok(Object.isFrozen(CATEGORIES));
assert.ok(Object.isFrozen(CATEGORIES));
});
});
t('the registry accepts a source in each new category and refuses one outside the contract', () => {
t('the registry accepts a source in each new category and refuses one outside the contract', () => {
- for (const category of ['housing', 'food', 'jobs']) {
+ for (const category of ['housing', 'food', 'jobs', 'code']) {
assert.strictEqual(validateSource(sourceDef({ key: `example-${category}`, category, endpoints: ['https://feeds.example.org/feed.xml'] })).category, category);
assert.strictEqual(validateSource(sourceDef({ key: `example-${category}`, category, endpoints: ['https://feeds.example.org/feed.xml'] })).category, category);
}
}
assert.throws(() => validateSource(sourceDef({ key: 'example-weather', category: 'weather', endpoints: ['https://feeds.example.org/feed.xml'] })), /category must be one of .*housing\|food\|jobs/);
assert.throws(() => validateSource(sourceDef({ key: 'example-weather', category: 'weather', endpoints: ['https://feeds.example.org/feed.xml'] })), /category must be one of .*housing\|food\|jobs/);
new file mode 100644
new file mode 100644
@@ -0,0 +1,100 @@
@@ -0,0 +1,100 @@
+'use strict';
+/**
+ * /api/v1/code: the repository rankings and the live search. Ingestion is driven by a stubbed GitHub
+ * client (no network); the history is written through the real run transaction, then older days are
+ * inserted so "hot" is computed from stars gained, not from totals.
+ */
+const assert = require('assert');
+const { boot, request, serviceToken, suite } = require('./helpers');
+const github = require('../server/adapters/github');
+
+const t = suite('code');
+let svc;
+const ITEMS = serviceToken('exchange', ['sources.item.read']);
+const SOURCES = serviceToken('exchange', ['sources.source.read']);
+const NOW = Date.now();
+const DAY = 86400000;
+
+function repo(full, { stars, language, created, topics = ['ai'] }) {
+ return {
+ full_name: full, html_url: `https://github.com/${full}`, description: `${full} description`,
+ stargazers_count: stars, forks_count: 12, open_issues_count: 3, language, topics,
+ license: { spdx_id: 'MIT', name: 'MIT License' }, created_at: created, pushed_at: new Date(NOW - 3600000).toISOString(),
+ owner: { login: full.split('/')[0], avatar_url: `https://avatars.githubusercontent.com/u/1?v=4` },
+ homepage: `https://${full.split('/')[0]}.example`, archived: false, fork: false,
+ };
+}
+
+const A = repo('acme/hot', { stars: 500, language: 'Rust', created: new Date(NOW - 3600000).toISOString() });
+const B = repo('beta/steady', { stars: 100, language: 'Python', created: new Date(NOW - 2 * DAY).toISOString(), topics: ['tools'] });
+
+t('boot and one GitHub run', async () => {
+ svc = await boot();
+ const seed = require('../seeds/sources.json').sources.find((s) => s.key === 'github');
+ await svc.registry.create(seed, 'test');
+ await svc.db.prepare("UPDATE sources SET enabled = 1, min_interval_ms = 0, last_request_at = NULL WHERE key = 'github'").run();
+ svc.ingest.github.hasToken = () => false;
+ svc.ingest.github.fetchQueries = async () => ({
+ items: [github.repoItem(A, 1000), github.repoItem(B, 1000)],
+ stars: [{ full_name: 'acme/hot', stars: 500 }, { full_name: 'beta/steady', stars: 100 }],
+ next: 2, queries: 2, lastStatus: 200, bytes: 100, rawBodyHash: 'a'.repeat(64), notBefore: 0, limited: false,
+ });
+ const out = await svc.ingest.run('github');
+ assert.strictEqual(out.runs[0].state, 'ok');
+ assert.strictEqual(out.runs[0].items_created, 2);
+ const stars = await svc.db.prepare('SELECT full_name, day, stars FROM github_stars ORDER BY full_name').all();
+ assert.deepStrictEqual(stars.map((s) => [s.full_name, s.stars]), [['acme/hot', 500], ['beta/steady', 100]]);
+ // Older days complete the history: +100/+5 in 1 day and +200/+10 in 7.
+ const put = svc.db.prepare('INSERT INTO github_stars (source_key, full_name, day, stars, captured_at) VALUES (?, ?, ?, ?, ?)');
+ await put.run('github', 'acme/hot', github.isoDay(NOW - DAY), 400, NOW - DAY);
+ await put.run('github', 'acme/hot', github.isoDay(NOW - 7 * DAY), 300, NOW - 7 * DAY);
+ await put.run('github', 'beta/steady', github.isoDay(NOW - DAY), 95, NOW - DAY);
+ await put.run('github', 'beta/steady', github.isoDay(NOW - 7 * DAY), 90, NOW - 7 * DAY);
+});
+
+t('the rankings need sources.item.read, carry the public item shape and rank hot by stars gained', async () => {
+ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/repos')).status, 401);
+ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/repos', { token: SOURCES })).status, 403);
+ const week = await request(svc.base, 'GET', '/api/v1/code/repos?list=week&limit=10', { token: ITEMS });
+ assert.strictEqual(week.status, 200);
+ assert.strictEqual(week.body.list, 'week');
+ assert.strictEqual(week.body.items.length, 2);
+ const one = week.body.items.find((x) => x.full_name === 'acme/hot');
+ for (const key of ['full_name', 'url', 'description', 'language', 'topics', 'stars', 'stars_gained_1d', 'stars_gained_7d', 'forks', 'open_issues', 'license', 'created', 'pushed', 'owner_avatar_url', 'homepage']) assert.ok(key in one, key);
+ assert.strictEqual(one.stars, 500);
+ assert.strictEqual(one.stars_gained_1d, 100);
+ assert.strictEqual(one.stars_gained_7d, 200);
+ assert.strictEqual(one.license, 'MIT');
+ const hot = await request(svc.base, 'GET', '/api/v1/code/repos?list=hot', { token: ITEMS });
+ assert.deepStrictEqual(hot.body.items.map((x) => x.full_name), ['acme/hot', 'beta/steady']);
+ assert.deepStrictEqual([hot.body.items[0].stars_gained_1d, hot.body.items[0].stars_gained_7d], [100, 200]);
+ const rust = await request(svc.base, 'GET', '/api/v1/code/repos?list=month&lang=Rust', { token: ITEMS });
+ assert.deepStrictEqual(rust.body.items.map((x) => x.full_name), ['acme/hot']);
+ const tools = await request(svc.base, 'GET', '/api/v1/code/repos?list=top&topic=tools', { token: ITEMS });
+ assert.deepStrictEqual(tools.body.items.map((x) => x.full_name), ['beta/steady']);
+ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/repos?list=weather', { token: ITEMS })).status, 422);
+});
+
+t('the live search passes q and language through, fills the gains, and answers 429 with Retry-After', async () => {
+ const seen = [];
+ svc.ingest.github.search = async (args) => {
+ seen.push(args);
+ return { items: [github.publicRepository(github.repoItem(A, 1000))], total: 1, incomplete: false, page: 1, sort: 'stars', q: args.q, lang: args.lang, cached: false, stale: false };
+ };
+ const r = await request(svc.base, 'GET', '/api/v1/code/search?q=stars:>500%20topic:ai&lang=Rust&sort=stars&page=1', { token: ITEMS });
+ assert.strictEqual(r.status, 200);
+ assert.deepStrictEqual(seen[0], { q: 'stars:>500 topic:ai', lang: 'Rust', sort: 'stars', page: 1 });
+ assert.strictEqual(r.body.items[0].full_name, 'acme/hot');
+ assert.strictEqual(r.body.items[0].stars_gained_1d, 100);
+ assert.match(r.headers.get('cache-control'), /max-age=600/);
+ assert.strictEqual((await request(svc.base, 'GET', '/api/v1/code/search?q=', { token: ITEMS })).status, 422);
+ svc.ingest.github.search = async () => { throw Object.assign(new Error('slow down'), { code: 'rate_limited', retryAfterMs: 30000 }); };
+ const limited = await request(svc.base, 'GET', '/api/v1/code/search?q=ai', { token: ITEMS });
+ assert.strictEqual(limited.status, 429);
+ assert.ok(Number(limited.headers.get('retry-after')) >= 30);
+ assert.strictEqual(limited.body.code, 'sources.rate_limited');
+});
+
+t('stop', async () => { await svc.stop(); });
+
+t.run();
new file mode 100644
new file mode 100644
@@ -0,0 +1,233 @@
@@ -0,0 +1,233 @@
+{
+ "total_count": 5757,
+ "incomplete_results": false,
+ "items": [
+ {
+ "id": 132750724,
+ "node_id": "MDEwOlJlcG9zaXRvcnkxMzI3NTA3MjQ=",
+ "name": "build-your-own-x",
+ "full_name": "codecrafters-io/build-your-own-x",
+ "private": false,
+ "owner": {
+ "login": "codecrafters-io",
+ "id": 58904235,
+ "node_id": "MDEyOk9yZ2FuaXphdGlvbjU4OTA0MjM1",
+ "avatar_url": "https://avatars.githubusercontent.com/u/58904235?v=4",
+ "gravatar_id": "",
+ "url": "https://api.github.com/users/codecrafters-io",
+ "html_url": "https://github.com/codecrafters-io",
+ "followers_url": "https://api.github.com/users/codecrafters-io/followers",
+ "following_url": "https://api.github.com/users/codecrafters-io/following{/other_user}",
+ "gists_url": "https://api.github.com/users/codecrafters-io/gists{/gist_id}",
+ "starred_url": "https://api.github.com/users/codecrafters-io/starred{/owner}{/repo}",
+ "subscriptions_url": "https://api.github.com/users/codecrafters-io/subscriptions",
+ "organizations_url": "https://api.github.com/users/codecrafters-io/orgs",
+ "repos_url": "https://api.github.com/users/codecrafters-io/repos",
+ "events_url": "https://api.github.com/users/codecrafters-io/events{/privacy}",
+ "received_events_url": "https://api.github.com/users/codecrafters-io/received_events",
+ "type": "Organization",
+ "user_view_type": "public",
+ "site_admin": false
+ },
+ "html_url": "https://github.com/codecrafters-io/build-your-own-x",
+ "description": "Master programming by recreating your favorite technologies from scratch.",
+ "fork": false,
+ "url": "https://api.github.com/repos/codecrafters-io/build-your-own-x",
+ "forks_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/forks",
+ "keys_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/keys{/key_id}",
+ "collaborators_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/collaborators{/collaborator}",
+ "teams_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/teams",
+ "hooks_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/hooks",
+ "issue_events_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/issues/events{/number}",
+ "events_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/events",
+ "assignees_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/assignees{/user}",
+ "branches_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/branches{/branch}",
+ "tags_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/tags",
+ "blobs_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/blobs{/sha}",
+ "git_tags_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/tags{/sha}",
+ "git_refs_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/refs{/sha}",
+ "trees_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/trees{/sha}",
+ "statuses_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/statuses/{sha}",
+ "languages_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/languages",
+ "stargazers_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/stargazers",
+ "contributors_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/contributors",
+ "subscribers_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/subscribers",
+ "subscription_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/subscription",
+ "commits_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/commits{/sha}",
+ "git_commits_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/git/commits{/sha}",
+ "comments_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/comments{/number}",
+ "issue_comment_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/issues/comments{/number}",
+ "contents_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/contents/{+path}",
+ "compare_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/compare/{base}...{head}",
+ "merges_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/merges",
+ "archive_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/{archive_format}{/ref}",
+ "downloads_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/downloads",
+ "issues_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/issues{/number}",
+ "pulls_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/pulls{/number}",
+ "milestones_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/milestones{/number}",
+ "notifications_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/notifications{?since,all,participating}",
+ "labels_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/labels{/name}",
+ "releases_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/releases{/id}",
+ "deployments_url": "https://api.github.com/repos/codecrafters-io/build-your-own-x/deployments",
+ "created_at": "2018-05-09T12:03:18Z",
+ "updated_at": "2026-11-12T07:55:05Z",
+ "pushed_at": "2026-11-08T16:38:15Z",
+ "git_url": "git://github.com/codecrafters-io/build-your-own-x.git",
+ "ssh_url": "[email protected]:codecrafters-io/build-your-own-x.git",
+ "clone_url": "https://github.com/codecrafters-io/build-your-own-x.git",
+ "svn_url": "https://github.com/codecrafters-io/build-your-own-x",
+ "homepage": "https://codecrafters.io",
+ "size": 1089,
+ "stargazers_count": 561961,
+ "watchers_count": 561961,
+ "language": "Markdown",
+ "has_issues": true,
+ "has_projects": false,
+ "has_downloads": false,
+ "has_wiki": false,
+ "has_pages": false,
+ "has_discussions": false,
+ "forks_count": 52587,
+ "mirror_url": null,
+ "archived": false,
+ "disabled": false,
+ "open_issues_count": 701,
+ "license": null,
+ "allow_forking": true,
+ "is_template": false,
+ "web_commit_signoff_required": false,
+ "has_pull_requests": true,
+ "pull_request_creation_policy": "all",
+ "topics": [
+ "awesome-list",
+ "free",
+ "programming",
+ "tutorial-code",
+ "tutorial-exercises",
+ "tutorials"
+ ],
+ "visibility": "public",
+ "forks": 52587,
+ "open_issues": 701,
+ "watchers": 561961,
+ "default_branch": "master",
+ "score": 1.0
+ },
+ {
+ "id": 21737465,
+ "node_id": "MDEwOlJlcG9zaXRvcnkyMTczNzQ2NQ==",
+ "name": "awesome",
+ "full_name": "sindresorhus/awesome",
+ "private": false,
+ "owner": {
+ "login": "sindresorhus",
+ "id": 170270,
+ "node_id": "MDQ6VXNlcjE3MDI3MA==",
+ "avatar_url": "https://avatars.githubusercontent.com/u/170270?v=4",
+ "gravatar_id": "",
+ "url": "https://api.github.com/users/sindresorhus",
+ "html_url": "https://github.com/sindresorhus",
+ "followers_url": "https://api.github.com/users/sindresorhus/followers",
+ "following_url": "https://api.github.com/users/sindresorhus/following{/other_user}",
+ "gists_url": "https://api.github.com/users/sindresorhus/gists{/gist_id}",
+ "starred_url": "https://api.github.com/users/sindresorhus/starred{/owner}{/repo}",
+ "subscriptions_url": "https://api.github.com/users/sindresorhus/subscriptions",
+ "organizations_url": "https://api.github.com/users/sindresorhus/orgs",
+ "repos_url": "https://api.github.com/users/sindresorhus/repos",
+ "events_url": "https://api.github.com/users/sindresorhus/events{/privacy}",
+ "received_events_url": "https://api.github.com/users/sindresorhus/received_events",
+ "type": "User",
+ "user_view_type": "public",
+ "site_admin": false
+ },
+ "html_url": "https://github.com/sindresorhus/awesome",
+ "description": "😎 Awesome lists about all kinds of interesting topics [NOTE: Pull requests are temporarily disabled until I have a chance to catch up with the existing ones]",
+ "fork": false,
+ "url": "https://api.github.com/repos/sindresorhus/awesome",
+ "forks_url": "https://api.github.com/repos/sindresorhus/awesome/forks",
+ "keys_url": "https://api.github.com/repos/sindresorhus/awesome/keys{/key_id}",
+ "collaborators_url": "https://api.github.com/repos/sindresorhus/awesome/collaborators{/collaborator}",
+ "teams_url": "https://api.github.com/repos/sindresorhus/awesome/teams",
+ "hooks_url": "https://api.github.com/repos/sindresorhus/awesome/hooks",
+ "issue_events_url": "https://api.github.com/repos/sindresorhus/awesome/issues/events{/number}",
+ "events_url": "https://api.github.com/repos/sindresorhus/awesome/events",
+ "assignees_url": "https://api.github.com/repos/sindresorhus/awesome/assignees{/user}",
+ "branches_url": "https://api.github.com/repos/sindresorhus/awesome/branches{/branch}",
+ "tags_url": "https://api.github.com/repos/sindresorhus/awesome/tags",
+ "blobs_url": "https://api.github.com/repos/sindresorhus/awesome/git/blobs{/sha}",
+ "git_tags_url": "https://api.github.com/repos/sindresorhus/awesome/git/tags{/sha}",
+ "git_refs_url": "https://api.github.com/repos/sindresorhus/awesome/git/refs{/sha}",
+ "trees_url": "https://api.github.com/repos/sindresorhus/awesome/git/trees{/sha}",
+ "statuses_url": "https://api.github.com/repos/sindresorhus/awesome/statuses/{sha}",
+ "languages_url": "https://api.github.com/repos/sindresorhus/awesome/languages",
+ "stargazers_url": "https://api.github.com/repos/sindresorhus/awesome/stargazers",
+ "contributors_url": "https://api.github.com/repos/sindresorhus/awesome/contributors",
+ "subscribers_url": "https://api.github.com/repos/sindresorhus/awesome/subscribers",
+ "subscription_url": "https://api.github.com/repos/sindresorhus/awesome/subscription",
+ "commits_url": "https://api.github.com/repos/sindresorhus/awesome/commits{/sha}",
+ "git_commits_url": "https://api.github.com/repos/sindresorhus/awesome/git/commits{/sha}",
+ "comments_url": "https://api.github.com/repos/sindresorhus/awesome/comments{/number}",
+ "issue_comment_url": "https://api.github.com/repos/sindresorhus/awesome/issues/comments{/number}",
+ "contents_url": "https://api.github.com/repos/sindresorhus/awesome/contents/{+path}",
+ "compare_url": "https://api.github.com/repos/sindresorhus/awesome/compare/{base}...{head}",
+ "merges_url": "https://api.github.com/repos/sindresorhus/awesome/merges",
+ "archive_url": "https://api.github.com/repos/sindresorhus/awesome/{archive_format}{/ref}",
+ "downloads_url": "https://api.github.com/repos/sindresorhus/awesome/downloads",
+ "issues_url": "https://api.github.com/repos/sindresorhus/awesome/issues{/number}",
+ "pulls_url": "https://api.github.com/repos/sindresorhus/awesome/pulls{/number}",
+ "milestones_url": "https://api.github.com/repos/sindresorhus/awesome/milestones{/number}",
+ "notifications_url": "https://api.github.com/repos/sindresorhus/awesome/notifications{?since,all,participating}",
+ "labels_url": "https://api.github.com/repos/sindresorhus/awesome/labels{/name}",
+ "releases_url": "https://api.github.com/repos/sindresorhus/awesome/releases{/id}",
+ "deployments_url": "https://api.github.com/repos/sindresorhus/awesome/deployments",
+ "created_at": "2014-07-11T13:42:37Z",
+ "updated_at": "2026-11-12T07:56:15Z",
+ "pushed_at": "2026-09-02T02:10:47Z",
+ "git_url": "git://github.com/sindresorhus/awesome.git",
+ "ssh_url": "[email protected]:sindresorhus/awesome.git",
+ "clone_url": "https://github.com/sindresorhus/awesome.git",
+ "svn_url": "https://github.com/sindresorhus/awesome",
+ "homepage": "",
+ "size": 1566,
+ "stargazers_count": 535531,
+ "watchers_count": 535531,
+ "language": null,
+ "has_issues": true,
+ "has_projects": false,
+ "has_downloads": false,
+ "has_wiki": false,
+ "has_pages": true,
+ "has_discussions": false,
+ "forks_count": 38323,
+ "mirror_url": null,
+ "archived": false,
+ "disabled": false,
+ "open_issues_count": 103,
+ "license": {
+ "key": "cc0-1.0",
+ "name": "Creative Commons Zero v1.0 Universal",
+ "spdx_id": "CC0-1.0",
+ "url": "https://api.github.com/licenses/cc0-1.0",
+ "node_id": "MDc6TGljZW5zZTY="
+ },
+ "allow_forking": true,
+ "is_template": false,
+ "web_commit_signoff_required": false,
+ "has_pull_requests": true,
+ "pull_request_creation_policy": "collaborators_only",
+ "topics": [
+ "awesome",
+ "awesome-list",
+ "lists",
+ "resources",
+ "unicorns"
+ ],
+ "visibility": "public",
+ "forks": 38323,
+ "open_issues": 103,
+ "watchers": 535531,
+ "default_branch": "main",
+ "score": 1.0
+ }
+ ]
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,143 @@
@@ -0,0 +1,143 @@
+'use strict';
+/**
+ * GitHub: the query plan, the recorded search answer mapped to items, the star-gain arithmetic from daily
+ * snapshots, and the client's cursor, cache and rate-limit behaviour. The fetcher is a stub returning a
+ * recorded api.github.com answer, so nothing here touches the network.
+ */
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const { suite } = require('./helpers');
+const github = require('../server/adapters/github');
+const { createGithub } = require('../server/github');
+
+const t = suite('github');
+const RECORDED = JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'github', 'search-top.json'), 'utf8'));
+const CONFIG = {
+ fetch: { userAgent: 'OpenVibeBot/test' },
+ items: { summaryMax: 1000 },
+ secrets: {},
+};
+const silent = { log() {}, warn() {}, error() {} };
+const allowedRobots = { check: async () => ({ allowed: true, reason: null, crawlDelaySec: 0 }) };
+const noSpace = { space: async () => {}, setCrawlDelay() {} };
+
+function fakeFetcher(handler) {
+ return { fetchUrl: async (url, opts) => handler(url, opts) };
+}
+
+t('the plan is today, this week, this month, recently pushed, all-time top, new, and twelve languages', () => {
+ const plan = github.plan(Date.parse('2026-11-12T12:00:00Z'));
+ assert.deepStrictEqual(plan.slice(0, 6).map((q) => q.key), ['trending', 'week', 'month', 'pushed', 'top', 'new']);
+ assert.strictEqual(plan[0].q, 'created:>2026-11-12');
+ assert.strictEqual(plan[1].q, 'created:>2026-11-05');
+ assert.strictEqual(plan[2].q, 'created:>2026-10-13');
+ assert.strictEqual(plan[3].q, 'pushed:>2026-10-13 stars:>100');
+ assert.strictEqual(plan.length, 6 + 12);
+ assert.deepStrictEqual(github.LANGUAGES.slice(0, 3), ['JavaScript', 'TypeScript', 'Python']);
+ assert.ok(plan.some((q) => q.q.includes('language:"Rust"')));
+});
+
+t('a recorded search answer becomes items with every stated field and null for the ones GitHub left out', () => {
+ const items = RECORDED.items.map((r) => github.repoItem(r, 1000)).filter(Boolean);
+ assert.strictEqual(items.length, 2);
+ const first = items[0];
+ assert.strictEqual(first.identity, 'codecrafters-io/build-your-own-x');
+ assert.strictEqual(first.canonical_url, 'https://github.com/codecrafters-io/build-your-own-x');
+ assert.strictEqual(first.fields.stars, 561961);
+ assert.strictEqual(first.fields.language, 'Markdown');
+ assert.ok(first.fields.owner_avatar_url.startsWith('https://avatars.githubusercontent.com/'));
+ assert.ok(Array.isArray(first.fields.topics));
+ assert.strictEqual(first.fields.license, null, 'no licence stated is null, never guessed');
+ assert.strictEqual(first.fields.created, new Date(RECORDED.items[0].created_at).toISOString());
+ const second = items[1];
+ assert.ok(second.fields.license, 'a stated licence is kept');
+ assert.ok(second.fields.homepage === null || second.fields.homepage.startsWith('https://'));
+ const pub = github.publicRepository(first, { days_1: 12, days_7: null });
+ assert.strictEqual(pub.stars_gained_1d, 12);
+ assert.strictEqual(pub.stars_gained_7d, null);
+ assert.strictEqual(pub.source, 'github');
+});
+
+t('a body without an items array is a parse error, never an empty success', () => {
+ assert.throws(() => github.parseSearch({ total_count: 0 }), /items array/);
+ assert.throws(() => github.parseSearch(null), /items array/);
+});
+
+t('stars gained come from daily snapshots; a window with no snapshot is null, never zero', () => {
+ const now = Date.parse('2026-11-12T12:00:00Z');
+ const history = [
+ { day: '2026-11-04', stars: 100 },
+ { day: '2026-11-10', stars: 180 },
+ { day: '2026-11-11', stars: 195 },
+ { day: '2026-11-12', stars: 210 },
+ ];
+ const g = github.gainsFrom(history, now);
+ assert.deepStrictEqual([g.stars, g.days_1, g.days_7], [210, 15, 110]);
+ const young = github.gainsFrom([{ day: '2026-11-12', stars: 14 }], now);
+ assert.deepStrictEqual([young.stars, young.days_1, young.days_7], [14, null, null]);
+ assert.deepStrictEqual(github.gainsFrom([], now), { stars: null, days_1: null, days_7: null });
+});
+
+t('one run asks its budget of queries from the stored cursor, keeps the stars and returns the next cursor', async () => {
+ const seen = [];
+ const client = createGithub({
+ config: CONFIG, robots: allowedRobots, spacer: noSpace, now: () => Date.parse('2026-11-12T12:00:00Z'), log: silent,
+ fetcher: fakeFetcher(async (url) => {
+ seen.push(url);
+ return { status: 200, body: Buffer.from(JSON.stringify(RECORDED)), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '9', 'x-ratelimit-reset': '1794471413' }, bytes: 100, finalUrl: url };
+ }),
+ });
+ const out = await client.fetchQueries({ source: { key: 'github', min_interval_ms: 0 }, cursor: {}, budget: 2 });
+ assert.strictEqual(seen.length, 2);
+ assert.match(seen[0], /created%3A%3E2026-11-12|created%3A%3E2026/);
+ assert.strictEqual(out.queries, 2);
+ assert.strictEqual(out.items.length, 2);
+ assert.deepStrictEqual(out.stars.map((s) => s.full_name).sort(), ['codecrafters-io/build-your-own-x', 'sindresorhus/awesome']);
+ assert.strictEqual(out.next, 2);
+ const next = await client.fetchQueries({ source: { key: 'github', min_interval_ms: 0 }, cursor: { next: out.next }, budget: 1 });
+ assert.strictEqual(next.queries, 1);
+ assert.strictEqual(next.next, 3);
+});
+
+t('a rate-limited answer backs off: the reset time is honoured and a partial batch keeps what it has', async () => {
+ let calls = 0;
+ const reset = Math.floor(Date.parse('2026-11-12T12:01:00Z') / 1000);
+ const client = createGithub({
+ config: CONFIG, robots: allowedRobots, spacer: noSpace, now: () => Date.parse('2026-11-12T12:00:00Z'), log: silent,
+ fetcher: fakeFetcher(async (url) => {
+ calls++;
+ if (calls === 1) return { status: 200, body: Buffer.from(JSON.stringify(RECORDED)), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '1', 'x-ratelimit-reset': String(reset) }, bytes: 100, finalUrl: url };
+ return { status: 403, body: Buffer.from('{"message":"API rate limit exceeded"}'), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '0', 'x-ratelimit-reset': String(reset) }, bytes: 40, finalUrl: url };
+ }),
+ });
+ const out = await client.fetchQueries({ source: { key: 'github', min_interval_ms: 0 }, budget: 3 });
+ assert.strictEqual(out.queries, 1, 'the batch stopped at the refusal');
+ assert.strictEqual(out.limited, true);
+ assert.strictEqual(out.items.length, 2);
+ assert.ok(out.notBefore >= Date.parse('2026-11-12T12:00:59Z'), 'the next run waits for GitHub reset');
+ assert.ok(!JSON.stringify(out).includes('Authorization'), 'the token never reaches a result');
+});
+
+t('a live search is cached for ten minutes and passes GitHub syntax through unchanged', async () => {
+ let calls = 0;
+ let seenUrl = '';
+ const client = createGithub({
+ config: CONFIG, robots: allowedRobots, spacer: noSpace, now: () => Date.parse('2026-11-12T12:00:00Z'), log: silent,
+ fetcher: fakeFetcher(async (url) => {
+ calls++;
+ seenUrl = url;
+ assert.match(decodeURIComponent(url).replace(/\+/g, ' '), /stars:>500 topic:ai language:"Rust"/);
+ return { status: 200, body: Buffer.from(JSON.stringify(RECORDED)), headers: { 'content-type': 'application/json', 'x-ratelimit-remaining': '8', 'x-ratelimit-reset': '1794471413' }, bytes: 100, finalUrl: url };
+ }),
+ });
+ const first = await client.search({ q: 'stars:>500 topic:ai', lang: 'Rust', sort: 'stars' });
+ const second = await client.search({ q: 'stars:>500 topic:ai', lang: 'Rust', sort: 'stars' });
+ assert.strictEqual(calls, 1, 'the second read is the cache');
+ assert.strictEqual(second.cached, true);
+ assert.strictEqual(first.items.length, 2);
+ assert.strictEqual(first.page, 1);
+ assert.match(decodeURIComponent(seenUrl), /sort=stars/);
+});
+
+t.run();