openvibers/openvibe.work Commits
Work: every ingested board but Remotive comes from OpenVibe.Sources (#274)
Changes
@@@ -58,19 +58,16 @@ WORK_EVENTS_URL=#WORK_INGEST=on#WORK_INGEST_INTERVAL_HOURS=6#WORK_INGEST_TIMEOUT_MS=15000--#WORK_INGEST_PAGES=3#WORK_INGEST_LIMIT=100#WORK_RETENTION_DAYS=30--# The boards' public APIs. Point these at a stand-in server to test the ingest; nothing else in the service fetches--# anything, and never a URL a caller typed.--#WORK_ARBEITNOW_URL=https://www.arbeitnow.com/api/job-board-api--# Arbeitnow through OpenVibe.Sources (server/ingest/from-sources.js): with WORK_ARBEITNOW_FROM=sources Work stops reading--# the board and follows Sources' arbeitnow-job-board items instead (pages 1-3; a job the board stops showing is retired--# after a week). Needs OV_OAUTH_CLIENT_SECRET (Network grants Work sources.item.read); Sources is on 127.0.0.1:4720.--#WORK_ARBEITNOW_FROM=sources--#OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720++# Remotive is the one board Work still fetches itself (its API terms grant a named, linked use; its robots.txt++# disallows /api/* so OpenVibe.Sources may not fetch it). Point this at a stand-in server to test the ingest;++# nothing else in the service fetches a URL, and never one a caller typed.#WORK_REMOTIVE_URL=https://remotive.com/api/remote-jobs--#WORK_REMOTEOK_URL=https://remoteok.com/api++# Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK come through OpenVibe.Sources++# (server/ingest/from-sources.js). Needs OV_OAUTH_CLIENT_SECRET (Network grants Work sources.item.read);++# Sources is on 127.0.0.1:4720.++#OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720# USAJOBS (D11 owner settings): the admin panel's work.data_sources namespace wins; these are the# fallback when no value was entered there. The email must match the key's account.#USAJOBS_API_KEY=@@@ -15,9 +15,10 @@ board, where the reader applies.- **Ingest:** `server/ingest/` (`index.js`, `from-sources.js`) and `server/jobs/` (`sources.js`,`store.js`, `text.js`, `hiring-line.js`, `job-types.js`, `places.js`, `companies.js`,`rederive-hn-titles.js`); run it with `npm run ingest` or the in-process interval.--- **Sources:** Arbeitnow, Remotive, RemoteOK and Hacker News "Who is hiring", each through its own-- public API and each with its terms applied on every page (`/sources`, `GET /api/v1/sources`);-- Arbeitnow can instead be read from OpenVibe.Sources (`WORK_ARBEITNOW_FROM=sources`).++- **Sources:** Arbeitnow, We Work Remotely, RemoteOK and Hacker News "Who is hiring" arrive through++ OpenVibe.Sources (`server/ingest/from-sources.js`); Remotive is the one board Work still fetches++ itself (its robots.txt disallows `/api/*`, its API terms grant the named, linked use). Every++ board's terms are applied on every page (`/sources`, `GET /api/v1/sources`).- **Docs/config:** README "Data sources and their terms", `SECURITY.md`, `.env.example`.## Owns, does not own@@@ -74,7 +74,7 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written bythe canonical subject. Work holds only `user:usr_…` on its own rows.- **Event delivery** — OpenVibe.Events: Work receives `network.account.export_requested` and `network.account.deleted`and creates its two subscriptions at boot; it does not own the topics or the delivery.--- **The listings themselves** — the boards (Arbeitnow, Remotive, RemoteOK and Hacker News "Who is hiring"): Work keeps a short excerpt and a link back,++- **The listings themselves** — the boards (Arbeitnow, We Work Remotely, RemoteOK, Hacker News "Who is hiring" and Remotive): Work keeps a short excerpt and a link back,never the full description, and reads each board only on the terms it publishes. A board that writes a listing asone pipe-separated line (Hacker News) has that line split into company, role(s), place, terms and pay(`server/jobs/hiring-line.js`); the line itself stays at the front of the excerpt.@@@ -90,9 +90,9 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by- **OpenVibe.Actor** — the person's agent: Work posts `POST /api/v1/tasks` with its own service token (capability`actor.task.create`, audience `openvibe.actor`) when a person asks for help preparing an application(`WORK_ACTOR_URL`, `WORK_ACTOR_PUBLIC_URL`, `WORK_ACTOR_TIMEOUT_MS`).--- **The job boards** — Arbeitnow, Remotive and RemoteOK are the only hosts Work itself fetches (`WORK_ARBEITNOW_URL`,-- `WORK_REMOTIVE_URL`, `WORK_REMOTEOK_URL`, `WORK_USER_AGENT`). We Work Remotely and Hacker News "Who is hiring" arrive-- through OpenVibe.Sources (`WORK_WWR_FROM=sources`, `WORK_HN_WHOISHIRING_FROM=sources`).++- **The job boards** — Remotive is the only host Work itself fetches (`WORK_REMOTIVE_URL`, `WORK_USER_AGENT`), because++ its API terms grant a named, linked use while its robots.txt disallows `/api/*`. Arbeitnow, RemoteOK, We Work Remotely++ and Hacker News "Who is hiring" arrive through OpenVibe.Sources (`server/ingest/from-sources.js`).- **PostgreSQL** (`DATABASE_URL`, `DATABASE_DIRECT_URL`; an embedded PGlite database in development via`WORK_PGLITE_DIR`) and **Valkey** for shared limit counters (`VALKEY_URL`, `VALKEY_PREFIX`).- **Packages**: `openvibe-contracts` v0.289.1, `openvibe-sdk` v0.150.0 (`db`, `auth`, `account-data`, `limits`,@@@ -151,9 +151,9 @@ Each board is read through its own public API, and its terms are applied on ever| Source | API | Terms, as applied here ||---|---|---|--| **Arbeitnow** | `GET https://www.arbeitnow.com/api/job-board-api?page=N` | Free to use. Arbeitnow is named on every listing and the listing links to its page on arbeitnow.com. Its first pages (configurable) are read. With `WORK_ARBEITNOW_FROM=sources` Work reads none: it follows OpenVibe.Sources' `arbeitnow-job-board` items (`server/ingest/from-sources.js`), and a job the board stops showing is retired by Sources after a week. |--| **Remotive** | `GET https://remotive.com/api/remote-jobs?limit=100` | Remotive must be named as the source and the listing linked. The API is called about four times a day — the six-hour interval is exactly that. |--| **RemoteOK** | `GET https://remoteok.com/api` | RemoteOK must be named as the source and the listing linked. The response's first element is a legal notice, not a listing, and is skipped. |++| **Arbeitnow** | `GET https://www.arbeitnow.com/api/job-board-api?page=N` | Read by OpenVibe.Sources (source `arbeitnow-job-board`), never by Work: its first three pages are fetched under the board's robots and terms, and Work follows that change feed (`server/ingest/from-sources.js`). Arbeitnow is named on every listing and the listing links to its page on arbeitnow.com; a job the board stops showing is retired by Sources after a week. |++| **Remotive** | `GET https://remotive.com/api/remote-jobs?limit=100` | Remotive must be named as the source and the listing linked. The API is called about four times a day — the six-hour interval is exactly that. It stays Work's own fetch because remotive.com/robots.txt disallows `/api/*`, so OpenVibe.Sources may not read it, while Remotive's API terms grant the named, linked use. |++| **RemoteOK** | `GET https://remoteok.com/api` | Read by OpenVibe.Sources (source `remoteok-job-board`), never by Work: the API is fetched under remoteok.com's robots.txt and its terms. RemoteOK must be named as the source and the listing linked; the response's first element is a legal notice, not a listing, and Sources' mapping skips it. |What is stored per listing: `source`, the board's own id, the original `url`, title, company, location, the remoteflag, tags, job type, salary text when the board gives one, when it was posted, when we fetched it, when it was first@@@ -171,9 +171,8 @@ Network), `BASE_URL`, `DATABASE_URL` and `DATABASE_DIRECT_URL`. The database isNetwork signing key, the OAuth client and Valkey are optional (the service says so, per check, on `/api/ready`).The ingest is **off** unless `WORK_INGEST=on`, so a development run and the test suite never call a job board. When it--is on, `WORK_INGEST_INTERVAL_HOURS` (6), `WORK_INGEST_PAGES` (3), `WORK_INGEST_LIMIT` (100), `WORK_INGEST_TIMEOUT_MS`--(15000), `WORK_RETENTION_DAYS` (30) and the per-source base URLs (`WORK_ARBEITNOW_URL`, `WORK_REMOTIVE_URL`,--`WORK_REMOTEOK_URL`) decide what it does.++is on, `WORK_INGEST_INTERVAL_HOURS` (6), `WORK_INGEST_LIMIT` (100), `WORK_INGEST_TIMEOUT_MS` (15000),++`WORK_RETENTION_DAYS` (30), `WORK_REMOTIVE_URL` and `OV_SOURCES_INTERNAL_URL` decide what it does.The agent pillar is driven by `WORK_ACTOR_URL` (default `http://127.0.0.1:4950`) — unset it (an empty value) and`POST …/assist` answers 503 `work.actor_unavailable` — and `WORK_ACTOR_PUBLIC_URL` (default `https://openvibe.actor`),@@@ -16,9 +16,9 @@"openvibe-shared v4.214.0","openvibe-publishing v1.7.0"],-- "summary": "OpenVibe.Work gathers job listings from three public job boards (Arbeitnow, Remotive, RemoteOK) under their terms — the source is named and the original listing is linked on every page, and only a short excerpt is kept. It searches them with facets, shows one listing with a schema.org JobPosting whose url is the original, and lets a signed-in person keep a search, see what is new since they last looked, get alerts through OpenVibe.Watch, track the jobs they apply to with a status history, and ask their OpenVibe.Actor agent to help prepare an application. Public reads need no token.",++ "summary": "OpenVibe.Work gathers job listings from five public job boards (Arbeitnow, We Work Remotely, RemoteOK, Hacker News and Remotive) under their terms — four arrive through OpenVibe.Sources, and Remotive is read directly because its robots.txt keeps Sources out — the source is named and the original listing is linked on every page, and only a short excerpt is kept. It searches them with facets, shows one listing with a schema.org JobPosting whose url is the original, and lets a signed-in person keep a search, see what is new since they last looked, get alerts through OpenVibe.Watch, track the jobs they apply to with a status history, and ask their OpenVibe.Actor agent to help prepare an application. Public reads need no token.","works": [-- "The ingest: three boards read on a timer (WORK_INGEST=on, off in tests and development), each with its own User-Agent, 15-second timeout, upsert by (source, source_id) and expiry after 30 days unseen",++ "The ingest: Remotive read directly and Arbeitnow, We Work Remotely, RemoteOK and Hacker News followed from OpenVibe.Sources (WORK_INGEST=on, off in tests and development), with upsert by (source, source_id) and expiry after 30 days unseen","The listings: provenance kept and shown (source name, link to the original, when it was fetched), a short plain-text excerpt, never the full description","Search: /jobs?q=&remote=&type=&location= newest first, 30 a page with a cursor, facets for remote and job type","One listing: /jobs/:id with the board's fields, the excerpt, a 'Read the full listing on <source>' link and a JobPosting JSON-LD whose url is the original",@@@ -126,8 +126,8 @@ async function createApp(opts = {}) {return createIndexNow({ host: config.baseUrl, key: '', fetch: fetchImpl, log: warn });}})();-- // Arbeitnow through OpenVibe.Sources (server/ingest/from-sources.js), off unless WORK_ARBEITNOW_FROM=sources; a test-- // passes its own Sources stand-in as opts.sourcesClient.++ // Arbeitnow, We Work Remotely, Hacker News and RemoteOK through OpenVibe.Sources (server/ingest/from-sources.js);++ // it is off without a Sources URL and the OAuth client secret, and a test passes its own stand-in as opts.sourcesClient.ctx.fromSources = createFromSources({ config, s, fetchImpl, log, source: opts.sourcesClient || null, afterRun: () => ctx.search.afterIngest(), indexnow: ctx.indexnow });ctx.ingest = createIngest({ config, s, fetchImpl, log, afterRun: () => ctx.search.afterIngest(), indexnow: ctx.indexnow, fromSources: ctx.fromSources });@@@ -77,25 +77,15 @@ function load(env = process.env) {timeoutMs: Math.max(1000, int(env.WORK_ACTOR_TIMEOUT_MS, 15000)),},-- // The public job boards the ingest reads (server/jobs/sources.js). The base URLs are configurable so a-- // test points them at a stand-in server; nothing else in the service ever fetches a URL, and never one a-- // caller typed. Their terms are quoted in the manifest that describes each source, and applied on every-- // page that shows a listing: the source is named and the original listing is linked.++ // The one public job board the ingest still reads itself (server/jobs/sources.js): Remotive's API terms grant a++ // named, linked use, while its robots.txt disallows /api/*, so Sources may not fetch it. Its base URL is++ // configurable so a test points it at a stand-in; nothing else in the service ever fetches a URL, and never one++ // a caller typed. Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK come through++ // OpenVibe.Sources (server/ingest/from-sources.js), which needs Sources (127.0.0.1:4720) and the OAuth client++ // secret (Network grants Work sources.item.read).sources: {-- arbeitnow: { url: trim(env.WORK_ARBEITNOW_URL || 'https://www.arbeitnow.com/api/job-board-api') },remotive: { url: trim(env.WORK_REMOTIVE_URL || 'https://remotive.com/api/remote-jobs') },-- remoteok: { url: trim(env.WORK_REMOTEOK_URL || 'https://remoteok.com/api') },-- // OpenVibe.Sources (server/ingest/from-sources.js): WORK_ARBEITNOW_FROM=sources takes Arbeitnow from Sources'-- // arbeitnow-job-board instead of the board (plan 2026-10-22, F2); it needs Sources (127.0.0.1:4720) and the-- // OAuth client secret (Network grants Work sources.item.read). Anything else keeps Work's own fetch.internalUrl: trim(env.OV_SOURCES_INTERNAL_URL == null ? 'http://127.0.0.1:4720' : env.OV_SOURCES_INTERNAL_URL),-- arbeitnowFrom: String(env.WORK_ARBEITNOW_FROM || '').trim().toLowerCase() === 'sources' ? 'sources' : 'board',-- // We Work Remotely reaches Work the same way (WORK_WWR_FROM=sources, the weworkremotely key in Sources);-- // it is off until the board is read through Sources, and Work never fetches the feed itself.-- weworkremotelyFrom: String(env.WORK_WWR_FROM || '').trim().toLowerCase() === 'sources' ? 'sources' : 'board',-- // Hacker News "Who is hiring" reaches Work the same way (WORK_HN_WHOISHIRING_FROM=sources, the-- // hn-whoishiring key in Sources); its resolver reads the thread, never Work itself.-- hnWhoishiringFrom: String(env.WORK_HN_WHOISHIRING_FROM || '').trim().toLowerCase() === 'sources' ? 'sources' : 'board',},// The disclosure rate the jobs page shows (server/jobs/store.js): of the last N listings we fetched, how// many carried a salary. N is configuration, so the number on the page is never typed in by hand.@@@ -111,8 +101,7 @@ function load(env = process.env) {enabled: String(env.WORK_INGEST || '').toLowerCase() === 'on',intervalMs: Math.max(1, num(env.WORK_INGEST_INTERVAL_HOURS, 6)) * 3_600_000,timeoutMs: Math.max(1000, int(env.WORK_INGEST_TIMEOUT_MS, 15_000)),-- // Arbeitnow's board is paginated; the first N pages are read. Remotive takes a limit.-- pages: Math.max(1, int(env.WORK_INGEST_PAGES, 3)),++ // Remotive takes a limit; the boards read through Sources carry their own mappings.limit: Math.max(1, int(env.WORK_INGEST_LIMIT, 100)),retentionDays: Math.max(1, int(env.WORK_RETENTION_DAYS, 30)),userAgent: env.WORK_USER_AGENT || `OpenVibeWork/0.1 (+${baseUrl})`,@@@ -1,7 +1,8 @@'use strict';/**-- * Job boards read through OpenVibe.Sources (plan 2026-10-22, track F2; We Work Remotely added 2026-10-30).++ * Job boards read through OpenVibe.Sources (plan 2026-10-22, track F; We Work Remotely added 2026-10-30, RemoteOK++ * 2026-11-12).** Sources fetches a board (under its robots and terms checks) and keeps each job as an item. Work follows Sources'* change feed (openvibe-publishing/ingest: the cursor in work_ingest_cursor, one transaction per page) and keeps one@@@ -12,12 +13,12 @@* The thin-listing gate (sources.toRow) is the same for all of them: no id, no title, no company or no http(s) link* back means the listing is not published.*-- * A job Sources retires (the board has not shown it for a week: retire_after_sec) or removes is expired here, like a-- * listing a board stopped showing. Work's own expiry sweep leaves these listings alone: an unchanged job is no change in-- * Sources' feed, so Work never hears it was seen again, and Sources' retirement is what says it is gone.++ * A job Sources retires (the board has not shown it for its own window: retire_after_sec) or removes is expired here,++ * like a listing a board stopped showing. Work's own expiry sweep leaves these listings alone: an unchanged job is no++ * change in Sources' feed, so Work never hears it was seen again, and Sources' retirement is what says it is gone.*-- * Pulls 30 seconds after start and then hourly. Each board has its own switch (WORK_ARBEITNOW_FROM=sources,-- * WORK_WWR_FROM=sources); a board that is not switched on is left to its own fetch, as before.++ * Pulls 30 seconds after start and then hourly. Remotive is the one board still read by Work itself: its robots.txt++ * disallows /api/*, so Sources may not fetch it, while its API terms explicitly grant a named, linked use.*/const ingest = require('openvibe-publishing/ingest');const sources = require('../jobs/sources');@@@ -138,6 +139,35 @@ function rawOfHn(item) {/** The start of a plain-text summary, with whitespace collapsed, for comparing it with the comment's first line. */const excerptLines = (text) => String(text || '').replace(/\s+/g, ' ').trim();++/** A salary bound as RemoteOK gives it (a number), or null; never a guess from a missing field. */++const amount = (v) => {++ const n = Number(v);++ return Number.isFinite(n) && n > 0 ? Math.round(n) : null;++};++++/**++ * A Sources item → the parsed row Work's own RemoteOK parse makes (server/jobs/sources.js): the board's id and URL,++ * its position as the title, and its salary bounds written the way the board's own fetch wrote them. Sources has++ * already turned the HTML description into a short plain-text excerpt, which is all Work stores.++ */++function rawOfRemoteOk(item) {++ const f = (item && item.fields) || {};++ const min = amount(f.salary_min);++ const max = amount(f.salary_max);++ return {++ source_id: item.identity,++ url: item.canonical_url,++ title: item.title,++ company: typeof f.company === 'string' ? f.company.trim() : null,++ location: typeof f.location === 'string' ? f.location.trim() || null : null,++ remote: true, // the board carries remote work only++ tags: list(f.tags),++ job_type: typeof f.job_type === 'string' ? f.job_type.trim() || null : null,++ salary: min && max ? `$${min.toLocaleString('en-US')}–$${max.toLocaleString('en-US')} a year` : null,++ posted_at: item.published_at || null,++ description_html: item.summary || '',++ };++}/*** The boards Work reads through OpenVibe.Sources: the Sources key an item arrives under, the id the listing is@@@ -147,22 +177,15 @@ const SOURCES = {[SOURCE_KEY]: { source_id: SOURCE_ID, kind: 'job', rawOf },'weworkremotely': { source_id: 'weworkremotely', kind: 'article', rawOf: rawOfWwr },'hn-whoishiring': { source_id: 'hn-whoishiring', kind: 'job', rawOf: rawOfHn },--};----/** The switch that says a board comes from Sources instead of from its own fetch. */--const SWITCH = {-- arbeitnow: (config) => (config.sources && config.sources.arbeitnowFrom) === 'sources',-- weworkremotely: (config) => (config.sources && config.sources.weworkremotelyFrom) === 'sources',-- 'hn-whoishiring': (config) => (config.sources && config.sources.hnWhoishiringFrom) === 'sources',++ 'remoteok-job-board': { source_id: 'remoteok', kind: 'job', rawOf: rawOfRemoteOk },};function createFromSources({ config, s, fetchImpl = globalThis.fetch, log = console, source = null, afterRun = null, indexnow = null }) {const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');-- // The boards that are switched on and therefore read through Sources. Each keeps its own listing id.-- const active = Object.entries(SOURCES).filter(([, e]) => (SWITCH[e.source_id] || (() => false))(config));++ // Every board in SOURCES is read through Sources; each keeps its own listing id.++ const active = Object.entries(SOURCES);const activeByKey = Object.fromEntries(active);-- const on = active.length > 0;-- const client = source || (on && url && config.oauth.clientSecret++ const client = source || (url && config.oauth.clientSecret? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'jobs' } }, fetchImpl }): null);const cursor = ingest.createChangeCursor(s.db, { prefix: 'work', now: s.now });@@@ -239,4 +262,4 @@ function createFromSources({ config, s, fetchImpl = globalThis.fetch, log = consreturn { enabled: Boolean(client), pull, start, stop, apply, SOURCE_ID, sourceIds: active.map(([, e]) => e.source_id) };}--module.exports = { createFromSources, rawOf, rawOfWwr, rawOfHn, SOURCES, SOURCE_ID, SOURCE_KEY, TITLE_RULE };++module.exports = { createFromSources, rawOf, rawOfWwr, rawOfHn, rawOfRemoteOk, SOURCES, SOURCE_ID, SOURCE_KEY, TITLE_RULE };@@@ -1,8 +1,9 @@'use strict';/**-- * The background ingest: every six hours, read each job board's public API (server/jobs/sources.js), keep what it-- * shows and retire what it stopped showing.++ * The background ingest: every six hours, read Remotive's public API (server/jobs/sources.js), keep what it shows and++ * retire what it stopped showing. Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK are read by++ * OpenVibe.Sources and followed in server/ingest/from-sources.js; no source in this module may fetch them.** It is off unless WORK_INGEST=on, so neither a test nor a development run ever calls a job board; when it is on,* each source keeps its own timer, its own last-fetch row and its own failures — one board being down never stops the@@@ -46,6 +47,8 @@ function createIngest({ config, s, fetchImpl = globalThis.fetch, log = console,async function runSource(sourceId) {const source = sources.byId(sourceId);if (!source) throw new Error(`no source ${sourceId}`);++ // A board OpenVibe.Sources feeds is not read here, not even by an operator naming it: Sources is its reader.++ if (source.viaSources) return { source: source.id, skipped: 'from_sources' };const startedAt = s.iso();const fetchedAt = startedAt;await store.beginFetch(s, source.id, startedAt);@@@ -51,42 +51,17 @@ function httpUrl(v) {} catch { return null; }}--const money = (n) => (n == null || n === '' || !Number.isFinite(Number(n)) || Number(n) <= 0 ? null : Math.round(Number(n)));--const SOURCES = [{id: 'arbeitnow',name: 'Arbeitnow',homepage: 'https://www.arbeitnow.com/',-- terms: 'Arbeitnow publishes its job board API free of charge. We attribute Arbeitnow by name and link every listing back to its page on arbeitnow.com.',++ terms: 'Arbeitnow publishes its job board API free of charge. OpenVibe.Sources reads its first three pages under the board\'s own robots and terms checks; Work follows that source (source key arbeitnow-job-board). We attribute Arbeitnow by name and link every listing back to its page on arbeitnow.com.',attribution: 'Listing from Arbeitnow',remoteDefault: false,-- requests(config) {-- const pages = [];-- for (let page = 1; page <= config.ingest.pages; page++) {-- const u = new URL(config.sources.arbeitnow.url);-- u.searchParams.set('page', String(page));-- pages.push(u.toString());-- }-- return pages;-- },-- // { data: [ { slug, company_name, title, description (HTML), remote, url, tags[], job_types[], location, created_at } ] }-- parse(json) {-- const rows = (json && Array.isArray(json.data)) ? json.data : [];-- return rows.map((r) => ({-- source_id: clip(r.slug || r.url, 200),-- url: httpUrl(r.url),-- title: clip(r.title, 300),-- company: clip(r.company_name, 200),-- location: clip(r.location, 200),-- remote: r.remote === true,-- tags: list(r.tags),-- job_type: list(r.job_types).join(', ') || null,-- salary: null, // this board gives no salary-- posted_at: instant(r.created_at),-- description_html: r.description,-- }));-- },++ viaSources: true, // read through OpenVibe.Sources, never by Work itself++ requests() { return []; },++ parse() { return []; }, // Work never calls the Arbeitnow API itself},{id: 'weworkremotely',@@@ -146,32 +121,12 @@ const SOURCES = [id: 'remoteok',name: 'RemoteOK',homepage: 'https://remoteok.com/',-- terms: 'RemoteOK asks that a use of its API names RemoteOK as the source and links back to the listing. The response\'s first element is a legal notice, not a listing, and is skipped.',++ terms: 'RemoteOK asks that a use of its API names RemoteOK as the source and links back to the listing. OpenVibe.Sources reads the API under those terms and its robots.txt; the response\'s first element is a legal notice, not a listing, and is skipped. Work follows that source (source key remoteok-job-board).',attribution: 'Listing from RemoteOK',remoteDefault: true,-- requests(config) { return [config.sources.remoteok.url]; },-- // The response is an array whose first element is the legal notice; every later element is a listing.-- parse(json) {-- const all = Array.isArray(json) ? json : [];-- const rows = all.filter((r) => r && (r.id != null || r.slug) && r.position);-- return rows.map((r) => {-- const min = money(r.salary_min);-- const max = money(r.salary_max);-- return {-- source_id: clip(r.id != null ? r.id : r.slug, 200),-- url: httpUrl(r.url || (r.slug ? `https://remoteok.com/remote-jobs/${r.slug}` : null)),-- title: clip(r.position, 300),-- company: clip(r.company, 200),-- location: clip(r.location, 200),-- remote: true,-- tags: list(r.tags),-- job_type: clip(r.job_type || r.type, 80) || null,-- salary: min && max ? `$${min.toLocaleString('en-US')}–$${max.toLocaleString('en-US')} a year` : null,-- posted_at: instant(r.date),-- description_html: r.description,-- };-- });-- },++ viaSources: true, // read through OpenVibe.Sources, never by Work itself++ requests() { return []; },++ parse() { return []; }, // Work never calls the RemoteOK API itself},];@@@ -18,7 +18,7 @@ const companies = require('../server/jobs/companies');assert.strictEqual(companies.slugOf('x'.repeat(200)).length, 80);});-- const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });++ const t = await boot();try {await t.ctx.ingest.runAll();const s = t.ctx.s;@@@ -1,12 +1,12 @@'use strict';/**-- * Arbeitnow through OpenVibe.Sources (server/ingest/from-sources.js; plan 2026-10-22, track F2), against a Sources-- * stand-in serving its change feed:++ * Arbeitnow, We Work Remotely, Hacker News and RemoteOK through OpenVibe.Sources (server/ingest/from-sources.js;++ * plan 2026-10-22, track F), against a Sources stand-in serving its change feed:* - each job item becomes the listing Work's own fetch made (same slug → same id and URL), with its excerpt, tags and* job types; other sources' items and jobs Work may not publish are held;* - the board is no longer read, and Work's expiry sweep leaves these listings to Sources;* - a job Sources retires (gone_from_source) expires here; the fetch record says what came through;-- * - without WORK_ARBEITNOW_FROM=sources nothing changes.++ * - without a Sources URL and client secret the follower stays off.*/const assert = require('assert');const { boot, check, done } = require('./helpers/boot');@@@ -36,7 +36,7 @@ const job = (identity, over = {}) => ({const clock = { t: Date.parse('2026-10-23T06:00:00Z') };const pinged = [];const indexnow = { enabled: false, pingSoon: (urls) => { pinged.push(...urls); return urls.length; } };-- const t = await boot({ now: () => clock.t, env: { WORK_ARBEITNOW_FROM: 'sources' }, sourcesClient, indexnow });++ const t = await boot({ now: () => clock.t, sourcesClient, indexnow });const s = t.ctx.s;const from = t.ctx.fromSources;@@@ -90,8 +90,8 @@ const job = (identity, over = {}) => ({assert.deepStrictEqual([(await from.pull()).applied, (await from.pull()).removed], [0, 0]);});-- await check('without WORK_ARBEITNOW_FROM=sources nothing follows Sources', async () => {-- const config = { ...t.config, sources: { ...t.config.sources, arbeitnowFrom: 'board', internalUrl: 'http://127.0.0.1:1' } };++ await check('without a Sources URL and client secret the follower stays off', async () => {++ const config = { ...t.config, sources: { ...t.config.sources, internalUrl: '' }, oauth: { ...t.config.oauth, clientSecret: '' } };const off = createFromSources({ config, s, log: { log() {}, warn() {} } });assert.strictEqual(off.enabled, false);assert.deepStrictEqual(await off.pull(), { skipped: true });@@@ -1,63 +1,23 @@'use strict';/**-- * Stand-in job boards for the ingest tests: one HTTP server with a path per board, answering the shape each real API-- * documents, with small fixtures written to look like the real thing (including RemoteOK's legal notice as the first-- * array element, and an Arbeitnow listing whose title carries a <script> tag, so the pages' escaping is provable).++ * A stand-in Remotive API for the one board Work still reads itself (test/ingest.test.js). Its payload is the shape++ * Remotive documents, with fixtures of ours (no real listing is copied here) including a description longer than the++ * excerpt Work keeps.++ *++ * The other boards are not here because Work does not fetch them: Arbeitnow, We Work Remotely, Hacker News "Who is++ * hiring" and RemoteOK are served by OpenVibe.Sources, and their tests use a Sources feed stand-in instead.** const b = await startBoards();-- * b.env WORK_ARBEITNOW_URL / WORK_REMOTIVE_URL / WORK_REMOTEOK_URL, for boot({ env })-- * b.state.arbeitnow = 'fail' one board answering HTTP 500 (default 'ok')-- * b.state.remotive = 'json' one board answering something that is not JSON-- * b.add('arbeitnow', row) a listing the board starts showing (see b.fixtures for the shape)++ * b.env WORK_REMOTIVE_URL, for boot({ env })++ * b.state.remotive = 'fail' the board answering HTTP 500 (default 'ok')++ * b.add('remotive', row) a listing the board starts showing (see b.fixtures)* b.requests [{ path, query, userAgent }] for every call-- *-- * The fixtures are ours, not a board's: no real listing is copied here.*/const http = require('http');/** One listing per board, in that board's own payload shape. */function fixtures() {return {-- arbeitnow: [-- {-- slug: 'senior-backend-engineer-berlin',-- company_name: 'Nordlicht Systems',-- title: 'Senior Backend Engineer (Node.js)',-- description: '<p>We run a fleet of Node.js services.</p><ul><li>PostgreSQL</li><li>Valkey</li></ul><p>You will own the billing pipeline end to end.</p>',-- remote: false,-- url: 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345',-- tags: ['nodejs', 'postgresql'],-- job_types: ['permanent'],-- location: 'Berlin, Germany',-- created_at: 1791500000,-- },-- {-- slug: 'frontend-engineer-remote-eu',-- company_name: 'Helles Werk',-- title: 'Frontend Engineer <script>alert("xss")</script>',-- description: '<div>Build the design system.<script>window.__pwned = true</script></div><div>TypeScript, CSS.</div>',-- remote: true,-- url: 'https://www.arbeitnow.com/view/frontend-engineer-remote-eu-23456',-- tags: ['typescript', 'css'],-- job_types: ['permanent', 'full_time'],-- location: 'Remote (EU)',-- created_at: 1791600000,-- },-- ],-- arbeitnowPage2: [-- {-- slug: 'data-analyst-munich',-- company_name: 'Alpen Daten',-- title: 'Data Analyst',-- description: '<p>SQL, dashboards and a lot of questions.</p>',-- remote: false,-- url: 'https://www.arbeitnow.com/view/data-analyst-munich-34567',-- tags: ['sql'],-- job_types: ['permanent'],-- location: 'Munich, Germany',-- created_at: 1791700000,-- },-- ],remotive: [{id: 1900123,@@@ -104,43 +64,14 @@ blameless. The office dog is called Waffle.</p>`,description: '<p>Write about developer tools.</p>',},],-- remoteok: [-- { legal: "RemoteOK's API is offered for personal use; please link back to the listing and name RemoteOK as the source." },-- {-- id: '1049001',-- slug: 'devops-engineer-skyward',-- company: 'Skyward Cloud',-- position: 'DevOps Engineer',-- tags: ['devops', 'kubernetes'],-- location: 'Worldwide',-- salary_min: 90000,-- salary_max: 130000,-- date: '2026-10-01T08:00:00+00:00',-- url: 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001',-- description: '<p>Kubernetes, Terraform, on-call.</p>',-- },-- {-- id: '1049002',-- slug: 'technical-writer-atlas',-- company: 'Atlas Docs',-- position: 'Technical Writer',-- tags: ['docs'],-- location: 'EU',-- salary_min: null,-- salary_max: null,-- date: '2026-09-25T08:00:00+00:00',-- url: 'https://remoteok.com/remote-jobs/remote-technical-writer-atlas-1049002',-- description: '<p>Document an API.</p>',-- },-- ],};}async function startBoards() {const data = fixtures();const requests = [];-- const state = { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' };-- const added = { arbeitnow: [], remotive: [], remoteok: [] };++ const state = { remotive: 'ok' };++ const added = { remotive: [] };const send = (res, status, body, type = 'application/json') => { res.writeHead(status, { 'content-type': type }); res.end(typeof body === 'string' ? body : JSON.stringify(body)); };@@@ -148,17 +79,10 @@ async function startBoards() {const url = new URL(req.url, 'http://stand-in');requests.push({ path: url.pathname, query: url.search, userAgent: req.headers['user-agent'] || '', accept: req.headers.accept || '' });const board = url.pathname.replace(/^\//, '');-- if (!['arbeitnow', 'remotive', 'remoteok'].includes(board)) return send(res, 404, { error: 'not a board' });-- if (state[board] === 'fail') return send(res, 500, { error: 'the board is having a bad day' }, 'text/plain');-- if (state[board] === 'html') return send(res, 200, '<html>not json</html>', 'text/html');-- if (board === 'arbeitnow') {-- const page = Number(url.searchParams.get('page') || 1);-- const rows = page === 1 ? data.arbeitnow : page === 2 ? data.arbeitnowPage2 : [];-- return send(res, 200, { data: [...rows, ...added.arbeitnow], links: { next: page < 2 ? `https://www.arbeitnow.com/api/job-board-api?page=${page + 1}` : null } });-- }-- if (board === 'remotive') return send(res, 200, { 'job-count': data.remotive.length + added.remotive.length, jobs: [...data.remotive, ...added.remotive] });-- // RemoteOK: the legal notice is element 0, exactly as the real API sends it.-- return send(res, 200, [...data.remoteok.slice(0, 1), ...data.remoteok.slice(1), ...added.remoteok]);++ if (board !== 'remotive') return send(res, 404, { error: 'not a board Work reads' });++ if (state.remotive === 'fail') return send(res, 500, { error: 'the board is having a bad day' }, 'text/plain');++ if (state.remotive === 'html') return send(res, 200, '<html>not json</html>', 'text/html');++ return send(res, 200, { 'job-count': data.remotive.length + added.remotive.length, jobs: [...data.remotive, ...added.remotive] });});await new Promise((r) => server.listen(0, '127.0.0.1', r));const base = `http://127.0.0.1:${server.address().port}`;@@@ -168,10 +92,10 @@ async function startBoards() {fixtures: data,state,requests,-- env: { WORK_ARBEITNOW_URL: `${base}/arbeitnow`, WORK_REMOTIVE_URL: `${base}/remotive`, WORK_REMOTEOK_URL: `${base}/remoteok` },-- /** Make a board start showing one more listing (the shape of that board's own payload). */++ env: { WORK_REMOTIVE_URL: `${base}/remotive` },++ /** Make the board start showing one more listing (the shape of its own payload). */add(board, row) { added[board].push(row); },-- reset() { Object.assign(state, { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' }); requests.length = 0; },++ reset() { Object.assign(state, { remotive: 'ok' }); requests.length = 0; },close: () => new Promise((r) => server.close(r)),};}@@@ -33,7 +33,7 @@ const hn = (identity, title, summary, over = {}) => ({(async () => {const clock = { t: Date.parse('2026-10-30T08:00:00Z') };-- const t = await boot({ now: () => clock.t, env: { WORK_HN_WHOISHIRING_FROM: 'sources' }, sourcesClient });++ const t = await boot({ now: () => clock.t, sourcesClient });const s = t.ctx.s;const from = t.ctx.fromSources;@@@ -1,9 +1,13 @@'use strict';/**-- * The background ingest against the stand-in boards (test/helpers/boards.js): what it keeps from each board and what-- * it refuses, that reading a board twice updates rather than duplicates, that a board which fails is recorded and-- * does not stop the others, that only a short plain-text excerpt is stored, and that a listing no board has shown for-- * the retention window expires out of the search and the pages.++ * The background ingest against Remotive's stand-in API (test/helpers/boards.js): what it keeps, that reading the++ * board twice updates rather than duplicates, that a failure is recorded and does not hide the listings already++ * gathered, that only a short plain-text excerpt is stored, and that a listing the board stopped showing expires out++ * of the search and the pages.++ *++ * Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK never reach this ingest: they come through++ * OpenVibe.Sources (server/ingest/from-sources.js, test/from-sources.test.js and test/remoteok-sources.test.js), so++ * every run lists them as skipped rather than reading them.*/const assert = require('assert');const { boot, check, done } = require('./helpers/boot');@@@ -15,73 +19,59 @@ const DAY = 86_400_000;const clock = { t: Date.parse('2026-10-01T06:00:00Z') };const pinged = [];const indexnow = { enabled: false, pingSoon: (urls) => { pinged.push(...urls); return urls.length; } };-- const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' }, indexnow });++ const t = await boot({ now: () => clock.t, indexnow });const ingest = t.ctx.ingest;const s = t.ctx.s;try {-- await check('reading the three boards keeps every listing with its source, its id and the original URL', async () => {++ await check('the read keeps every Remotive listing, and the boards Sources feeds are skipped', async () => {const out = await ingest.runAll();-- // We Work Remotely is skipped here: it reaches Work through OpenVibe.Sources, never from its own feed.assert.deepStrictEqual(out.map((o) => o.source), ['arbeitnow', 'weworkremotely', 'hn-whoishiring', 'remotive', 'remoteok']);-- assert.ok(out.filter((o) => !['weworkremotely', 'hn-whoishiring'].includes(o.source)).every((o) => o.ok), JSON.stringify(out.map((o) => o.error)));-- assert.strictEqual(out.find((o) => o.source === 'weworkremotely').skipped, 'from_sources');-- assert.strictEqual(out.find((o) => o.source === 'hn-whoishiring').skipped, 'from_sources');-- // Arbeitnow's two pages (2 + 1), Remotive's two jobs, RemoteOK's two listings.++ assert.ok(out.filter((o) => ['arbeitnow', 'weworkremotely', 'hn-whoishiring', 'remoteok'].includes(o.source)).every((o) => o.skipped === 'from_sources'));++ assert.strictEqual(out.find((o) => o.source === 'remotive').ok, true, JSON.stringify(out));const counts = await store.countsBySource(s);-- assert.strictEqual(counts.get('arbeitnow').count, 3);-- assert.strictEqual(counts.get("remotive").count, 3);-- assert.strictEqual(counts.get('remoteok').count, 2);++ assert.strictEqual(counts.get('remotive').count, 3);++ assert.ok(!counts.has('arbeitnow') && !counts.has('remoteok'), 'a board Sources feeds was fetched by Work');-- const page = (await t.get('/api/v1/jobs?q=DevOps')).json();++ const page = (await t.get('/api/v1/jobs?q=Platform')).json();assert.strictEqual(page.jobs.length, 1);const job = page.jobs[0];-- assert.strictEqual(job.source, 'remoteok');-- assert.strictEqual(job.source_name, 'RemoteOK');-- assert.strictEqual(job.url, 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001');-- assert.strictEqual(job.company, 'Skyward Cloud');-- assert.strictEqual(job.salary, '$90,000–$130,000 a year');++ assert.strictEqual(job.source, 'remotive');++ assert.strictEqual(job.source_name, 'Remotive');++ assert.strictEqual(job.url, 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789');++ assert.strictEqual(job.company, 'Meridian Rail');++ assert.strictEqual(job.salary, '€70,000 – €85,000');assert.strictEqual(job.remote, true);-- assert.deepStrictEqual(job.tags, ['devops', 'kubernetes']);-- assert.ok(job.posted_at.startsWith('2026-10-01'));++ assert.deepStrictEqual(job.tags, ['kubernetes', 'terraform', 'Software Development']);++ assert.ok(job.posted_at.startsWith('2026-09-29'));-- // RemoteOK's first array element is a legal notice, not a listing: nothing was made of it.const all = (await t.get('/api/v1/jobs?limit=100')).json();-- assert.strictEqual(all.count, 8);-- assert.ok(!JSON.stringify(all.jobs).includes('personal use'), 'the legal notice became a listing');++ assert.strictEqual(all.count, 3);// Search engines hear about each new listing page and the sitemap (IndexNow), absolute and on this host.const jobUrls = pinged.filter((u) => u !== 'https://openvibe.work/sitemap.xml');-- assert.strictEqual(jobUrls.length, 8, pinged.join(' '));++ assert.strictEqual(jobUrls.length, 3, pinged.join(' '));assert.deepStrictEqual(jobUrls.sort(), all.jobs.map((j) => `https://openvibe.work/jobs/${j.id}`).sort());assert.ok(pinged.includes('https://openvibe.work/sitemap.xml'));});-- await check('every request used this service\'s User-Agent, a JSON Accept, and only the configured hosts', async () => {-- assert.ok(t.boards.requests.length >= 4);++ await check('the one request used this service\'s User-Agent, a JSON Accept, and only the configured host', async () => {++ assert.ok(t.boards.requests.length >= 1);for (const r of t.boards.requests) {assert.strictEqual(r.userAgent, 'OpenVibeWork/0.1 (+https://openvibe.work)', r.path);assert.match(r.accept, /application\/json/);}-- assert.deepStrictEqual([...new Set(t.boards.requests.map((r) => r.path))].sort(), ['/arbeitnow', '/remoteok', '/remotive']);-- // Arbeitnow is paginated as configured (WORK_INGEST_PAGES=2); Remotive is asked for a limit.-- assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=1'));-- assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=2'));++ assert.deepStrictEqual([...new Set(t.boards.requests.map((r) => r.path))], ['/remotive']);assert.ok(t.boards.requests.some((r) => r.path === '/remotive' && r.query === '?limit=100'));});await check('an HTML description becomes a short plain-text excerpt: no tags, no script code, under 320 characters', async () => {-- const job = (await t.get('/api/v1/jobs?q=Frontend')).json().jobs[0];++ const job = (await t.get('/api/v1/jobs?q=Platform')).json().jobs[0];assert.ok(!/[<>]/.test(job.excerpt), job.excerpt);-- assert.ok(!/window\.__pwned/.test(job.excerpt), 'script content was kept');-- assert.ok(job.excerpt.includes('Build the design system'), job.excerpt);++ assert.ok(job.excerpt.includes('Kubernetes clusters'), job.excerpt);assert.ok(job.excerpt.length <= 320, String(job.excerpt.length));-- const page = (await t.get('/api/v1/jobs?q=Senior Backend')).json().jobs[0];-- assert.ok(page.excerpt.includes('PostgreSQL'), page.excerpt);-- assert.ok(page.excerpt.includes('fleet of Node.js services'), page.excerpt);-- // The full description is nowhere in the database: only the excerpt is stored.const dump = await t.dbDump();-- assert.ok(!dump.includes('billing pipeline end to end</p>'), 'the full description was stored');++ assert.ok(!dump.includes('The office dog is called Waffle'), 'the full description was stored');});await check('the text helper strips markup and entities, and cuts a long excerpt on a word', () => {@@@ -106,54 +96,49 @@ const DAY = 86_400_000;assert.ok(!long.includes(' ') && /^\S/.test(long), 'the excerpt is cut mid-word or padded');});-- await check('reading a board again updates the listing in place: no duplicate, same id, newer fetch', async () => {-- const before = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];++ await check('reading the board again updates the listing in place: no duplicate, same id, newer fetch', async () => {++ const before = (await t.get('/api/v1/jobs?q=Platform')).json().jobs[0];const seenBefore = before.fetched_at;clock.t += 2 * 3600_000;// The board's own text changed under the same id, as a board edits a listing.-- t.boards.fixtures.remoteok[1].position = 'DevOps Engineer (Platform)';-- const remoteok = (await ingest.runAll()).find((o) => o.source === 'remoteok');++ t.boards.fixtures.remotive[1].title = 'Platform Engineer (Ops)';++ const remotive = (await ingest.runAll()).find((o) => o.source === 'remotive');pinged.length = 0;-- assert.deepStrictEqual([remoteok.ok, remoteok.listings, remoteok.added, remoteok.updated], [true, 2, 0, 2]);++ assert.deepStrictEqual([remotive.ok, remotive.listings, remotive.added, remotive.updated], [true, 3, 0, 3]);assert.deepStrictEqual(pinged, [], 'a listing read again is not new: no IndexNow ping');-- const after = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];++ const after = (await t.get('/api/v1/jobs?q=Platform')).json().jobs[0];assert.strictEqual(after.id, before.id, 'the id must not change when a board is read again');-- assert.strictEqual(after.title, 'DevOps Engineer (Platform)');++ assert.strictEqual(after.title, 'Platform Engineer (Ops)');assert.ok(after.fetched_at > seenBefore, 'fetched_at did not move');-- assert.strictEqual((await t.get("/api/v1/jobs?limit=100")).json().count, 8);-- t.boards.fixtures.remoteok[1].position = 'DevOps Engineer';++ assert.strictEqual((await t.get('/api/v1/jobs?limit=100')).json().count, 3);++ t.boards.fixtures.remotive[1].title = 'Platform Engineer';});await check('a listing the board stopped showing expires out, leaving the others', async () => {-- // Arbeitnow's page 2 goes quiet: one listing was there, is not any more.-- const gone = t.boards.fixtures.arbeitnowPage2.shift();++ const gone = t.boards.fixtures.remotive.shift();clock.t += 31 * DAY;await ingest.runAll();-- assert.strictEqual((await store.countsBySource(s)).get('arbeitnow').count, 2);-- assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 0);-- const row = await s.db.maybe("SELECT * FROM work_listings WHERE source_id = $1 AND expired", [gone.slug]);++ assert.strictEqual((await store.countsBySource(s)).get('remotive').count, 2);++ assert.strictEqual((await t.get('/api/v1/jobs?q=Support')).json().count, 0);++ const row = await s.db.maybe('SELECT * FROM work_listings WHERE source_id = $1 AND expired', [String(gone.id)]);assert.ok(row, 'the expired listing was deleted rather than kept');assert.strictEqual((await t.get(`/jobs/${row.id}`)).status, 404);// And it comes back if the board shows it again.-- t.boards.fixtures.arbeitnowPage2.push(gone);++ t.boards.fixtures.remotive.push(gone);await ingest.runAll();-- assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 1);++ assert.strictEqual((await t.get('/api/v1/jobs?q=Support')).json().count, 1);});-- await check('a board that fails is recorded, says why, and does not stop the others', async () => {++ await check('a failed read is recorded, says why, and never empties the board', async () => {clock.t += 3600_000;t.boards.state.remotive = 'fail';-- t.boards.state.remoteok = 'html';const out = await ingest.runAll();assert.strictEqual(out.find((o) => o.source === 'remotive').ok, false);-- assert.strictEqual(out.find((o) => o.source === 'arbeitnow').ok, true, 'one board failing stopped another');const src = (await t.get('/api/v1/sources')).json();const remotive = src.sources.find((x) => x.id === 'remotive');assert.strictEqual(remotive.last_fetch.ok, false);assert.match(remotive.last_fetch.error, /500/);assert.ok(remotive.last_fetch.finished_at, 'the failure was not timestamped');-- assert.match(src.sources.find((x) => x.id === 'remoteok').last_fetch.error, /not JSON/);-- // What was already gathered stays: a failed read never empties the board.assert.strictEqual(remotive.listings, 3);t.boards.reset();});@@@ -71,21 +71,22 @@ const jobTypes = require('../server/jobs/job-types');assert.strictEqual(jobTypes.canonicalKey(''), null);});-- const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });++ const t = await boot();try {await t.ctx.ingest.runAll();// Two listings in the boards' own words: a German one and a French one.-- t.boards.add('arbeitnow', {-- slug: 'support-werkstudent-berlin',-- company_name: 'Nordlicht Systems',++ t.boards.add('remotive', {++ id: 1900950,++ url: 'https://remotive.com/remote-jobs/support/support-werkstudent-1900950',title: 'Support Werkstudent',-- description: '<p>Support in Berlin.</p>',-- remote: false,-- url: 'https://www.arbeitnow.com/view/support-werkstudent-berlin-45678',++ company_name: 'Nordlicht Systems',++ category: 'Customer Support',tags: ['support'],-- job_types: ['Werkstudent', 'hilfstätigkeit'],-- location: 'Berlin, Germany',-- created_at: 1791800000,++ job_type: 'Werkstudent',++ publication_date: '2026-10-02T09:00:00',++ candidate_required_location: 'Berlin, Germany',++ salary: '',++ description: '<p>Support in Berlin.</p>',});t.boards.add('remotive', {id: 1900900, url: 'https://remotive.com/remote-jobs/other/consultant-cdi-1900900',@@@ -124,7 +125,7 @@ const jobTypes = require('../server/jobs/job-types');assert.ok(partTime.jobs.some((j) => /Support Werkstudent/.test(j.title)), 'the German listing is not part time');// And it is counted under both.const fullTime = (await t.get('/api/v1/jobs?type=full-time')).json();-- assert.ok(fullTime.jobs.some((j) => /Frontend Engineer/.test(j.title)));++ assert.ok(fullTime.jobs.some((j) => /Platform Engineer/.test(j.title)));});await check('a type in a board\'s own words still works, as the plain search it always was', async () => {@@@ -13,20 +13,28 @@ const ldBlocks = (html) => [...html.matchAll(/<script type="application\/ld\+jsoconst first = async (t, path) => (await t.get(path)).json();(async () => {-- const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });++ const t = await boot();try {++ // A listing with a city and a hostile title, so the home's city list and the escaping are exercised.++ t.boards.add('remotive', {++ id: 1900970, url: 'https://remotive.com/remote-jobs/software-dev/frontend-engineer-1900970',++ title: 'Frontend Engineer <script>alert("xss")</script>', company_name: 'Helles Werk',++ category: 'Software Development', tags: ['typescript', 'css'], job_type: 'permanent, full_time',++ publication_date: '2026-10-03T09:00:00', candidate_required_location: 'Berlin, Germany', salary: '',++ description: '<div>Build the design system.<script>window.__pwned = true</script></div><div>TypeScript, CSS.</div>',++ });await t.ctx.ingest.runAll();await check('search: text matches the title, the company and the tags; results are newest first', async () => {-- const byTitle = await first(t, '/api/v1/jobs?q=devops');-- assert.deepStrictEqual(byTitle.jobs.map((j) => j.title), ['DevOps Engineer']);-- const byCompany = await first(t, '/api/v1/jobs?q=Skyward');-- assert.deepStrictEqual(byCompany.jobs.map((j) => j.company), ['Skyward Cloud']);-- // The term is found in a tag and in the excerpt, so both listings answer.++ const byTitle = await first(t, '/api/v1/jobs?q=platform');++ assert.deepStrictEqual(byTitle.jobs.map((j) => j.title), ['Platform Engineer']);++ const byCompany = await first(t, '/api/v1/jobs?q=Meridian');++ assert.deepStrictEqual(byCompany.jobs.map((j) => j.company), ['Meridian Rail']);++ // The term is found in a tag and in the excerpt.const byTag = await first(t, '/api/v1/jobs?q=terraform');-- assert.deepStrictEqual(byTag.jobs.map((j) => j.company).sort(), ['Meridian Rail', 'Skyward Cloud']);++ assert.deepStrictEqual(byTag.jobs.map((j) => j.company), ['Meridian Rail']);// Case does not matter, and a % a caller typed is a character, not a wildcard.-- assert.strictEqual((await first(t, '/api/v1/jobs?q=DEVOPS')).count, 1);++ assert.strictEqual((await first(t, '/api/v1/jobs?q=PLATFORM')).count, 1);assert.strictEqual((await first(t, '/api/v1/jobs?q=%25')).count, 0);const all = await first(t, '/api/v1/jobs?limit=100');const dates = all.jobs.map((j) => j.posted_at);@@@ -37,18 +45,19 @@ const first = async (t, path) => (await t.get(path)).json();const remote = await first(t, '/api/v1/jobs?remote=true');assert.ok(remote.count > 0);assert.ok(remote.jobs.every((j) => j.remote === true));++ // Remotive's board carries remote work only, so nothing is on-site.const onSite = await first(t, '/api/v1/jobs?remote=false');-- assert.ok(onSite.count > 0);-- assert.ok(onSite.jobs.every((j) => j.remote === false));-- assert.strictEqual(remote.count + onSite.count, (await first(t, '/api/v1/jobs?limit=100')).count);++ assert.strictEqual(onSite.count, 0);++ const everything = (await first(t, '/api/v1/jobs?limit=100')).count;++ assert.strictEqual(remote.count, everything);const contract = await first(t, '/api/v1/jobs?type=contract');assert.deepStrictEqual(contract.jobs.map((j) => j.title), ['Content Marketer']);assert.strictEqual(contract.facets.matches, 1, 'the count must be what the filters match');-- assert.strictEqual(contract.facets.scope, 8, 'the type facet must not narrow its own scope');++ assert.strictEqual(contract.facets.scope, everything, 'the type facet must not narrow its own scope');const germany = await first(t, '/api/v1/jobs?location=Germany');-- assert.deepStrictEqual(germany.jobs.map((j) => j.title).sort(), ['Data Analyst', 'Senior Backend Engineer (Node.js)']);++ assert.deepStrictEqual(germany.jobs.map((j) => j.title), ['Frontend Engineer <script>alert("xss")</script>']);const combined = await first(t, '/api/v1/jobs?q=engineer&remote=true');assert.ok(combined.count >= 1 && combined.jobs.every((j) => /engineer/i.test(j.title)));@@@ -77,9 +86,9 @@ const first = async (t, path) => (await t.get(path)).json();const html = (await t.get('/jobs')).text;assert.strictEqual((await t.get('/jobs')).status, 200);for (const [name, url] of [-- ['Arbeitnow', 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345'],++ ['Remotive', 'https://remotive.com/remote-jobs/software-dev/support-engineer-1900123'],['Remotive', 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789'],-- ['RemoteOK', 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001'],++ ['Remotive', 'https://remotive.com/remote-jobs/marketing/content-marketer-1900456'],]) {assert.ok(html.includes(name), `the source ${name} is not named on the list`);assert.ok(html.includes(`href="${url}"`), `the original listing ${url} is not linked on the list`);@@@ -92,28 +101,28 @@ const first = async (t, path) => (await t.get(path)).json();});await check('a search is an Atom feed with the same filters, and its page offers alerts through OpenVibe.Watch', async () => {-- const r = await t.get('/jobs.atom?q=devops&remote=true');++ const r = await t.get('/jobs.atom?q=platform&remote=true');assert.strictEqual(r.status, 200);assert.match(r.headers.get('content-type'), /^application\/atom\+xml/);assert.strictEqual(r.headers.get('cache-control'), 'public, max-age=300');assert.match(r.text, /^<\?xml version="1\.0" encoding="utf-8"\?>\n<feed xmlns="http:\/\/www\.w3\.org\/2005\/Atom">/);const entries = [...r.text.matchAll(/<entry>([\s\S]*?)<\/entry>/g)].map((m) => m[1]);assert.strictEqual(entries.length, 1, 'only the matching listing');-- assert.match(entries[0], /<title>DevOps Engineer at Skyward Cloud<\/title>/);++ assert.match(entries[0], /<title>Platform Engineer at Meridian Rail<\/title>/);assert.match(entries[0], /<link rel="alternate" type="text\/html" href="[^"]+\/jobs\/[^"]+"\/>/);-- assert.match(entries[0], /<author><name>RemoteOK<\/name><\/author>/);-- assert.match(r.text, /<link rel="self" href="[^"]*\/jobs\.atom\?q=devops&remote=true"\/>/);++ assert.match(entries[0], /<author><name>Remotive<\/name><\/author>/);++ assert.match(r.text, /<link rel="self" href="[^"]*\/jobs\.atom\?q=platform&remote=true"\/>/);const all = await t.get('/jobs.atom');const ids = [...all.text.matchAll(/<updated>([^<]+)<\/updated><author>/g)].map((m) => m[1]);assert.ok(ids.length >= 3 && ids.join() === [...ids].sort().reverse().join(), 'every listing, newest first');// The page links its feed, and Get alerts opens Watch's feed form with it filled in.-- const page = (await t.get('/jobs?q=devops&remote=true')).text;-- assert.ok(page.includes('<link rel="alternate" type="application/atom+xml" title="Jobs “devops” remote" href="/jobs.atom?q=devops&remote=true">'), 'the feed link in the head');++ const page = (await t.get('/jobs?q=platform&remote=true')).text;++ assert.ok(page.includes('<link rel="alternate" type="application/atom+xml" title="Jobs “platform” remote" href="/jobs.atom?q=platform&remote=true">'), 'the feed link in the head');const alert = /href="(https:\/\/openvibe\.watch\/watches\/new\?[^"]+)">Get alerts for this search/.exec(page);assert.ok(alert, 'the alerts link');const u = new URL(alert[1].replace(/&/g, '&'));assert.deepStrictEqual([u.searchParams.get('template'), new URL(u.searchParams.get('url')).pathname + new URL(u.searchParams.get('url')).search, u.searchParams.get('name')],-- ['feed', '/jobs.atom?q=devops&remote=true', 'Jobs: “devops” remote']);++ ['feed', '/jobs.atom?q=platform&remote=true', 'Jobs: “platform” remote']);});await check('one listing: the board\'s fields, the excerpt, the source and "read the full listing on" link', async () => {@@@ -138,19 +147,19 @@ const first = async (t, path) => (await t.get(path)).json();});await check('the listing page carries a valid JobPosting whose url is the original, not this page', async () => {-- const job = (await first(t, '/api/v1/jobs?q=DevOps')).jobs[0];++ const job = (await first(t, '/api/v1/jobs?q=Platform Engineer')).jobs[0];const html = (await t.get(`/jobs/${job.id}`)).text;assert.strictEqual((html.match(/application\/ld\+json/g) || []).length, 1, 'one JSON-LD script per page');const blocks = ldBlocks(html);const posting = blocks.find((b) => b['@type'] === 'JobPosting');assert.ok(posting, `no JobPosting node: ${JSON.stringify(blocks.map((b) => b['@type']))}`);-- assert.strictEqual(posting.title, 'DevOps Engineer');++ assert.strictEqual(posting.title, 'Platform Engineer');assert.strictEqual(posting.url, job.url);assert.notStrictEqual(posting.url, `https://openvibe.work/jobs/${job.id}`);-- assert.strictEqual(posting.hiringOrganization.name, 'Skyward Cloud');++ assert.strictEqual(posting.hiringOrganization.name, 'Meridian Rail');assert.strictEqual(posting.jobLocationType, 'TELECOMMUTE');-- assert.match(posting.datePosted, /^2026-10-01T/);-- assert.strictEqual(posting.identifier.name, 'remoteok');++ assert.match(posting.datePosted, /^2026-09-29T/);++ assert.strictEqual(posting.identifier.name, 'remotive');assert.ok(posting.description.length <= 320);assert.ok(posting['@context'] === 'https://schema.org');// The search page lists many listings but is not one job, so it carries no JobPosting.@@@ -188,7 +197,7 @@ const first = async (t, path) => (await t.get(path)).json();const api = await first(t, '/api/v1/sources');assert.deepStrictEqual(api.sources.map((s) => s.id).sort(), ['arbeitnow', 'hn-whoishiring', 'remoteok', 'remotive', 'weworkremotely']);// A board OpenVibe.Sources feeds has no read of its own here, so it is not asked for one.-- for (const src of api.sources.filter((x) => !['weworkremotely', 'hn-whoishiring'].includes(x.id))) {++ for (const src of api.sources.filter((x) => x.id === 'remotive')) {assert.ok(src.terms.length > 40, `${src.id} has no terms`);assert.match(src.terms, /link|attribute|mention|name/i, `${src.id}'s terms do not state the attribution rule`);assert.strictEqual(src.last_fetch.ok, true);@@@ -228,7 +237,7 @@ const first = async (t, path) => (await t.get(path)).json();assert.ok(r.text.includes('Find work.'), 'the hero');assert.match(r.text, /action="\/jobs"/, 'the search box');assert.ok(r.text.includes('Latest remote jobs'));-- assert.ok(r.text.includes('DevOps Engineer'), 'no listing on the home page');++ assert.ok(r.text.includes('Platform Engineer'), 'no listing on the home page');for (const [name, url] of [['Arbeitnow', 'https://www.arbeitnow.com/'], ['Remotive', 'https://remotive.com/'], ['RemoteOK', 'https://remoteok.com/']]) {assert.ok(r.text.includes(`href="${url}"`), `the source ${name} is not linked on the home page`);}@@@ -250,7 +259,7 @@ const first = async (t, path) => (await t.get(path)).json();});await check('search is limited per caller, and past it the API says so with Retry-After', async () => {-- const busy = await boot({ callerLimits: true, limitsNow: () => Date.now(), env: { WORK_INGEST_PAGES: '1' } });++ const busy = await boot({ callerLimits: true, limitsNow: () => Date.now() });try {await busy.ctx.ingest.runAll();let limited = null;@@@ -32,10 +32,17 @@ const cardOf = (html, id) => {};(async () => {-- const t = await boot({ env: { WORK_INGEST_PAGES: '2', WORK_DISCLOSURE_SAMPLE: '100' } });++ const t = await boot({ env: { WORK_DISCLOSURE_SAMPLE: '100' } });const kim = t.network.addUser('kim');const SAME = { 'sec-fetch-site': 'same-origin' };try {++ // A second listing with two types in the board's own words, so both kinds are on the page.++ t.boards.add('remotive', {++ id: 1900960, url: 'https://remotive.com/remote-jobs/software-dev/frontend-engineer-1900960',++ title: 'Frontend Engineer', company_name: 'Helles Werk', category: 'Software Development', tags: ['typescript', 'css'],++ job_type: 'permanent, full_time', publication_date: '2026-10-01T10:00:00',++ candidate_required_location: 'Remote (EU)', salary: '', description: '<p>Build the design system.</p>',++ });await t.ctx.ingest.runAll();await check('outcome 1: the salary as the board wrote it, or that the board wrote none', async () => {@@@ -77,7 +84,7 @@ const cardOf = (html, id) => {assert.ok(page.includes(`Of the last ${expected.n} listings we fetched, ${expected.salaried} included a salary (${pct}%)`),'the disclosure rate is not the one the database gives');// The sample size is configuration, not a literal on the page: with a smaller sample the line follows.-- const small = await boot({ env: { WORK_INGEST_PAGES: '2', WORK_DISCLOSURE_SAMPLE: '3' } });++ const small = await boot({ env: { WORK_DISCLOSURE_SAMPLE: '3' } });try {await small.ctx.ingest.runAll();const want = await small.ctx.s.db.one(nnew file mode 100644@@@ -0,0 +1,77 @@++'use strict';++/**++ * RemoteOK through OpenVibe.Sources (server/ingest/from-sources.js; plan 2026-10-22, track F): the board's API is++ * read by Sources (source remoteok-job-board) under its robots and terms, and Work follows the change feed. A listing++ * must become the row Work's own RemoteOK fetch made: the same id (the board's own id, so the same listing URL), the++ * position as its title, the company and place, tags, and the salary bounds written the way the board's fetch wrote++ * them. A job Sources retires expires here; Work never calls remoteok.com itself.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const store = require('../server/jobs/store');++++const feed = [];++let seq = 0;++function push(item) { feed.push({ ...item, change_seq: ++seq }); }++const sourcesClient = {++ enabled: true,++ async listItems({ after = 0, limit = 100 } = {}) {++ const page = feed.filter((i) => i.change_seq > after).slice(0, limit);++ return { items: page, next_after: page.length ? page[page.length - 1].change_seq : after, more: false };++ },++};++const job = (identity, over = {}) => ({++ id: `itm_remoteok_${identity}`, source_key: 'remoteok-job-board', category: 'jobs', kind: 'job', identity,++ canonical_url: `https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-${identity}`,++ title: 'DevOps Engineer', summary: 'Kubernetes, Terraform, on-call.',++ authors: [], published_at: '2026-11-10T08:00:00.000Z', source_updated_at: null,++ fields: {++ company: 'Skyward Cloud', location: 'Worldwide', job_type: null,++ salary_min: 90000, salary_max: 130000, tags: ['devops', 'kubernetes'],++ },++ revision: 1, provenance: { retrieved_at: '2026-11-12T06:00:00.000Z' }, removed: null, ...over,++});++++(async () => {++ const clock = { t: Date.parse('2026-11-12T06:00:00Z') };++ const t = await boot({ now: () => clock.t, sourcesClient });++ const s = t.ctx.s;++ const from = t.ctx.fromSources;++++ try {++ await check('RemoteOK is one of the boards followed, not one Work fetches', async () => {++ assert.ok(from.sourceIds.includes('remoteok'), from.sourceIds.join(', '));++ const out = await t.ctx.ingest.runAll();++ assert.deepStrictEqual(out.find((o) => o.source === 'remoteok'), { source: 'remoteok', skipped: 'from_sources' });++ });++++ await check('a Sources item becomes the listing RemoteOK\'s own fetch made', async () => {++ push(job('1049001'));++ const out = await from.pull();++ assert.deepStrictEqual([out.applied, out.hold, out.removed], [1, 0, 0]);++++ const id = store.idOf('remoteok', '1049001');++ const row = await s.db.maybe('SELECT * FROM work_listings WHERE id = $1', [id]);++ assert.deepStrictEqual(++ [row.source, row.source_id, row.title, row.company, row.location, row.remote, row.tags, row.job_type, row.salary, row.expired],++ ['remoteok', '1049001', 'DevOps Engineer', 'Skyward Cloud', 'Worldwide', true, 'devops, kubernetes', null, '$90,000–$130,000 a year', false]);++ assert.strictEqual(row.url, 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001');++ assert.strictEqual(row.excerpt, 'Kubernetes, Terraform, on-call.');++ const page = (await t.get('/api/v1/jobs?q=DevOps')).json();++ assert.deepStrictEqual(page.jobs.map((j) => [j.id, j.source, j.salary]), [[id, 'remoteok', '$90,000–$130,000 a year']]);++ assert.strictEqual((await store.sourceFetches(s)).get('remoteok').listings, 1);++ });++++ await check('a RemoteOK job Sources retires expires here', async () => {++ push(job('1049001', { revision: 2, removed: { at: '2026-11-20T06:00:00.000Z', reason: 'gone_from_source' } }));++ const out = await from.pull();++ assert.deepStrictEqual([out.applied, out.removed], [0, 1]);++ const row = await s.db.maybe('SELECT expired FROM work_listings WHERE source_id = $1', ['1049001']);++ assert.strictEqual(row.expired, true);++ assert.strictEqual((await t.get('/api/v1/jobs?q=DevOps')).json().jobs.length, 0);++ });++ } finally {++ await t.close();++ }++ done();++})().catch((err) => { console.error(err); process.exit(1); });@@@ -13,7 +13,7 @@ const { boot, check, done } = require('./helpers/boot');// The clock starts at the real now: the session is a Network token, and this service checks it against its own// clock, so a test that moves time must move only a little or the token stops verifying.const clock = { t: Date.now() };-- const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });++ const t = await boot({ now: () => clock.t });const kim = t.network.addUser('kim');const lee = t.network.addUser('lee');const SAME = { 'sec-fetch-site': 'same-origin' };@@@ -43,14 +43,14 @@ const { boot, check, done } = require('./helpers/boot');const empty = await t.get('/api/v1/saved-searches', { as: kim, json: { q: ' ' }, headers: SAME });assert.strictEqual(empty.status, 422);assert.strictEqual(empty.json().code, 'work.saved_searches.empty');-- const remote = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'devops', remote: 'yes', location: 'Worldwide' }, headers: SAME });++ const remote = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'platform', remote: 'yes' }, headers: SAME });assert.strictEqual(remote.status, 201, remote.text);const saved = remote.json();-- assert.deepStrictEqual(saved.filters, { remote: 'true', type: '', location: 'worldwide' });-- assert.strictEqual(saved.search_path, '/jobs?q=devops&remote=true&location=worldwide');++ assert.deepStrictEqual(saved.filters, { remote: 'true', type: '', location: '' });++ assert.strictEqual(saved.search_path, '/jobs?q=platform&remote=true');// Running it again gives the same listings the search would.const run = (await t.get(saved.search_path)).text;-- assert.ok(run.includes('DevOps Engineer'));++ assert.ok(run.includes('Platform Engineer'));});await check('a signed-in write from another site is refused; a Bearer token is not (it is that person\'s own)', async () => {@@@ -14,7 +14,7 @@ const DAY = 86_400_000;(async () => {const clock = { t: Date.parse('2026-10-01T06:00:00Z') };-- const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });++ const t = await boot({ now: () => clock.t });const { s, search, ingest } = t.ctx;const outbox = async () => (await s.db.many('SELECT envelope FROM event_outbox ORDER BY id')).map((r) => r.envelope);const valid = (env) => {@@@ -38,36 +38,35 @@ const DAY = 86_400_000;await search.sweep(); // the run's own sweep is fire-and-forget; this one waits (and shares or repeats it)const envs = (await outbox()).map(valid);const listings = await s.db.many('SELECT * FROM work_listings ORDER BY id');-- assert.strictEqual(listings.length, 8);-- assert.strictEqual(envs.length, 8, 'one document per listing, sent once');++ assert.strictEqual(listings.length, 3);++ assert.strictEqual(envs.length, 3, 'one document per listing, sent once');assert.ok(envs.every((e) => e.event_type === 'work.index_document.upserted'));const byId = new Map(envs.map((e) => [e.payload.id, e.payload]));-- const devops = listings.find((r) => r.title.includes('DevOps'));-- const doc = byId.get(devops.id);-- assert.strictEqual(doc.canonical_url, `https://openvibe.work/jobs/${devops.id}`);-- assert.strictEqual(doc.title, `${devops.title} at ${devops.company}`);++ const platform = listings.find((r) => r.title.includes('Platform'));++ const doc = byId.get(platform.id);++ assert.strictEqual(doc.canonical_url, `https://openvibe.work/jobs/${platform.id}`);++ assert.strictEqual(doc.title, `${platform.title} at ${platform.company}`);assert.strictEqual(doc.authorship, 'imported');assert.strictEqual(doc.visibility, 'public');assert.deepStrictEqual(doc.indexability, { decision: 'index', reasons: [] });-- assert.strictEqual(doc.facets.source, 'remoteok');++ assert.strictEqual(doc.facets.source, 'remotive');assert.strictEqual(doc.facets.remote, true);-- assert.match(doc.title, /Skyward Cloud/);++ assert.match(doc.title, /Meridian Rail/);// The snippet reads the summary and body together: the body leaves out the title, company, place and pay.-- assert.ok(!doc.body.includes(devops.title) && !doc.body.includes('Skyward Cloud') && !doc.body.includes('$90,000'), `no repeat: ${doc.body.slice(0, 120)}`);++ assert.ok(!doc.body.includes(platform.title) && !doc.body.includes('Meridian Rail') && !doc.body.includes('€70,000'), `no repeat: ${doc.body.slice(0, 120)}`);const { meta } = require('../server/render/listing');-- assert.strictEqual(meta({ ...devops, location: 'Berlin, Berlin, Deutschland' }).location, 'Berlin, Deutschland', 'a repeated place part is said once');-- assert.match(doc.summary, /\$90,000–\$130,000 a year/);++ assert.strictEqual(meta({ ...platform, location: 'Berlin, Berlin, Deutschland' }).location, 'Berlin, Deutschland', 'a repeated place part is said once');++ assert.match(doc.summary, /€70,000 – €85,000/);assert.deepStrictEqual(doc.provenance.map((p) => [p.service, p.type, p.label, p.url]),-- [['work', 'board_listing', 'RemoteOK', devops.url]], 'the board and the original listing travel with it');-- assert.ok(!JSON.stringify(envs).includes('personal use'), 'RemoteOK\'s legal notice is not a document');++ [['work', 'board_listing', 'Remotive', platform.url]], 'the board and the original listing travel with it');});await check('a second sweep sends nothing; a changed listing is sent again at the next revision', async () => {const before = (await outbox()).length;const again = await search.sweep();-- assert.deepStrictEqual(again.job, { seen: 8, sent: 0, removed: 0, failed: 0 });++ assert.deepStrictEqual(again.job, { seen: 3, sent: 0, removed: 0, failed: 0 });assert.strictEqual((await outbox()).length, before);-- const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'arbeitnow' ORDER BY id LIMIT 1");++ const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' ORDER BY id LIMIT 1");await s.db.exec('UPDATE work_listings SET salary = $1 WHERE id = $2', ['€70,000 a year', one.id]);const res = await search.sweep();assert.strictEqual(res.job.sent, 1);@@@ -78,7 +77,7 @@ const DAY = 86_400_000;});await check('a listing that expires (its page answers 404) becomes a tombstone, once', async () => {-- const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' ORDER BY id LIMIT 1");++ const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' ORDER BY id DESC LIMIT 1");await s.db.exec('UPDATE work_listings SET expired = true WHERE id = $1', [one.id]);assert.strictEqual((await t.get(`/jobs/${one.id}`)).status, 404);const res = await search.sweep();@@@ -88,7 +87,7 @@ const DAY = 86_400_000;assert.deepStrictEqual(last.payload, { type: 'job', id: one.id, revision: 2 });assert.strictEqual((await search.sweep()).job.sent, 0, 'a tombstone is not re-sent');// A row deleted outright is tombstoned too (exists()).-- const gone = await s.db.one("SELECT * FROM work_listings WHERE source = 'arbeitnow' AND expired = false ORDER BY id DESC LIMIT 1");++ const gone = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' AND expired = false ORDER BY id DESC LIMIT 1");await s.db.exec('DELETE FROM work_listings WHERE id = $1', [gone.id]);assert.strictEqual((await search.sweep()).job.removed, 1);});@@@ -36,7 +36,7 @@ const wwr = (identity, over = {}) => ({(async () => {const clock = { t: Date.parse('2026-10-30T08:00:00Z') };-- const t = await boot({ now: () => clock.t, env: { WORK_WWR_FROM: 'sources' }, sourcesClient });++ const t = await boot({ now: () => clock.t, sourcesClient });const s = t.ctx.s;const from = t.ctx.fromSources;
@@ -58,19 +58,16 @@ WORK_EVENTS_URL=
@@ -58,19 +58,16 @@ WORK_EVENTS_URL=
#WORK_INGEST=on
#WORK_INGEST=on
#WORK_INGEST_INTERVAL_HOURS=6
#WORK_INGEST_INTERVAL_HOURS=6
#WORK_INGEST_TIMEOUT_MS=15000
#WORK_INGEST_TIMEOUT_MS=15000
-#WORK_INGEST_PAGES=3
#WORK_INGEST_LIMIT=100
#WORK_INGEST_LIMIT=100
#WORK_RETENTION_DAYS=30
#WORK_RETENTION_DAYS=30
-# The boards' public APIs. Point these at a stand-in server to test the ingest; nothing else in the service fetches
-# anything, and never a URL a caller typed.
-#WORK_ARBEITNOW_URL=https://www.arbeitnow.com/api/job-board-api
-# Arbeitnow through OpenVibe.Sources (server/ingest/from-sources.js): with WORK_ARBEITNOW_FROM=sources Work stops reading
-# the board and follows Sources' arbeitnow-job-board items instead (pages 1-3; a job the board stops showing is retired
-# after a week). Needs OV_OAUTH_CLIENT_SECRET (Network grants Work sources.item.read); Sources is on 127.0.0.1:4720.
-#WORK_ARBEITNOW_FROM=sources
-#OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720
+# Remotive is the one board Work still fetches itself (its API terms grant a named, linked use; its robots.txt
+# disallows /api/* so OpenVibe.Sources may not fetch it). Point this at a stand-in server to test the ingest;
+# nothing else in the service fetches a URL, and never one a caller typed.
#WORK_REMOTIVE_URL=https://remotive.com/api/remote-jobs
#WORK_REMOTIVE_URL=https://remotive.com/api/remote-jobs
-#WORK_REMOTEOK_URL=https://remoteok.com/api
+# Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK come through OpenVibe.Sources
+# (server/ingest/from-sources.js). Needs OV_OAUTH_CLIENT_SECRET (Network grants Work sources.item.read);
+# Sources is on 127.0.0.1:4720.
+#OV_SOURCES_INTERNAL_URL=http://127.0.0.1:4720
# USAJOBS (D11 owner settings): the admin panel's work.data_sources namespace wins; these are the
# USAJOBS (D11 owner settings): the admin panel's work.data_sources namespace wins; these are the
# fallback when no value was entered there. The email must match the key's account.
# fallback when no value was entered there. The email must match the key's account.
#USAJOBS_API_KEY=
#USAJOBS_API_KEY=
@@ -15,9 +15,10 @@ board, where the reader applies.
@@ -15,9 +15,10 @@ board, where the reader applies.
- **Ingest:** `server/ingest/` (`index.js`, `from-sources.js`) and `server/jobs/` (`sources.js`,
- **Ingest:** `server/ingest/` (`index.js`, `from-sources.js`) and `server/jobs/` (`sources.js`,
`store.js`, `text.js`, `hiring-line.js`, `job-types.js`, `places.js`, `companies.js`,
`store.js`, `text.js`, `hiring-line.js`, `job-types.js`, `places.js`, `companies.js`,
`rederive-hn-titles.js`); run it with `npm run ingest` or the in-process interval.
`rederive-hn-titles.js`); run it with `npm run ingest` or the in-process interval.
-- **Sources:** Arbeitnow, Remotive, RemoteOK and Hacker News "Who is hiring", each through its own
- public API and each with its terms applied on every page (`/sources`, `GET /api/v1/sources`);
- Arbeitnow can instead be read from OpenVibe.Sources (`WORK_ARBEITNOW_FROM=sources`).
+- **Sources:** Arbeitnow, We Work Remotely, RemoteOK and Hacker News "Who is hiring" arrive through
+ OpenVibe.Sources (`server/ingest/from-sources.js`); Remotive is the one board Work still fetches
+ itself (its robots.txt disallows `/api/*`, its API terms grant the named, linked use). Every
+ board's terms are applied on every page (`/sources`, `GET /api/v1/sources`).
- **Docs/config:** README "Data sources and their terms", `SECURITY.md`, `.env.example`.
- **Docs/config:** README "Data sources and their terms", `SECURITY.md`, `.env.example`.
## Owns, does not own
## Owns, does not own
@@ -74,7 +74,7 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by
@@ -74,7 +74,7 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by
the canonical subject. Work holds only `user:usr_…` on its own rows.
the canonical subject. Work holds only `user:usr_…` on its own rows.
- **Event delivery** — OpenVibe.Events: Work receives `network.account.export_requested` and `network.account.deleted`
- **Event delivery** — OpenVibe.Events: Work receives `network.account.export_requested` and `network.account.deleted`
and creates its two subscriptions at boot; it does not own the topics or the delivery.
and creates its two subscriptions at boot; it does not own the topics or the delivery.
-- **The listings themselves** — the boards (Arbeitnow, Remotive, RemoteOK and Hacker News "Who is hiring"): Work keeps a short excerpt and a link back,
+- **The listings themselves** — the boards (Arbeitnow, We Work Remotely, RemoteOK, Hacker News "Who is hiring" and Remotive): Work keeps a short excerpt and a link back,
never the full description, and reads each board only on the terms it publishes. A board that writes a listing as
never the full description, and reads each board only on the terms it publishes. A board that writes a listing as
one pipe-separated line (Hacker News) has that line split into company, role(s), place, terms and pay
one pipe-separated line (Hacker News) has that line split into company, role(s), place, terms and pay
(`server/jobs/hiring-line.js`); the line itself stays at the front of the excerpt.
(`server/jobs/hiring-line.js`); the line itself stays at the front of the excerpt.
@@ -90,9 +90,9 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by
@@ -90,9 +90,9 @@ Its own PostgreSQL tables, created by [migrations/](migrations/) and written by
- **OpenVibe.Actor** — the person's agent: Work posts `POST /api/v1/tasks` with its own service token (capability
- **OpenVibe.Actor** — the person's agent: Work posts `POST /api/v1/tasks` with its own service token (capability
`actor.task.create`, audience `openvibe.actor`) when a person asks for help preparing an application
`actor.task.create`, audience `openvibe.actor`) when a person asks for help preparing an application
(`WORK_ACTOR_URL`, `WORK_ACTOR_PUBLIC_URL`, `WORK_ACTOR_TIMEOUT_MS`).
(`WORK_ACTOR_URL`, `WORK_ACTOR_PUBLIC_URL`, `WORK_ACTOR_TIMEOUT_MS`).
-- **The job boards** — Arbeitnow, Remotive and RemoteOK are the only hosts Work itself fetches (`WORK_ARBEITNOW_URL`,
- `WORK_REMOTIVE_URL`, `WORK_REMOTEOK_URL`, `WORK_USER_AGENT`). We Work Remotely and Hacker News "Who is hiring" arrive
- through OpenVibe.Sources (`WORK_WWR_FROM=sources`, `WORK_HN_WHOISHIRING_FROM=sources`).
+- **The job boards** — Remotive is the only host Work itself fetches (`WORK_REMOTIVE_URL`, `WORK_USER_AGENT`), because
+ its API terms grant a named, linked use while its robots.txt disallows `/api/*`. Arbeitnow, RemoteOK, We Work Remotely
+ and Hacker News "Who is hiring" arrive through OpenVibe.Sources (`server/ingest/from-sources.js`).
- **PostgreSQL** (`DATABASE_URL`, `DATABASE_DIRECT_URL`; an embedded PGlite database in development via
- **PostgreSQL** (`DATABASE_URL`, `DATABASE_DIRECT_URL`; an embedded PGlite database in development via
`WORK_PGLITE_DIR`) and **Valkey** for shared limit counters (`VALKEY_URL`, `VALKEY_PREFIX`).
`WORK_PGLITE_DIR`) and **Valkey** for shared limit counters (`VALKEY_URL`, `VALKEY_PREFIX`).
- **Packages**: `openvibe-contracts` v0.289.1, `openvibe-sdk` v0.150.0 (`db`, `auth`, `account-data`, `limits`,
- **Packages**: `openvibe-contracts` v0.289.1, `openvibe-sdk` v0.150.0 (`db`, `auth`, `account-data`, `limits`,
@@ -151,9 +151,9 @@ Each board is read through its own public API, and its terms are applied on ever
@@ -151,9 +151,9 @@ Each board is read through its own public API, and its terms are applied on ever
| Source | API | Terms, as applied here |
| Source | API | Terms, as applied here |
|---|---|---|
|---|---|---|
-| **Arbeitnow** | `GET https://www.arbeitnow.com/api/job-board-api?page=N` | Free to use. Arbeitnow is named on every listing and the listing links to its page on arbeitnow.com. Its first pages (configurable) are read. With `WORK_ARBEITNOW_FROM=sources` Work reads none: it follows OpenVibe.Sources' `arbeitnow-job-board` items (`server/ingest/from-sources.js`), and a job the board stops showing is retired by Sources after a week. |
-| **Remotive** | `GET https://remotive.com/api/remote-jobs?limit=100` | Remotive must be named as the source and the listing linked. The API is called about four times a day — the six-hour interval is exactly that. |
-| **RemoteOK** | `GET https://remoteok.com/api` | RemoteOK must be named as the source and the listing linked. The response's first element is a legal notice, not a listing, and is skipped. |
+| **Arbeitnow** | `GET https://www.arbeitnow.com/api/job-board-api?page=N` | Read by OpenVibe.Sources (source `arbeitnow-job-board`), never by Work: its first three pages are fetched under the board's robots and terms, and Work follows that change feed (`server/ingest/from-sources.js`). Arbeitnow is named on every listing and the listing links to its page on arbeitnow.com; a job the board stops showing is retired by Sources after a week. |
+| **Remotive** | `GET https://remotive.com/api/remote-jobs?limit=100` | Remotive must be named as the source and the listing linked. The API is called about four times a day — the six-hour interval is exactly that. It stays Work's own fetch because remotive.com/robots.txt disallows `/api/*`, so OpenVibe.Sources may not read it, while Remotive's API terms grant the named, linked use. |
+| **RemoteOK** | `GET https://remoteok.com/api` | Read by OpenVibe.Sources (source `remoteok-job-board`), never by Work: the API is fetched under remoteok.com's robots.txt and its terms. RemoteOK must be named as the source and the listing linked; the response's first element is a legal notice, not a listing, and Sources' mapping skips it. |
What is stored per listing: `source`, the board's own id, the original `url`, title, company, location, the remote
What is stored per listing: `source`, the board's own id, the original `url`, title, company, location, the remote
flag, tags, job type, salary text when the board gives one, when it was posted, when we fetched it, when it was first
flag, tags, job type, salary text when the board gives one, when it was posted, when we fetched it, when it was first
@@ -171,9 +171,8 @@ Network), `BASE_URL`, `DATABASE_URL` and `DATABASE_DIRECT_URL`. The database is
@@ -171,9 +171,8 @@ Network), `BASE_URL`, `DATABASE_URL` and `DATABASE_DIRECT_URL`. The database is
Network signing key, the OAuth client and Valkey are optional (the service says so, per check, on `/api/ready`).
Network signing key, the OAuth client and Valkey are optional (the service says so, per check, on `/api/ready`).
The ingest is **off** unless `WORK_INGEST=on`, so a development run and the test suite never call a job board. When it
The ingest is **off** unless `WORK_INGEST=on`, so a development run and the test suite never call a job board. When it
-is on, `WORK_INGEST_INTERVAL_HOURS` (6), `WORK_INGEST_PAGES` (3), `WORK_INGEST_LIMIT` (100), `WORK_INGEST_TIMEOUT_MS`
-(15000), `WORK_RETENTION_DAYS` (30) and the per-source base URLs (`WORK_ARBEITNOW_URL`, `WORK_REMOTIVE_URL`,
-`WORK_REMOTEOK_URL`) decide what it does.
+is on, `WORK_INGEST_INTERVAL_HOURS` (6), `WORK_INGEST_LIMIT` (100), `WORK_INGEST_TIMEOUT_MS` (15000),
+`WORK_RETENTION_DAYS` (30), `WORK_REMOTIVE_URL` and `OV_SOURCES_INTERNAL_URL` decide what it does.
The agent pillar is driven by `WORK_ACTOR_URL` (default `http://127.0.0.1:4950`) — unset it (an empty value) and
The agent pillar is driven by `WORK_ACTOR_URL` (default `http://127.0.0.1:4950`) — unset it (an empty value) and
`POST …/assist` answers 503 `work.actor_unavailable` — and `WORK_ACTOR_PUBLIC_URL` (default `https://openvibe.actor`),
`POST …/assist` answers 503 `work.actor_unavailable` — and `WORK_ACTOR_PUBLIC_URL` (default `https://openvibe.actor`),
@@ -16,9 +16,9 @@
@@ -16,9 +16,9 @@
"openvibe-shared v4.214.0",
"openvibe-shared v4.214.0",
"openvibe-publishing v1.7.0"
"openvibe-publishing v1.7.0"
],
],
- "summary": "OpenVibe.Work gathers job listings from three public job boards (Arbeitnow, Remotive, RemoteOK) under their terms — the source is named and the original listing is linked on every page, and only a short excerpt is kept. It searches them with facets, shows one listing with a schema.org JobPosting whose url is the original, and lets a signed-in person keep a search, see what is new since they last looked, get alerts through OpenVibe.Watch, track the jobs they apply to with a status history, and ask their OpenVibe.Actor agent to help prepare an application. Public reads need no token.",
+ "summary": "OpenVibe.Work gathers job listings from five public job boards (Arbeitnow, We Work Remotely, RemoteOK, Hacker News and Remotive) under their terms — four arrive through OpenVibe.Sources, and Remotive is read directly because its robots.txt keeps Sources out — the source is named and the original listing is linked on every page, and only a short excerpt is kept. It searches them with facets, shows one listing with a schema.org JobPosting whose url is the original, and lets a signed-in person keep a search, see what is new since they last looked, get alerts through OpenVibe.Watch, track the jobs they apply to with a status history, and ask their OpenVibe.Actor agent to help prepare an application. Public reads need no token.",
"works": [
"works": [
- "The ingest: three boards read on a timer (WORK_INGEST=on, off in tests and development), each with its own User-Agent, 15-second timeout, upsert by (source, source_id) and expiry after 30 days unseen",
+ "The ingest: Remotive read directly and Arbeitnow, We Work Remotely, RemoteOK and Hacker News followed from OpenVibe.Sources (WORK_INGEST=on, off in tests and development), with upsert by (source, source_id) and expiry after 30 days unseen",
"The listings: provenance kept and shown (source name, link to the original, when it was fetched), a short plain-text excerpt, never the full description",
"The listings: provenance kept and shown (source name, link to the original, when it was fetched), a short plain-text excerpt, never the full description",
"Search: /jobs?q=&remote=&type=&location= newest first, 30 a page with a cursor, facets for remote and job type",
"Search: /jobs?q=&remote=&type=&location= newest first, 30 a page with a cursor, facets for remote and job type",
"One listing: /jobs/:id with the board's fields, the excerpt, a 'Read the full listing on <source>' link and a JobPosting JSON-LD whose url is the original",
"One listing: /jobs/:id with the board's fields, the excerpt, a 'Read the full listing on <source>' link and a JobPosting JSON-LD whose url is the original",
@@ -126,8 +126,8 @@ async function createApp(opts = {}) {
@@ -126,8 +126,8 @@ async function createApp(opts = {}) {
return createIndexNow({ host: config.baseUrl, key: '', fetch: fetchImpl, log: warn });
return createIndexNow({ host: config.baseUrl, key: '', fetch: fetchImpl, log: warn });
}
}
})();
})();
- // Arbeitnow through OpenVibe.Sources (server/ingest/from-sources.js), off unless WORK_ARBEITNOW_FROM=sources; a test
- // passes its own Sources stand-in as opts.sourcesClient.
+ // Arbeitnow, We Work Remotely, Hacker News and RemoteOK through OpenVibe.Sources (server/ingest/from-sources.js);
+ // it is off without a Sources URL and the OAuth client secret, and a test passes its own stand-in as opts.sourcesClient.
ctx.fromSources = createFromSources({ config, s, fetchImpl, log, source: opts.sourcesClient || null, afterRun: () => ctx.search.afterIngest(), indexnow: ctx.indexnow });
ctx.fromSources = createFromSources({ config, s, fetchImpl, log, source: opts.sourcesClient || null, afterRun: () => ctx.search.afterIngest(), indexnow: ctx.indexnow });
ctx.ingest = createIngest({ config, s, fetchImpl, log, afterRun: () => ctx.search.afterIngest(), indexnow: ctx.indexnow, fromSources: ctx.fromSources });
ctx.ingest = createIngest({ config, s, fetchImpl, log, afterRun: () => ctx.search.afterIngest(), indexnow: ctx.indexnow, fromSources: ctx.fromSources });
@@ -77,25 +77,15 @@ function load(env = process.env) {
@@ -77,25 +77,15 @@ function load(env = process.env) {
timeoutMs: Math.max(1000, int(env.WORK_ACTOR_TIMEOUT_MS, 15000)),
timeoutMs: Math.max(1000, int(env.WORK_ACTOR_TIMEOUT_MS, 15000)),
},
},
- // The public job boards the ingest reads (server/jobs/sources.js). The base URLs are configurable so a
- // test points them at a stand-in server; nothing else in the service ever fetches a URL, and never one a
- // caller typed. Their terms are quoted in the manifest that describes each source, and applied on every
- // page that shows a listing: the source is named and the original listing is linked.
+ // The one public job board the ingest still reads itself (server/jobs/sources.js): Remotive's API terms grant a
+ // named, linked use, while its robots.txt disallows /api/*, so Sources may not fetch it. Its base URL is
+ // configurable so a test points it at a stand-in; nothing else in the service ever fetches a URL, and never one
+ // a caller typed. Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK come through
+ // OpenVibe.Sources (server/ingest/from-sources.js), which needs Sources (127.0.0.1:4720) and the OAuth client
+ // secret (Network grants Work sources.item.read).
sources: {
sources: {
- arbeitnow: { url: trim(env.WORK_ARBEITNOW_URL || 'https://www.arbeitnow.com/api/job-board-api') },
remotive: { url: trim(env.WORK_REMOTIVE_URL || 'https://remotive.com/api/remote-jobs') },
remotive: { url: trim(env.WORK_REMOTIVE_URL || 'https://remotive.com/api/remote-jobs') },
- remoteok: { url: trim(env.WORK_REMOTEOK_URL || 'https://remoteok.com/api') },
- // OpenVibe.Sources (server/ingest/from-sources.js): WORK_ARBEITNOW_FROM=sources takes Arbeitnow from Sources'
- // arbeitnow-job-board instead of the board (plan 2026-10-22, F2); it needs Sources (127.0.0.1:4720) and the
- // OAuth client secret (Network grants Work sources.item.read). Anything else keeps Work's own fetch.
internalUrl: trim(env.OV_SOURCES_INTERNAL_URL == null ? 'http://127.0.0.1:4720' : env.OV_SOURCES_INTERNAL_URL),
internalUrl: trim(env.OV_SOURCES_INTERNAL_URL == null ? 'http://127.0.0.1:4720' : env.OV_SOURCES_INTERNAL_URL),
- arbeitnowFrom: String(env.WORK_ARBEITNOW_FROM || '').trim().toLowerCase() === 'sources' ? 'sources' : 'board',
- // We Work Remotely reaches Work the same way (WORK_WWR_FROM=sources, the weworkremotely key in Sources);
- // it is off until the board is read through Sources, and Work never fetches the feed itself.
- weworkremotelyFrom: String(env.WORK_WWR_FROM || '').trim().toLowerCase() === 'sources' ? 'sources' : 'board',
- // Hacker News "Who is hiring" reaches Work the same way (WORK_HN_WHOISHIRING_FROM=sources, the
- // hn-whoishiring key in Sources); its resolver reads the thread, never Work itself.
- hnWhoishiringFrom: String(env.WORK_HN_WHOISHIRING_FROM || '').trim().toLowerCase() === 'sources' ? 'sources' : 'board',
},
},
// The disclosure rate the jobs page shows (server/jobs/store.js): of the last N listings we fetched, how
// The disclosure rate the jobs page shows (server/jobs/store.js): of the last N listings we fetched, how
// many carried a salary. N is configuration, so the number on the page is never typed in by hand.
// many carried a salary. N is configuration, so the number on the page is never typed in by hand.
@@ -111,8 +101,7 @@ function load(env = process.env) {
@@ -111,8 +101,7 @@ function load(env = process.env) {
enabled: String(env.WORK_INGEST || '').toLowerCase() === 'on',
enabled: String(env.WORK_INGEST || '').toLowerCase() === 'on',
intervalMs: Math.max(1, num(env.WORK_INGEST_INTERVAL_HOURS, 6)) * 3_600_000,
intervalMs: Math.max(1, num(env.WORK_INGEST_INTERVAL_HOURS, 6)) * 3_600_000,
timeoutMs: Math.max(1000, int(env.WORK_INGEST_TIMEOUT_MS, 15_000)),
timeoutMs: Math.max(1000, int(env.WORK_INGEST_TIMEOUT_MS, 15_000)),
- // Arbeitnow's board is paginated; the first N pages are read. Remotive takes a limit.
- pages: Math.max(1, int(env.WORK_INGEST_PAGES, 3)),
+ // Remotive takes a limit; the boards read through Sources carry their own mappings.
limit: Math.max(1, int(env.WORK_INGEST_LIMIT, 100)),
limit: Math.max(1, int(env.WORK_INGEST_LIMIT, 100)),
retentionDays: Math.max(1, int(env.WORK_RETENTION_DAYS, 30)),
retentionDays: Math.max(1, int(env.WORK_RETENTION_DAYS, 30)),
userAgent: env.WORK_USER_AGENT || `OpenVibeWork/0.1 (+${baseUrl})`,
userAgent: env.WORK_USER_AGENT || `OpenVibeWork/0.1 (+${baseUrl})`,
@@ -1,7 +1,8 @@
@@ -1,7 +1,8 @@
'use strict';
'use strict';
/**
/**
- * Job boards read through OpenVibe.Sources (plan 2026-10-22, track F2; We Work Remotely added 2026-10-30).
+ * Job boards read through OpenVibe.Sources (plan 2026-10-22, track F; We Work Remotely added 2026-10-30, RemoteOK
+ * 2026-11-12).
*
*
* Sources fetches a board (under its robots and terms checks) and keeps each job as an item. Work follows Sources'
* Sources fetches a board (under its robots and terms checks) and keeps each job as an item. Work follows Sources'
* change feed (openvibe-publishing/ingest: the cursor in work_ingest_cursor, one transaction per page) and keeps one
* change feed (openvibe-publishing/ingest: the cursor in work_ingest_cursor, one transaction per page) and keeps one
@@ -12,12 +13,12 @@
@@ -12,12 +13,12 @@
* The thin-listing gate (sources.toRow) is the same for all of them: no id, no title, no company or no http(s) link
* The thin-listing gate (sources.toRow) is the same for all of them: no id, no title, no company or no http(s) link
* back means the listing is not published.
* back means the listing is not published.
*
*
- * A job Sources retires (the board has not shown it for a week: retire_after_sec) or removes is expired here, like a
- * listing a board stopped showing. Work's own expiry sweep leaves these listings alone: an unchanged job is no change in
- * Sources' feed, so Work never hears it was seen again, and Sources' retirement is what says it is gone.
+ * A job Sources retires (the board has not shown it for its own window: retire_after_sec) or removes is expired here,
+ * like a listing a board stopped showing. Work's own expiry sweep leaves these listings alone: an unchanged job is no
+ * change in Sources' feed, so Work never hears it was seen again, and Sources' retirement is what says it is gone.
*
*
- * Pulls 30 seconds after start and then hourly. Each board has its own switch (WORK_ARBEITNOW_FROM=sources,
- * WORK_WWR_FROM=sources); a board that is not switched on is left to its own fetch, as before.
+ * Pulls 30 seconds after start and then hourly. Remotive is the one board still read by Work itself: its robots.txt
+ * disallows /api/*, so Sources may not fetch it, while its API terms explicitly grant a named, linked use.
*/
*/
const ingest = require('openvibe-publishing/ingest');
const ingest = require('openvibe-publishing/ingest');
const sources = require('../jobs/sources');
const sources = require('../jobs/sources');
@@ -138,6 +139,35 @@ function rawOfHn(item) {
@@ -138,6 +139,35 @@ function rawOfHn(item) {
/** The start of a plain-text summary, with whitespace collapsed, for comparing it with the comment's first line. */
/** The start of a plain-text summary, with whitespace collapsed, for comparing it with the comment's first line. */
const excerptLines = (text) => String(text || '').replace(/\s+/g, ' ').trim();
const excerptLines = (text) => String(text || '').replace(/\s+/g, ' ').trim();
+/** A salary bound as RemoteOK gives it (a number), or null; never a guess from a missing field. */
+const amount = (v) => {
+ const n = Number(v);
+ return Number.isFinite(n) && n > 0 ? Math.round(n) : null;
+};
+
+/**
+ * A Sources item → the parsed row Work's own RemoteOK parse makes (server/jobs/sources.js): the board's id and URL,
+ * its position as the title, and its salary bounds written the way the board's own fetch wrote them. Sources has
+ * already turned the HTML description into a short plain-text excerpt, which is all Work stores.
+ */
+function rawOfRemoteOk(item) {
+ const f = (item && item.fields) || {};
+ const min = amount(f.salary_min);
+ const max = amount(f.salary_max);
+ return {
+ source_id: item.identity,
+ url: item.canonical_url,
+ title: item.title,
+ company: typeof f.company === 'string' ? f.company.trim() : null,
+ location: typeof f.location === 'string' ? f.location.trim() || null : null,
+ remote: true, // the board carries remote work only
+ tags: list(f.tags),
+ job_type: typeof f.job_type === 'string' ? f.job_type.trim() || null : null,
+ salary: min && max ? `$${min.toLocaleString('en-US')}–$${max.toLocaleString('en-US')} a year` : null,
+ posted_at: item.published_at || null,
+ description_html: item.summary || '',
+ };
+}
/**
/**
* The boards Work reads through OpenVibe.Sources: the Sources key an item arrives under, the id the listing is
* The boards Work reads through OpenVibe.Sources: the Sources key an item arrives under, the id the listing is
@@ -147,22 +177,15 @@ const SOURCES = {
@@ -147,22 +177,15 @@ const SOURCES = {
[SOURCE_KEY]: { source_id: SOURCE_ID, kind: 'job', rawOf },
[SOURCE_KEY]: { source_id: SOURCE_ID, kind: 'job', rawOf },
'weworkremotely': { source_id: 'weworkremotely', kind: 'article', rawOf: rawOfWwr },
'weworkremotely': { source_id: 'weworkremotely', kind: 'article', rawOf: rawOfWwr },
'hn-whoishiring': { source_id: 'hn-whoishiring', kind: 'job', rawOf: rawOfHn },
'hn-whoishiring': { source_id: 'hn-whoishiring', kind: 'job', rawOf: rawOfHn },
-};
-
-/** The switch that says a board comes from Sources instead of from its own fetch. */
-const SWITCH = {
- arbeitnow: (config) => (config.sources && config.sources.arbeitnowFrom) === 'sources',
- weworkremotely: (config) => (config.sources && config.sources.weworkremotelyFrom) === 'sources',
- 'hn-whoishiring': (config) => (config.sources && config.sources.hnWhoishiringFrom) === 'sources',
+ 'remoteok-job-board': { source_id: 'remoteok', kind: 'job', rawOf: rawOfRemoteOk },
};
};
function createFromSources({ config, s, fetchImpl = globalThis.fetch, log = console, source = null, afterRun = null, indexnow = null }) {
function createFromSources({ config, s, fetchImpl = globalThis.fetch, log = console, source = null, afterRun = null, indexnow = null }) {
const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');
const url = String((config.sources && config.sources.internalUrl) || '').replace(/\/+$/, '');
- // The boards that are switched on and therefore read through Sources. Each keeps its own listing id.
- const active = Object.entries(SOURCES).filter(([, e]) => (SWITCH[e.source_id] || (() => false))(config));
+ // Every board in SOURCES is read through Sources; each keeps its own listing id.
+ const active = Object.entries(SOURCES);
const activeByKey = Object.fromEntries(active);
const activeByKey = Object.fromEntries(active);
- const on = active.length > 0;
- const client = source || (on && url && config.oauth.clientSecret
+ const client = source || (url && config.oauth.clientSecret
? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'jobs' } }, fetchImpl })
? ingest.createSourcesClient({ config: { ...config, sources: { internalUrl: url, category: 'jobs' } }, fetchImpl })
: null);
: null);
const cursor = ingest.createChangeCursor(s.db, { prefix: 'work', now: s.now });
const cursor = ingest.createChangeCursor(s.db, { prefix: 'work', now: s.now });
@@ -239,4 +262,4 @@ function createFromSources({ config, s, fetchImpl = globalThis.fetch, log = cons
@@ -239,4 +262,4 @@ function createFromSources({ config, s, fetchImpl = globalThis.fetch, log = cons
return { enabled: Boolean(client), pull, start, stop, apply, SOURCE_ID, sourceIds: active.map(([, e]) => e.source_id) };
return { enabled: Boolean(client), pull, start, stop, apply, SOURCE_ID, sourceIds: active.map(([, e]) => e.source_id) };
}
}
-module.exports = { createFromSources, rawOf, rawOfWwr, rawOfHn, SOURCES, SOURCE_ID, SOURCE_KEY, TITLE_RULE };
+module.exports = { createFromSources, rawOf, rawOfWwr, rawOfHn, rawOfRemoteOk, SOURCES, SOURCE_ID, SOURCE_KEY, TITLE_RULE };
@@ -1,8 +1,9 @@
@@ -1,8 +1,9 @@
'use strict';
'use strict';
/**
/**
- * The background ingest: every six hours, read each job board's public API (server/jobs/sources.js), keep what it
- * shows and retire what it stopped showing.
+ * The background ingest: every six hours, read Remotive's public API (server/jobs/sources.js), keep what it shows and
+ * retire what it stopped showing. Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK are read by
+ * OpenVibe.Sources and followed in server/ingest/from-sources.js; no source in this module may fetch them.
*
*
* It is off unless WORK_INGEST=on, so neither a test nor a development run ever calls a job board; when it is on,
* It is off unless WORK_INGEST=on, so neither a test nor a development run ever calls a job board; when it is on,
* each source keeps its own timer, its own last-fetch row and its own failures — one board being down never stops the
* each source keeps its own timer, its own last-fetch row and its own failures — one board being down never stops the
@@ -46,6 +47,8 @@ function createIngest({ config, s, fetchImpl = globalThis.fetch, log = console,
@@ -46,6 +47,8 @@ function createIngest({ config, s, fetchImpl = globalThis.fetch, log = console,
async function runSource(sourceId) {
async function runSource(sourceId) {
const source = sources.byId(sourceId);
const source = sources.byId(sourceId);
if (!source) throw new Error(`no source ${sourceId}`);
if (!source) throw new Error(`no source ${sourceId}`);
+ // A board OpenVibe.Sources feeds is not read here, not even by an operator naming it: Sources is its reader.
+ if (source.viaSources) return { source: source.id, skipped: 'from_sources' };
const startedAt = s.iso();
const startedAt = s.iso();
const fetchedAt = startedAt;
const fetchedAt = startedAt;
await store.beginFetch(s, source.id, startedAt);
await store.beginFetch(s, source.id, startedAt);
@@ -51,42 +51,17 @@ function httpUrl(v) {
@@ -51,42 +51,17 @@ function httpUrl(v) {
} catch { return null; }
} catch { return null; }
}
}
-const money = (n) => (n == null || n === '' || !Number.isFinite(Number(n)) || Number(n) <= 0 ? null : Math.round(Number(n)));
-
const SOURCES = [
const SOURCES = [
{
{
id: 'arbeitnow',
id: 'arbeitnow',
name: 'Arbeitnow',
name: 'Arbeitnow',
homepage: 'https://www.arbeitnow.com/',
homepage: 'https://www.arbeitnow.com/',
- terms: 'Arbeitnow publishes its job board API free of charge. We attribute Arbeitnow by name and link every listing back to its page on arbeitnow.com.',
+ terms: 'Arbeitnow publishes its job board API free of charge. OpenVibe.Sources reads its first three pages under the board\'s own robots and terms checks; Work follows that source (source key arbeitnow-job-board). We attribute Arbeitnow by name and link every listing back to its page on arbeitnow.com.',
attribution: 'Listing from Arbeitnow',
attribution: 'Listing from Arbeitnow',
remoteDefault: false,
remoteDefault: false,
- requests(config) {
- const pages = [];
- for (let page = 1; page <= config.ingest.pages; page++) {
- const u = new URL(config.sources.arbeitnow.url);
- u.searchParams.set('page', String(page));
- pages.push(u.toString());
- }
- return pages;
- },
- // { data: [ { slug, company_name, title, description (HTML), remote, url, tags[], job_types[], location, created_at } ] }
- parse(json) {
- const rows = (json && Array.isArray(json.data)) ? json.data : [];
- return rows.map((r) => ({
- source_id: clip(r.slug || r.url, 200),
- url: httpUrl(r.url),
- title: clip(r.title, 300),
- company: clip(r.company_name, 200),
- location: clip(r.location, 200),
- remote: r.remote === true,
- tags: list(r.tags),
- job_type: list(r.job_types).join(', ') || null,
- salary: null, // this board gives no salary
- posted_at: instant(r.created_at),
- description_html: r.description,
- }));
- },
+ viaSources: true, // read through OpenVibe.Sources, never by Work itself
+ requests() { return []; },
+ parse() { return []; }, // Work never calls the Arbeitnow API itself
},
},
{
{
id: 'weworkremotely',
id: 'weworkremotely',
@@ -146,32 +121,12 @@ const SOURCES = [
@@ -146,32 +121,12 @@ const SOURCES = [
id: 'remoteok',
id: 'remoteok',
name: 'RemoteOK',
name: 'RemoteOK',
homepage: 'https://remoteok.com/',
homepage: 'https://remoteok.com/',
- terms: 'RemoteOK asks that a use of its API names RemoteOK as the source and links back to the listing. The response\'s first element is a legal notice, not a listing, and is skipped.',
+ terms: 'RemoteOK asks that a use of its API names RemoteOK as the source and links back to the listing. OpenVibe.Sources reads the API under those terms and its robots.txt; the response\'s first element is a legal notice, not a listing, and is skipped. Work follows that source (source key remoteok-job-board).',
attribution: 'Listing from RemoteOK',
attribution: 'Listing from RemoteOK',
remoteDefault: true,
remoteDefault: true,
- requests(config) { return [config.sources.remoteok.url]; },
- // The response is an array whose first element is the legal notice; every later element is a listing.
- parse(json) {
- const all = Array.isArray(json) ? json : [];
- const rows = all.filter((r) => r && (r.id != null || r.slug) && r.position);
- return rows.map((r) => {
- const min = money(r.salary_min);
- const max = money(r.salary_max);
- return {
- source_id: clip(r.id != null ? r.id : r.slug, 200),
- url: httpUrl(r.url || (r.slug ? `https://remoteok.com/remote-jobs/${r.slug}` : null)),
- title: clip(r.position, 300),
- company: clip(r.company, 200),
- location: clip(r.location, 200),
- remote: true,
- tags: list(r.tags),
- job_type: clip(r.job_type || r.type, 80) || null,
- salary: min && max ? `$${min.toLocaleString('en-US')}–$${max.toLocaleString('en-US')} a year` : null,
- posted_at: instant(r.date),
- description_html: r.description,
- };
- });
- },
+ viaSources: true, // read through OpenVibe.Sources, never by Work itself
+ requests() { return []; },
+ parse() { return []; }, // Work never calls the RemoteOK API itself
},
},
];
];
@@ -18,7 +18,7 @@ const companies = require('../server/jobs/companies');
@@ -18,7 +18,7 @@ const companies = require('../server/jobs/companies');
assert.strictEqual(companies.slugOf('x'.repeat(200)).length, 80);
assert.strictEqual(companies.slugOf('x'.repeat(200)).length, 80);
});
});
- const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });
+ const t = await boot();
try {
try {
await t.ctx.ingest.runAll();
await t.ctx.ingest.runAll();
const s = t.ctx.s;
const s = t.ctx.s;
@@ -1,12 +1,12 @@
@@ -1,12 +1,12 @@
'use strict';
'use strict';
/**
/**
- * Arbeitnow through OpenVibe.Sources (server/ingest/from-sources.js; plan 2026-10-22, track F2), against a Sources
- * stand-in serving its change feed:
+ * Arbeitnow, We Work Remotely, Hacker News and RemoteOK through OpenVibe.Sources (server/ingest/from-sources.js;
+ * plan 2026-10-22, track F), against a Sources stand-in serving its change feed:
* - each job item becomes the listing Work's own fetch made (same slug → same id and URL), with its excerpt, tags and
* - each job item becomes the listing Work's own fetch made (same slug → same id and URL), with its excerpt, tags and
* job types; other sources' items and jobs Work may not publish are held;
* job types; other sources' items and jobs Work may not publish are held;
* - the board is no longer read, and Work's expiry sweep leaves these listings to Sources;
* - the board is no longer read, and Work's expiry sweep leaves these listings to Sources;
* - a job Sources retires (gone_from_source) expires here; the fetch record says what came through;
* - a job Sources retires (gone_from_source) expires here; the fetch record says what came through;
- * - without WORK_ARBEITNOW_FROM=sources nothing changes.
+ * - without a Sources URL and client secret the follower stays off.
*/
*/
const assert = require('assert');
const assert = require('assert');
const { boot, check, done } = require('./helpers/boot');
const { boot, check, done } = require('./helpers/boot');
@@ -36,7 +36,7 @@ const job = (identity, over = {}) => ({
@@ -36,7 +36,7 @@ const job = (identity, over = {}) => ({
const clock = { t: Date.parse('2026-10-23T06:00:00Z') };
const clock = { t: Date.parse('2026-10-23T06:00:00Z') };
const pinged = [];
const pinged = [];
const indexnow = { enabled: false, pingSoon: (urls) => { pinged.push(...urls); return urls.length; } };
const indexnow = { enabled: false, pingSoon: (urls) => { pinged.push(...urls); return urls.length; } };
- const t = await boot({ now: () => clock.t, env: { WORK_ARBEITNOW_FROM: 'sources' }, sourcesClient, indexnow });
+ const t = await boot({ now: () => clock.t, sourcesClient, indexnow });
const s = t.ctx.s;
const s = t.ctx.s;
const from = t.ctx.fromSources;
const from = t.ctx.fromSources;
@@ -90,8 +90,8 @@ const job = (identity, over = {}) => ({
@@ -90,8 +90,8 @@ const job = (identity, over = {}) => ({
assert.deepStrictEqual([(await from.pull()).applied, (await from.pull()).removed], [0, 0]);
assert.deepStrictEqual([(await from.pull()).applied, (await from.pull()).removed], [0, 0]);
});
});
- await check('without WORK_ARBEITNOW_FROM=sources nothing follows Sources', async () => {
- const config = { ...t.config, sources: { ...t.config.sources, arbeitnowFrom: 'board', internalUrl: 'http://127.0.0.1:1' } };
+ await check('without a Sources URL and client secret the follower stays off', async () => {
+ const config = { ...t.config, sources: { ...t.config.sources, internalUrl: '' }, oauth: { ...t.config.oauth, clientSecret: '' } };
const off = createFromSources({ config, s, log: { log() {}, warn() {} } });
const off = createFromSources({ config, s, log: { log() {}, warn() {} } });
assert.strictEqual(off.enabled, false);
assert.strictEqual(off.enabled, false);
assert.deepStrictEqual(await off.pull(), { skipped: true });
assert.deepStrictEqual(await off.pull(), { skipped: true });
@@ -1,63 +1,23 @@
@@ -1,63 +1,23 @@
'use strict';
'use strict';
/**
/**
- * Stand-in job boards for the ingest tests: one HTTP server with a path per board, answering the shape each real API
- * documents, with small fixtures written to look like the real thing (including RemoteOK's legal notice as the first
- * array element, and an Arbeitnow listing whose title carries a <script> tag, so the pages' escaping is provable).
+ * A stand-in Remotive API for the one board Work still reads itself (test/ingest.test.js). Its payload is the shape
+ * Remotive documents, with fixtures of ours (no real listing is copied here) including a description longer than the
+ * excerpt Work keeps.
+ *
+ * The other boards are not here because Work does not fetch them: Arbeitnow, We Work Remotely, Hacker News "Who is
+ * hiring" and RemoteOK are served by OpenVibe.Sources, and their tests use a Sources feed stand-in instead.
*
*
* const b = await startBoards();
* const b = await startBoards();
- * b.env WORK_ARBEITNOW_URL / WORK_REMOTIVE_URL / WORK_REMOTEOK_URL, for boot({ env })
- * b.state.arbeitnow = 'fail' one board answering HTTP 500 (default 'ok')
- * b.state.remotive = 'json' one board answering something that is not JSON
- * b.add('arbeitnow', row) a listing the board starts showing (see b.fixtures for the shape)
+ * b.env WORK_REMOTIVE_URL, for boot({ env })
+ * b.state.remotive = 'fail' the board answering HTTP 500 (default 'ok')
+ * b.add('remotive', row) a listing the board starts showing (see b.fixtures)
* b.requests [{ path, query, userAgent }] for every call
* b.requests [{ path, query, userAgent }] for every call
- *
- * The fixtures are ours, not a board's: no real listing is copied here.
*/
*/
const http = require('http');
const http = require('http');
/** One listing per board, in that board's own payload shape. */
/** One listing per board, in that board's own payload shape. */
function fixtures() {
function fixtures() {
return {
return {
- arbeitnow: [
- {
- slug: 'senior-backend-engineer-berlin',
- company_name: 'Nordlicht Systems',
- title: 'Senior Backend Engineer (Node.js)',
- description: '<p>We run a fleet of Node.js services.</p><ul><li>PostgreSQL</li><li>Valkey</li></ul><p>You will own the billing pipeline end to end.</p>',
- remote: false,
- url: 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345',
- tags: ['nodejs', 'postgresql'],
- job_types: ['permanent'],
- location: 'Berlin, Germany',
- created_at: 1791500000,
- },
- {
- slug: 'frontend-engineer-remote-eu',
- company_name: 'Helles Werk',
- title: 'Frontend Engineer <script>alert("xss")</script>',
- description: '<div>Build the design system.<script>window.__pwned = true</script></div><div>TypeScript, CSS.</div>',
- remote: true,
- url: 'https://www.arbeitnow.com/view/frontend-engineer-remote-eu-23456',
- tags: ['typescript', 'css'],
- job_types: ['permanent', 'full_time'],
- location: 'Remote (EU)',
- created_at: 1791600000,
- },
- ],
- arbeitnowPage2: [
- {
- slug: 'data-analyst-munich',
- company_name: 'Alpen Daten',
- title: 'Data Analyst',
- description: '<p>SQL, dashboards and a lot of questions.</p>',
- remote: false,
- url: 'https://www.arbeitnow.com/view/data-analyst-munich-34567',
- tags: ['sql'],
- job_types: ['permanent'],
- location: 'Munich, Germany',
- created_at: 1791700000,
- },
- ],
remotive: [
remotive: [
{
{
id: 1900123,
id: 1900123,
@@ -104,43 +64,14 @@ blameless. The office dog is called Waffle.</p>`,
@@ -104,43 +64,14 @@ blameless. The office dog is called Waffle.</p>`,
description: '<p>Write about developer tools.</p>',
description: '<p>Write about developer tools.</p>',
},
},
],
],
- remoteok: [
- { legal: "RemoteOK's API is offered for personal use; please link back to the listing and name RemoteOK as the source." },
- {
- id: '1049001',
- slug: 'devops-engineer-skyward',
- company: 'Skyward Cloud',
- position: 'DevOps Engineer',
- tags: ['devops', 'kubernetes'],
- location: 'Worldwide',
- salary_min: 90000,
- salary_max: 130000,
- date: '2026-10-01T08:00:00+00:00',
- url: 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001',
- description: '<p>Kubernetes, Terraform, on-call.</p>',
- },
- {
- id: '1049002',
- slug: 'technical-writer-atlas',
- company: 'Atlas Docs',
- position: 'Technical Writer',
- tags: ['docs'],
- location: 'EU',
- salary_min: null,
- salary_max: null,
- date: '2026-09-25T08:00:00+00:00',
- url: 'https://remoteok.com/remote-jobs/remote-technical-writer-atlas-1049002',
- description: '<p>Document an API.</p>',
- },
- ],
};
};
}
}
async function startBoards() {
async function startBoards() {
const data = fixtures();
const data = fixtures();
const requests = [];
const requests = [];
- const state = { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' };
- const added = { arbeitnow: [], remotive: [], remoteok: [] };
+ const state = { remotive: 'ok' };
+ const added = { remotive: [] };
const send = (res, status, body, type = 'application/json') => { res.writeHead(status, { 'content-type': type }); res.end(typeof body === 'string' ? body : JSON.stringify(body)); };
const send = (res, status, body, type = 'application/json') => { res.writeHead(status, { 'content-type': type }); res.end(typeof body === 'string' ? body : JSON.stringify(body)); };
@@ -148,17 +79,10 @@ async function startBoards() {
@@ -148,17 +79,10 @@ async function startBoards() {
const url = new URL(req.url, 'http://stand-in');
const url = new URL(req.url, 'http://stand-in');
requests.push({ path: url.pathname, query: url.search, userAgent: req.headers['user-agent'] || '', accept: req.headers.accept || '' });
requests.push({ path: url.pathname, query: url.search, userAgent: req.headers['user-agent'] || '', accept: req.headers.accept || '' });
const board = url.pathname.replace(/^\//, '');
const board = url.pathname.replace(/^\//, '');
- if (!['arbeitnow', 'remotive', 'remoteok'].includes(board)) return send(res, 404, { error: 'not a board' });
- if (state[board] === 'fail') return send(res, 500, { error: 'the board is having a bad day' }, 'text/plain');
- if (state[board] === 'html') return send(res, 200, '<html>not json</html>', 'text/html');
- if (board === 'arbeitnow') {
- const page = Number(url.searchParams.get('page') || 1);
- const rows = page === 1 ? data.arbeitnow : page === 2 ? data.arbeitnowPage2 : [];
- return send(res, 200, { data: [...rows, ...added.arbeitnow], links: { next: page < 2 ? `https://www.arbeitnow.com/api/job-board-api?page=${page + 1}` : null } });
- }
- if (board === 'remotive') return send(res, 200, { 'job-count': data.remotive.length + added.remotive.length, jobs: [...data.remotive, ...added.remotive] });
- // RemoteOK: the legal notice is element 0, exactly as the real API sends it.
- return send(res, 200, [...data.remoteok.slice(0, 1), ...data.remoteok.slice(1), ...added.remoteok]);
+ if (board !== 'remotive') return send(res, 404, { error: 'not a board Work reads' });
+ if (state.remotive === 'fail') return send(res, 500, { error: 'the board is having a bad day' }, 'text/plain');
+ if (state.remotive === 'html') return send(res, 200, '<html>not json</html>', 'text/html');
+ return send(res, 200, { 'job-count': data.remotive.length + added.remotive.length, jobs: [...data.remotive, ...added.remotive] });
});
});
await new Promise((r) => server.listen(0, '127.0.0.1', r));
await new Promise((r) => server.listen(0, '127.0.0.1', r));
const base = `http://127.0.0.1:${server.address().port}`;
const base = `http://127.0.0.1:${server.address().port}`;
@@ -168,10 +92,10 @@ async function startBoards() {
@@ -168,10 +92,10 @@ async function startBoards() {
fixtures: data,
fixtures: data,
state,
state,
requests,
requests,
- env: { WORK_ARBEITNOW_URL: `${base}/arbeitnow`, WORK_REMOTIVE_URL: `${base}/remotive`, WORK_REMOTEOK_URL: `${base}/remoteok` },
- /** Make a board start showing one more listing (the shape of that board's own payload). */
+ env: { WORK_REMOTIVE_URL: `${base}/remotive` },
+ /** Make the board start showing one more listing (the shape of its own payload). */
add(board, row) { added[board].push(row); },
add(board, row) { added[board].push(row); },
- reset() { Object.assign(state, { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' }); requests.length = 0; },
+ reset() { Object.assign(state, { remotive: 'ok' }); requests.length = 0; },
close: () => new Promise((r) => server.close(r)),
close: () => new Promise((r) => server.close(r)),
};
};
}
}
@@ -33,7 +33,7 @@ const hn = (identity, title, summary, over = {}) => ({
@@ -33,7 +33,7 @@ const hn = (identity, title, summary, over = {}) => ({
(async () => {
(async () => {
const clock = { t: Date.parse('2026-10-30T08:00:00Z') };
const clock = { t: Date.parse('2026-10-30T08:00:00Z') };
- const t = await boot({ now: () => clock.t, env: { WORK_HN_WHOISHIRING_FROM: 'sources' }, sourcesClient });
+ const t = await boot({ now: () => clock.t, sourcesClient });
const s = t.ctx.s;
const s = t.ctx.s;
const from = t.ctx.fromSources;
const from = t.ctx.fromSources;
@@ -1,9 +1,13 @@
@@ -1,9 +1,13 @@
'use strict';
'use strict';
/**
/**
- * The background ingest against the stand-in boards (test/helpers/boards.js): what it keeps from each board and what
- * it refuses, that reading a board twice updates rather than duplicates, that a board which fails is recorded and
- * does not stop the others, that only a short plain-text excerpt is stored, and that a listing no board has shown for
- * the retention window expires out of the search and the pages.
+ * The background ingest against Remotive's stand-in API (test/helpers/boards.js): what it keeps, that reading the
+ * board twice updates rather than duplicates, that a failure is recorded and does not hide the listings already
+ * gathered, that only a short plain-text excerpt is stored, and that a listing the board stopped showing expires out
+ * of the search and the pages.
+ *
+ * Arbeitnow, We Work Remotely, Hacker News "Who is hiring" and RemoteOK never reach this ingest: they come through
+ * OpenVibe.Sources (server/ingest/from-sources.js, test/from-sources.test.js and test/remoteok-sources.test.js), so
+ * every run lists them as skipped rather than reading them.
*/
*/
const assert = require('assert');
const assert = require('assert');
const { boot, check, done } = require('./helpers/boot');
const { boot, check, done } = require('./helpers/boot');
@@ -15,73 +19,59 @@ const DAY = 86_400_000;
@@ -15,73 +19,59 @@ const DAY = 86_400_000;
const clock = { t: Date.parse('2026-10-01T06:00:00Z') };
const clock = { t: Date.parse('2026-10-01T06:00:00Z') };
const pinged = [];
const pinged = [];
const indexnow = { enabled: false, pingSoon: (urls) => { pinged.push(...urls); return urls.length; } };
const indexnow = { enabled: false, pingSoon: (urls) => { pinged.push(...urls); return urls.length; } };
- const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' }, indexnow });
+ const t = await boot({ now: () => clock.t, indexnow });
const ingest = t.ctx.ingest;
const ingest = t.ctx.ingest;
const s = t.ctx.s;
const s = t.ctx.s;
try {
try {
- await check('reading the three boards keeps every listing with its source, its id and the original URL', async () => {
+ await check('the read keeps every Remotive listing, and the boards Sources feeds are skipped', async () => {
const out = await ingest.runAll();
const out = await ingest.runAll();
- // We Work Remotely is skipped here: it reaches Work through OpenVibe.Sources, never from its own feed.
assert.deepStrictEqual(out.map((o) => o.source), ['arbeitnow', 'weworkremotely', 'hn-whoishiring', 'remotive', 'remoteok']);
assert.deepStrictEqual(out.map((o) => o.source), ['arbeitnow', 'weworkremotely', 'hn-whoishiring', 'remotive', 'remoteok']);
- assert.ok(out.filter((o) => !['weworkremotely', 'hn-whoishiring'].includes(o.source)).every((o) => o.ok), JSON.stringify(out.map((o) => o.error)));
- assert.strictEqual(out.find((o) => o.source === 'weworkremotely').skipped, 'from_sources');
- assert.strictEqual(out.find((o) => o.source === 'hn-whoishiring').skipped, 'from_sources');
- // Arbeitnow's two pages (2 + 1), Remotive's two jobs, RemoteOK's two listings.
+ assert.ok(out.filter((o) => ['arbeitnow', 'weworkremotely', 'hn-whoishiring', 'remoteok'].includes(o.source)).every((o) => o.skipped === 'from_sources'));
+ assert.strictEqual(out.find((o) => o.source === 'remotive').ok, true, JSON.stringify(out));
const counts = await store.countsBySource(s);
const counts = await store.countsBySource(s);
- assert.strictEqual(counts.get('arbeitnow').count, 3);
- assert.strictEqual(counts.get("remotive").count, 3);
- assert.strictEqual(counts.get('remoteok').count, 2);
+ assert.strictEqual(counts.get('remotive').count, 3);
+ assert.ok(!counts.has('arbeitnow') && !counts.has('remoteok'), 'a board Sources feeds was fetched by Work');
- const page = (await t.get('/api/v1/jobs?q=DevOps')).json();
+ const page = (await t.get('/api/v1/jobs?q=Platform')).json();
assert.strictEqual(page.jobs.length, 1);
assert.strictEqual(page.jobs.length, 1);
const job = page.jobs[0];
const job = page.jobs[0];
- assert.strictEqual(job.source, 'remoteok');
- assert.strictEqual(job.source_name, 'RemoteOK');
- assert.strictEqual(job.url, 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001');
- assert.strictEqual(job.company, 'Skyward Cloud');
- assert.strictEqual(job.salary, '$90,000–$130,000 a year');
+ assert.strictEqual(job.source, 'remotive');
+ assert.strictEqual(job.source_name, 'Remotive');
+ assert.strictEqual(job.url, 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789');
+ assert.strictEqual(job.company, 'Meridian Rail');
+ assert.strictEqual(job.salary, '€70,000 – €85,000');
assert.strictEqual(job.remote, true);
assert.strictEqual(job.remote, true);
- assert.deepStrictEqual(job.tags, ['devops', 'kubernetes']);
- assert.ok(job.posted_at.startsWith('2026-10-01'));
+ assert.deepStrictEqual(job.tags, ['kubernetes', 'terraform', 'Software Development']);
+ assert.ok(job.posted_at.startsWith('2026-09-29'));
- // RemoteOK's first array element is a legal notice, not a listing: nothing was made of it.
const all = (await t.get('/api/v1/jobs?limit=100')).json();
const all = (await t.get('/api/v1/jobs?limit=100')).json();
- assert.strictEqual(all.count, 8);
- assert.ok(!JSON.stringify(all.jobs).includes('personal use'), 'the legal notice became a listing');
+ assert.strictEqual(all.count, 3);
// Search engines hear about each new listing page and the sitemap (IndexNow), absolute and on this host.
// Search engines hear about each new listing page and the sitemap (IndexNow), absolute and on this host.
const jobUrls = pinged.filter((u) => u !== 'https://openvibe.work/sitemap.xml');
const jobUrls = pinged.filter((u) => u !== 'https://openvibe.work/sitemap.xml');
- assert.strictEqual(jobUrls.length, 8, pinged.join(' '));
+ assert.strictEqual(jobUrls.length, 3, pinged.join(' '));
assert.deepStrictEqual(jobUrls.sort(), all.jobs.map((j) => `https://openvibe.work/jobs/${j.id}`).sort());
assert.deepStrictEqual(jobUrls.sort(), all.jobs.map((j) => `https://openvibe.work/jobs/${j.id}`).sort());
assert.ok(pinged.includes('https://openvibe.work/sitemap.xml'));
assert.ok(pinged.includes('https://openvibe.work/sitemap.xml'));
});
});
- await check('every request used this service\'s User-Agent, a JSON Accept, and only the configured hosts', async () => {
- assert.ok(t.boards.requests.length >= 4);
+ await check('the one request used this service\'s User-Agent, a JSON Accept, and only the configured host', async () => {
+ assert.ok(t.boards.requests.length >= 1);
for (const r of t.boards.requests) {
for (const r of t.boards.requests) {
assert.strictEqual(r.userAgent, 'OpenVibeWork/0.1 (+https://openvibe.work)', r.path);
assert.strictEqual(r.userAgent, 'OpenVibeWork/0.1 (+https://openvibe.work)', r.path);
assert.match(r.accept, /application\/json/);
assert.match(r.accept, /application\/json/);
}
}
- assert.deepStrictEqual([...new Set(t.boards.requests.map((r) => r.path))].sort(), ['/arbeitnow', '/remoteok', '/remotive']);
- // Arbeitnow is paginated as configured (WORK_INGEST_PAGES=2); Remotive is asked for a limit.
- assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=1'));
- assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=2'));
+ assert.deepStrictEqual([...new Set(t.boards.requests.map((r) => r.path))], ['/remotive']);
assert.ok(t.boards.requests.some((r) => r.path === '/remotive' && r.query === '?limit=100'));
assert.ok(t.boards.requests.some((r) => r.path === '/remotive' && r.query === '?limit=100'));
});
});
await check('an HTML description becomes a short plain-text excerpt: no tags, no script code, under 320 characters', async () => {
await check('an HTML description becomes a short plain-text excerpt: no tags, no script code, under 320 characters', async () => {
- const job = (await t.get('/api/v1/jobs?q=Frontend')).json().jobs[0];
+ const job = (await t.get('/api/v1/jobs?q=Platform')).json().jobs[0];
assert.ok(!/[<>]/.test(job.excerpt), job.excerpt);
assert.ok(!/[<>]/.test(job.excerpt), job.excerpt);
- assert.ok(!/window\.__pwned/.test(job.excerpt), 'script content was kept');
- assert.ok(job.excerpt.includes('Build the design system'), job.excerpt);
+ assert.ok(job.excerpt.includes('Kubernetes clusters'), job.excerpt);
assert.ok(job.excerpt.length <= 320, String(job.excerpt.length));
assert.ok(job.excerpt.length <= 320, String(job.excerpt.length));
- const page = (await t.get('/api/v1/jobs?q=Senior Backend')).json().jobs[0];
- assert.ok(page.excerpt.includes('PostgreSQL'), page.excerpt);
- assert.ok(page.excerpt.includes('fleet of Node.js services'), page.excerpt);
- // The full description is nowhere in the database: only the excerpt is stored.
const dump = await t.dbDump();
const dump = await t.dbDump();
- assert.ok(!dump.includes('billing pipeline end to end</p>'), 'the full description was stored');
+ assert.ok(!dump.includes('The office dog is called Waffle'), 'the full description was stored');
});
});
await check('the text helper strips markup and entities, and cuts a long excerpt on a word', () => {
await check('the text helper strips markup and entities, and cuts a long excerpt on a word', () => {
@@ -106,54 +96,49 @@ const DAY = 86_400_000;
@@ -106,54 +96,49 @@ const DAY = 86_400_000;
assert.ok(!long.includes(' ') && /^\S/.test(long), 'the excerpt is cut mid-word or padded');
assert.ok(!long.includes(' ') && /^\S/.test(long), 'the excerpt is cut mid-word or padded');
});
});
- await check('reading a board again updates the listing in place: no duplicate, same id, newer fetch', async () => {
- const before = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];
+ await check('reading the board again updates the listing in place: no duplicate, same id, newer fetch', async () => {
+ const before = (await t.get('/api/v1/jobs?q=Platform')).json().jobs[0];
const seenBefore = before.fetched_at;
const seenBefore = before.fetched_at;
clock.t += 2 * 3600_000;
clock.t += 2 * 3600_000;
// The board's own text changed under the same id, as a board edits a listing.
// The board's own text changed under the same id, as a board edits a listing.
- t.boards.fixtures.remoteok[1].position = 'DevOps Engineer (Platform)';
- const remoteok = (await ingest.runAll()).find((o) => o.source === 'remoteok');
+ t.boards.fixtures.remotive[1].title = 'Platform Engineer (Ops)';
+ const remotive = (await ingest.runAll()).find((o) => o.source === 'remotive');
pinged.length = 0;
pinged.length = 0;
- assert.deepStrictEqual([remoteok.ok, remoteok.listings, remoteok.added, remoteok.updated], [true, 2, 0, 2]);
+ assert.deepStrictEqual([remotive.ok, remotive.listings, remotive.added, remotive.updated], [true, 3, 0, 3]);
assert.deepStrictEqual(pinged, [], 'a listing read again is not new: no IndexNow ping');
assert.deepStrictEqual(pinged, [], 'a listing read again is not new: no IndexNow ping');
- const after = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];
+ const after = (await t.get('/api/v1/jobs?q=Platform')).json().jobs[0];
assert.strictEqual(after.id, before.id, 'the id must not change when a board is read again');
assert.strictEqual(after.id, before.id, 'the id must not change when a board is read again');
- assert.strictEqual(after.title, 'DevOps Engineer (Platform)');
+ assert.strictEqual(after.title, 'Platform Engineer (Ops)');
assert.ok(after.fetched_at > seenBefore, 'fetched_at did not move');
assert.ok(after.fetched_at > seenBefore, 'fetched_at did not move');
- assert.strictEqual((await t.get("/api/v1/jobs?limit=100")).json().count, 8);
- t.boards.fixtures.remoteok[1].position = 'DevOps Engineer';
+ assert.strictEqual((await t.get('/api/v1/jobs?limit=100')).json().count, 3);
+ t.boards.fixtures.remotive[1].title = 'Platform Engineer';
});
});
await check('a listing the board stopped showing expires out, leaving the others', async () => {
await check('a listing the board stopped showing expires out, leaving the others', async () => {
- // Arbeitnow's page 2 goes quiet: one listing was there, is not any more.
- const gone = t.boards.fixtures.arbeitnowPage2.shift();
+ const gone = t.boards.fixtures.remotive.shift();
clock.t += 31 * DAY;
clock.t += 31 * DAY;
await ingest.runAll();
await ingest.runAll();
- assert.strictEqual((await store.countsBySource(s)).get('arbeitnow').count, 2);
- assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 0);
- const row = await s.db.maybe("SELECT * FROM work_listings WHERE source_id = $1 AND expired", [gone.slug]);
+ assert.strictEqual((await store.countsBySource(s)).get('remotive').count, 2);
+ assert.strictEqual((await t.get('/api/v1/jobs?q=Support')).json().count, 0);
+ const row = await s.db.maybe('SELECT * FROM work_listings WHERE source_id = $1 AND expired', [String(gone.id)]);
assert.ok(row, 'the expired listing was deleted rather than kept');
assert.ok(row, 'the expired listing was deleted rather than kept');
assert.strictEqual((await t.get(`/jobs/${row.id}`)).status, 404);
assert.strictEqual((await t.get(`/jobs/${row.id}`)).status, 404);
// And it comes back if the board shows it again.
// And it comes back if the board shows it again.
- t.boards.fixtures.arbeitnowPage2.push(gone);
+ t.boards.fixtures.remotive.push(gone);
await ingest.runAll();
await ingest.runAll();
- assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 1);
+ assert.strictEqual((await t.get('/api/v1/jobs?q=Support')).json().count, 1);
});
});
- await check('a board that fails is recorded, says why, and does not stop the others', async () => {
+ await check('a failed read is recorded, says why, and never empties the board', async () => {
clock.t += 3600_000;
clock.t += 3600_000;
t.boards.state.remotive = 'fail';
t.boards.state.remotive = 'fail';
- t.boards.state.remoteok = 'html';
const out = await ingest.runAll();
const out = await ingest.runAll();
assert.strictEqual(out.find((o) => o.source === 'remotive').ok, false);
assert.strictEqual(out.find((o) => o.source === 'remotive').ok, false);
- assert.strictEqual(out.find((o) => o.source === 'arbeitnow').ok, true, 'one board failing stopped another');
const src = (await t.get('/api/v1/sources')).json();
const src = (await t.get('/api/v1/sources')).json();
const remotive = src.sources.find((x) => x.id === 'remotive');
const remotive = src.sources.find((x) => x.id === 'remotive');
assert.strictEqual(remotive.last_fetch.ok, false);
assert.strictEqual(remotive.last_fetch.ok, false);
assert.match(remotive.last_fetch.error, /500/);
assert.match(remotive.last_fetch.error, /500/);
assert.ok(remotive.last_fetch.finished_at, 'the failure was not timestamped');
assert.ok(remotive.last_fetch.finished_at, 'the failure was not timestamped');
- assert.match(src.sources.find((x) => x.id === 'remoteok').last_fetch.error, /not JSON/);
- // What was already gathered stays: a failed read never empties the board.
assert.strictEqual(remotive.listings, 3);
assert.strictEqual(remotive.listings, 3);
t.boards.reset();
t.boards.reset();
});
});
@@ -71,21 +71,22 @@ const jobTypes = require('../server/jobs/job-types');
@@ -71,21 +71,22 @@ const jobTypes = require('../server/jobs/job-types');
assert.strictEqual(jobTypes.canonicalKey(''), null);
assert.strictEqual(jobTypes.canonicalKey(''), null);
});
});
- const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });
+ const t = await boot();
try {
try {
await t.ctx.ingest.runAll();
await t.ctx.ingest.runAll();
// Two listings in the boards' own words: a German one and a French one.
// Two listings in the boards' own words: a German one and a French one.
- t.boards.add('arbeitnow', {
- slug: 'support-werkstudent-berlin',
- company_name: 'Nordlicht Systems',
+ t.boards.add('remotive', {
+ id: 1900950,
+ url: 'https://remotive.com/remote-jobs/support/support-werkstudent-1900950',
title: 'Support Werkstudent',
title: 'Support Werkstudent',
- description: '<p>Support in Berlin.</p>',
- remote: false,
- url: 'https://www.arbeitnow.com/view/support-werkstudent-berlin-45678',
+ company_name: 'Nordlicht Systems',
+ category: 'Customer Support',
tags: ['support'],
tags: ['support'],
- job_types: ['Werkstudent', 'hilfstätigkeit'],
- location: 'Berlin, Germany',
- created_at: 1791800000,
+ job_type: 'Werkstudent',
+ publication_date: '2026-10-02T09:00:00',
+ candidate_required_location: 'Berlin, Germany',
+ salary: '',
+ description: '<p>Support in Berlin.</p>',
});
});
t.boards.add('remotive', {
t.boards.add('remotive', {
id: 1900900, url: 'https://remotive.com/remote-jobs/other/consultant-cdi-1900900',
id: 1900900, url: 'https://remotive.com/remote-jobs/other/consultant-cdi-1900900',
@@ -124,7 +125,7 @@ const jobTypes = require('../server/jobs/job-types');
@@ -124,7 +125,7 @@ const jobTypes = require('../server/jobs/job-types');
assert.ok(partTime.jobs.some((j) => /Support Werkstudent/.test(j.title)), 'the German listing is not part time');
assert.ok(partTime.jobs.some((j) => /Support Werkstudent/.test(j.title)), 'the German listing is not part time');
// And it is counted under both.
// And it is counted under both.
const fullTime = (await t.get('/api/v1/jobs?type=full-time')).json();
const fullTime = (await t.get('/api/v1/jobs?type=full-time')).json();
- assert.ok(fullTime.jobs.some((j) => /Frontend Engineer/.test(j.title)));
+ assert.ok(fullTime.jobs.some((j) => /Platform Engineer/.test(j.title)));
});
});
await check('a type in a board\'s own words still works, as the plain search it always was', async () => {
await check('a type in a board\'s own words still works, as the plain search it always was', async () => {
@@ -13,20 +13,28 @@ const ldBlocks = (html) => [...html.matchAll(/<script type="application\/ld\+jso
@@ -13,20 +13,28 @@ const ldBlocks = (html) => [...html.matchAll(/<script type="application\/ld\+jso
const first = async (t, path) => (await t.get(path)).json();
const first = async (t, path) => (await t.get(path)).json();
(async () => {
(async () => {
- const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });
+ const t = await boot();
try {
try {
+ // A listing with a city and a hostile title, so the home's city list and the escaping are exercised.
+ t.boards.add('remotive', {
+ id: 1900970, url: 'https://remotive.com/remote-jobs/software-dev/frontend-engineer-1900970',
+ title: 'Frontend Engineer <script>alert("xss")</script>', company_name: 'Helles Werk',
+ category: 'Software Development', tags: ['typescript', 'css'], job_type: 'permanent, full_time',
+ publication_date: '2026-10-03T09:00:00', candidate_required_location: 'Berlin, Germany', salary: '',
+ description: '<div>Build the design system.<script>window.__pwned = true</script></div><div>TypeScript, CSS.</div>',
+ });
await t.ctx.ingest.runAll();
await t.ctx.ingest.runAll();
await check('search: text matches the title, the company and the tags; results are newest first', async () => {
await check('search: text matches the title, the company and the tags; results are newest first', async () => {
- const byTitle = await first(t, '/api/v1/jobs?q=devops');
- assert.deepStrictEqual(byTitle.jobs.map((j) => j.title), ['DevOps Engineer']);
- const byCompany = await first(t, '/api/v1/jobs?q=Skyward');
- assert.deepStrictEqual(byCompany.jobs.map((j) => j.company), ['Skyward Cloud']);
- // The term is found in a tag and in the excerpt, so both listings answer.
+ const byTitle = await first(t, '/api/v1/jobs?q=platform');
+ assert.deepStrictEqual(byTitle.jobs.map((j) => j.title), ['Platform Engineer']);
+ const byCompany = await first(t, '/api/v1/jobs?q=Meridian');
+ assert.deepStrictEqual(byCompany.jobs.map((j) => j.company), ['Meridian Rail']);
+ // The term is found in a tag and in the excerpt.
const byTag = await first(t, '/api/v1/jobs?q=terraform');
const byTag = await first(t, '/api/v1/jobs?q=terraform');
- assert.deepStrictEqual(byTag.jobs.map((j) => j.company).sort(), ['Meridian Rail', 'Skyward Cloud']);
+ assert.deepStrictEqual(byTag.jobs.map((j) => j.company), ['Meridian Rail']);
// Case does not matter, and a % a caller typed is a character, not a wildcard.
// Case does not matter, and a % a caller typed is a character, not a wildcard.
- assert.strictEqual((await first(t, '/api/v1/jobs?q=DEVOPS')).count, 1);
+ assert.strictEqual((await first(t, '/api/v1/jobs?q=PLATFORM')).count, 1);
assert.strictEqual((await first(t, '/api/v1/jobs?q=%25')).count, 0);
assert.strictEqual((await first(t, '/api/v1/jobs?q=%25')).count, 0);
const all = await first(t, '/api/v1/jobs?limit=100');
const all = await first(t, '/api/v1/jobs?limit=100');
const dates = all.jobs.map((j) => j.posted_at);
const dates = all.jobs.map((j) => j.posted_at);
@@ -37,18 +45,19 @@ const first = async (t, path) => (await t.get(path)).json();
@@ -37,18 +45,19 @@ const first = async (t, path) => (await t.get(path)).json();
const remote = await first(t, '/api/v1/jobs?remote=true');
const remote = await first(t, '/api/v1/jobs?remote=true');
assert.ok(remote.count > 0);
assert.ok(remote.count > 0);
assert.ok(remote.jobs.every((j) => j.remote === true));
assert.ok(remote.jobs.every((j) => j.remote === true));
+ // Remotive's board carries remote work only, so nothing is on-site.
const onSite = await first(t, '/api/v1/jobs?remote=false');
const onSite = await first(t, '/api/v1/jobs?remote=false');
- assert.ok(onSite.count > 0);
- assert.ok(onSite.jobs.every((j) => j.remote === false));
- assert.strictEqual(remote.count + onSite.count, (await first(t, '/api/v1/jobs?limit=100')).count);
+ assert.strictEqual(onSite.count, 0);
+ const everything = (await first(t, '/api/v1/jobs?limit=100')).count;
+ assert.strictEqual(remote.count, everything);
const contract = await first(t, '/api/v1/jobs?type=contract');
const contract = await first(t, '/api/v1/jobs?type=contract');
assert.deepStrictEqual(contract.jobs.map((j) => j.title), ['Content Marketer']);
assert.deepStrictEqual(contract.jobs.map((j) => j.title), ['Content Marketer']);
assert.strictEqual(contract.facets.matches, 1, 'the count must be what the filters match');
assert.strictEqual(contract.facets.matches, 1, 'the count must be what the filters match');
- assert.strictEqual(contract.facets.scope, 8, 'the type facet must not narrow its own scope');
+ assert.strictEqual(contract.facets.scope, everything, 'the type facet must not narrow its own scope');
const germany = await first(t, '/api/v1/jobs?location=Germany');
const germany = await first(t, '/api/v1/jobs?location=Germany');
- assert.deepStrictEqual(germany.jobs.map((j) => j.title).sort(), ['Data Analyst', 'Senior Backend Engineer (Node.js)']);
+ assert.deepStrictEqual(germany.jobs.map((j) => j.title), ['Frontend Engineer <script>alert("xss")</script>']);
const combined = await first(t, '/api/v1/jobs?q=engineer&remote=true');
const combined = await first(t, '/api/v1/jobs?q=engineer&remote=true');
assert.ok(combined.count >= 1 && combined.jobs.every((j) => /engineer/i.test(j.title)));
assert.ok(combined.count >= 1 && combined.jobs.every((j) => /engineer/i.test(j.title)));
@@ -77,9 +86,9 @@ const first = async (t, path) => (await t.get(path)).json();
@@ -77,9 +86,9 @@ const first = async (t, path) => (await t.get(path)).json();
const html = (await t.get('/jobs')).text;
const html = (await t.get('/jobs')).text;
assert.strictEqual((await t.get('/jobs')).status, 200);
assert.strictEqual((await t.get('/jobs')).status, 200);
for (const [name, url] of [
for (const [name, url] of [
- ['Arbeitnow', 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345'],
+ ['Remotive', 'https://remotive.com/remote-jobs/software-dev/support-engineer-1900123'],
['Remotive', 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789'],
['Remotive', 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789'],
- ['RemoteOK', 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001'],
+ ['Remotive', 'https://remotive.com/remote-jobs/marketing/content-marketer-1900456'],
]) {
]) {
assert.ok(html.includes(name), `the source ${name} is not named on the list`);
assert.ok(html.includes(name), `the source ${name} is not named on the list`);
assert.ok(html.includes(`href="${url}"`), `the original listing ${url} is not linked on the list`);
assert.ok(html.includes(`href="${url}"`), `the original listing ${url} is not linked on the list`);
@@ -92,28 +101,28 @@ const first = async (t, path) => (await t.get(path)).json();
@@ -92,28 +101,28 @@ const first = async (t, path) => (await t.get(path)).json();
});
});
await check('a search is an Atom feed with the same filters, and its page offers alerts through OpenVibe.Watch', async () => {
await check('a search is an Atom feed with the same filters, and its page offers alerts through OpenVibe.Watch', async () => {
- const r = await t.get('/jobs.atom?q=devops&remote=true');
+ const r = await t.get('/jobs.atom?q=platform&remote=true');
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.match(r.headers.get('content-type'), /^application\/atom\+xml/);
assert.match(r.headers.get('content-type'), /^application\/atom\+xml/);
assert.strictEqual(r.headers.get('cache-control'), 'public, max-age=300');
assert.strictEqual(r.headers.get('cache-control'), 'public, max-age=300');
assert.match(r.text, /^<\?xml version="1\.0" encoding="utf-8"\?>\n<feed xmlns="http:\/\/www\.w3\.org\/2005\/Atom">/);
assert.match(r.text, /^<\?xml version="1\.0" encoding="utf-8"\?>\n<feed xmlns="http:\/\/www\.w3\.org\/2005\/Atom">/);
const entries = [...r.text.matchAll(/<entry>([\s\S]*?)<\/entry>/g)].map((m) => m[1]);
const entries = [...r.text.matchAll(/<entry>([\s\S]*?)<\/entry>/g)].map((m) => m[1]);
assert.strictEqual(entries.length, 1, 'only the matching listing');
assert.strictEqual(entries.length, 1, 'only the matching listing');
- assert.match(entries[0], /<title>DevOps Engineer at Skyward Cloud<\/title>/);
+ assert.match(entries[0], /<title>Platform Engineer at Meridian Rail<\/title>/);
assert.match(entries[0], /<link rel="alternate" type="text\/html" href="[^"]+\/jobs\/[^"]+"\/>/);
assert.match(entries[0], /<link rel="alternate" type="text\/html" href="[^"]+\/jobs\/[^"]+"\/>/);
- assert.match(entries[0], /<author><name>RemoteOK<\/name><\/author>/);
- assert.match(r.text, /<link rel="self" href="[^"]*\/jobs\.atom\?q=devops&remote=true"\/>/);
+ assert.match(entries[0], /<author><name>Remotive<\/name><\/author>/);
+ assert.match(r.text, /<link rel="self" href="[^"]*\/jobs\.atom\?q=platform&remote=true"\/>/);
const all = await t.get('/jobs.atom');
const all = await t.get('/jobs.atom');
const ids = [...all.text.matchAll(/<updated>([^<]+)<\/updated><author>/g)].map((m) => m[1]);
const ids = [...all.text.matchAll(/<updated>([^<]+)<\/updated><author>/g)].map((m) => m[1]);
assert.ok(ids.length >= 3 && ids.join() === [...ids].sort().reverse().join(), 'every listing, newest first');
assert.ok(ids.length >= 3 && ids.join() === [...ids].sort().reverse().join(), 'every listing, newest first');
// The page links its feed, and Get alerts opens Watch's feed form with it filled in.
// The page links its feed, and Get alerts opens Watch's feed form with it filled in.
- const page = (await t.get('/jobs?q=devops&remote=true')).text;
- assert.ok(page.includes('<link rel="alternate" type="application/atom+xml" title="Jobs “devops” remote" href="/jobs.atom?q=devops&remote=true">'), 'the feed link in the head');
+ const page = (await t.get('/jobs?q=platform&remote=true')).text;
+ assert.ok(page.includes('<link rel="alternate" type="application/atom+xml" title="Jobs “platform” remote" href="/jobs.atom?q=platform&remote=true">'), 'the feed link in the head');
const alert = /href="(https:\/\/openvibe\.watch\/watches\/new\?[^"]+)">Get alerts for this search/.exec(page);
const alert = /href="(https:\/\/openvibe\.watch\/watches\/new\?[^"]+)">Get alerts for this search/.exec(page);
assert.ok(alert, 'the alerts link');
assert.ok(alert, 'the alerts link');
const u = new URL(alert[1].replace(/&/g, '&'));
const u = new URL(alert[1].replace(/&/g, '&'));
assert.deepStrictEqual([u.searchParams.get('template'), new URL(u.searchParams.get('url')).pathname + new URL(u.searchParams.get('url')).search, u.searchParams.get('name')],
assert.deepStrictEqual([u.searchParams.get('template'), new URL(u.searchParams.get('url')).pathname + new URL(u.searchParams.get('url')).search, u.searchParams.get('name')],
- ['feed', '/jobs.atom?q=devops&remote=true', 'Jobs: “devops” remote']);
+ ['feed', '/jobs.atom?q=platform&remote=true', 'Jobs: “platform” remote']);
});
});
await check('one listing: the board\'s fields, the excerpt, the source and "read the full listing on" link', async () => {
await check('one listing: the board\'s fields, the excerpt, the source and "read the full listing on" link', async () => {
@@ -138,19 +147,19 @@ const first = async (t, path) => (await t.get(path)).json();
@@ -138,19 +147,19 @@ const first = async (t, path) => (await t.get(path)).json();
});
});
await check('the listing page carries a valid JobPosting whose url is the original, not this page', async () => {
await check('the listing page carries a valid JobPosting whose url is the original, not this page', async () => {
- const job = (await first(t, '/api/v1/jobs?q=DevOps')).jobs[0];
+ const job = (await first(t, '/api/v1/jobs?q=Platform Engineer')).jobs[0];
const html = (await t.get(`/jobs/${job.id}`)).text;
const html = (await t.get(`/jobs/${job.id}`)).text;
assert.strictEqual((html.match(/application\/ld\+json/g) || []).length, 1, 'one JSON-LD script per page');
assert.strictEqual((html.match(/application\/ld\+json/g) || []).length, 1, 'one JSON-LD script per page');
const blocks = ldBlocks(html);
const blocks = ldBlocks(html);
const posting = blocks.find((b) => b['@type'] === 'JobPosting');
const posting = blocks.find((b) => b['@type'] === 'JobPosting');
assert.ok(posting, `no JobPosting node: ${JSON.stringify(blocks.map((b) => b['@type']))}`);
assert.ok(posting, `no JobPosting node: ${JSON.stringify(blocks.map((b) => b['@type']))}`);
- assert.strictEqual(posting.title, 'DevOps Engineer');
+ assert.strictEqual(posting.title, 'Platform Engineer');
assert.strictEqual(posting.url, job.url);
assert.strictEqual(posting.url, job.url);
assert.notStrictEqual(posting.url, `https://openvibe.work/jobs/${job.id}`);
assert.notStrictEqual(posting.url, `https://openvibe.work/jobs/${job.id}`);
- assert.strictEqual(posting.hiringOrganization.name, 'Skyward Cloud');
+ assert.strictEqual(posting.hiringOrganization.name, 'Meridian Rail');
assert.strictEqual(posting.jobLocationType, 'TELECOMMUTE');
assert.strictEqual(posting.jobLocationType, 'TELECOMMUTE');
- assert.match(posting.datePosted, /^2026-10-01T/);
- assert.strictEqual(posting.identifier.name, 'remoteok');
+ assert.match(posting.datePosted, /^2026-09-29T/);
+ assert.strictEqual(posting.identifier.name, 'remotive');
assert.ok(posting.description.length <= 320);
assert.ok(posting.description.length <= 320);
assert.ok(posting['@context'] === 'https://schema.org');
assert.ok(posting['@context'] === 'https://schema.org');
// The search page lists many listings but is not one job, so it carries no JobPosting.
// The search page lists many listings but is not one job, so it carries no JobPosting.
@@ -188,7 +197,7 @@ const first = async (t, path) => (await t.get(path)).json();
@@ -188,7 +197,7 @@ const first = async (t, path) => (await t.get(path)).json();
const api = await first(t, '/api/v1/sources');
const api = await first(t, '/api/v1/sources');
assert.deepStrictEqual(api.sources.map((s) => s.id).sort(), ['arbeitnow', 'hn-whoishiring', 'remoteok', 'remotive', 'weworkremotely']);
assert.deepStrictEqual(api.sources.map((s) => s.id).sort(), ['arbeitnow', 'hn-whoishiring', 'remoteok', 'remotive', 'weworkremotely']);
// A board OpenVibe.Sources feeds has no read of its own here, so it is not asked for one.
// A board OpenVibe.Sources feeds has no read of its own here, so it is not asked for one.
- for (const src of api.sources.filter((x) => !['weworkremotely', 'hn-whoishiring'].includes(x.id))) {
+ for (const src of api.sources.filter((x) => x.id === 'remotive')) {
assert.ok(src.terms.length > 40, `${src.id} has no terms`);
assert.ok(src.terms.length > 40, `${src.id} has no terms`);
assert.match(src.terms, /link|attribute|mention|name/i, `${src.id}'s terms do not state the attribution rule`);
assert.match(src.terms, /link|attribute|mention|name/i, `${src.id}'s terms do not state the attribution rule`);
assert.strictEqual(src.last_fetch.ok, true);
assert.strictEqual(src.last_fetch.ok, true);
@@ -228,7 +237,7 @@ const first = async (t, path) => (await t.get(path)).json();
@@ -228,7 +237,7 @@ const first = async (t, path) => (await t.get(path)).json();
assert.ok(r.text.includes('Find work.'), 'the hero');
assert.ok(r.text.includes('Find work.'), 'the hero');
assert.match(r.text, /action="\/jobs"/, 'the search box');
assert.match(r.text, /action="\/jobs"/, 'the search box');
assert.ok(r.text.includes('Latest remote jobs'));
assert.ok(r.text.includes('Latest remote jobs'));
- assert.ok(r.text.includes('DevOps Engineer'), 'no listing on the home page');
+ assert.ok(r.text.includes('Platform Engineer'), 'no listing on the home page');
for (const [name, url] of [['Arbeitnow', 'https://www.arbeitnow.com/'], ['Remotive', 'https://remotive.com/'], ['RemoteOK', 'https://remoteok.com/']]) {
for (const [name, url] of [['Arbeitnow', 'https://www.arbeitnow.com/'], ['Remotive', 'https://remotive.com/'], ['RemoteOK', 'https://remoteok.com/']]) {
assert.ok(r.text.includes(`href="${url}"`), `the source ${name} is not linked on the home page`);
assert.ok(r.text.includes(`href="${url}"`), `the source ${name} is not linked on the home page`);
}
}
@@ -250,7 +259,7 @@ const first = async (t, path) => (await t.get(path)).json();
@@ -250,7 +259,7 @@ const first = async (t, path) => (await t.get(path)).json();
});
});
await check('search is limited per caller, and past it the API says so with Retry-After', async () => {
await check('search is limited per caller, and past it the API says so with Retry-After', async () => {
- const busy = await boot({ callerLimits: true, limitsNow: () => Date.now(), env: { WORK_INGEST_PAGES: '1' } });
+ const busy = await boot({ callerLimits: true, limitsNow: () => Date.now() });
try {
try {
await busy.ctx.ingest.runAll();
await busy.ctx.ingest.runAll();
let limited = null;
let limited = null;
@@ -32,10 +32,17 @@ const cardOf = (html, id) => {
@@ -32,10 +32,17 @@ const cardOf = (html, id) => {
};
};
(async () => {
(async () => {
- const t = await boot({ env: { WORK_INGEST_PAGES: '2', WORK_DISCLOSURE_SAMPLE: '100' } });
+ const t = await boot({ env: { WORK_DISCLOSURE_SAMPLE: '100' } });
const kim = t.network.addUser('kim');
const kim = t.network.addUser('kim');
const SAME = { 'sec-fetch-site': 'same-origin' };
const SAME = { 'sec-fetch-site': 'same-origin' };
try {
try {
+ // A second listing with two types in the board's own words, so both kinds are on the page.
+ t.boards.add('remotive', {
+ id: 1900960, url: 'https://remotive.com/remote-jobs/software-dev/frontend-engineer-1900960',
+ title: 'Frontend Engineer', company_name: 'Helles Werk', category: 'Software Development', tags: ['typescript', 'css'],
+ job_type: 'permanent, full_time', publication_date: '2026-10-01T10:00:00',
+ candidate_required_location: 'Remote (EU)', salary: '', description: '<p>Build the design system.</p>',
+ });
await t.ctx.ingest.runAll();
await t.ctx.ingest.runAll();
await check('outcome 1: the salary as the board wrote it, or that the board wrote none', async () => {
await check('outcome 1: the salary as the board wrote it, or that the board wrote none', async () => {
@@ -77,7 +84,7 @@ const cardOf = (html, id) => {
@@ -77,7 +84,7 @@ const cardOf = (html, id) => {
assert.ok(page.includes(`Of the last ${expected.n} listings we fetched, ${expected.salaried} included a salary (${pct}%)`),
assert.ok(page.includes(`Of the last ${expected.n} listings we fetched, ${expected.salaried} included a salary (${pct}%)`),
'the disclosure rate is not the one the database gives');
'the disclosure rate is not the one the database gives');
// The sample size is configuration, not a literal on the page: with a smaller sample the line follows.
// The sample size is configuration, not a literal on the page: with a smaller sample the line follows.
- const small = await boot({ env: { WORK_INGEST_PAGES: '2', WORK_DISCLOSURE_SAMPLE: '3' } });
+ const small = await boot({ env: { WORK_DISCLOSURE_SAMPLE: '3' } });
try {
try {
await small.ctx.ingest.runAll();
await small.ctx.ingest.runAll();
const want = await small.ctx.s.db.one(
const want = await small.ctx.s.db.one(
new file mode 100644
new file mode 100644
@@ -0,0 +1,77 @@
@@ -0,0 +1,77 @@
+'use strict';
+/**
+ * RemoteOK through OpenVibe.Sources (server/ingest/from-sources.js; plan 2026-10-22, track F): the board's API is
+ * read by Sources (source remoteok-job-board) under its robots and terms, and Work follows the change feed. A listing
+ * must become the row Work's own RemoteOK fetch made: the same id (the board's own id, so the same listing URL), the
+ * position as its title, the company and place, tags, and the salary bounds written the way the board's fetch wrote
+ * them. A job Sources retires expires here; Work never calls remoteok.com itself.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const store = require('../server/jobs/store');
+
+const feed = [];
+let seq = 0;
+function push(item) { feed.push({ ...item, change_seq: ++seq }); }
+const sourcesClient = {
+ enabled: true,
+ async listItems({ after = 0, limit = 100 } = {}) {
+ const page = feed.filter((i) => i.change_seq > after).slice(0, limit);
+ return { items: page, next_after: page.length ? page[page.length - 1].change_seq : after, more: false };
+ },
+};
+const job = (identity, over = {}) => ({
+ id: `itm_remoteok_${identity}`, source_key: 'remoteok-job-board', category: 'jobs', kind: 'job', identity,
+ canonical_url: `https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-${identity}`,
+ title: 'DevOps Engineer', summary: 'Kubernetes, Terraform, on-call.',
+ authors: [], published_at: '2026-11-10T08:00:00.000Z', source_updated_at: null,
+ fields: {
+ company: 'Skyward Cloud', location: 'Worldwide', job_type: null,
+ salary_min: 90000, salary_max: 130000, tags: ['devops', 'kubernetes'],
+ },
+ revision: 1, provenance: { retrieved_at: '2026-11-12T06:00:00.000Z' }, removed: null, ...over,
+});
+
+(async () => {
+ const clock = { t: Date.parse('2026-11-12T06:00:00Z') };
+ const t = await boot({ now: () => clock.t, sourcesClient });
+ const s = t.ctx.s;
+ const from = t.ctx.fromSources;
+
+ try {
+ await check('RemoteOK is one of the boards followed, not one Work fetches', async () => {
+ assert.ok(from.sourceIds.includes('remoteok'), from.sourceIds.join(', '));
+ const out = await t.ctx.ingest.runAll();
+ assert.deepStrictEqual(out.find((o) => o.source === 'remoteok'), { source: 'remoteok', skipped: 'from_sources' });
+ });
+
+ await check('a Sources item becomes the listing RemoteOK\'s own fetch made', async () => {
+ push(job('1049001'));
+ const out = await from.pull();
+ assert.deepStrictEqual([out.applied, out.hold, out.removed], [1, 0, 0]);
+
+ const id = store.idOf('remoteok', '1049001');
+ const row = await s.db.maybe('SELECT * FROM work_listings WHERE id = $1', [id]);
+ assert.deepStrictEqual(
+ [row.source, row.source_id, row.title, row.company, row.location, row.remote, row.tags, row.job_type, row.salary, row.expired],
+ ['remoteok', '1049001', 'DevOps Engineer', 'Skyward Cloud', 'Worldwide', true, 'devops, kubernetes', null, '$90,000–$130,000 a year', false]);
+ assert.strictEqual(row.url, 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001');
+ assert.strictEqual(row.excerpt, 'Kubernetes, Terraform, on-call.');
+ const page = (await t.get('/api/v1/jobs?q=DevOps')).json();
+ assert.deepStrictEqual(page.jobs.map((j) => [j.id, j.source, j.salary]), [[id, 'remoteok', '$90,000–$130,000 a year']]);
+ assert.strictEqual((await store.sourceFetches(s)).get('remoteok').listings, 1);
+ });
+
+ await check('a RemoteOK job Sources retires expires here', async () => {
+ push(job('1049001', { revision: 2, removed: { at: '2026-11-20T06:00:00.000Z', reason: 'gone_from_source' } }));
+ const out = await from.pull();
+ assert.deepStrictEqual([out.applied, out.removed], [0, 1]);
+ const row = await s.db.maybe('SELECT expired FROM work_listings WHERE source_id = $1', ['1049001']);
+ assert.strictEqual(row.expired, true);
+ assert.strictEqual((await t.get('/api/v1/jobs?q=DevOps')).json().jobs.length, 0);
+ });
+ } finally {
+ await t.close();
+ }
+ done();
+})().catch((err) => { console.error(err); process.exit(1); });
@@ -13,7 +13,7 @@ const { boot, check, done } = require('./helpers/boot');
@@ -13,7 +13,7 @@ const { boot, check, done } = require('./helpers/boot');
// The clock starts at the real now: the session is a Network token, and this service checks it against its own
// The clock starts at the real now: the session is a Network token, and this service checks it against its own
// clock, so a test that moves time must move only a little or the token stops verifying.
// clock, so a test that moves time must move only a little or the token stops verifying.
const clock = { t: Date.now() };
const clock = { t: Date.now() };
- const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });
+ const t = await boot({ now: () => clock.t });
const kim = t.network.addUser('kim');
const kim = t.network.addUser('kim');
const lee = t.network.addUser('lee');
const lee = t.network.addUser('lee');
const SAME = { 'sec-fetch-site': 'same-origin' };
const SAME = { 'sec-fetch-site': 'same-origin' };
@@ -43,14 +43,14 @@ const { boot, check, done } = require('./helpers/boot');
@@ -43,14 +43,14 @@ const { boot, check, done } = require('./helpers/boot');
const empty = await t.get('/api/v1/saved-searches', { as: kim, json: { q: ' ' }, headers: SAME });
const empty = await t.get('/api/v1/saved-searches', { as: kim, json: { q: ' ' }, headers: SAME });
assert.strictEqual(empty.status, 422);
assert.strictEqual(empty.status, 422);
assert.strictEqual(empty.json().code, 'work.saved_searches.empty');
assert.strictEqual(empty.json().code, 'work.saved_searches.empty');
- const remote = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'devops', remote: 'yes', location: 'Worldwide' }, headers: SAME });
+ const remote = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'platform', remote: 'yes' }, headers: SAME });
assert.strictEqual(remote.status, 201, remote.text);
assert.strictEqual(remote.status, 201, remote.text);
const saved = remote.json();
const saved = remote.json();
- assert.deepStrictEqual(saved.filters, { remote: 'true', type: '', location: 'worldwide' });
- assert.strictEqual(saved.search_path, '/jobs?q=devops&remote=true&location=worldwide');
+ assert.deepStrictEqual(saved.filters, { remote: 'true', type: '', location: '' });
+ assert.strictEqual(saved.search_path, '/jobs?q=platform&remote=true');
// Running it again gives the same listings the search would.
// Running it again gives the same listings the search would.
const run = (await t.get(saved.search_path)).text;
const run = (await t.get(saved.search_path)).text;
- assert.ok(run.includes('DevOps Engineer'));
+ assert.ok(run.includes('Platform Engineer'));
});
});
await check('a signed-in write from another site is refused; a Bearer token is not (it is that person\'s own)', async () => {
await check('a signed-in write from another site is refused; a Bearer token is not (it is that person\'s own)', async () => {
@@ -14,7 +14,7 @@ const DAY = 86_400_000;
@@ -14,7 +14,7 @@ const DAY = 86_400_000;
(async () => {
(async () => {
const clock = { t: Date.parse('2026-10-01T06:00:00Z') };
const clock = { t: Date.parse('2026-10-01T06:00:00Z') };
- const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });
+ const t = await boot({ now: () => clock.t });
const { s, search, ingest } = t.ctx;
const { s, search, ingest } = t.ctx;
const outbox = async () => (await s.db.many('SELECT envelope FROM event_outbox ORDER BY id')).map((r) => r.envelope);
const outbox = async () => (await s.db.many('SELECT envelope FROM event_outbox ORDER BY id')).map((r) => r.envelope);
const valid = (env) => {
const valid = (env) => {
@@ -38,36 +38,35 @@ const DAY = 86_400_000;
@@ -38,36 +38,35 @@ const DAY = 86_400_000;
await search.sweep(); // the run's own sweep is fire-and-forget; this one waits (and shares or repeats it)
await search.sweep(); // the run's own sweep is fire-and-forget; this one waits (and shares or repeats it)
const envs = (await outbox()).map(valid);
const envs = (await outbox()).map(valid);
const listings = await s.db.many('SELECT * FROM work_listings ORDER BY id');
const listings = await s.db.many('SELECT * FROM work_listings ORDER BY id');
- assert.strictEqual(listings.length, 8);
- assert.strictEqual(envs.length, 8, 'one document per listing, sent once');
+ assert.strictEqual(listings.length, 3);
+ assert.strictEqual(envs.length, 3, 'one document per listing, sent once');
assert.ok(envs.every((e) => e.event_type === 'work.index_document.upserted'));
assert.ok(envs.every((e) => e.event_type === 'work.index_document.upserted'));
const byId = new Map(envs.map((e) => [e.payload.id, e.payload]));
const byId = new Map(envs.map((e) => [e.payload.id, e.payload]));
- const devops = listings.find((r) => r.title.includes('DevOps'));
- const doc = byId.get(devops.id);
- assert.strictEqual(doc.canonical_url, `https://openvibe.work/jobs/${devops.id}`);
- assert.strictEqual(doc.title, `${devops.title} at ${devops.company}`);
+ const platform = listings.find((r) => r.title.includes('Platform'));
+ const doc = byId.get(platform.id);
+ assert.strictEqual(doc.canonical_url, `https://openvibe.work/jobs/${platform.id}`);
+ assert.strictEqual(doc.title, `${platform.title} at ${platform.company}`);
assert.strictEqual(doc.authorship, 'imported');
assert.strictEqual(doc.authorship, 'imported');
assert.strictEqual(doc.visibility, 'public');
assert.strictEqual(doc.visibility, 'public');
assert.deepStrictEqual(doc.indexability, { decision: 'index', reasons: [] });
assert.deepStrictEqual(doc.indexability, { decision: 'index', reasons: [] });
- assert.strictEqual(doc.facets.source, 'remoteok');
+ assert.strictEqual(doc.facets.source, 'remotive');
assert.strictEqual(doc.facets.remote, true);
assert.strictEqual(doc.facets.remote, true);
- assert.match(doc.title, /Skyward Cloud/);
+ assert.match(doc.title, /Meridian Rail/);
// The snippet reads the summary and body together: the body leaves out the title, company, place and pay.
// The snippet reads the summary and body together: the body leaves out the title, company, place and pay.
- assert.ok(!doc.body.includes(devops.title) && !doc.body.includes('Skyward Cloud') && !doc.body.includes('$90,000'), `no repeat: ${doc.body.slice(0, 120)}`);
+ assert.ok(!doc.body.includes(platform.title) && !doc.body.includes('Meridian Rail') && !doc.body.includes('€70,000'), `no repeat: ${doc.body.slice(0, 120)}`);
const { meta } = require('../server/render/listing');
const { meta } = require('../server/render/listing');
- assert.strictEqual(meta({ ...devops, location: 'Berlin, Berlin, Deutschland' }).location, 'Berlin, Deutschland', 'a repeated place part is said once');
- assert.match(doc.summary, /\$90,000–\$130,000 a year/);
+ assert.strictEqual(meta({ ...platform, location: 'Berlin, Berlin, Deutschland' }).location, 'Berlin, Deutschland', 'a repeated place part is said once');
+ assert.match(doc.summary, /€70,000 – €85,000/);
assert.deepStrictEqual(doc.provenance.map((p) => [p.service, p.type, p.label, p.url]),
assert.deepStrictEqual(doc.provenance.map((p) => [p.service, p.type, p.label, p.url]),
- [['work', 'board_listing', 'RemoteOK', devops.url]], 'the board and the original listing travel with it');
- assert.ok(!JSON.stringify(envs).includes('personal use'), 'RemoteOK\'s legal notice is not a document');
+ [['work', 'board_listing', 'Remotive', platform.url]], 'the board and the original listing travel with it');
});
});
await check('a second sweep sends nothing; a changed listing is sent again at the next revision', async () => {
await check('a second sweep sends nothing; a changed listing is sent again at the next revision', async () => {
const before = (await outbox()).length;
const before = (await outbox()).length;
const again = await search.sweep();
const again = await search.sweep();
- assert.deepStrictEqual(again.job, { seen: 8, sent: 0, removed: 0, failed: 0 });
+ assert.deepStrictEqual(again.job, { seen: 3, sent: 0, removed: 0, failed: 0 });
assert.strictEqual((await outbox()).length, before);
assert.strictEqual((await outbox()).length, before);
- const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'arbeitnow' ORDER BY id LIMIT 1");
+ const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' ORDER BY id LIMIT 1");
await s.db.exec('UPDATE work_listings SET salary = $1 WHERE id = $2', ['€70,000 a year', one.id]);
await s.db.exec('UPDATE work_listings SET salary = $1 WHERE id = $2', ['€70,000 a year', one.id]);
const res = await search.sweep();
const res = await search.sweep();
assert.strictEqual(res.job.sent, 1);
assert.strictEqual(res.job.sent, 1);
@@ -78,7 +77,7 @@ const DAY = 86_400_000;
@@ -78,7 +77,7 @@ const DAY = 86_400_000;
});
});
await check('a listing that expires (its page answers 404) becomes a tombstone, once', async () => {
await check('a listing that expires (its page answers 404) becomes a tombstone, once', async () => {
- const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' ORDER BY id LIMIT 1");
+ const one = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' ORDER BY id DESC LIMIT 1");
await s.db.exec('UPDATE work_listings SET expired = true WHERE id = $1', [one.id]);
await s.db.exec('UPDATE work_listings SET expired = true WHERE id = $1', [one.id]);
assert.strictEqual((await t.get(`/jobs/${one.id}`)).status, 404);
assert.strictEqual((await t.get(`/jobs/${one.id}`)).status, 404);
const res = await search.sweep();
const res = await search.sweep();
@@ -88,7 +87,7 @@ const DAY = 86_400_000;
@@ -88,7 +87,7 @@ const DAY = 86_400_000;
assert.deepStrictEqual(last.payload, { type: 'job', id: one.id, revision: 2 });
assert.deepStrictEqual(last.payload, { type: 'job', id: one.id, revision: 2 });
assert.strictEqual((await search.sweep()).job.sent, 0, 'a tombstone is not re-sent');
assert.strictEqual((await search.sweep()).job.sent, 0, 'a tombstone is not re-sent');
// A row deleted outright is tombstoned too (exists()).
// A row deleted outright is tombstoned too (exists()).
- const gone = await s.db.one("SELECT * FROM work_listings WHERE source = 'arbeitnow' AND expired = false ORDER BY id DESC LIMIT 1");
+ const gone = await s.db.one("SELECT * FROM work_listings WHERE source = 'remotive' AND expired = false ORDER BY id DESC LIMIT 1");
await s.db.exec('DELETE FROM work_listings WHERE id = $1', [gone.id]);
await s.db.exec('DELETE FROM work_listings WHERE id = $1', [gone.id]);
assert.strictEqual((await search.sweep()).job.removed, 1);
assert.strictEqual((await search.sweep()).job.removed, 1);
});
});
@@ -36,7 +36,7 @@ const wwr = (identity, over = {}) => ({
@@ -36,7 +36,7 @@ const wwr = (identity, over = {}) => ({
(async () => {
(async () => {
const clock = { t: Date.parse('2026-10-30T08:00:00Z') };
const clock = { t: Date.parse('2026-10-30T08:00:00Z') };
- const t = await boot({ now: () => clock.t, env: { WORK_WWR_FROM: 'sources' }, sourcesClient });
+ const t = await boot({ now: () => clock.t, sourcesClient });
const s = t.ctx.s;
const s = t.ctx.s;
const from = t.ctx.fromSources;
const from = t.ctx.fromSources;