openvibers/openvibe.work Commits
OpenVibe.Work: job listings with their provenance, search and saved searches
Changes
@@@ -22,7 +22,31 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000OV_OAUTH_CLIENT_ID=workOV_OAUTH_CLIENT_SECRET=# OV_OAUTH_REDIRECT_URI=https://openvibe.work/auth/callback++# OV_NETWORK_ISSUER=https://openvibe.network++# The audience a person's own session token carries.++# OV_SESSION_AUDIENCE=openvibe.network++# The audience app, agent and service tokens for this service's API carry.# WORK_AUDIENCE=openvibe.work# optional: defaults to true in production# COOKIE_SECURE=true++++# The job-board ingest (server/ingest/). Off unless WORK_INGEST=on: a development run and the test suite read no board.++# Set it on the host, or no listing ever arrives. Each source is read at most every WORK_INGEST_INTERVAL_HOURS (6),++# which is exactly the four calls a day Remotive's terms allow.++#WORK_INGEST=on++#WORK_INGEST_INTERVAL_HOURS=6++#WORK_INGEST_TIMEOUT_MS=15000++#WORK_INGEST_PAGES=3++#WORK_INGEST_LIMIT=100++#WORK_RETENTION_DAYS=30++# The boards' public APIs. Point these at a stand-in server to test the ingest; nothing else in the service fetches++# anything, and never a URL a caller typed.++#WORK_ARBEITNOW_URL=https://www.arbeitnow.com/api/job-board-api++#WORK_REMOTIVE_URL=https://remotive.com/api/remote-jobs++#WORK_REMOTEOK_URL=https://remoteok.com/api++# The User-Agent the boards see. Defaults to OpenVibeWork/0.1 (+<BASE_URL>).++#WORK_USER_AGENT=++++# An embedded PGlite directory in development, when DATABASE_URL is unset.++#WORK_PGLITE_DIR=@@@ -2,6 +2,33 @@What changed in OpenVibe.Work, newest first. Each site also publishes its patch notes at /updates.++## 0.2.0 — 2026-10-08++++**Job listings, searchable, with their provenance — and saved searches.**++++- **Listings, gathered from open boards.** A background ingest reads Arbeitnow, Remotive and RemoteOK through their++ public APIs (server/jobs/sources.js, server/ingest/). Each board's terms are applied as written: the source is named++ and the original listing is linked on every page that shows it, and only a short plain-text excerpt is stored — never++ the full description. Every listing keeps the board's own id and URL, and a listing without both a title, a company,++ an id and an http(s) link back is refused rather than published. The ingest is off unless `WORK_INGEST=on`, calls a++ source at most every six hours (exactly the four a day Remotive's terms allow), upserts on `(source, source_id)`, and++ expires a listing no board has shown for 30 days.++- **Search** at `/jobs` (`q`, `remote`, `type`, `location`), newest first, 30 a page with a cursor, with facets for++ remote and job type. One listing at `/jobs/:id`, with a schema.org `JobPosting` whose `url` is the original.++- **Saved searches** for a signed-in person: save the search you are looking at, and `/saved` shows each one with the++ count of listings that appeared since you last looked (opening the page is what clears it).++- **The API** at `/api/v1`: `GET /jobs`, `GET /jobs/:id`, `GET /sources` (each board's terms as applied and its last++ fetch), and `POST`/`GET`/`DELETE /saved-searches`. Reading is public; a saved search is a person's.++- **Pages**: a home page with the search box, the latest remote listings and each source with its count; `/sources`,++ which states each board's terms as applied; and `/docs` for the API. A sitemap carrying the newest listings, and++ `llms.txt` pointing a crawler at the listings and the sources.++- **Limits**: the search and the saved-search writes have per-caller budgets of their own++ (`work.jobs.search`, `work.saved_searches.write`).++- **New tables**: `work_listings`, `work_saved_searches`, `work_source_fetches` (`migrations/0002_listings.sql`).++- **New tools**: `npm run ingest` reads every board once and exits.++- Also: `test/skeleton.test.js` now skips itself in a generated service (this repository, whose skeleton placeholders++ were already rewritten), saying so, rather than asserting against a rewrite that can no longer happen.++## 0.1.0 — 2026-10-08- **First release:** the service starts from the OpenVibe skeleton — sign-in with OpenVibe.Network (OAuth 2 + PKCE), server-rendered pages through the OpenVibe Frame, the `/api/v1` mount with per-caller limits, crawl artifacts, PostgreSQL migrations, the deploy files and the test suite.@@@ -2,42 +2,79 @@> Find work, with an agent on your side.--**Status:** skeleton. It runs, it is tested, and its pages and API are live — but the product itself is not written yet.++**Status:** the listings and the search are live (listings gathered from three public job boards, search, saved++searches, an API and the pages around them). The agent pillar — help preparing and tracking an application — is not++written yet.**Domain:** `openvibe.work` · **Port:** 4960 · **Service id:** `work` · **Env prefix:** `WORK`**License:** AGPL-3.0 (same as every OpenVibe service).--## What is already here++## What it does++++Job listings gathered from public job boards, each one carrying its provenance — the board it came from and a link++back to the original listing — searchable, filterable, and readable without an account. A signed-in person can keep a++search and see how many listings have appeared since they last looked. Nothing is republished: Work stores a short++plain-text excerpt and sends the reader to the board for the full listing and to apply.++++## What is here| Piece | Where | What it does ||---|---|---|| Config | [server/config.js](server/config.js) | Every value from the environment; `load(env)` is pure so tests build a config without touching `process.env` || App | [server/app.js](server/app.js) | helmet (CSP, frame-ancestors none), the Network session middleware, `/auth`, the legal pages, static assets, the `/api/v1` mount, the 404 and the error handler || Sign-in | [server/auth/sso.js](server/auth/sso.js) | OAuth 2 authorization code with PKCE (S256) against OpenVibe.Network; httpOnly `work_at` / `work_rt` cookies; `/auth/me` for the shared navbar |--| Signing key | [server/auth/keys.js](server/auth/keys.js) | The Network JWKS through `openvibe-sdk/auth`, kept fresh, verified offline |--| Who is calling | [server/http/principal.js](server/http/principal.js) | `req.principal`: a person, an app/agent/service with a capability, or anonymous |--| API | [server/http/api.js](server/http/api.js) | `GET /api/v1/ping`; the router, the problem+json errors and the guards are wired for the product's routes |--| Pages | [server/http/pages.js](server/http/pages.js), [server/render/](server/render/) | The home page and `/updates`, server-rendered through `openvibe-shared/shell`, readable without JavaScript |--| Discovery | [server/http/discovery.js](server/http/discovery.js) | `robots.txt`, `sitemap.xml`, `llms.txt`, `llms-full.txt` and the home page's JSON-LD |--| Limits | [server/http/caller-limits.js](server/http/caller-limits.js), [deploy/nginx/](deploy/nginx/) | Per-caller limits at the API and per-address limits at nginx |++| Sources | [server/jobs/sources.js](server/jobs/sources.js) | The three boards: where each is fetched, its terms, how its payload becomes a listing, and what is refused |++| Ingest | [server/ingest/](server/ingest/) | The background read (timer, User-Agent, 15 s timeout, upsert by source + the board's id, expiry) |++| Listings | [server/jobs/store.js](server/jobs/store.js), [migrations/0002_listings.sql](migrations/0002_listings.sql) | Search, facets, the cursor, saved searches, the fetch record |++| Text | [server/jobs/text.js](server/jobs/text.js) | HTML → plain text, and the ~300-character excerpt that is all we keep |++| API | [server/http/api.js](server/http/api.js) | `/api/v1`: jobs, one job, the sources, saved searches; problem+json errors |++| Pages | [server/http/pages.js](server/http/pages.js), [server/render/](server/render/) | `/`, `/jobs`, `/jobs/:id`, `/saved`, `/sources`, `/docs`, `/updates` — server-rendered, no JavaScript needed |++| Discovery | [server/http/discovery.js](server/http/discovery.js) | `robots.txt`, `sitemap.xml` (the newest listings too), `llms.txt`, `llms-full.txt`, JSON-LD |++| Limits | [server/http/caller-limits.js](server/http/caller-limits.js), [deploy/nginx/](deploy/nginx/) | Per-caller limits at the API (the search and the saved-search writes have their own) and per-address limits at nginx || Health | [server/observability.js](server/observability.js) | `/api/health`, a truthful `/api/ready` (the database is required) and Prometheus metrics on loopback only || Database | [server/db.js](server/db.js), [migrations/](migrations/) | PostgreSQL through `openvibe-sdk/db`; `NNNN_*.sql` applied at boot; PGlite in development |--| Process | [server/index.js](server/index.js) | Listens on `PORT`, and stops gracefully through `openvibe-sdk/service` |++| Process | [server/index.js](server/index.js) | Listens on `PORT`, starts the ingest when it is on, and stops gracefully through `openvibe-sdk/service` || Deploy | [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), [deploy/systemd/openvibe-work.service](deploy/systemd/openvibe-work.service) | nginx vhost and systemd unit (port 4960, `/opt/openvibe.work`, `/etc/openvibe/work.env`) |--| Tests | [test/](test/) | `npm test`: every `test/*.test.js` in its own process, on a temp PGlite database with an in-process mock of OpenVibe.Network |++| Tests | [test/](test/) | `npm test`: every `test/*.test.js` in its own process, on a temp PGlite database, with a mock OpenVibe.Network and a stand-in for each job board |## API| Route | Who | ||---|---|---|| `GET /api/v1/ping` | anyone | `{ ok: true, service: "work" }` |++| `GET /api/v1/jobs` | anyone | Search: `q`, `remote`, `type`, `location`, `limit` (to 100), `cursor` (the `next` of the previous page). Newest first, 30 a page, with facets |++| `GET /api/v1/jobs/:id` | anyone | One listing: the board's fields, the excerpt, the source, and `url` — the original to apply on |++| `GET /api/v1/sources` | anyone | The boards, each one's terms as applied, its listing count and its last fetch |++| `GET /api/v1/saved-searches` | a person | Your saved searches, each with `new_count` — new since you last looked |++| `POST /api/v1/saved-searches` | a person | `{ q?, remote?, type?, location? }` (at least one). 201 with the saved search |++| `DELETE /api/v1/saved-searches/:id` | a person | Forget one. 204 |++++**Who can call it:** reading the listings is public. A saved search belongs to a person, so those three routes need a++signed-in person — their Network token as a Bearer, or this site's session — and answer 401 otherwise. An app or agent++token cannot hold one (403): a saved search is not a project's. A write made with the session cookie must come from++`openvibe.work` itself (`request.cross_site` otherwise).--Everything else the site serves is a page. The product adds its routes in [server/http/api.js](server/http/api.js), naming--each route's capability in [server/http/principal.js](server/http/principal.js) (`CAPABILITIES`) and its numbers in--[server/http/caller-limits.js](server/http/caller-limits.js) (`BUDGETS`).++**Errors** are RFC 9457 `application/problem+json` with a stable `code` (`work.jobs.not_found`,++`work.saved_searches.empty`, `work.saved_searches.limit`, `rate_limited`, …).--**Who can call it:** a person, with their Network token as a Bearer or this site's session; or an app, agent or service--with a Network token for audience `openvibe.work` that holds the route's capability. A write made with the session cookie--must come from `openvibe.work` itself. **Errors** are RFC 9457 `application/problem+json` with a stable `code`.++## Data sources and their terms++++Each board is read through its own public API, and its terms are applied on every page that shows one of its listings++— they are also served by `GET /api/v1/sources` and written out on [/sources](server/http/pages.js).++++| Source | API | Terms, as applied here |++|---|---|---|++| **Arbeitnow** | `GET https://www.arbeitnow.com/api/job-board-api?page=N` | Free to use. Arbeitnow is named on every listing and the listing links to its page on arbeitnow.com. Its first pages (configurable) are read. |++| **Remotive** | `GET https://remotive.com/api/remote-jobs?limit=100` | Remotive must be named as the source and the listing linked. The API is called about four times a day — the six-hour interval is exactly that. |++| **RemoteOK** | `GET https://remoteok.com/api` | RemoteOK must be named as the source and the listing linked. The response's first element is a legal notice, not a listing, and is skipped. |++++What is stored per listing: `source`, the board's own id, the original `url`, title, company, location, the remote++flag, tags, job type, salary text when the board gives one, when it was posted, when we fetched it, when it was first++seen and last seen, and a **short plain-text excerpt** — never the full description. The listing page says *"Read the++full listing on <source>"* and links there; that is where a reader applies.++++Listings are upserted on `(source, source_id)` and **expire** — hidden from the search, the pages and the sitemap —++when no board has shown one for 30 days. An ingest never invents a listing: one without a title, a company, an id or++an http(s) URL to link back to is refused.## Configuration@@@ -45,12 +82,18 @@ See [.env.example](.env.example). Required in production: `OV_OAUTH_CLIENT_SECRENetwork), `BASE_URL`, `DATABASE_URL` and `DATABASE_DIRECT_URL`. The database is the only required readiness check; theNetwork signing key, the OAuth client and Valkey are optional (the service says so, per check, on `/api/ready`).++The ingest is **off** unless `WORK_INGEST=on`, so a development run and the test suite never call a job board. When it++is on, `WORK_INGEST_INTERVAL_HOURS` (6), `WORK_INGEST_PAGES` (3), `WORK_INGEST_LIMIT` (100), `WORK_INGEST_TIMEOUT_MS`++(15000), `WORK_RETENTION_DAYS` (30) and the per-source base URLs (`WORK_ARBEITNOW_URL`, `WORK_REMOTIVE_URL`,++`WORK_REMOTEOK_URL`) decide what it does.++## Development```bashnpm install--fnm exec --using=22 npm test # every test/*.test.js, on temp PGlite databases with a mock Network--fnm exec --using=22 npm run dev # http://localhost:4960++fnm exec --using=22 npm test # every test/*.test.js, on temp PGlite databases with a mock Network and stand-in boards++fnm exec --using=22 npm run dev # http://localhost:4960 (no ingest unless WORK_INGEST=on)++fnm exec --using=22 npm run ingest # read every board once, then exit```Without `DATABASE_URL` development uses an embedded PGlite database in `data/pglite` (one process only). `npm run@@@ -60,6 +103,7 @@ test:pg` runs the same suite through PostgreSQL and PgBouncer (see [.github/work- **Deploy:** `sudo ovhost deploy work` on the host (git checkout at `/opt/openvibe.work`, unit`openvibe-work.service` on 127.0.0.1:4960, env `/etc/openvibe/work.env`, database `ov_work` on the data role).++ Set `WORK_INGEST=on` in `/etc/openvibe/work.env` or no listing will ever arrive.- **nginx:** [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), installed with `ov-vhost-install`.- **Rollback:** ovhost puts the previous sha back by itself when `/api/ready` does not answer after the restart.- Register the service and its capabilities in **OpenVibe.Contracts** (`contracts-service: work` in CI) and with@@@ -72,8 +116,17 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md).- Session tokens are httpOnly cookies; a FedCM assertion or an app or service token is never a session.- Secrets live only in the env file; only environment variable names appear in code and docs, and no secret is logged.- Request bodies are never logged.--- Nothing in a request may decide a URL this service fetches: a caller's URL goes to OpenVibe.Tools, whose own guard-- decides what may be fetched.++- Nothing in a request may decide a URL this service fetches: the ingest fetches only the three base URLs from the++ configuration, with this service's User-Agent, a timeout and a size cap, and stores only http(s) listing URLs.++- A board's text is data, never markup: every value goes through the `html` tagged template, and a listing URL that++ is not http(s) is refused rather than linked.++++## Not yet++++- **The agent**: drafting, tailoring and tracking an application, with the person's approval at each step.++- **Alerts**: being told when a listing that fits appears — that comes through OpenVibe.Watch with the agent pillar.++ Today a saved search is a search you do not have to type again, plus a count of what is new.++- More boards, and a per-source schedule finer than "the same six hours for everyone".---@@@ -1,8 +1,8 @@{-- "stage": "skeleton",++ "stage": "beta","code": true,"wave": "W20",-- "plan": "OpenVibe.Work starts from the OpenVibe service skeleton; the product's own plan is written here when it is",++ "plan": "Job listings gathered from open boards with their provenance, search and saved searches; the agent pillar comes next.","runtime": "node 22, express 4, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9","port": "4960","domain": "openvibe.work",@@@ -14,21 +14,23 @@"openvibe-sdk v0.35.2","openvibe-shared v2.14.0"],-- "summary": "OpenVibe.Work — Find work, with an agent on your side. A new OpenVibe service on the shared skeleton, not deployed yet: sign-in with OpenVibe.Network (OAuth 2 + PKCE, httpOnly session cookies), server-rendered pages through the OpenVibe Frame, the /api/v1 mount with per-caller limits, crawl artifacts, PostgreSQL migrations applied at boot, the nginx and systemd deploy files and a test suite on PGlite with a mock Network.",++ "summary": "OpenVibe.Work gathers job listings from three public job boards (Arbeitnow, Remotive, RemoteOK) under their terms — the source is named and the original listing is linked on every page, and only a short excerpt is kept. It searches them with facets, shows one listing with a schema.org JobPosting whose url is the original, and lets a signed-in person keep a search and see what is new since they last looked. Public reads need no token.","works": [-- "Sign-in with OpenVibe.Network (PKCE S256, server-side code exchange, httpOnly session cookies)",-- "Server-rendered pages: the home page and /updates, through the OpenVibe Frame, complete without JavaScript",-- "Crawl artifacts: robots.txt, sitemap.xml, llms.txt, llms-full.txt and the home page's JSON-LD",-- "The API mount at /api/v1 with the problem+json error shape, the per-address limits and the per-caller limits",-- "Capability verification for app, agent and service tokens (audience openvibe.work) and for a person's own token",-- "Truthful readiness (/api/ready: the database is required) and Prometheus metrics on loopback",-- "PostgreSQL with migrations/NNNN_*.sql (openvibe-sdk/db); PGlite in development",-- "Deploy files: deploy/nginx/openvibe.work.conf and deploy/systemd/openvibe-work.service",-- "Tests on a temp PGlite database with an in-process mock of OpenVibe.Network"++ "The ingest: three boards read on a timer (WORK_INGEST=on, off in tests and development), each with its own User-Agent, 15-second timeout, upsert by (source, source_id) and expiry after 30 days unseen",++ "The listings: provenance kept and shown (source name, link to the original, when it was fetched), a short plain-text excerpt, never the full description",++ "Search: /jobs?q=&remote=&type=&location= newest first, 30 a page with a cursor, facets for remote and job type",++ "One listing: /jobs/:id with the board's fields, the excerpt, a 'Read the full listing on <source>' link and a JobPosting JSON-LD whose url is the original",++ "Saved searches for a signed-in person, each with the count of listings new since they last looked; /saved is where that count is cleared",++ "The API (/api/v1): jobs, one job, sources with their terms and last fetch, and saved searches",++ "The pages: home, /jobs, /jobs/:id, /saved, /sources, /docs and /updates — server-rendered, complete without JavaScript",++ "Discovery: a sitemap carrying the newest listings, llms.txt, robots.txt (saved searches excluded) and JSON-LD",++ "Per-caller limits on the search and on saved-search writes; problem+json errors with stable codes",++ "Tests: the ingest, the search, the pages and the saved searches against stand-in boards, on a temp PGlite database with a mock Network"],"notYet": [-- "The product itself: its tables and migrations, its capabilities, its routes in server/http/api.js and its pages in server/http/pages.js",-- "Its service and capability manifests in openvibe-contracts (contracts-service is work)",++ "The agent pillar: drafting, tailoring and tracking an application with the person's approval at each step",++ "Alerts: being told when a listing that fits appears — that comes through OpenVibe.Watch with the agent pillar; today a saved search is a repeatable search plus a count of what is new",++ "Its service and capability manifests in openvibe-contracts (contracts-service is work): the API is live, but no <work>.jobs.* capability is registered yet, so app and agent tokens cannot call it","A deployment: the service is not registered with OpenVibe.Services nor deployed by OpenVibe.Host"],"capabilitiesRegistered": false,@@@ -37,9 +39,15 @@"placeholderInSites": false,"features": ["Network sign-in",++ "job listings with provenance",++ "search and facets",++ "saved searches",++ "sources and their terms",++ "api","server-rendered pages",-- "api mount",++ "discovery files","limits and readiness",++ "background ingest","deploy files"],"updated": "2026-10-08"@@@ -96,6 +96,16 @@ server {proxy_pass http://127.0.0.1:4960;}++ # Saving or forgetting a saved search is a write made with a session cookie: the strict form zone, but reading++ # /saved is an ordinary page and keeps the page budget (the browser reloads it freely).++ location ~ ^/saved(/[A-Za-z0-9_-]+/delete)?$ {++ limit_except GET HEAD {++ limit_req zone=work_forms burst=10 nodelay;++ limit_req_status 429;++ }++ proxy_pass http://127.0.0.1:4960;++ }++# ── Prometheus metrics: host-local scrapers only (http://127.0.0.1:4960/metrics) ──location = /metrics { return 404; }nnew file mode 100644@@@ -0,0 +1,56 @@++-- phase: expand++-- OpenVibe.Work: the job listings gathered from the open job boards (server/jobs/sources.js), the searches a person++-- saved and when they last looked at each, and what each source's last fetch did.++--++-- A listing keeps its provenance: the source it came from and the original URL it must always link back to. Only a++-- short plain-text excerpt is stored — never the board's full description, which stays on the board. Nothing here is++-- ever a credential, a key or a token.++++CREATE TABLE work_listings (++ id text COLLATE "C" PRIMARY KEY, -- job_<base32>, derived from (source, source_id): stable++ source text COLLATE "C" NOT NULL, -- arbeitnow | remotive | remoteok++ source_id text COLLATE "C" NOT NULL, -- the board's own id for it++ url text NOT NULL, -- the original listing; always linked, never rehosted++ title text NOT NULL,++ company text NOT NULL,++ location text,++ remote boolean NOT NULL DEFAULT false,++ tags text NOT NULL DEFAULT '', -- comma-joined, for display++ job_type text COLLATE "C", -- the board's own words, lowercased where it repeats one++ salary text, -- the board's own salary text, when it gives one++ excerpt text NOT NULL, -- first ~300 characters of the description, plain text++ search_text text NOT NULL, -- lower(title, company, tags, location, excerpt)++ posted_at text COLLATE "C", -- ISO 8601 UTC, when the board says so++ sort_at text COLLATE "C" NOT NULL, -- posted_at, or fetched_at when the board gave none++ fetched_at text COLLATE "C" NOT NULL, -- when the ingest read the board++ first_seen_at text COLLATE "C" NOT NULL,++ last_seen_at text COLLATE "C" NOT NULL, -- touched on every fetch that still lists it++ expired boolean NOT NULL DEFAULT false -- not seen for the retention window: hidden, kept++);++-- One row per (source, its id): the ingest upserts on this.++CREATE UNIQUE INDEX work_listings_by_source ON work_listings (source, source_id);++CREATE INDEX work_listings_latest ON work_listings (expired, sort_at DESC, id DESC);++CREATE INDEX work_listings_by_last_seen ON work_listings (last_seen_at);++++-- A saved search belongs to the person who saved it (user:usr_…). The filters are the same four the search takes.++CREATE TABLE work_saved_searches (++ id text COLLATE "C" PRIMARY KEY, -- ssc_<ULID>++ subject text COLLATE "C" NOT NULL,++ query text NOT NULL DEFAULT '',++ remote text COLLATE "C" NOT NULL DEFAULT '', -- '' (any) | 'true' | 'false'++ type text COLLATE "C" NOT NULL DEFAULT '',++ location text NOT NULL DEFAULT '',++ created_at text COLLATE "C" NOT NULL,++ last_seen_at text COLLATE "C" NOT NULL -- listings first seen after this are "new"++);++CREATE INDEX work_saved_searches_by_subject ON work_saved_searches (subject, created_at DESC);++++-- What each source's last fetch did, so /sources can say when it last ran and how it went.++CREATE TABLE work_source_fetches (++ source text COLLATE "C" PRIMARY KEY,++ started_at text COLLATE "C" NOT NULL,++ finished_at text COLLATE "C",++ ok boolean NOT NULL DEFAULT false,++ listings integer NOT NULL DEFAULT 0,++ error text++);@@@ -1,12 +1,13 @@{"name": "openvibe-work",-- "version": "0.1.0",++ "version": "0.2.0","private": true,"description": "OpenVibe.Work — Find work, with an agent on your side.","main": "server/index.js","scripts": {"start": "node server/index.js","dev": "NODE_ENV=development node server/index.js",++ "ingest": "node scripts/ingest.js","test": "node test/run.js","test:pg": "WORK_TEST_STORE=pg node test/run.js"},@@@ -42,3 +42,43 @@ form.inline { display: inline; }.page .sc-ic .ov-icon { --ovi-size: 28px !important; }button.sc-btn { background: transparent; }button.sc-btn.sc-primary { border: 0; }++++/* ── The listings ───────────────────────────────────────── */++.job-search { display: grid; grid-template-columns: minmax(0, 2fr) minmax(0, 1fr) auto minmax(0, 1fr) auto; gap: 8px; align-items: center; margin: 12px 0; }++.job-search input, .job-search select { font: inherit; width: 100%; padding: 7px 9px; border-radius: 8px; border: 1px solid var(--border, #1f2d47); background: var(--bg-secondary, #0f1729); color: inherit; }++.job-clear { font-size: .9rem; }++@media (max-width: 720px) { .job-search { grid-template-columns: 1fr 1fr; } }++.job-facets { display: flex; flex-wrap: wrap; gap: 8px 24px; align-items: baseline; margin: 8px 0 4px; font-size: .92rem; }++.facets { list-style: none; display: flex; flex-wrap: wrap; gap: 6px; margin: 0; padding: 0; align-items: baseline; }++.facets .facet-label { color: var(--text-muted, #7386a3); font-size: .85rem; }++.facets a { text-decoration: none; padding: 1px 8px; border: 1px solid var(--border, #1f2d47); border-radius: 999px; font-size: .85rem; }++.facets li.on a { border-color: var(--accent, #3b82f6); color: var(--text-primary, #e6edf7); }++.facet-n { color: var(--text-muted, #7386a3); }++++.job-list { list-style: none; margin: 8px 0 0; padding: 0; }++.job-item { border-top: 1px solid var(--border, #1f2d47); padding: 14px 0; }++.job-item:first-child { border-top: 0; }++.job-title { margin: 0 0 4px; }++.job-meta { margin: 0 0 6px; color: var(--text-secondary, #96a7c2); font-size: .92rem; }++.job-company { font-weight: 600; color: var(--text-primary, #e6edf7); }++.job-excerpt { margin: 0 0 8px; max-width: 78ch; }++.job-tags { list-style: none; display: flex; flex-wrap: wrap; gap: 6px; margin: 0 0 8px; padding: 0; }++.job-tags li { font-size: .78rem; border: 1px solid var(--border-light, #2c3d5c); border-radius: 999px; padding: 0 8px; color: var(--text-muted, #7386a3); }++.job-provenance { margin: 0; font-size: .88rem; }++.job-source { font-weight: 600; }++.job-pager { display: flex; gap: 12px; align-items: center; margin: 18px 0 8px; }++.job-facts { display: grid; grid-template-columns: max-content minmax(0, 1fr); gap: 4px 16px; margin: 16px 0; font-size: .92rem; }++.job-facts dt { color: var(--text-secondary, #96a7c2); }++.job-facts dd { margin: 0; overflow-wrap: anywhere; }++.job-apply { margin: 16px 0; }++.job-excerpt-block { max-width: 78ch; }++++.saved-list { list-style: none; margin: 12px 0; padding: 0; }++.saved-item { border-top: 1px solid var(--border, #1f2d47); padding: 12px 0; }++.saved-query { margin: 0 0 4px; }++.saved-form fieldset { border: 1px solid var(--border, #1f2d47); border-radius: 10px; padding: 12px; }++.saved-form legend { padding: 0 6px; color: var(--text-secondary, #96a7c2); }++.saved-form label { display: inline-flex; flex-direction: column; gap: 3px; margin: 0 12px 10px 0; font-size: .88rem; color: var(--text-secondary, #96a7c2); }++.saved-form input, .saved-form select { font: inherit; padding: 6px 8px; border-radius: 8px; border: 1px solid var(--border, #1f2d47); background: var(--bg-secondary, #0f1729); color: inherit; }++.source-block { border-top: 1px solid var(--border, #1f2d47); padding-top: 12px; margin-top: 18px; }++.source-links { margin: 6px 0 12px; }nnew file mode 100644@@@ -0,0 +1,33 @@++#!/usr/bin/env node++'use strict';++++/**++ * Read every job board once, right now, and exit — the same work the timer does (server/ingest/), for an operator who++ * wants the listings without waiting for the next interval, or wants to see what a board answers.++ *++ * node scripts/ingest.js every source++ * node scripts/ingest.js arbeitnow one source++ *++ * It is not gated on WORK_INGEST: running it by hand is the point. It still only fetches the configured base URLs.++ */++const configLib = require('../server/config');++const { openStore } = require('../server/db');++const { createIngest } = require('../server/ingest');++++(async () => {++ const config = configLib.load();++ const s = await openStore(config, { log: console });++ const ingest = createIngest({ config, s, log: console });++ try {++ const want = process.argv.slice(2);++ const out = want.length++ ? await Promise.all(want.map((id) => ingest.runSource(id)))++ : await ingest.runAll();++ for (const r of out) {++ process.stdout.write(`${r.ok ? 'ok ' : 'FAIL'} ${r.source || ''}: ${r.listings} listing(s)${r.error ? ` — ${r.error}` : ''}\n`);++ }++ process.exitCode = out.every((r) => r.ok) ? 0 : 1;++ } finally {++ await s.close();++ }++})().catch((err) => { console.error('[OpenVibe.Work] ingest failed:', err && err.message ? err.message : err); process.exit(1); });@@@ -30,6 +30,7 @@ const { createApi } = require('./http/api');const { createPageRoutes } = require('./http/pages');const { createServiceReadiness } = require('./observability');const { createCallerLimits } = require('./http/caller-limits');++const { createIngest } = require('./ingest');const { assetVersion, send } = require('./render/layout');const { html } = require('./render/html');@@@ -62,6 +63,9 @@ async function createApp(opts = {}) {const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);ctx.valkey = valkey;ctx.limits = createCallerLimits({ config, now: opts.limitsNow || (() => Date.now()), registry: metrics.registry, log, enabled: opts.callerLimits !== false, valkey });++ // The job-board ingest. Built here so a test can run one source against its stand-in server, but started only by++ // server/index.js, and only when WORK_INGEST=on: nothing fetches a board in a test or a development run.++ ctx.ingest = createIngest({ config, s, fetchImpl, log });app.use(contracts.http.middleware());app.use(helmet({@@@ -9,6 +9,7 @@require('dotenv').config();const trim = (s) => String(s || '').replace(/\/+$/, '');const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);++const num = (v, def) => (v != null && v !== '' && Number.isFinite(Number(v)) ? Number(v) : def);function load(env = process.env) {const nodeEnv = env.NODE_ENV || 'development';@@@ -52,6 +53,29 @@ function load(env = process.env) {sessionAudience: env.OV_SESSION_AUDIENCE || 'openvibe.network',},cookies: { secure: env.COOKIE_SECURE ? env.COOKIE_SECURE === 'true' : isProduction },++++ // The public job boards the ingest reads (server/jobs/sources.js). The base URLs are configurable so a++ // test points them at a stand-in server; nothing else in the service ever fetches a URL, and never one a++ // caller typed. Their terms are quoted in the manifest that describes each source, and applied on every++ // page that shows a listing: the source is named and the original listing is linked.++ sources: {++ arbeitnow: { url: trim(env.WORK_ARBEITNOW_URL || 'https://www.arbeitnow.com/api/job-board-api') },++ remotive: { url: trim(env.WORK_REMOTIVE_URL || 'https://remotive.com/api/remote-jobs') },++ remoteok: { url: trim(env.WORK_REMOTEOK_URL || 'https://remoteok.com/api') },++ },++ // The background ingest (server/ingest/): off unless WORK_INGEST=on, so a test and a development run++ // never call a job board. Once on, each source is fetched at most every intervalMs (Remotive's terms++ // allow about four calls a day: six hours is exactly four), and listings not seen for retentionDays go.++ ingest: {++ enabled: String(env.WORK_INGEST || '').toLowerCase() === 'on',++ intervalMs: Math.max(1, num(env.WORK_INGEST_INTERVAL_HOURS, 6)) * 3_600_000,++ timeoutMs: Math.max(1000, int(env.WORK_INGEST_TIMEOUT_MS, 15_000)),++ // Arbeitnow's board is paginated; the first N pages are read. Remotive takes a limit.++ pages: Math.max(1, int(env.WORK_INGEST_PAGES, 3)),++ limit: Math.max(1, int(env.WORK_INGEST_LIMIT, 100)),++ retentionDays: Math.max(1, int(env.WORK_RETENTION_DAYS, 30)),++ userAgent: env.WORK_USER_AGENT || `OpenVibeWork/0.1 (+${baseUrl})`,++ },};}@@@ -3,19 +3,31 @@/*** /api/v1 — OpenVibe.Work's API.*-- * GET /ping public liveness: { ok: true, service }++ * GET /ping public liveness: { ok: true, service }++ * GET /jobs public search the listings (q, remote, type, location, limit, cursor) with facets++ * GET /jobs/:id public one listing, with its source and the original URL to apply on++ * GET /sources public the boards we gather from, their terms as applied, and the last fetch++ * GET /saved-searches person your saved searches, each with the count new since you last looked++ * POST /saved-searches person save the search you are looking at++ * DELETE /saved-searches/:id person forget one++ *++ * Reads are public: a listing is public data with its provenance. A saved search is a person's, so those three routes++ * need a signed-in person (their token as a Bearer, or this site's session) and answer 401 otherwise — a saved search++ * belongs to nobody else. Every route carries its per-caller numbers (../http/caller-limits.js).*-- * The product adds its own routes here. Every route that reads or changes a person's data names the capability it-- * needs (`principal.requireCapability('<work>.thing.read')`, listed in CAPABILITIES in ./principal.js) and takes-- * its per-caller numbers (`limits.reads('<work>.thing.read')` or `limits.budget('<name>')`, ./caller-limits.js).* A refusal is an RFC 9457 problem+json with a stable code.*/const express = require('express');++const contracts = require('openvibe-contracts');const { asyncRouter } = require('./router');++const { sameOrigin } = require('./principal');++const store = require('../jobs/store');++const sources = require('../jobs/sources');function createApi(ctx) {-- const { config, principal, limits } = ctx;++ const { config, s, principal, limits } = ctx;const r = asyncRouter();++ const problem = (req, res, status, code, detail, extra) => contracts.http.sendProblem(res, status, code, { detail, ctx: req.ov, extra });r.use(express.json({ limit: '64kb' }));r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });@@@ -24,6 +36,102 @@ function createApi(ctx) {// The one route the skeleton ships. It is public and counted per caller with the default read numbers.r.get('/ping', limits.reads('work.api.read'), (_req, res) => res.json({ ok: true, service: config.service }));++ const wire = (row) => store.toWire(row, { source: sources.byId(row.source) });++ const person = (req, res) => {++ if (req.principal.kind === 'user') {++ // A write made with the session cookie must come from this site: another site cannot make a signed-in++ // visitor save or forget a search. (A Bearer token is the person's own and is not tied to an origin.)++ if (req.principal.viaSession && req.method !== 'GET' && req.method !== 'HEAD' && !sameOrigin(req, config.baseUrl)) {++ problem(req, res, 403, 'request.cross_site', 'A signed-in request that changes something must come from openvibe.work itself.');++ return null;++ }++ return req.principal.requester;++ }++ // A person's data is theirs: an app or agent token is not a person, and anonymous is not signed in.++ problem(req, res, req.principal.kind === 'anonymous' ? 401 : 403, req.principal.kind === 'anonymous' ? 'token.required' : 'work.saved_searches.person_only',++ req.principal.kind === 'anonymous'++ ? 'Sign in at openvibe.work to keep saved searches, or send a person\'s Network token as a Bearer.'++ : 'Saved searches belong to a person; an app or agent token cannot hold one. A person\'s token can.');++ return null;++ };++++ // ── The listings ────────────────────────────────────────++ r.get('/jobs', limits.budget('work.jobs.search'), async (req, res) => {++ const filters = store.filtersOf(req.query);++ const cursor = store.decodeCursor(req.query.cursor);++ if (req.query.cursor && !cursor) return problem(req, res, 422, 'work.jobs.cursor_invalid', 'cursor: the value the previous page returned as `next`, or absent');++ const limit = Math.min(store.MAX_PAGE, Math.max(1, parseInt(req.query.limit, 10) || store.PAGE_SIZE));++ const page = await store.search(s, filters, { limit, cursor });++ res.json({++ jobs: page.rows.map(wire),++ next: store.encodeCursor(page.next),++ count: page.rows.length,++ filters,++ facets: page.facets,++ sources: sources.SOURCES.map((src) => ({ id: src.id, name: src.name, url: src.homepage })),++ });++ });++++ r.get('/jobs/:id', limits.reads('work.api.read'), async (req, res) => {++ const row = await store.getListing(s, req.params.id);++ if (!row) return problem(req, res, 404, 'work.jobs.not_found', 'No such listing. It may have expired from the board it came from.');++ res.json(wire(row));++ });++++ // ── The sources ─────────────────────────────────────────++ r.get('/sources', limits.reads('work.api.read'), async (_req, res) => {++ const fetches = await store.sourceFetches(s);++ const counts = await store.countsBySource(s);++ res.json({++ sources: sources.SOURCES.map((src) => {++ const f = fetches.get(src.id) || null;++ return {++ id: src.id,++ name: src.name,++ url: src.homepage,++ terms: src.terms,++ attribution: src.attribution,++ listings: (counts.get(src.id) || {}).count || 0,++ last_fetch: f ? { started_at: f.started_at, finished_at: f.finished_at || null, ok: !!f.ok, listings: Number(f.listings) || 0, error: f.error || null } : null,++ };++ }),++ cadence_hours: Math.round(config.ingest.intervalMs / 3_600_000),++ retention_days: config.ingest.retentionDays,++ });++ });++++ // ── Saved searches ──────────────────────────────────────++ const savedWire = async (row) => store.toSavedWire(row, { newCount: await store.newCount(s, row) });++++ r.get('/saved-searches', limits.reads('work.api.read'), async (req, res) => {++ const subject = person(req, res);++ if (!subject) return undefined;++ const rows = await store.listSaved(s, subject);++ return res.json({ saved_searches: await Promise.all(rows.map(savedWire)) });++ });++++ r.post('/saved-searches', limits.budget('work.saved_searches.write'), async (req, res) => {++ const subject = person(req, res);++ if (!subject) return undefined;++ const b = req.body || {};++ const filters = store.filtersOf({ q: b.q != null ? b.q : b.query, remote: b.remote, type: b.type, location: b.location });++ if (!filters.q && !filters.remote && !filters.type && !filters.location) {++ return problem(req, res, 422, 'work.saved_searches.empty', 'A saved search needs at least one of q, remote, type or location — otherwise it is the whole board.');++ }++ const out = await store.createSaved(s, subject, filters, { id: s.newId('ssc'), now: s.iso() });++ if (out.error === 'too_many') return problem(req, res, 429, 'work.saved_searches.limit', `You can keep ${store.MAX_SAVED} saved searches; delete one first.`);++ res.status(201).set('Location', `/api/v1/saved-searches/${out.row.id}`).json(await savedWire(out.row));++ return undefined;++ });++++ r.delete('/saved-searches/:id', limits.budget('work.saved_searches.write'), async (req, res) => {++ const subject = person(req, res);++ if (!subject) return undefined;++ const gone = await store.deleteSaved(s, subject, req.params.id);++ if (!gone) return problem(req, res, 404, 'work.saved_searches.not_found', 'You have no saved search with that id.');++ return res.status(204).end();++ });++return r;}@@@ -31,8 +31,14 @@ function caller(req) {* not declared here. The empty object is the skeleton's starting point:** 'work.thing.create': { minute: 6, hour: 60 },++ *++ * Searching the listings reads a local index, but it is the route a scraper would sit on and the one worth a++ * number of its own; saving a search writes a row, so it gets a tighter one.*/--const BUDGETS = {};++const BUDGETS = {++ 'work.jobs.search': { minute: 60, hour: 600 },++ 'work.saved_searches.write': { minute: 20, hour: 200 },++};function createCallerLimits({ config, now = () => Date.now(), registry = null, log = console, enabled = true, valkey = null }) {const refused = registry@@@ -3,22 +3,30 @@/*** Crawl artifacts for openvibe.work, built with openvibe-shared/seo: robots.txt, sitemap.xml, llms.txt and* llms-full.txt, and the home page's JSON-LD. The public pages are for search engines and AI crawlers; sign-in and-- * the API are not.++ * the API are not, and neither are a person's saved searches.*-- * The product extends PAGE_TEXT and publicPages() with its own pages; the routes and headers here stay.++ * The sitemap lists the site's own pages plus the newest listings (their links back to the board are what a crawler++ * wants to follow), capped at SITEMAP_JOBS. llms-full.txt describes the site's own pages, not every listing: a++ * crawler wanting the listings has /jobs and the sitemap.*/const fs = require('fs');const path = require('path');const seo = require('openvibe-shared/seo');const cache = require('openvibe-shared/cache-policy');const { asyncRouter } = require('./router');++const store = require('../jobs/store');++const sources = require('../jobs/sources');const SITE_NAME = 'OpenVibe.Work';--const DESCRIPTION = 'OpenVibe.Work — Find work, with an agent on your side.';--const DISALLOW = ['/auth/', '/api/'];++const DESCRIPTION = 'OpenVibe.Work — Find work, with an agent on your side. Job listings from open boards with their source, searchable and free to read.';++const DISALLOW = ['/auth/', '/api/', '/saved'];++const SITEMAP_JOBS = 1000;const PAGE_TEXT = {-- '/': ['OpenVibe.Work home', 'OpenVibe.Work: Find work, with an agent on your side.'],++ '/': ['OpenVibe.Work home', 'OpenVibe.Work gathers job listings from public job boards, names the source of every one and links back to the original listing. Search is free and needs no account.'],++ '/jobs': ['Jobs', 'Search job listings gathered from Arbeitnow, Remotive and RemoteOK, newest first, filterable by words, workplace, job type and location.'],++ '/sources': ['Sources', 'The public job boards OpenVibe.Work gathers from, the terms each is used under as applied here, and when each was last read.'],++ '/docs': ['API', 'The OpenVibe.Work API: search the listings, read one with its provenance, list the sources and their terms, keep saved searches.'],'/updates': ['What shipped on OpenVibe.Work', 'This site\'s update log, from the network changelog feed.'],};@@@ -33,19 +41,22 @@ function siteUpdated() {function homeJsonLd(config) {const site = String(config.baseUrl).replace(/\/+$/, '');return [-- seo.jsonLd.website({ name: SITE_NAME, url: site, description: DESCRIPTION }),-- seo.jsonLd.softwareApp({ name: SITE_NAME, url: site, description: DESCRIPTION, category: 'BusinessApplication', keywords: 'openvibe' }),++ seo.jsonLd.website({ name: SITE_NAME, url: site, description: DESCRIPTION, searchUrl: `${site}/jobs?q={q}` }),++ seo.jsonLd.softwareApp({ name: SITE_NAME, url: site, description: DESCRIPTION, category: 'BusinessApplication', keywords: 'openvibe, jobs, job search, remote jobs' }),seo.jsonLd.webPage({ name: SITE_NAME, url: `${site}/`, description: DESCRIPTION, siteUrl: site }),];}const publicPages = () => [-- { path: '/', changefreq: 'weekly', priority: 1.0 },++ { path: '/', changefreq: 'daily', priority: 1.0 },++ { path: '/jobs', changefreq: 'hourly', priority: 0.9 },++ { path: '/sources', changefreq: 'weekly', priority: 0.6 },++ { path: '/docs', changefreq: 'monthly', priority: 0.5 },{ path: '/updates', changefreq: 'daily', priority: 0.5 },];function createDiscoveryRoutes(ctx) {-- const { config } = ctx;++ const { config, s } = ctx;const r = asyncRouter();const site = String(config.baseUrl).replace(/\/+$/, '');const abs = (p) => `${site}${p}`;@@@ -53,21 +64,25 @@ function createDiscoveryRoutes(ctx) {r.get('/robots.txt', (_req, res) => {res.type('text/plain').set('Cache-Control', TEXT).send(-- '# openvibe.work: the public pages are for search and AI crawlers; sign-in and the API are not.\n'++ '# openvibe.work: the public pages are for search and AI crawlers; sign-in, the API and saved searches are not.\n'+ seo.robotsTxt({ sitemaps: [abs('/sitemap.xml')], disallow: DISALLOW }));});r.get('/llms.txt', (_req, res) => {res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({name: SITE_NAME,-- summary: 'OpenVibe.Work: Find work, with an agent on your side.',-- details: 'Every page is server-rendered and readable without JavaScript.',++ summary: DESCRIPTION,++ details: 'Every page is server-rendered and readable without JavaScript. Listings come from public job boards under their terms: the source is named and the original listing is linked; only a short excerpt is kept here.',sections: [{ title: 'Start here', links: [{ title: 'OpenVibe.Work', url: abs('/'), note: 'Find work, with an agent on your side.' },++ { title: 'Search the listings', url: abs('/jobs'), note: 'Filter by words, workplace, job type and location.' },++ { title: 'Where the listings come from', url: abs('/sources'), note: `${sources.SOURCES.map((x) => x.name).join(', ')}, with the terms each is used under.` },{ title: 'What shipped on OpenVibe.Work', url: abs('/updates') },] },-- { title: 'Machine-readable', links: [++ { title: 'For machines', links: [++ { title: 'The API', url: abs('/docs'), note: 'GET /api/v1/jobs, /jobs/:id, /sources — public.' },++ { title: 'The API, live', url: abs('/api/v1/jobs'), note: 'JSON listings with their source and the original URL.' },{ title: 'Sitemap', url: abs('/sitemap.xml') },{ title: 'Full text for language models', url: abs('/llms-full.txt') },{ title: 'Release metadata (JSON)', url: abs('/release.json') },@@@ -84,20 +99,23 @@ function createDiscoveryRoutes(ctx) {const pages = publicPages().map((p) => ({ url: p.path, title: PAGE_TEXT[p.path][0], text: PAGE_TEXT[p.path][1] }));res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsFull({site: SITE_NAME,-- summary: 'Every public page of OpenVibe.Work, one line each.',++ summary: 'Every public page of OpenVibe.Work, one paragraph each; the listings themselves are at /jobs.',base: site,maxBytes: 64 * 1024,sections: [{ title: 'Pages', pages }],}));});-- r.get('/sitemap.xml', (_req, res) => {++ r.get('/sitemap.xml', async (_req, res) => {const lastmod = siteUpdated();const urls = publicPages().map((e) => ({ loc: abs(e.path), ...(lastmod ? { lastmod } : {}), changefreq: e.changefreq, priority: e.priority }));++ // The newest listings, so a crawler reaches each board's own page through ours. No lastmod: these entries turn++ // over with every ingest, and the site's own freshness is the pages above.++ for (const row of await store.recentIds(s, SITEMAP_JOBS)) urls.push({ loc: abs(`/jobs/${row.id}`), changefreq: 'daily', priority: 0.4 });res.type('application/xml').set('Cache-Control', TEXT).send(seo.sitemapXml(urls));});return r;}--module.exports = { createDiscoveryRoutes, homeJsonLd, publicPages, DESCRIPTION, SITE_NAME };++module.exports = { createDiscoveryRoutes, homeJsonLd, publicPages, DESCRIPTION, SITE_NAME, DISALLOW, PAGE_TEXT };@@@ -1,11 +1,13 @@'use strict';/**-- * Public pages: the home page and the update log. Crawl artifacts (robots.txt, sitemap.xml, llms.txt,-- * llms-full.txt, JSON-LD) are http/discovery.js.++ * Public pages: the home page, the search (/jobs), one listing (/jobs/:id), the saved searches (/saved), the sources++ * (/sources) and the update log (/updates). Crawl artifacts (robots.txt, sitemap.xml, llms.txt, llms-full.txt,++ * JSON-LD) are http/discovery.js.*-- * Every page works without JavaScript and is server-rendered through openvibe-shared/shell (render/layout.js).-- * The product replaces the home page's sections and adds its own routes here.++ * Every page is server-rendered through openvibe-shared/shell and works without JavaScript: the search is a GET form,++ * saving and deleting a saved search are plain POSTs. What is shown is read, never restated — the listings come from++ * the database, the sources and their terms from server/jobs/sources.js.*/const ovServe = require('openvibe-shared/serve');const frame = require('openvibe-shared/frame');@@@ -13,58 +15,343 @@ const showcase = require('openvibe-shared/showcase');const cache = require('openvibe-shared/cache-policy');const { asyncRouter } = require('./router');const { createDiscoveryRoutes, homeJsonLd } = require('./discovery');--const { html, raw } = require('../render/html');++const { sameOrigin } = require('./principal');++const { html, raw, table, notice, time } = require('../render/html');const { send } = require('../render/layout');++const listing = require('../render/listing');++const store = require('../jobs/store');++const sources = require('../jobs/sources');const SITE_NAME = 'OpenVibe.Work';const TAGLINE = 'Find work, with an agent on your side.';function createPageRoutes(ctx) {-- const { config } = ctx;++ const { config, s } = ctx;const r = asyncRouter();const PUBLIC_CACHE = cache.htmlHeaders({ maxAge: 300 });const page = (req, res, o, status = 200) => send(res, status, { viewer: req.viewer, config, path: req.originalUrl, ...o });const signedIn = (req) => req.viewer && req.viewer.kind === 'user' && req.viewer.subject;++ const subjectOf = (req) => `user:${req.viewer.subject}`;++ const sourceOf = (row) => sources.byId(row.source);++++ // ── Search: one form, one set of filters, one place they are read ──++ const filterForm = (filters, { facets = null, action = '/jobs' } = {}) => html`<form class="job-search" method="get" action="${action}" role="search">++<label class="sr-only" for="q">Search listings</label>++<input type="search" id="q" name="q" value="${filters.q}" placeholder="Job title, skill, company" maxlength="120">++<label class="sr-only" for="location">Location</label>++<input type="text" id="location" name="location" value="${filters.location}" placeholder="Location" maxlength="120">++<label class="sr-only" for="remote">Remote</label>++<select id="remote" name="remote">++<option value=""${filters.remote === '' ? raw(' selected') : ''}>Remote or on-site</option>++<option value="true"${filters.remote === 'true' ? raw(' selected') : ''}>Remote only</option>++<option value="false"${filters.remote === 'false' ? raw(' selected') : ''}>On-site only</option>++</select>++<label class="sr-only" for="type">Job type</label>++<input type="text" id="type" name="type" value="${filters.type}" placeholder="Job type" maxlength="60">++<button class="sc-btn sc-primary" type="submit">Search</button>++${(filters.q || filters.remote || filters.type || filters.location) ? html`<a class="job-clear" href="${action}">Clear</a>` : ''}++</form>++${facets ? facetBar(filters, facets, action) : ''}`;++++ /** The facets: what is actually in the result set, as links that set one filter and keep the rest. */++ const facetBar = (filters, facets, action) => {++ const link = (patch, label, active, count) => html`<li${active ? raw(' class="on"') : ''}><a href="${withFilters(filters, { ...patch, cursor: null }, action)}">${label}${count == null ? '' : html` <span class="facet-n">${count}</span>`}</a></li>`;++ return html`<div class="job-facets">++<p class="facet-head"><strong>${facets.matches}</strong> listing${facets.matches === 1 ? '' : 's'} match${facets.matches === 1 ? 'es' : ''} this search</p>++<ul class="facets" aria-label="Remote"><li class="facet-label">Workplace</li>++${link({ remote: null }, 'Any', filters.remote === '', null)}++${link({ remote: 'true' }, 'Remote', filters.remote === 'true', facets.remote.true)}++${link({ remote: 'false' }, 'On-site', filters.remote === 'false', facets.remote.false)}++</ul>++${facets.type.length ? html`<ul class="facets" aria-label="Job type"><li class="facet-label">Type</li>++${facets.type.map((t) => link({ type: t.type }, t.type, filters.type === t.type, t.count))}++</ul>` : ''}++</div>`;++ };++++ const passFilters = (f) => {++ const out = {};++ if (f.q) out.q = f.q;++ if (f.remote) out.remote = f.remote;++ if (f.type) out.type = f.type;++ if (f.location) out.location = f.location;++ return out;++ };++ /** A URL for one filter patch, over the current query string. Null removes a filter. */++ function withFilters(filters, patch, path) {++ const merged = { ...passFilters(filters), ...patch };++ const params = new URLSearchParams();++ for (const [k, v] of Object.entries(merged)) if (v != null && v !== '') params.set(k, String(v));++ const q = params.toString();++ return q ? `${path}?${q}` : path;++ }++++ /** Which of the three sources a listing came from, named; the ingest keeps to these and nothing else. */++ const sourceLine = (row) => html`${listing.sourceLink(row, sourceOf(row))}`;// ── Home ─────────────────────────────────────────────────-- r.get('/', (req, res) => {++ r.get('/', async (req, res) => {++ const [latest, counts, rows] = await Promise.all([store.listRecent(s, 6, { remote: true }), store.countsBySource(s), store.search(s, {}, { limit: 1 })]);++ const total = rows.facets.matches;++ const fetches = await store.sourceFetches(s);const hero = showcase.hero({-- eyebrow: `${SITE_NAME} · ${TAGLINE}`,-- title: SITE_NAME,-- accent: TAGLINE,-- lede: `${SITE_NAME} is a new OpenVibe service. This is its skeleton: the platform plumbing below is in place, and the product is what goes on top of it.`,-- actions: signedIn(req)-- ? [{ label: 'Your account', href: '/auth/me' }, { label: 'What shipped', href: '/updates' }]-- : [{ label: 'Sign in with OpenVibe', href: '/auth/login?next=%2F', primary: true }, { label: 'What shipped', href: '/updates' }],-- note: 'Open source (AGPL-3.0). Sign-in is OpenVibe.Network; a person\'s token stays in an httpOnly cookie.',++ eyebrow: `${SITE_NAME} · job listings with their provenance`,++ title: 'Find work.',++ accent: 'Every listing shows where it came from.',++ lede: `${SITE_NAME} gathers job listings from public job boards, keeps the board's own link on every one, and lets you search them. No paywall, no account needed to look: sign in only to keep a search. An agent that helps you apply comes next.`,++ actions: [{ label: 'Search the listings', href: '#search', primary: true }, { label: 'Where they come from', href: '/sources' }],++ note: 'Open source (AGPL-3.0). Listings are shown with the source named and a link to the original — read the full listing there.',});page(req, res, {index: true, cache: signedIn(req) ? null : PUBLIC_CACHE,jsonLd: homeJsonLd(config),styles: [showcase.STYLESHEET],body: html`${raw(hero)}--${raw(showcase.features({-- title: 'What is already here',-- lede: 'The parts every OpenVibe service needs, so a new product starts from them instead of rebuilding them.',-- items: [-- { icon: 'ov:account', title: 'Sign-in with OpenVibe.Network', text: 'OAuth 2 with PKCE (S256), server-side code exchange, httpOnly session cookies, and offline token verification against the Network signing key.' },-- { icon: 'ov:page', title: 'Server-rendered pages', text: 'The OpenVibe Frame (navbar and footer), the shared theme, one stylesheet, crawl artifacts and JSON-LD. Every page is complete without JavaScript.' },-- { icon: 'ov:tools', title: 'An API mount and capabilities', text: 'GET /api/v1/ping today; the router, the problem+json errors and the app/person capability guard are wired and waiting for the product\'s routes.' },-- { icon: 'ov:gauge', title: 'Limits and readiness', text: 'Per-address and per-caller rate limits, /api/health, a truthful /api/ready and Prometheus metrics on loopback.' },-- { icon: 'ov:db', title: 'PostgreSQL and migrations', text: 'openvibe-sdk/db, migrations/NNNN_*.sql applied at boot, PGlite in development, and a store with an injectable clock.' },-- { icon: 'ov:deploy', title: 'Deploy files and tests', text: 'nginx and systemd units, an environment template, and a test suite on PGlite with a mock Network, all through npm test.' },-- ],-- }))}++<section class="sc-sec" id="search" aria-labelledby="h-search"><h2 id="h-search">Search the listings</h2>++${filterForm({ q: '', remote: '', type: '', location: '' }, { action: '/jobs' })}++<p class="muted">${total} listing${total === 1 ? '' : 's'} from ${sources.SOURCES.length} sources, newest first.</p></section>++<section class="sc-sec" aria-labelledby="h-latest"><h2 id="h-latest">Latest remote jobs</h2>++${listing.listingList(latest, { sourceOf, empty: 'No listings yet — the ingest has not run.' })}++<p><a class="sc-btn" href="/jobs?remote=true">All remote listings</a></p></section>++<section class="sc-sec" aria-labelledby="h-sources"><h2 id="h-sources">Where the listings come from</h2>++<p>Each source below is a public job board read through its own API, under its own terms. Every listing on this site names its source and links to the original; we keep a short excerpt and send you to the board for the rest.</p>++${table(['Source', 'Listings', 'Last read', 'Terms'], sources.SOURCES.map((src) => {++ const f = fetches.get(src.id);++ return [++ html`<a href="${src.homepage}" rel="noopener">${src.name}</a>`,++ String((counts.get(src.id) || {}).count || 0),++ f && f.finished_at ? html`${time(f.finished_at)}${f.ok ? '' : html` <span class="muted small">(the last read failed)</span>`}` : html`<span class="muted">not read yet</span>`,++ html`${src.attribution} — <a href="/sources#${src.id}">terms as applied</a>`,++ ];++ }))}</section>${raw(showcase.steps({-- title: 'How a new service starts',++ title: 'How it works',items: [-- { title: 'Generate it', text: 'node scripts/new-service.js replaces the skeleton\'s tokens (service id, name, domain, port, tagline, env prefix) in every file and file name.' },-- { title: 'Install and run', text: 'npm install, then npm run dev: the service listens on its port with an embedded PGlite database, if no DATABASE_URL is set.' },-- { title: 'Make it a product', text: 'Add the product\'s tables and migrations, its capabilities, its routes in server/http/api.js and its pages in server/http/pages.js.' },-- { title: 'Grant and deploy', text: 'Register the service and its capabilities with OpenVibe.Contracts and OpenVibe.Services, then install the unit and the vhost.' },++ { title: 'Gathered from open boards', text: 'A background ingest reads each board\'s public API every few hours, keeps the listing\'s own id and URL, and stores a short excerpt — never the full description.' },++ { title: 'Search it, with the source on it', text: 'Filter by text, workplace, job type and location. Every result names the board it came from and links back to it.' },++ { title: 'Read it on the board', text: 'The listing page shows the excerpt and sends you to the original listing to read it in full and apply. Nothing is republished.' },++ { title: 'Keep a search', text: 'Sign in to save a search; /saved shows each one with the count of listings that appeared since you last looked.' },++ { title: 'Nothing behind a paywall', text: 'Reading and searching is free and needs no account. There is no premium tier on the listings, and listings expire when a board stops showing them.' },++ { title: 'An agent, next', text: 'The agent that drafts and tracks applications is the next pillar; alerts through OpenVibe.Watch come with it.' },],}))}--${raw(showcase.cta({ title: 'Read the API', text: 'GET /api/v1/ping is the whole API today. The rest is what the product adds.', actions: [{ label: 'The API', href: '/api/v1/ping' }, { label: 'What shipped', href: '/updates' }] }))}`,++${raw(showcase.cta({ title: 'Prefer the API?', text: 'The same search, the sources and their terms, and your saved searches over HTTP. Reading is public.', actions: [{ label: 'The API', href: '/docs' }, { label: 'The sources', href: '/sources' }] }))}`,++ });++ });++++ // ── Search ───────────────────────────────────────────────++ r.get('/jobs', async (req, res) => {++ const filters = store.filtersOf(req.query);++ const cursor = store.decodeCursor(req.query.cursor);++ // A cursor we did not write is not a position in the result set; start from the top rather than guess.++ const p = await store.search(s, filters, { cursor });++ const nextUrl = p.next ? withFilters(filters, { cursor: store.encodeCursor(p.next) }, '/jobs') : null;++ const described = [filters.q ? `“${filters.q}”` : '', filters.remote === 'true' ? 'remote' : filters.remote === 'false' ? 'on-site' : '', filters.type ? filters.type : '', filters.location ? `in ${filters.location}` : ''].filter(Boolean).join(' ');++ page(req, res, {++ index: true, cache: signedIn(req) ? null : PUBLIC_CACHE,++ title: described ? `Jobs: ${described}` : 'Jobs',++ description: `Job listings from open boards${described ? ` matching ${described}` : ''}, newest first, each with the source it came from and a link to the original listing.`,++ crumbs: [{ label: 'Jobs' }],++ styles: [showcase.STYLESHEET],++ body: html`<h1>Jobs</h1>++${filterForm(filters, { facets: p.facets })}++${listing.listingList(p.rows, { sourceOf, empty: 'No listings match this search. Try fewer words, or clear a filter and search again.' })}++<nav class="job-pager" aria-label="Pages">++${cursor ? html`<a class="sc-btn" href="${withFilters(filters, { cursor: null }, '/jobs')}">First page</a>` : ''}++${nextUrl ? html`<a class="sc-btn sc-primary" href="${nextUrl}">Older listings</a>` : html`<span class="muted">That is every listing matching this search.</span>`}++</nav>++<p class="muted small">Read the full listing on its source: each entry links back to the board it came from.</p>`,++ });++ });++++ // ── One listing ──────────────────────────────────────────++ r.get('/jobs/:id', async (req, res) => {++ const row = await store.getListing(s, req.params.id);++ if (!row) {++ return page(req, res, {++ title: 'Listing not found',++ body: html`<h1>Listing not found</h1><p>No listing here with that id. A listing is removed once the board it came from stops showing it, and old links stop working.</p><p><a href="/jobs">Search the listings</a></p>`,++ }, 404);++ }++ const src = sourceOf(row);++ const l = listing.meta(row);++ page(req, res, {++ index: true, cache: PUBLIC_CACHE,++ title: `${l.title} at ${l.company}`,++ description: listing.plainExcerpt(row).slice(0, 300) || `${l.title} at ${l.company}, from ${src.name}.`,++ crumbs: [{ label: 'Jobs', href: '/jobs' }, { label: l.title }],++ styles: [showcase.STYLESHEET],++ jsonLd: [listing.jobPosting(row, { siteUrl: config.baseUrl })],++ body: html`<article class="job">++<h1>${l.title}</h1>++<p class="job-meta"><span class="job-company">${l.company}</span> · <span class="job-location">${l.location || 'Location not given'}</span>${l.remote ? ' · Remote' : ''}${l.job_type ? html` · <span class="job-type">${l.job_type}</span>` : ''}</p>++${l.salary ? html`<p class="job-salary"><strong>Salary:</strong> ${l.salary} <span class="muted small">(as the board gives it)</span></p>` : ''}++${l.tags.length ? html`<ul class="job-tags">${l.tags.map((t) => html`<li>${t}</li>`)}</ul>` : ''}++<section class="job-excerpt-block" aria-labelledby="h-excerpt"><h2 id="h-excerpt">Excerpt</h2>++<p>${l.excerpt}</p>++<p class="muted small">This is a short excerpt only. The full listing stays on the board.</p></section>++<p class="job-apply"><a class="sc-btn sc-primary" href="${l.url}" rel="nofollow noopener external" target="_blank">Read the full listing on ${src.name}</a></p>++<dl class="job-facts">++<dt>Source</dt><dd>${sourceLine(row)} — <a href="/sources#${src.id}">its terms as applied</a></dd>++${l.posted_at ? html`<dt>Posted</dt><dd>${time(l.posted_at)}</dd>` : ''}++<dt>Read from the board</dt><dd>${time(row.fetched_at)}</dd>++<dt>Board's id</dt><dd><code>${row.source_id}</code></dd>++</dl>++<p class="muted small">Found a problem with this listing, or want it gone? It is the board's listing, not ours — the source above is where it lives. In code: <code>GET /api/v1/jobs/${row.id}</code>.</p>++</article>`,++ });++ });++++ // ── Saved searches ───────────────────────────────────────++ function savedForm(req, filters, { problem = null } = {}) {++ return html`${problem ? notice(problem, 'warn') : ''}++<form class="saved-form" method="post" action="/saved">++<fieldset><legend>Keep this search</legend>++<label>Words <input type="text" name="q" value="${filters.q}" maxlength="120" placeholder="Job title, skill, company"></label>++<label>Location <input type="text" name="location" value="${filters.location}" maxlength="120"></label>++<label>Job type <input type="text" name="type" value="${filters.type}" maxlength="60"></label>++<label>Workplace <select name="remote">++<option value=""${filters.remote === '' ? raw(' selected') : ''}>Any</option>++<option value="true"${filters.remote === 'true' ? raw(' selected') : ''}>Remote only</option>++<option value="false"${filters.remote === 'false' ? raw(' selected') : ''}>On-site only</option>++</select></label>++<button class="sc-btn sc-primary" type="submit">Save this search</button>++</fieldset>++</form>`;++ }++++ const describeSaved = (row) => {++ const bits = [row.query ? `“${row.query}”` : '', row.remote === 'true' ? 'remote' : row.remote === 'false' ? 'on-site' : '', row.type ? row.type : '', row.location ? `in ${row.location}` : ''].filter(Boolean);++ return bits.length ? bits.join(' ') : 'everything';++ };++++ r.get('/saved', async (req, res) => {++ if (!signedIn(req)) {++ return page(req, res, {++ title: 'Saved searches',++ body: html`<h1>Saved searches</h1>++<p>Sign in to keep a search and see what has appeared since you last looked. Searching itself needs no account.</p>++<p><a class="sc-btn sc-primary" href="/auth/login?next=%2Fsaved">Sign in with OpenVibe</a> <a class="sc-btn" href="/jobs">Search the listings</a></p>`,++ });++ }++ const subject = subjectOf(req);++ const rows = await store.listSaved(s, subject);++ // The counts are read before last_seen_at moves: opening this page is what "looking" means.++ const withCounts = await Promise.all(rows.map(async (row) => ({ row, count: await store.newCount(s, row) })));++ const now = s.iso();++ for (const row of rows) await store.touchSaved(s, subject, row.id, now);++ const initial = store.filtersOf(req.query);++ page(req, res, {++ title: 'Saved searches', crumbs: [{ label: 'Saved searches' }],++ body: html`<h1>Saved searches</h1>++<p>${rows.length ? html`You are keeping ${rows.length} search${rows.length === 1 ? '' : 'es'}. The count is how many listings appeared since you last opened this page.` : 'You are not keeping any search yet.'}</p>++${withCounts.length ? html`<ul class="saved-list">${withCounts.map(({ row, count }) => html`<li class="saved-item">++<p class="saved-query"><a href="${store.toSavedWire(row).search_path}">${describeSaved(row)}</a> <span class="badge${count ? ' ok' : ''}">${count ? `${count} new` : 'nothing new'}</span></p>++<p class="muted small">Saved ${time(row.created_at)} · last looked ${row.last_seen_at ? time(row.last_seen_at) : 'never'}</p>++<form method="post" action="/saved/${row.id}/delete" class="inline"><button class="sc-btn" type="submit">Forget it</button></form>++</li>`)}</ul>` : ''}++${savedForm(req, initial)}++<p class="muted small">Alerts — being told when a listing that fits appears — come later, through OpenVibe.Watch. For now a saved search is a search you do not have to type again, and a count of what is new.</p>++<p class="muted small">In code: <code>GET /api/v1/saved-searches</code>.</p>`,++ });++ });++++ r.post('/saved', require('express').urlencoded({ extended: false, limit: '8kb' }), async (req, res) => {++ res.set('Cache-Control', cache.htmlHeaders({ private: true }));++ if (!signedIn(req)) return res.redirect(303, '/auth/login?next=%2Fsaved');++ // A signed-in write must come from this site, as everywhere else in this service.++ if (!sameOrigin(req, config.baseUrl)) return res.status(403).type('text/plain').send('Saving a search must come from openvibe.work itself.');++ const filters = store.filtersOf(req.body || {});++ // A search with nothing in it is the whole board, which is not something worth keeping (the API refuses it++ // too). Say so on the page rather than saving it.++ if (!filters.q && !filters.remote && !filters.type && !filters.location) {++ return page(req, res, {++ title: 'Saved searches', crumbs: [{ label: 'Saved searches' }],++ body: html`<h1>Saved searches</h1>++${notice('Give the search at least one thing to look for — words, a location, a job type or a workplace — or there is nothing to save.', 'warn')}++<p><a href="/saved">Back to your saved searches</a></p>`,++ }, 422);++ }++ const out = await store.createSaved(s, subjectOf(req), filters, { id: s.newId('ssc'), now: s.iso() });++ if (out.error === 'too_many') {++ const rows = await store.listSaved(s, subjectOf(req));++ return page(req, res, { title: 'Saved searches', body: html`<h1>Saved searches</h1>${notice(`You can keep ${store.MAX_SAVED} saved searches. Forget one below, then save this again.`, 'warn')}<p><a href="/saved">Back to your saved searches</a> (${rows.length})</p>` }, 429);++ }++ return res.redirect(303, '/saved');++ });++++ r.post('/saved/:id/delete', async (req, res) => {++ res.set('Cache-Control', cache.htmlHeaders({ private: true }));++ if (!signedIn(req)) return res.redirect(303, '/auth/login?next=%2Fsaved');++ if (!sameOrigin(req, config.baseUrl)) return res.status(403).type('text/plain').send('Forgetting a search must come from openvibe.work itself.');++ await store.deleteSaved(s, subjectOf(req), req.params.id);++ res.redirect(303, '/saved');++ });++++ // ── The sources and their terms ──────────────────────────++ r.get('/sources', async (req, res) => {++ const [fetches, counts] = await Promise.all([store.sourceFetches(s), store.countsBySource(s)]);++ page(req, res, {++ index: true, cache: PUBLIC_CACHE,++ title: 'Sources',++ description: 'The public job boards OpenVibe.Work gathers listings from, the terms each one is used under as applied here, and when each was last read.',++ crumbs: [{ label: 'Sources' }],++ body: html`<h1>Sources</h1>++<p>Every listing here comes from one of these boards, read through its public API. We keep the board's own id and URL for each listing, show a short excerpt, and link to the original on every page that shows it. Nothing is republished, and no board is read more often than its terms allow.</p>++${sources.SOURCES.map((src) => {++ const f = fetches.get(src.id);++ return html`<section class="source-block" id="${src.id}" aria-labelledby="h-${src.id}">++<h2 id="h-${src.id}"><a href="${src.homepage}" rel="noopener">${src.name}</a></h2>++<p class="muted small">${(counts.get(src.id) || {}).count || 0} live listing(s) from this board right now.</p>++<h3>Its terms, as applied here</h3>++<p>${src.terms}</p>++<p class="source-attribution">Every listing from this board is shown as: <strong>${src.attribution}</strong>, linked back to the original. On this site that means:</p>++<ul class="source-links"><li>List and detail pages name ${src.name} and link the listing to the original URL on the board.</li>++<li>The listing page carries a <em>“Read the full listing on ${src.name}”</em> link, and the full description is not stored or shown.</li>++<li>${src.name} listings also carry their source in the JSON-LD on the listing page.</li></ul>++<h3>What each read does</h3>++<dl class="job-facts">++<dt>API</dt><dd><code>${src.requests(config)[0]}</code></dd>++<dt>Last read</dt><dd>${f && f.started_at ? html`${time(f.started_at)}${f.ok ? ' — succeeded' : ' — failed'}${f.error ? html` <span class="muted small">(${f.error})</span>` : ''}` : html`<span class="muted">not read yet</span>`}</dd>++<dt>Reads every</dt><dd>${Math.round(config.ingest.intervalMs / 3_600_000)} hours ${src.id === 'remotive' ? '(its terms allow about four a day)' : ''}</dd>++<dt>Listings kept</dt><dd>${(counts.get(src.id) || {}).count || 0}, retired when the board stops showing one for ${config.ingest.retentionDays} days</dd>++</dl>++</section>`;++ })}++<p class="muted">Machine-readable, with each source's terms and last fetch: <a href="/api/v1/sources"><code>GET /api/v1/sources</code></a>.</p>`,++ });++ });++++ // ── The API reference ────────────────────────────────────++ r.get('/docs', (req, res) => {++ const b = config.baseUrl;++ const search = `curl -s '${b}/api/v1/jobs?q=engineer&remote=true'++# → { "jobs": [ { "id": "job_…", "title": "…", "company": "…", "url": "https://…", "source": "remotive", "source_name": "Remotive", "posted_at": "…", "excerpt": "…" } ], "next": null, "facets": { … } }`;++ page(req, res, {++ index: true, cache: PUBLIC_CACHE,++ title: 'API',++ description: 'The OpenVibe.Work API: search the listings, read one with its provenance, list the sources and their terms, and keep saved searches. Reading is public.',++ crumbs: [{ label: 'API' }],++ body: html`<h1>The API</h1>++<p>Everything the site does is this API. Reading is public; a saved search is a person's.</p>++<h2 id="who">Who can call it</h2>++<ul>++<li><strong>Anyone</strong>: <code>GET /api/v1/jobs</code>, <code>GET /api/v1/jobs/:id</code> and <code>GET /api/v1/sources</code> need no token.</li>++<li><strong>A person</strong>: the saved searches need a signed-in person — their Network token as <code>Authorization: Bearer</code>, or this site's sign-in. They belong to that person alone.</li>++</ul>++${table(['Route', 'Who', 'What it does'], [++ [html`<code>GET /api/v1/jobs</code>`, 'public', html`Search: <code>?q=</code>, <code>?remote=true|false</code>, <code>?type=</code>, <code>?location=</code>, <code>?limit=</code> (to ${store.MAX_PAGE}), <code>?cursor=</code> (the <code>next</code> the previous page returned). Newest first, ${store.PAGE_SIZE} a page, with facets for remote and job type.`],++ [html`<code>GET /api/v1/jobs/:id</code>`, 'public', 'One listing: the board\'s fields, the excerpt, the source, and <code>url</code> — the original listing to apply on.'],++ [html`<code>GET /api/v1/sources</code>`, 'public', 'The boards, each one\'s terms as applied, its listing count and when it was last read.'],++ [html`<code>GET /api/v1/saved-searches</code>`, 'person', 'Your saved searches, each with <code>new_count</code> — listings that appeared since you last looked.'],++ [html`<code>POST /api/v1/saved-searches</code>`, 'person', html`Save one: <code>{ q?, remote?, type?, location? }</code>. At least one filter. 201 with the saved search.`],++ [html`<code>DELETE /api/v1/saved-searches/:id</code>`, 'person', 'Forget one. 204.'],++ ])}++${raw(showcase.code({ title: 'Search, then read one', samples: [{ label: 'bash', lang: 'bash', code: search }] }))}++<h2 id="terms">The terms we apply</h2>++<p>Each source is read under its own terms, which are stated on <a href="/sources">/sources</a> and in <code>GET /api/v1/sources</code>: the source is named, the original listing is linked, and only a short excerpt is stored. If you build on this API, carry the attribution and the link back with the listing.</p>++<h2 id="limits">Limits</h2>++<p>Search is limited per caller (per person, per app, else per address); past it the API answers 429 <code>rate_limited</code> with <code>Retry-After</code>. Errors are RFC 9457 <code>application/problem+json</code> with a stable <code>code</code>.</p>`,});});@@@ -80,4 +367,4 @@ ${raw(showcase.cta({ title: 'Read the API', text: 'GET /api/v1/ping is the wholereturn r;}--module.exports = { createPageRoutes };++module.exports = { createPageRoutes, SITE_NAME, TAGLINE };@@@ -8,7 +8,7 @@ const express = require('express');function asyncRouter() {const r = express.Router();-- for (const method of ['get', 'post']) {++ for (const method of ['get', 'post', 'delete', 'put', 'patch']) {const orig = r[method].bind(r);r[method] = (path, ...handlers) => orig(path, ...handlers.map((h) => (h.length === 4 ? h : (req, res, next) => {try {@@@ -14,7 +14,7 @@ const { gracefulStop } = require('openvibe-sdk/service');function createLifecycle({ server, ctx, exit, signals, timers = [] }) {return gracefulStop({name: 'OpenVibe.Work', server, deadlineExitCode: 0, exit, signals, deadlineMs: 10_000,-- close: [() => { for (const t of timers) clearInterval(t); }, () => ctx.keys.client.stop(), () => ctx.s.close()],++ close: [() => { for (const t of timers) clearInterval(t); }, () => ctx.ingest.stop(), () => ctx.keys.client.stop(), () => ctx.s.close()],});}@@@ -27,6 +27,7 @@ async function start() {});server.keepAliveTimeout = 65_000;ctx.keys.client.start();++ ctx.ingest.start();createLifecycle({ server, ctx });return { server, ctx };nnew file mode 100644@@@ -0,0 +1,117 @@++'use strict';++++/**++ * The background ingest: every six hours, read each job board's public API (server/jobs/sources.js), keep what it++ * shows and retire what it stopped showing.++ *++ * It is off unless WORK_INGEST=on, so neither a test nor a development run ever calls a job board; when it is on,++ * each source keeps its own timer, its own last-fetch row and its own failures — one board being down never stops the++ * others, and the failure is recorded and shown on /sources rather than hidden.++ *++ * Every outbound request goes to a base URL from the configuration (never to a URL anyone typed), with this++ * service's User-Agent, a 15-second timeout and a size cap, and only http(s).++ */++const sources = require('../jobs/sources');++const store = require('../jobs/store');++++const MAX_BYTES = 8 * 1024 * 1024;++const START_DELAY_MS = 5_000;++++async function fetchJson(url, { fetchImpl, timeoutMs, userAgent }) {++ let res;++ try {++ res = await fetchImpl(url, {++ headers: { 'user-agent': userAgent, accept: 'application/json' },++ redirect: 'follow',++ signal: AbortSignal.timeout(timeoutMs),++ });++ } catch (err) {++ throw new Error(err && err.name === 'TimeoutError' ? `no answer in ${Math.round(timeoutMs / 1000)}s` : 'could not be reached');++ }++ if (!res.ok) throw new Error(`HTTP ${res.status} from the board`);++ const text = await res.text();++ if (text.length > MAX_BYTES) throw new Error('the board answered with more than 8 MB');++ try { return JSON.parse(text); } catch { throw new Error('the board answered with something that is not JSON'); }++}++++function createIngest({ config, s, fetchImpl = globalThis.fetch, log = console }) {++ const timers = new Map();++ let running = false;++++ /** Read one source end to end and record what happened. Never throws: the caller is a timer. */++ async function runSource(sourceId) {++ const source = sources.byId(sourceId);++ if (!source) throw new Error(`no source ${sourceId}`);++ const startedAt = s.iso();++ const fetchedAt = startedAt;++ await store.beginFetch(s, source.id, startedAt);++ let listings = 0;++ try {++ const seen = new Map();++ for (const url of source.requests(config)) {++ const json = await fetchJson(url, { fetchImpl, timeoutMs: config.ingest.timeoutMs, userAgent: config.ingest.userAgent });++ for (const raw of source.parse(json)) {++ const row = sources.toRow(raw, { sourceId: source.id, fetchedAt, idOf: store.idOf });++ // Two pages can carry the same listing; the later one wins, exactly as the board would show it.++ if (row) seen.set(row.source_id, row);++ }++ }++ const rows = [...seen.values()];++ const out = await store.upsertListings(s, rows, { sourceId: source.id, fetchedAt });++ listings = out.seen;++ await store.finishFetch(s, source.id, { ok: true, listings, error: null, finishedAt: s.iso() });++ log.log(`[Ingest] ${source.name}: ${listings} listings (${out.added} new, ${out.updated} updated)`);++ return { ok: true, listings, added: out.added, updated: out.updated, rows };++ } catch (err) {++ const detail = (err && err.message) || String(err);++ await store.finishFetch(s, source.id, { ok: false, listings, error: detail, finishedAt: s.iso() });++ log.warn(`[Ingest] ${source.name} failed: ${detail}`);++ return { ok: false, error: detail, listings };++ }++ }++++ /** Every source, then the expiry sweep: a listing no board has shown for the retention window goes. */++ async function runAll() {++ if (running) return [];++ running = true;++ try {++ const out = [];++ for (const source of sources.SOURCES) out.push({ source: source.id, ...(await runSource(source.id)) });++ try {++ const gone = await store.expire(s, config.ingest.retentionDays, s.now());++ if (gone) log.log(`[Ingest] expired ${gone} listings not seen for ${config.ingest.retentionDays} days`);++ } catch (err) {++ log.warn(`[Ingest] expiry failed: ${(err && err.message) || err}`);++ }++ return out;++ } finally { running = false; }++ }++++ /**++ * Start the timers: one per source, so a slow board delays only itself. The first run waits a few seconds so it++ * never competes with the process starting up or with a deploy's readiness check.++ */++ function start() {++ if (!config.ingest.enabled || timers.size) return false;++ for (const source of sources.SOURCES) {++ const timer = setInterval(() => { runSource(source.id).catch(() => {}); }, config.ingest.intervalMs);++ timer.unref();++ timers.set(source.id, timer);++ }++ const kick = setTimeout(() => { runAll().catch(() => {}); }, START_DELAY_MS);++ kick.unref();++ timers.set('start', kick);++ log.log(`[Ingest] on: ${sources.ids().join(', ')} every ${Math.round(config.ingest.intervalMs / 3_600_000)}h`);++ return true;++ }++++ function stop() {++ for (const timer of timers.values()) clearInterval(timer);++ for (const timer of timers.values()) clearTimeout(timer);++ timers.clear();++ }++++ return { start, stop, runSource, runAll, timers };++}++++module.exports = { createIngest };nnew file mode 100644@@@ -0,0 +1,191 @@++'use strict';++++/**++ * The job boards OpenVibe.Work gathers from, one entry each: where it fetches, the terms it must apply, and how its++ * payload becomes a listing. Nothing here is invented: a field a board does not give stays empty, and every listing++ * keeps the board's own id and the original URL to link back to.++ *++ * Adding a source is adding an entry here and nothing else — the ingest, the search and the pages read this list.++ * Each entry is:++ *++ * id the short name stored on every listing, and shown to the reader++ * name the board's name, as it wants to be attributed++ * homepage the board itself++ * terms its terms of use for the API, in the board's own terms, as applied here++ * attribution the line every page that shows one of its listings must carry++ * requests(config) the URLs to fetch, in order++ * parse(json) the payload → normalized listings (see normalize below)++ * remoteDefault what a listing from this board is when the board says nothing++ *++ * A normalized listing is:++ * { source_id, url, title, company, location, remote, tags[], job_type, salary, posted_at, description_html }++ */++++const { excerpt } = require('./text');++++const clip = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);++const list = (v, n = 12) => (Array.isArray(v) ? v : [v]).filter((x) => x != null && String(x).trim()).map((x) => clip(x, 60)).slice(0, n);++++/** An ISO 8601 UTC instant, or null. Boards mix Unix seconds, Unix milliseconds and date strings. */++function instant(v) {++ if (v == null || v === '') return null;++ if (typeof v === 'number' || /^\d+$/.test(String(v))) {++ const n = Number(v);++ // Seconds, milliseconds or microseconds since the epoch — whichever lands in a sane range.++ const ms = n > 1e14 ? n / 1000 : n > 1e11 ? n : n * 1000;++ const d = new Date(ms);++ return Number.isNaN(d.getTime()) ? null : d.toISOString();++ }++ const d = new Date(String(v));++ return Number.isNaN(d.getTime()) ? null : d.toISOString();++}++++/**++ * The listing's URL, or null. Only http(s) is kept: anything else (a javascript: URL, a data: URL) is not a link++ * back to a listing and is dropped with the listing rather than shown.++ */++function httpUrl(v) {++ try {++ const u = new URL(String(v));++ return u.protocol === 'http:' || u.protocol === 'https:' ? u.toString().slice(0, 2000) : null;++ } catch { return null; }++}++++const money = (n) => (n == null || n === '' || !Number.isFinite(Number(n)) || Number(n) <= 0 ? null : Math.round(Number(n)));++++const SOURCES = [++ {++ id: 'arbeitnow',++ name: 'Arbeitnow',++ homepage: 'https://www.arbeitnow.com/',++ terms: 'Arbeitnow publishes its job board API free of charge. We attribute Arbeitnow by name and link every listing back to its page on arbeitnow.com.',++ attribution: 'Listing from Arbeitnow',++ remoteDefault: false,++ requests(config) {++ const pages = [];++ for (let page = 1; page <= config.ingest.pages; page++) {++ const u = new URL(config.sources.arbeitnow.url);++ u.searchParams.set('page', String(page));++ pages.push(u.toString());++ }++ return pages;++ },++ // { data: [ { slug, company_name, title, description (HTML), remote, url, tags[], job_types[], location, created_at } ] }++ parse(json) {++ const rows = (json && Array.isArray(json.data)) ? json.data : [];++ return rows.map((r) => ({++ source_id: clip(r.slug || r.url, 200),++ url: httpUrl(r.url),++ title: clip(r.title, 300),++ company: clip(r.company_name, 200),++ location: clip(r.location, 200),++ remote: r.remote === true,++ tags: list(r.tags),++ job_type: list(r.job_types).join(', ') || null,++ salary: null, // this board gives no salary++ posted_at: instant(r.created_at),++ description_html: r.description,++ }));++ },++ },++ {++ id: 'remotive',++ name: 'Remotive',++ homepage: 'https://remotive.com/',++ terms: 'To use the Remotive API you must mention Remotive as the source and link to the listing. The API is read at most about four times a day, which is what the six-hour ingest interval is.',++ attribution: 'Listing from Remotive',++ remoteDefault: true, // every Remotive listing is remote by definition++ requests(config) {++ const u = new URL(config.sources.remotive.url);++ u.searchParams.set('limit', String(config.ingest.limit));++ return [u.toString()];++ },++ // { jobs: [ { id, url, title, company_name, category, tags[], job_type, publication_date, candidate_required_location, salary, description (HTML) } ] }++ parse(json) {++ const rows = (json && Array.isArray(json.jobs)) ? json.jobs : [];++ return rows.map((j) => ({++ source_id: clip(j.id, 200),++ url: httpUrl(j.url),++ title: clip(j.title, 300),++ company: clip(j.company_name, 200),++ location: clip(j.candidate_required_location, 200),++ remote: true,++ // Remotive labels a job with both tags and one category; a listing row has one labels column, so both++ // are kept there, in the board's own words rather than our reading of them.++ tags: list([].concat(j.tags || [], j.category || [])),++ job_type: clip(j.job_type, 80) || null,++ salary: clip(j.salary, 200) || null,++ posted_at: instant(j.publication_date),++ description_html: j.description,++ }));++ },++ },++ {++ id: 'remoteok',++ name: 'RemoteOK',++ homepage: 'https://remoteok.com/',++ terms: 'RemoteOK asks that a use of its API names RemoteOK as the source and links back to the listing. The response\'s first element is a legal notice, not a listing, and is skipped.',++ attribution: 'Listing from RemoteOK',++ remoteDefault: true,++ requests(config) { return [config.sources.remoteok.url]; },++ // The response is an array whose first element is the legal notice; every later element is a listing.++ parse(json) {++ const all = Array.isArray(json) ? json : [];++ const rows = all.filter((r) => r && (r.id != null || r.slug) && r.position);++ return rows.map((r) => {++ const min = money(r.salary_min);++ const max = money(r.salary_max);++ return {++ source_id: clip(r.id != null ? r.id : r.slug, 200),++ url: httpUrl(r.url || (r.slug ? `https://remoteok.com/remote-jobs/${r.slug}` : null)),++ title: clip(r.position, 300),++ company: clip(r.company, 200),++ location: clip(r.location, 200),++ remote: true,++ tags: list(r.tags),++ job_type: clip(r.job_type || r.type, 80) || null,++ salary: min && max ? `$${min.toLocaleString('en-US')}–$${max.toLocaleString('en-US')} a year` : null,++ posted_at: instant(r.date),++ description_html: r.description,++ };++ });++ },++ },++];++++const byId = (id) => SOURCES.find((s) => s.id === id) || null;++const ids = () => SOURCES.map((s) => s.id);++++/**++ * A parsed row → the columns of work_listings, or null when it is not a listing we may keep. What we refuse: a row++ * with no id, no title, no company or no http(s) URL to link back to — without those a listing has no provenance and++ * must not be published. The excerpt is what is stored; the board keeps the full description.++ */++function toRow(raw, { sourceId, fetchedAt, idOf }) {++ // The URL is checked again here even though each parse does it: this is the last gate before anything is stored,++ // and a listing without an http(s) link back to its board has no provenance and is not published.++ const url = raw && httpUrl(raw.url);++ if (!raw || !raw.source_id || !raw.title || !raw.company || !url) return null;++ const tags = list(raw.tags);++ const row = {++ id: idOf(sourceId, raw.source_id),++ source: sourceId,++ source_id: String(raw.source_id),++ url,++ title: String(raw.title),++ company: String(raw.company),++ location: raw.location || null,++ remote: !!raw.remote,++ tags: tags.join(', '),++ job_type: raw.job_type || null,++ salary: raw.salary || null,++ excerpt: excerpt(raw.description_html).slice(0, 400),++ posted_at: raw.posted_at || null,++ sort_at: raw.posted_at || fetchedAt,++ fetched_at: fetchedAt,++ };++ row.search_text = [row.title, row.company, row.tags, row.location, row.excerpt, row.salary].filter(Boolean).join(' ').toLowerCase().slice(0, 4000);++ return row;++}++++module.exports = { SOURCES, byId, ids, toRow };nnew file mode 100644@@@ -0,0 +1,290 @@++'use strict';++++/**++ * The listings, the searches a person saved and what each source's last fetch did (migrations/0002_listings.sql).++ * Every function takes the store (server/db.js createStore) first, as everywhere else in this service.++ *++ * A listing is never restated: the row holds the board's own fields, a short excerpt and the original URL. Search is++ * a plain case-insensitive substring over one lowercased column — a job board of this size does not need a text++ * index, and the LIKE pattern is a bound parameter, never built from what a caller typed.++ */++const crypto = require('crypto');++++const PAGE_SIZE = 30;++const MAX_PAGE = 100;++const TYPE_FACETS = 12;++++// ── ids ─────────────────────────────────────────────────────++// A listing's id is derived from (source, source_id), so the same listing keeps the same URL on this site however++// often the board is re-read, and two sources can never collide. 26 base32 characters, the shape of a ULID.++const B32 = '0123456789abcdefghjkmnpqrstvwxyz';++function idOf(source, sourceId) {++ const digest = crypto.createHash('sha256').update(`${source}:${sourceId}`).digest();++ let bits = 0;++ let value = 0;++ let out = '';++ for (const byte of digest) {++ value = (value << 8) | byte;++ bits += 8;++ while (bits >= 5 && out.length < 26) {++ out += B32[(value >>> (bits - 5)) & 31];++ bits -= 5;++ }++ if (out.length >= 26) break;++ }++ return `job_${out}`;++}++const LISTING_ID = /^job_[0-9abcdefghjkmnpqrstvwxyz]{26}$/;++++// ── search ──────────────────────────────────────────────────++const lower = (s) => String(s == null ? '' : s).trim().toLowerCase().slice(0, 120);++/** A LIKE pattern that matches the term literally: % and _ a caller typed are characters, not wildcards. */++const like = (term) => `%${term.replace(/[\\%_]/g, '\\$&')}%`;++++/** The four filters a search takes, from anything (a query string, an API call, a saved search row). */++function filtersOf(input = {}) {++ const remote = String((input.remote == null ? '' : input.remote)).toLowerCase();++ return {++ // A search comes from a query string (`q`), an API body or a saved search row (`query`).++ q: lower(input.q == null ? input.query : input.q),++ remote: remote === 'true' || remote === 'yes' || remote === '1' ? 'true' : remote === 'false' || remote === 'no' || remote === '0' ? 'false' : '',++ type: lower(input.type),++ location: lower(input.location),++ };++}++++const num = (v) => (v == null ? 0 : Number(v));++const parseTags = (s) => String(s || '').split(',').map((t) => t.trim()).filter(Boolean);++++/** A row → the listing as the pages and the API see it. `url` is always the board's own. */++function toWire(row, { source } = {}) {++ return {++ id: row.id,++ source: row.source,++ source_name: (source && source.name) || row.source,++ source_url: (source && source.homepage) || null,++ url: row.url,++ title: row.title,++ company: row.company,++ location: row.location || null,++ remote: !!row.remote,++ tags: parseTags(row.tags),++ job_type: row.job_type || null,++ salary: row.salary || null,++ excerpt: row.excerpt,++ posted_at: row.posted_at || null,++ fetched_at: row.fetched_at,++ first_seen_at: row.first_seen_at,++ attribution: (source && source.attribution) || null,++ };++}++++// The filters a caller set, as bound parameters — never string-built. Everything is a LIKE on a lowercased column,++// and the term's own % and _ are literal characters.++const PARAM = {++ q: (n) => `($${n} = '' OR search_text LIKE $${n} ESCAPE '\\')`,++ remote: (n) => `($${n} = '' OR remote = ($${n} = 'true'))`,++ type: (n) => `($${n} = '' OR lower(coalesce(job_type, '')) LIKE $${n} ESCAPE '\\')`,++ location: (n) => `($${n} = '' OR lower(coalesce(location, '')) LIKE $${n} ESCAPE '\\')`,++};++const valueOf = { q: (f) => (f.q ? like(f.q) : ''), remote: (f) => f.remote, type: (f) => (f.type ? like(f.type) : ''), location: (f) => (f.location ? like(f.location) : '') };++++/**++ * One page of listings, newest first, plus the facets for the current scope. The cursor is the position of the last++ * row (sort_at, id); a page is the rows strictly after it, so a listing that arrives mid-scroll does not shift it.++ */++async function search(s, filters, { limit = PAGE_SIZE, cursor = null } = {}) {++ const f = filtersOf(filters);++ const n = Math.min(MAX_PAGE, Math.max(1, Number(limit) || PAGE_SIZE));++ const args = [];++ const where = ['expired = false'];++ for (const name of ['q', 'remote', 'type', 'location']) {++ args.push(valueOf[name](f));++ where.push(PARAM[name](args.length));++ }++ const scope = where.join(' AND ');++ // How many listings the reader's filters actually match, counted exactly rather than inferred from the page.++ const matches = Number(await s.db.value(`SELECT count(*) AS n FROM work_listings WHERE ${scope}`, args));++ const at = cursor && cursor.sort_at && cursor.id ? cursor : null;++ if (at) {++ args.push(at.sort_at, at.id);++ where.push(`(sort_at, id) < ($${args.length - 1}, $${args.length})`);++ }++ const rows = await s.db.many(++ `SELECT * FROM work_listings WHERE ${where.join(' AND ')} ORDER BY sort_at DESC, id DESC LIMIT ${n + 1}`, args);++ const pageRows = rows.slice(0, n);++ const last = pageRows[pageRows.length - 1];++ return {++ rows: pageRows,++ next: rows.length > n && last ? { sort_at: last.sort_at, id: last.id } : null,++ facets: { matches, ...(await facets(s, f)) },++ };++}++++/**++ * The counts the filter bar shows: how many are remote and not, and the job types present, for the text and location++ * in force. The remote and type filters are deliberately left out of their own facet — applied, they would show only++ * the answer the reader already chose.++ */++async function facets(s, f) {++ const args = [valueOf.q(f), valueOf.location(f)];++ const scope = `expired = false AND ${PARAM.q(1)} AND ${PARAM.location(2)}`;++ const remoteRows = await s.db.many(`SELECT remote, count(*) AS n FROM work_listings WHERE ${scope} GROUP BY remote`, args);++ const typeRows = await s.db.many(++ `SELECT lower(job_type) AS t, count(*) AS n FROM work_listings++ WHERE ${scope} AND coalesce(job_type, '') <> ''++ GROUP BY lower(job_type) ORDER BY n DESC, t ASC LIMIT ${TYPE_FACETS}`, args);++ const remote = { true: 0, false: 0 };++ for (const r of remoteRows) remote[r.remote ? 'true' : 'false'] = num(r.n);++ // `scope` counts the words and the location only — the population each facet below divides up.++ return { remote, type: typeRows.map((r) => ({ type: r.t, count: num(r.n) })), scope: remote.true + remote.false };++}++++const getListing = (s, id) => (LISTING_ID.test(String(id)) ? s.db.maybe('SELECT * FROM work_listings WHERE id = $1 AND expired = false', [id]) : Promise.resolve(null));++++/** How many live listings each source has, and when one was last seen — the home page's provenance line. */++async function countsBySource(s) {++ const rows = await s.db.many(++ `SELECT source, count(*) AS n, max(last_seen_at) AS last_seen_at++ FROM work_listings WHERE expired = false GROUP BY source`);++ const out = new Map(rows.map((r) => [r.source, { count: num(r.n), last_seen_at: r.last_seen_at }]));++ return out;++}++++/** The newest live listings; `remote: true` is the remote-only selection the home page shows. */++const listRecent = (s, limit = 5, { remote = false } = {}) => s.db.many(++ `SELECT * FROM work_listings WHERE expired = false${remote ? ' AND remote = true' : ''} ORDER BY sort_at DESC, id DESC LIMIT $1`,++ [Math.min(50, Math.max(1, limit))]);++++/** The ids of the newest live listings, for the sitemap: its own cap, so a big board cannot bloat the file. */++const recentIds = (s, limit = 1000) => s.db.many(++ `SELECT id FROM work_listings WHERE expired = false ORDER BY sort_at DESC, id DESC LIMIT $1`, [Math.min(1000, Math.max(1, limit))]);++++/** The listings first seen after a saved search's last look — what "new since you last looked" counts. */++function newCountSql() {++ return `SELECT count(*) AS n FROM work_listings++ WHERE expired = false AND first_seen_at > $1++ AND ${PARAM.q(2)} AND ${PARAM.remote(3)} AND ${PARAM.type(4)} AND ${PARAM.location(5)}`;++}++++// ── ingest writes ───────────────────────────────────────────++/**++ * Upsert one source's listings. A listing the board still shows is updated in place and its last_seen_at moves; one++ * that reappears after expiring comes back (expired = false). Returns { seen, added, updated }.++ */++async function upsertListings(s, rows, { sourceId, fetchedAt }) {++ let added = 0;++ let updated = 0;++ await s.tx(async () => {++ for (const row of rows) {++ const out = await s.db.maybe(++ `INSERT INTO work_listings (id, source, source_id, url, title, company, location, remote, tags, job_type, salary, excerpt, search_text, posted_at, sort_at, fetched_at, first_seen_at, last_seen_at, expired)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $16, $16, false)++ ON CONFLICT (source, source_id) DO UPDATE SET++ url = EXCLUDED.url, title = EXCLUDED.title, company = EXCLUDED.company, location = EXCLUDED.location,++ remote = EXCLUDED.remote, tags = EXCLUDED.tags, job_type = EXCLUDED.job_type, salary = EXCLUDED.salary,++ excerpt = EXCLUDED.excerpt, search_text = EXCLUDED.search_text, posted_at = EXCLUDED.posted_at,++ sort_at = EXCLUDED.sort_at, fetched_at = EXCLUDED.fetched_at, last_seen_at = EXCLUDED.last_seen_at,++ expired = false++ RETURNING (xmax = 0) AS inserted`,++ [row.id, row.source, row.source_id, row.url, row.title, row.company, row.location, row.remote, row.tags, row.job_type, row.salary, row.excerpt, row.search_text, row.posted_at, row.sort_at, fetchedAt]);++ if (out && out.inserted) added++; else updated++;++ }++ });++ return { seen: rows.length, added, updated };++}++++/** A source's fetch record: started now, finished when it finished, with what happened. */++const beginFetch = (s, sourceId, startedAt) => s.db.query(++ `INSERT INTO work_source_fetches (source, started_at, finished_at, ok, listings, error) VALUES ($1, $2, NULL, false, 0, NULL)++ ON CONFLICT (source) DO UPDATE SET started_at = EXCLUDED.started_at, finished_at = NULL, ok = false, listings = 0, error = NULL`,++ [sourceId, startedAt]);++++const finishFetch = (s, sourceId, { ok, listings, error, finishedAt }) => s.db.query(++ 'UPDATE work_source_fetches SET finished_at = $2, ok = $3, listings = $4, error = $5 WHERE source = $1',++ [sourceId, finishedAt, ok, listings, error ? String(error).slice(0, 300) : null]);++++async function sourceFetches(s) {++ const rows = await s.db.many('SELECT * FROM work_source_fetches');++ return new Map(rows.map((r) => [r.source, r]));++}++++/**++ * Listings the boards stopped showing go. A listing not seen for `days` is expired rather than deleted: the row keeps++ * its history, the search and the pages stop offering it, and a board that shows it again brings it back.++ */++async function expire(s, days, now) {++ const cutoff = new Date(now - days * 86_400_000).toISOString();++ return s.db.exec('UPDATE work_listings SET expired = true WHERE expired = false AND last_seen_at < $1', [cutoff]);++}++++// ── saved searches ──────────────────────────────────────────++const SAVED_ID = /^ssc_[0-9A-HJKMNP-TV-Z]{26}$/;++const MAX_SAVED = 50;++++const listSaved = (s, subject) => s.db.many('SELECT * FROM work_saved_searches WHERE subject = $1 ORDER BY created_at DESC, id DESC', [subject]);++++function toSavedWire(row, { newCount = 0 } = {}) {++ const query = queryOf(row);++ return {++ id: row.id,++ query: row.query || '',++ filters: { remote: row.remote || '', type: row.type || '', location: row.location || '' },++ created_at: row.created_at,++ last_seen_at: row.last_seen_at,++ new_count: Number(newCount) || 0,++ // Where to run it again: the person's own search, reproducible from the saved row alone.++ search_path: `/jobs${query ? `?${query}` : ''}`,++ };++}++++async function createSaved(s, subject, filters, { id, now }) {++ const f = filtersOf(filters);++ const count = await s.db.value('SELECT count(*) AS n FROM work_saved_searches WHERE subject = $1', [subject]);++ if (Number(count) >= MAX_SAVED) return { error: 'too_many' };++ const row = await s.db.maybe(++ `INSERT INTO work_saved_searches (id, subject, query, remote, type, location, created_at, last_seen_at)++ VALUES ($1, $2, $3, $4, $5, $6, $7, $7) RETURNING *`,++ [id, subject, f.q, f.remote, f.type, f.location, now]);++ return { row };++}++++const getSaved = (s, subject, id) => (SAVED_ID.test(String(id)) ? s.db.maybe('SELECT * FROM work_saved_searches WHERE id = $1 AND subject = $2', [id, subject]) : Promise.resolve(null));++const deleteSaved = (s, subject, id) => (SAVED_ID.test(String(id)) ? s.db.exec('DELETE FROM work_saved_searches WHERE id = $1 AND subject = $2', [id, subject]) : Promise.resolve(0));++++/** The count of listings that first appeared after the saved search was last looked at. */++async function newCount(s, row) {++ const f = filtersOf(row);++ return Number(await s.db.value(newCountSql(), [row.last_seen_at, valueOf.q(f), valueOf.remote(f), valueOf.type(f), valueOf.location(f)]));++}++++/** Mark a saved search as looked at (the moment its count stops being "new"). */++const touchSaved = (s, subject, id, at) => s.db.query('UPDATE work_saved_searches SET last_seen_at = $3 WHERE id = $1 AND subject = $2', [id, subject, at]);++++/** The query string a saved search runs, for the "run it again" link and the API's search_url. */++function queryOf(row) {++ const params = new URLSearchParams();++ if (row.query) params.set('q', row.query);++ if (row.remote) params.set('remote', row.remote);++ if (row.type) params.set('type', row.type);++ if (row.location) params.set('location', row.location);++ return params.toString();++}++++// ── the cursor ──────────────────────────────────────────────++const encodeCursor = (c) => (c ? Buffer.from(JSON.stringify([c.sort_at, c.id]), 'utf8').toString('base64url') : null);++function decodeCursor(v) {++ if (typeof v !== 'string' || !v || v.length > 200) return null;++ try {++ const [at, id] = JSON.parse(Buffer.from(v, 'base64url').toString('utf8'));++ // Both parts go into a SQL comparison; only the two shapes this service writes are accepted.++ if (typeof at !== 'string' || !/^\d{4}-\d{2}-\d{2}T[\d:.]+Z$/.test(at) || !LISTING_ID.test(String(id))) return null;++ return { sort_at: at, id: String(id) };++ } catch { return null; }++}++++module.exports = {++ search, facets, getListing, countsBySource, listRecent, recentIds, upsertListings, beginFetch, finishFetch, sourceFetches,++ expire, listSaved, createSaved, getSaved, deleteSaved, newCount, touchSaved, toWire, toSavedWire, filtersOf,++ idOf, encodeCursor, decodeCursor, PAGE_SIZE, MAX_PAGE, MAX_SAVED, LISTING_ID,++};nnew file mode 100644@@@ -0,0 +1,53 @@++'use strict';++++/**++ * The boards send a listing's description as HTML. We never republish it: we store a short plain-text excerpt and++ * link back to the listing for the rest. These are the two steps that make that safe — strip the markup, then cut on++ * a word boundary — and they are deliberately conservative: script and style content goes with the tags, and anything++ * that still looks like markup after the strip is removed rather than shown.++ */++++const EXCERPT_CHARS = 300;++++const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', '#39': "'", '#34': '"', '#38': '&', '#60': '<', '#62': '>' };++++function decode(text) {++ return String(text).replace(/&(#x?[0-9a-f]+|[a-z]+);/gi, (m, body) => {++ const key = body.toLowerCase();++ if (Object.prototype.hasOwnProperty.call(ENTITIES, key)) return ENTITIES[key];++ if (key[0] === '#') {++ const code = key[1] === 'x' ? parseInt(key.slice(2), 16) : parseInt(key.slice(1), 10);++ // Only real, printable code points: a control character or a surrogate is not text anyone wrote.++ if (Number.isFinite(code) && code >= 32 && code <= 0x10ffff && !(code >= 0xd800 && code <= 0xdfff)) {++ try { return String.fromCodePoint(code); } catch { return ' '; }++ }++ }++ return ' ';++ });++}++++/** The description as plain text: no tags, no script or style content, whitespace collapsed. */++function stripHtml(html) {++ let s = String(html == null ? '' : html);++ s = s.replace(/<!--[\s\S]*?-->/g, ' ');++ // A script or style element's content is code, not prose; drop it with its tags.++ s = s.replace(/<(script|style|noscript|template|svg)\b[^>]*>[\s\S]*?<\/\1\s*>/gi, ' ');++ // Block boundaries become a space so words either side do not run together.++ s = s.replace(/<\/?(br|p|div|li|ul|ol|tr|td|th|h[1-6]|section|article|header|footer|blockquote|pre|figure|hr)\b[^>]*>/gi, ' ');++ s = s.replace(/<[^>]*>/g, ' ');++ s = decode(s);++ // An unmatched '<' left by broken markup is not something a listing should show.++ s = s.replace(/<[^>]*$/g, ' ').replace(/[<>]/g, ' ');++ return s.replace(/\s+/g, ' ').trim();++}++++/** The excerpt: the first ~300 characters of the plain text, cut on a word, with a trailing ellipsis. */++function excerpt(html, max = EXCERPT_CHARS) {++ const text = stripHtml(html);++ if (text.length <= max) return text;++ const cut = text.slice(0, max);++ const at = cut.lastIndexOf(' ');++ return `${(at > max * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.\-–—]+$/, '')}…`;++}++++module.exports = { stripHtml, excerpt, EXCERPT_CHARS };@@@ -24,6 +24,9 @@ const TAGLINE = 'Find work, with an agent on your side.';const PUBLIC_DIR = path.join(__dirname, '..', '..', 'public');// The product's own navigation: one entry per public page it serves.const NAV = [++ { label: 'Jobs', href: '/jobs' },++ { label: 'Sources', href: '/sources' },++ { label: 'API', href: '/docs' },{ label: 'What shipped', href: '/updates' },];nnew file mode 100644@@@ -0,0 +1,96 @@++'use strict';++++/**++ * How a listing is shown, in one place so the list and the detail page cannot drift apart: the fields the board++ * gave, the short excerpt, and the provenance every listing carries — the source named and the original linked.++ *++ * Everything dynamic goes through the `html` tagged template, so a board's text (a title with <script> in it, a++ * company name with quotes) is escaped where it is shown, and a URL is only ever a link when it is the http(s) URL++ * the ingest kept.++ */++const { html, badge } = require('./html');++const { stripHtml } = require('../jobs/text');++++const MONTHS = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];++const DAYS = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];++++/** A date out of an ISO instant, in plain words; '' when there is none. */++function dateWords(iso) {++ const d = iso ? new Date(iso) : null;++ if (!d || Number.isNaN(d.getTime())) return '';++ return `${d.getUTCDate()} ${MONTHS[d.getUTCMonth()]} ${d.getUTCFullYear()}`;++}++++const meta = (row) => ({++ id: row.id,++ url: row.url,++ title: row.title,++ company: row.company,++ location: row.location || '',++ remote: !!row.remote,++ tags: String(row.tags || '').split(',').map((t) => t.trim()).filter(Boolean),++ job_type: row.job_type || '',++ salary: row.salary || '',++ excerpt: row.excerpt,++ posted_at: row.posted_at || null,++ source: row.source,++});++++/** The source's name, linked to the original listing: the attribution its terms require, in one element. */++function sourceLink(row, source) {++ const name = (source && source.name) || row.source;++ return html`<a class="job-source" href="${row.url}" rel="nofollow noopener external" target="_blank">${name}</a>`;++}++++/** The one-line provenance: where it came from, linked back, and when we read that. */++function provenance(row, source) {++ return html`<p class="job-provenance"><span class="muted">Listing from</span> ${sourceLink(row, source)}${row.posted_at ? html` <span class="muted">· posted ${dateWords(row.posted_at)}</span>` : ''}</p>`;++}++++/** One listing in a list: title, who and where, the excerpt, the tags, and the link back to the board. */++function listingItem(row, { source }) {++ const l = meta(row);++ return html`<li class="job-item" id="${l.id}">++<h3 class="job-title"><a href="/jobs/${l.id}">${l.title}</a></h3>++<p class="job-meta"><span class="job-company">${l.company}</span> · <span class="job-location">${l.location || 'Location not given'}</span>${l.remote ? html` · ${badge('Remote', 'ok')}` : ''}${l.job_type ? html` · <span class="job-type">${l.job_type}</span>` : ''}${l.salary ? html` · <span class="job-salary">${l.salary}</span>` : ''}</p>++<p class="job-excerpt">${l.excerpt}</p>++${l.tags.length ? html`<ul class="job-tags">${l.tags.slice(0, 8).map((t) => html`<li>${t}</li>`)}</ul>` : ''}++${provenance(row, source)}++</li>`;++}++++/** A list of listings, or a line saying there is nothing to show. */++function listingList(rows, { sourceOf, empty = 'No listings match this search.' } = {}) {++ if (!rows.length) return html`<p class="muted">${empty}</p>`;++ return html`<ul class="job-list">${rows.map((row) => listingItem(row, { source: sourceOf(row) }))}</ul>`;++}++++/**++ * schema.org JobPosting for one listing, from the fields the board gave and nothing else: the url is the original++ * listing (never this page), and the description is the excerpt we hold — the full text stays on the board.++ */++function jobPosting(row, { siteUrl }) {++ const l = meta(row);++ const node = {++ '@context': 'https://schema.org',++ '@type': 'JobPosting',++ title: l.title,++ description: l.excerpt,++ url: l.url,++ identifier: { '@type': 'PropertyValue', name: l.source, value: row.source_id || l.id },++ hiringOrganization: { '@type': 'Organization', name: l.company },++ directApply: false,++ publisher: { '@type': 'Organization', name: 'OpenVibe.Work', url: siteUrl },++ };++ if (l.posted_at) node.datePosted = l.posted_at;++ if (l.job_type) node.employmentType = l.job_type;++ if (l.remote) node.jobLocationType = 'TELECOMMUTE';++ if (l.location) node.jobLocation = { '@type': 'Place', address: { '@type': 'PostalAddress', addressLocality: l.location } };++ if (l.salary) node.baseSalary = { '@type': 'MonetaryAmount', description: l.salary };++ return node;++}++++/** The excerpt as plain text, for the meta description and the API. */++const plainExcerpt = (row) => stripHtml(row.excerpt || '');++++module.exports = { listingItem, listingList, sourceLink, provenance, jobPosting, dateWords, plainExcerpt, meta };@@@ -20,8 +20,14 @@ const READ = 'work.api.read';const rosa = t.network.addUser('rosa');const sam = t.network.addUser('sam');-- await check('BUDGETS starts empty: the product declares its own expensive routes', () => {-- assert.deepStrictEqual(BUDGETS, {});++ await check('the product declares its own expensive routes in BUDGETS, and only those', () => {++ // Searching the listings and saving a search: the two routes the product gives numbers of their own.++ assert.deepStrictEqual(Object.keys(BUDGETS).sort(), ['work.jobs.search', 'work.saved_searches.write']);++ for (const [name, own] of Object.entries(BUDGETS)) {++ assert.ok(name.startsWith('work.'), name);++ assert.ok(own.minute > 0 && own.hour >= own.minute, name);++ }++ assert.throws(() => t.ctx.limits.budget('work.thing.create'), /no budget named/);});await check('a read: 3 a minute per address, then 429 rate_limited with Retry-After; another address passes', async () => {@@@ -55,7 +55,7 @@ const PRIVATE = ['/auth/', '/api/'];for (const p of ['/auth/']) assert.ok(!r.text.includes(`https://openvibe.work${p}`), `private path in llms.txt: ${p}`);});-- await check('llms-full.txt: 200 text/plain, the site title, and every page the sitemap lists', async () => {++ await check('llms-full.txt: 200 text/plain, the site title, and every page the sitemap lists that is the site\'s own', async () => {const r = await t.get('/llms-full.txt');assert.strictEqual(r.status, 200);assert.match(r.headers.get('content-type'), /^text\/plain/);@@@ -64,8 +64,12 @@ const PRIVATE = ['/auth/', '/api/'];const sitemap = await t.get('/sitemap.xml');const locs = [...sitemap.text.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);assert.ok(locs.length >= 2, `sitemap too small: ${locs.length}`);-- for (const loc of locs) assert.ok(r.text.includes(loc), `llms-full.txt is missing ${loc}`);++ // The site's own pages are described in full. A listing URL is in the sitemap (so a crawler follows it to++ // the board) but deliberately not in llms-full.txt: there is no full text here to give, only an excerpt.++ const own = locs.filter((l) => !/^https:\/\/openvibe\.work\/jobs\/job_/.test(l));++ for (const loc of own) assert.ok(r.text.includes(loc), `llms-full.txt is missing ${loc}`);assert.ok(r.text.includes('URL: https://openvibe.work/updates\n'), 'no /updates entry');++ assert.ok(r.text.includes('https://openvibe.work/jobs'), 'the listings are not named');for (const p of PRIVATE) assert.ok(!r.text.includes(`https://openvibe.work${p}`), `private path in llms-full.txt: ${p}`);});nnew file mode 100644@@@ -0,0 +1,179 @@++'use strict';++/**++ * Stand-in job boards for the ingest tests: one HTTP server with a path per board, answering the shape each real API++ * documents, with small fixtures written to look like the real thing (including RemoteOK's legal notice as the first++ * array element, and an Arbeitnow listing whose title carries a <script> tag, so the pages' escaping is provable).++ *++ * const b = await startBoards();++ * b.env WORK_ARBEITNOW_URL / WORK_REMOTIVE_URL / WORK_REMOTEOK_URL, for boot({ env })++ * b.state.arbeitnow = 'fail' one board answering HTTP 500 (default 'ok')++ * b.state.remotive = 'json' one board answering something that is not JSON++ * b.add('arbeitnow', row) a listing the board starts showing (see b.fixtures for the shape)++ * b.requests [{ path, query, userAgent }] for every call++ *++ * The fixtures are ours, not a board's: no real listing is copied here.++ */++const http = require('http');++++/** One listing per board, in that board's own payload shape. */++function fixtures() {++ return {++ arbeitnow: [++ {++ slug: 'senior-backend-engineer-berlin',++ company_name: 'Nordlicht Systems',++ title: 'Senior Backend Engineer (Node.js)',++ description: '<p>We run a fleet of Node.js services.</p><ul><li>PostgreSQL</li><li>Valkey</li></ul><p>You will own the billing pipeline end to end.</p>',++ remote: false,++ url: 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345',++ tags: ['nodejs', 'postgresql'],++ job_types: ['permanent'],++ location: 'Berlin, Germany',++ created_at: 1791500000,++ },++ {++ slug: 'frontend-engineer-remote-eu',++ company_name: 'Helles Werk',++ title: 'Frontend Engineer <script>alert("xss")</script>',++ description: '<div>Build the design system.<script>window.__pwned = true</script></div><div>TypeScript, CSS.</div>',++ remote: true,++ url: 'https://www.arbeitnow.com/view/frontend-engineer-remote-eu-23456',++ tags: ['typescript', 'css'],++ job_types: ['permanent', 'full_time'],++ location: 'Remote (EU)',++ created_at: 1791600000,++ },++ ],++ arbeitnowPage2: [++ {++ slug: 'data-analyst-munich',++ company_name: 'Alpen Daten',++ title: 'Data Analyst',++ description: '<p>SQL, dashboards and a lot of questions.</p>',++ remote: false,++ url: 'https://www.arbeitnow.com/view/data-analyst-munich-34567',++ tags: ['sql'],++ job_types: ['permanent'],++ location: 'Munich, Germany',++ created_at: 1791700000,++ },++ ],++ remotive: [++ {++ id: 1900123,++ url: 'https://remotive.com/remote-jobs/software-dev/support-engineer-1900123',++ title: 'Support Engineer',++ company_name: 'Cassiopeia',++ category: 'Software Development',++ tags: ['customer support', 'saas'],++ job_type: 'full_time',++ publication_date: '2026-09-30T09:12:00',++ candidate_required_location: 'Europe',++ salary: '€45,000 – €55,000',++ description: '<p>Answer customers, write runbooks.</p><p>Postgres and a sharp eye.</p>',++ },++ {++ id: 1900789,++ url: 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789',++ title: 'Platform Engineer',++ company_name: 'Meridian Rail',++ category: 'Software Development',++ tags: ['kubernetes', 'terraform'],++ job_type: 'full_time',++ publication_date: '2026-09-29T07:45:00',++ candidate_required_location: 'Remote (EU)',++ salary: '€70,000 – €85,000',++ // Longer than the excerpt we keep, so a test can prove the tail is never stored or shown.++ description: `<p>You will run the platform other engineers build on.</p>++<p>That means Kubernetes clusters, Terraform modules, a queue nobody has to think about, and an on-call rotation that++people actually join voluntarily. We look after the boring parts of shipping software so product teams do not have to.</p>++<p>You will work with people across four time zones, mostly in writing. We keep our runbooks honest and our postmortems++blameless. The office dog is called Waffle.</p>`,++ },++ {++ id: 1900456,++ url: 'https://remotive.com/remote-jobs/marketing/content-marketer-1900456',++ title: 'Content Marketer',++ company_name: 'Longform Ltd',++ category: 'Marketing',++ tags: ['writing'],++ job_type: 'contract',++ publication_date: '2026-09-28T14:00:00',++ candidate_required_location: 'Worldwide',++ salary: '',++ description: '<p>Write about developer tools.</p>',++ },++ ],++ remoteok: [++ { legal: "RemoteOK's API is offered for personal use; please link back to the listing and name RemoteOK as the source." },++ {++ id: '1049001',++ slug: 'devops-engineer-skyward',++ company: 'Skyward Cloud',++ position: 'DevOps Engineer',++ tags: ['devops', 'kubernetes'],++ location: 'Worldwide',++ salary_min: 90000,++ salary_max: 130000,++ date: '2026-10-01T08:00:00+00:00',++ url: 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001',++ description: '<p>Kubernetes, Terraform, on-call.</p>',++ },++ {++ id: '1049002',++ slug: 'technical-writer-atlas',++ company: 'Atlas Docs',++ position: 'Technical Writer',++ tags: ['docs'],++ location: 'EU',++ salary_min: null,++ salary_max: null,++ date: '2026-09-25T08:00:00+00:00',++ url: 'https://remoteok.com/remote-jobs/remote-technical-writer-atlas-1049002',++ description: '<p>Document an API.</p>',++ },++ ],++ };++}++++async function startBoards() {++ const data = fixtures();++ const requests = [];++ const state = { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' };++ const added = { arbeitnow: [], remotive: [], remoteok: [] };++++ const send = (res, status, body, type = 'application/json') => { res.writeHead(status, { 'content-type': type }); res.end(typeof body === 'string' ? body : JSON.stringify(body)); };++++ const server = http.createServer((req, res) => {++ const url = new URL(req.url, 'http://stand-in');++ requests.push({ path: url.pathname, query: url.search, userAgent: req.headers['user-agent'] || '', accept: req.headers.accept || '' });++ const board = url.pathname.replace(/^\//, '');++ if (!['arbeitnow', 'remotive', 'remoteok'].includes(board)) return send(res, 404, { error: 'not a board' });++ if (state[board] === 'fail') return send(res, 500, { error: 'the board is having a bad day' }, 'text/plain');++ if (state[board] === 'html') return send(res, 200, '<html>not json</html>', 'text/html');++ if (board === 'arbeitnow') {++ const page = Number(url.searchParams.get('page') || 1);++ const rows = page === 1 ? data.arbeitnow : page === 2 ? data.arbeitnowPage2 : [];++ return send(res, 200, { data: [...rows, ...added.arbeitnow], links: { next: page < 2 ? `https://www.arbeitnow.com/api/job-board-api?page=${page + 1}` : null } });++ }++ if (board === 'remotive') return send(res, 200, { 'job-count': data.remotive.length + added.remotive.length, jobs: [...data.remotive, ...added.remotive] });++ // RemoteOK: the legal notice is element 0, exactly as the real API sends it.++ return send(res, 200, [...data.remoteok.slice(0, 1), ...data.remoteok.slice(1), ...added.remoteok]);++ });++ await new Promise((r) => server.listen(0, '127.0.0.1', r));++ const base = `http://127.0.0.1:${server.address().port}`;++++ return {++ url: base,++ fixtures: data,++ state,++ requests,++ env: { WORK_ARBEITNOW_URL: `${base}/arbeitnow`, WORK_REMOTIVE_URL: `${base}/remotive`, WORK_REMOTEOK_URL: `${base}/remoteok` },++ /** Make a board start showing one more listing (the shape of that board's own payload). */++ add(board, row) { added[board].push(row); },++ reset() { Object.assign(state, { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' }); requests.length = 0; },++ close: () => new Promise((r) => server.close(r)),++ };++}++++module.exports = { startBoards, fixtures };@@@ -10,10 +10,15 @@ process.env.WORK_API_RATE_LIMIT_PER_MIN = process.env.WORK_API_RATE_LIMIT_PER_MI* const t = await boot();* t.get(path, { as: user, bearer, json, form, headers, method, cookie })* t.signIn(user) → cookie header value (work_at = a Network user token, as after /auth/callback)-- * t.logs(), t.dbDump()++ * t.boards (the stand-in job boards: state, add, requests), t.logs(), t.dbDump()++ *++ * The three job-board base URLs always point at the stand-in (test/helpers/boards.js), so an ingest run in a test++ * never touches the internet. The ingest itself is off unless a test asks for it: it runs when a test calls++ * t.ctx.ingest.runSource(…) or t.ctx.ingest.runAll().*/const http = require('http');const { startNetwork } = require('./mocks');++const { startBoards } = require('./boards');const captured = [];for (const m of ['log', 'info', 'warn', 'error']) {@@@ -23,10 +28,12 @@ for (const m of ['log', 'info', 'warn', 'error']) {async function boot(opts = {}) {const network = await startNetwork(opts.network || {});++ const boards = await startBoards();const env = {NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.work', TRUST_PROXY: '1',OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,OV_OAUTH_CLIENT_ID: 'work', OV_OAUTH_CLIENT_SECRET: 'work-secret', COOKIE_SECURE: 'false',++ ...boards.env,...(opts.env || {}),};for (const [k, v] of Object.entries(opts.env || {})) if (v === null) delete env[k];@@@ -66,12 +73,14 @@ async function boot(opts = {}) {}const t = {-- base, network, config, ctx: built.ctx, get, signIn, dbDump,++ base, network, boards, config, ctx: built.ctx, get, signIn, dbDump,logs: () => captured.join('\n'),async close() {++ built.ctx.ingest.stop();await new Promise((r) => server.close(r));built.ctx.keys.client.stop();await testdb.close();++ await boards.close();await network.close();},};nnew file mode 100644@@@ -0,0 +1,173 @@++'use strict';++/**++ * The background ingest against the stand-in boards (test/helpers/boards.js): what it keeps from each board and what++ * it refuses, that reading a board twice updates rather than duplicates, that a board which fails is recorded and++ * does not stop the others, that only a short plain-text excerpt is stored, and that a listing no board has shown for++ * the retention window expires out of the search and the pages.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++const store = require('../server/jobs/store');++++const DAY = 86_400_000;++++(async () => {++ const clock = { t: Date.parse('2026-10-01T06:00:00Z') };++ const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });++ const ingest = t.ctx.ingest;++ const s = t.ctx.s;++++ try {++ await check('reading the three boards keeps every listing with its source, its id and the original URL', async () => {++ const out = await ingest.runAll();++ assert.deepStrictEqual(out.map((o) => o.source), ['arbeitnow', 'remotive', 'remoteok']);++ assert.ok(out.every((o) => o.ok), JSON.stringify(out.map((o) => o.error)));++ // Arbeitnow's two pages (2 + 1), Remotive's two jobs, RemoteOK's two listings.++ const counts = await store.countsBySource(s);++ assert.strictEqual(counts.get('arbeitnow').count, 3);++ assert.strictEqual(counts.get("remotive").count, 3);++ assert.strictEqual(counts.get('remoteok').count, 2);++++ const page = (await t.get('/api/v1/jobs?q=DevOps')).json();++ assert.strictEqual(page.jobs.length, 1);++ const job = page.jobs[0];++ assert.strictEqual(job.source, 'remoteok');++ assert.strictEqual(job.source_name, 'RemoteOK');++ assert.strictEqual(job.url, 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001');++ assert.strictEqual(job.company, 'Skyward Cloud');++ assert.strictEqual(job.salary, '$90,000–$130,000 a year');++ assert.strictEqual(job.remote, true);++ assert.deepStrictEqual(job.tags, ['devops', 'kubernetes']);++ assert.ok(job.posted_at.startsWith('2026-10-01'));++++ // RemoteOK's first array element is a legal notice, not a listing: nothing was made of it.++ const all = (await t.get('/api/v1/jobs?limit=100')).json();++ assert.strictEqual(all.count, 8);++ assert.ok(!JSON.stringify(all.jobs).includes('personal use'), 'the legal notice became a listing');++ });++++ await check('every request used this service\'s User-Agent, a JSON Accept, and only the configured hosts', async () => {++ assert.ok(t.boards.requests.length >= 4);++ for (const r of t.boards.requests) {++ assert.strictEqual(r.userAgent, 'OpenVibeWork/0.1 (+https://openvibe.work)', r.path);++ assert.match(r.accept, /application\/json/);++ }++ assert.deepStrictEqual([...new Set(t.boards.requests.map((r) => r.path))].sort(), ['/arbeitnow', '/remoteok', '/remotive']);++ // Arbeitnow is paginated as configured (WORK_INGEST_PAGES=2); Remotive is asked for a limit.++ assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=1'));++ assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=2'));++ assert.ok(t.boards.requests.some((r) => r.path === '/remotive' && r.query === '?limit=100'));++ });++++ await check('an HTML description becomes a short plain-text excerpt: no tags, no script code, under 320 characters', async () => {++ const job = (await t.get('/api/v1/jobs?q=Frontend')).json().jobs[0];++ assert.ok(!/[<>]/.test(job.excerpt), job.excerpt);++ assert.ok(!/window\.__pwned/.test(job.excerpt), 'script content was kept');++ assert.ok(job.excerpt.includes('Build the design system'), job.excerpt);++ assert.ok(job.excerpt.length <= 320, String(job.excerpt.length));++ const page = (await t.get('/api/v1/jobs?q=Senior Backend')).json().jobs[0];++ assert.ok(page.excerpt.includes('PostgreSQL'), page.excerpt);++ assert.ok(page.excerpt.includes('fleet of Node.js services'), page.excerpt);++ // The full description is nowhere in the database: only the excerpt is stored.++ const dump = await t.dbDump();++ assert.ok(!dump.includes('billing pipeline end to end</p>'), 'the full description was stored');++ });++++ await check('the text helper strips markup and entities, and cuts a long excerpt on a word', () => {++ const { stripHtml, excerpt, EXCERPT_CHARS } = require('../server/jobs/text');++ assert.strictEqual(stripHtml('<p>Hello <b>world</b> — it's 5 & 6 <ok></p>'), "Hello world — it's 5 & 6 ok");++ assert.strictEqual(stripHtml('<div>a</div><div>b</div>'), 'a b');++ assert.strictEqual(stripHtml('<style>p{color:red}</style><p>real</p>'), 'real');++ assert.strictEqual(stripHtml('<script>alert(1)</script>'), '');++ assert.strictEqual(stripHtml(null), '');++ // A doubly-encoded tag decodes to text and is then dropped, never re-formed into markup.++ assert.ok(!stripHtml('<script>alert(1)</script>').includes('<'));++ assert.strictEqual(excerpt('<p>short</p>'), 'short');++ const long = excerpt(`<p>${'word '.repeat(120)}</p>`);++ assert.ok(long.length <= EXCERPT_CHARS + 1, String(long.length));++ assert.ok(long.endsWith('…'));++ assert.ok(!long.includes(' ') && /^\S/.test(long), 'the excerpt is cut mid-word or padded');++ });++++ await check('reading a board again updates the listing in place: no duplicate, same id, newer fetch', async () => {++ const before = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];++ const seenBefore = before.fetched_at;++ clock.t += 2 * 3600_000;++ // The board's own text changed under the same id, as a board edits a listing.++ t.boards.fixtures.remoteok[1].position = 'DevOps Engineer (Platform)';++ const remoteok = (await ingest.runAll()).find((o) => o.source === 'remoteok');++ assert.deepStrictEqual([remoteok.ok, remoteok.listings, remoteok.added, remoteok.updated], [true, 2, 0, 2]);++ const after = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];++ assert.strictEqual(after.id, before.id, 'the id must not change when a board is read again');++ assert.strictEqual(after.title, 'DevOps Engineer (Platform)');++ assert.ok(after.fetched_at > seenBefore, 'fetched_at did not move');++ assert.strictEqual((await t.get("/api/v1/jobs?limit=100")).json().count, 8);++ t.boards.fixtures.remoteok[1].position = 'DevOps Engineer';++ });++++ await check('a listing the board stopped showing expires out, leaving the others', async () => {++ // Arbeitnow's page 2 goes quiet: one listing was there, is not any more.++ const gone = t.boards.fixtures.arbeitnowPage2.shift();++ clock.t += 31 * DAY;++ await ingest.runAll();++ assert.strictEqual((await store.countsBySource(s)).get('arbeitnow').count, 2);++ assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 0);++ const row = await s.db.maybe("SELECT * FROM work_listings WHERE source_id = $1 AND expired", [gone.slug]);++ assert.ok(row, 'the expired listing was deleted rather than kept');++ assert.strictEqual((await t.get(`/jobs/${row.id}`)).status, 404);++ // And it comes back if the board shows it again.++ t.boards.fixtures.arbeitnowPage2.push(gone);++ await ingest.runAll();++ assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 1);++ });++++ await check('a board that fails is recorded, says why, and does not stop the others', async () => {++ clock.t += 3600_000;++ t.boards.state.remotive = 'fail';++ t.boards.state.remoteok = 'html';++ const out = await ingest.runAll();++ assert.strictEqual(out.find((o) => o.source === 'remotive').ok, false);++ assert.strictEqual(out.find((o) => o.source === 'arbeitnow').ok, true, 'one board failing stopped another');++ const src = (await t.get('/api/v1/sources')).json();++ const remotive = src.sources.find((x) => x.id === 'remotive');++ assert.strictEqual(remotive.last_fetch.ok, false);++ assert.match(remotive.last_fetch.error, /500/);++ assert.ok(remotive.last_fetch.finished_at, 'the failure was not timestamped');++ assert.match(src.sources.find((x) => x.id === 'remoteok').last_fetch.error, /not JSON/);++ // What was already gathered stays: a failed read never empties the board.++ assert.strictEqual(remotive.listings, 3);++ t.boards.reset();++ });++++ await check('the ingest is off unless WORK_INGEST=on, so a test or a development run reads no board', async () => {++ assert.strictEqual(t.config.ingest.enabled, false);++ assert.strictEqual(ingest.start(), false);++ const other = await boot({ env: { WORK_INGEST: 'on', WORK_INGEST_INTERVAL_HOURS: '6' } });++ try {++ assert.strictEqual(other.config.ingest.enabled, true);++ assert.strictEqual(other.config.ingest.intervalMs, 6 * 3600_000);++ assert.strictEqual(other.ctx.ingest.start(), true);++ assert.strictEqual(other.ctx.ingest.timers.size, 4, 'one timer per source plus the first run');++ other.ctx.ingest.stop();++ } finally { await other.close(); }++ });++++ await check('a page that is not a listing is refused rather than published', async () => {++ // A row with no URL to link back to, or a URL that is not http(s), has no provenance: it is not kept.++ const { toRow } = require('../server/jobs/sources');++ const base = { source_id: 'x', title: 't', company: 'c', url: 'https://example.org/x', tags: [], description_html: '<p>hi</p>' };++ assert.strictEqual(toRow({ ...base, url: 'javascript:alert(1)' }, { sourceId: 'remotive', fetchedAt: 'x', idOf: store.idOf }), null);++ assert.strictEqual(toRow({ ...base, url: null }, { sourceId: 'remotive', fetchedAt: 'x', idOf: store.idOf }), null);++ assert.strictEqual(toRow({ ...base, title: '' }, { sourceId: 'remotive', fetchedAt: 'x', idOf: store.idOf }), null);++ const row = toRow(base, { sourceId: 'remotive', fetchedAt: '2026-10-01T00:00:00.000Z', idOf: store.idOf });++ assert.strictEqual(row.sort_at, '2026-10-01T00:00:00.000Z', 'a listing with no date sorts by when we read it');++ });++++ await check('a listing\'s id is derived from its source and the board\'s id, so it is stable', () => {++ assert.strictEqual(store.idOf('remotive', '1900123'), store.idOf('remotive', '1900123'));++ assert.notStrictEqual(store.idOf('remotive', '1900123'), store.idOf('remoteok', '1900123'));++ assert.match(store.idOf('arbeitnow', 'x'), store.LISTING_ID);++ });++ } finally { await t.close(); }++ done();++})().catch((e) => { console.error(e); process.exit(1); });nnew file mode 100644@@@ -0,0 +1,226 @@++'use strict';++/**++ * The listings as a reader and as a crawler meet them: the search and its filters and facets, the cursor, one listing++ * with its provenance and its JSON-LD, the escaping of a hostile title, and the discovery files that carry the++ * listings (sitemap) and the sources (llms.txt).++ *++ * The listings come from the stand-in boards through the real ingest, so what is searched is what was really stored.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++++const ldBlocks = (html) => [...html.matchAll(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/g)].map((m) => JSON.parse(m[1]));++const first = async (t, path) => (await t.get(path)).json();++++(async () => {++ const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });++ try {++ await t.ctx.ingest.runAll();++++ await check('search: text matches the title, the company and the tags; results are newest first', async () => {++ const byTitle = await first(t, '/api/v1/jobs?q=devops');++ assert.deepStrictEqual(byTitle.jobs.map((j) => j.title), ['DevOps Engineer']);++ const byCompany = await first(t, '/api/v1/jobs?q=Skyward');++ assert.deepStrictEqual(byCompany.jobs.map((j) => j.company), ['Skyward Cloud']);++ // The term is found in a tag and in the excerpt, so both listings answer.++ const byTag = await first(t, '/api/v1/jobs?q=terraform');++ assert.deepStrictEqual(byTag.jobs.map((j) => j.company).sort(), ['Meridian Rail', 'Skyward Cloud']);++ // Case does not matter, and a % a caller typed is a character, not a wildcard.++ assert.strictEqual((await first(t, '/api/v1/jobs?q=DEVOPS')).count, 1);++ assert.strictEqual((await first(t, '/api/v1/jobs?q=%25')).count, 0);++ const all = await first(t, '/api/v1/jobs?limit=100');++ const dates = all.jobs.map((j) => j.posted_at);++ assert.deepStrictEqual(dates, [...dates].sort().reverse(), 'newest first');++ });++++ await check('filters: remote, job type and location, with the facets counting the same scope', async () => {++ const remote = await first(t, '/api/v1/jobs?remote=true');++ assert.ok(remote.count > 0);++ assert.ok(remote.jobs.every((j) => j.remote === true));++ const onSite = await first(t, '/api/v1/jobs?remote=false');++ assert.ok(onSite.count > 0);++ assert.ok(onSite.jobs.every((j) => j.remote === false));++ assert.strictEqual(remote.count + onSite.count, (await first(t, '/api/v1/jobs?limit=100')).count);++++ const contract = await first(t, '/api/v1/jobs?type=contract');++ assert.deepStrictEqual(contract.jobs.map((j) => j.title), ['Content Marketer']);++ assert.strictEqual(contract.facets.matches, 1, 'the count must be what the filters match');++ assert.strictEqual(contract.facets.scope, 8, 'the type facet must not narrow its own scope');++++ const germany = await first(t, '/api/v1/jobs?location=Germany');++ assert.deepStrictEqual(germany.jobs.map((j) => j.title).sort(), ['Data Analyst', 'Senior Backend Engineer (Node.js)']);++++ const combined = await first(t, '/api/v1/jobs?q=engineer&remote=true');++ assert.ok(combined.count >= 1 && combined.jobs.every((j) => /engineer/i.test(j.title)));++ const facetTypes = (await first(t, '/api/v1/jobs')).facets.type;++ assert.ok(facetTypes.length > 0 && facetTypes.every((x) => x.type && x.count > 0), JSON.stringify(facetTypes));++ });++++ await check('the cursor pages through without repeating or losing a listing', async () => {++ const seen = [];++ let cursor = null;++ for (let i = 0; i < 5; i++) {++ const page = await first(t, `/api/v1/jobs?limit=3${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''}`);++ seen.push(...page.jobs.map((j) => j.id));++ cursor = page.next;++ if (!cursor) break;++ }++ const all = (await first(t, '/api/v1/jobs?limit=100')).jobs.map((j) => j.id);++ assert.strictEqual(seen.length, all.length);++ assert.deepStrictEqual([...new Set(seen)].length, all.length, 'a listing was repeated');++ assert.deepStrictEqual(seen, all);++ // A cursor we did not write is refused rather than guessed at.++ assert.strictEqual((await t.get('/api/v1/jobs?cursor=nonsense')).status, 422);++ });++++ await check('the list page shows every listing with its source named and linked back to the board', async () => {++ const html = (await t.get('/jobs')).text;++ assert.strictEqual((await t.get('/jobs')).status, 200);++ for (const [name, url] of [++ ['Arbeitnow', 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345'],++ ['Remotive', 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789'],++ ['RemoteOK', 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001'],++ ]) {++ assert.ok(html.includes(name), `the source ${name} is not named on the list`);++ assert.ok(html.includes(`href="${url}"`), `the original listing ${url} is not linked on the list`);++ }++ assert.match(html, /rel="nofollow noopener external"/, 'the link back must be marked as external');++ // Filters and facets are on the page, and paging is a plain link (no JavaScript).++ assert.ok(html.includes('name="remote"') && html.includes('name="location"'), 'the filter form is missing');++ assert.match(html, /class="facets"/);++ assert.match(html, /href="\/jobs\?[^"]*remote=true"/, 'a facet link is missing');++ });++++ await check('one listing: the board\'s fields, the excerpt, the source and "read the full listing on" link', async () => {++ const job = (await first(t, '/api/v1/jobs?q=Platform Engineer')).jobs[0];++ const r = await t.get(`/jobs/${job.id}`);++ assert.strictEqual(r.status, 200);++ const html = r.text;++ assert.ok(html.includes('Platform Engineer'));++ assert.ok(html.includes('Meridian Rail'));++ assert.ok(html.includes('Remote (EU)'));++ assert.ok(html.includes('€70,000 – €85,000'), 'the salary the board gave');++ assert.ok(html.includes('<li>kubernetes</li>'), 'the tags');++ assert.ok(html.includes('Read the full listing on Remotive'), 'no apply link');++ assert.ok(html.includes(`href="${job.url}"`), 'the apply link must be the original URL');++ assert.match(html, /<dt>Source<\/dt>/, 'the source is not stated');++ // The excerpt is shown; the tail of the description was never stored, so it cannot be shown.++ assert.ok(html.includes('You will run the platform other engineers build on'), 'the excerpt is missing');++ assert.ok(!html.includes('Waffle'), 'the full description was republished');++ assert.ok(html.length < 60_000, `the page is ${html.length} bytes: too much of the description is on it`);++ assert.ok(html.includes('/sources#remotive'), 'no link to the source\'s terms as applied');++ });++++ await check('the listing page carries a valid JobPosting whose url is the original, not this page', async () => {++ const job = (await first(t, '/api/v1/jobs?q=DevOps')).jobs[0];++ const html = (await t.get(`/jobs/${job.id}`)).text;++ const blocks = ldBlocks(html);++ const posting = blocks.find((b) => b['@type'] === 'JobPosting');++ assert.ok(posting, `no JobPosting node: ${JSON.stringify(blocks.map((b) => b['@type']))}`);++ assert.strictEqual(posting.title, 'DevOps Engineer');++ assert.strictEqual(posting.url, job.url);++ assert.notStrictEqual(posting.url, `https://openvibe.work/jobs/${job.id}`);++ assert.strictEqual(posting.hiringOrganization.name, 'Skyward Cloud');++ assert.strictEqual(posting.jobLocationType, 'TELECOMMUTE');++ assert.match(posting.datePosted, /^2026-10-01T/);++ assert.strictEqual(posting.identifier.name, 'remoteok');++ assert.ok(posting.description.length <= 320);++ assert.ok(posting['@context'] === 'https://schema.org');++ // Every JSON-LD block on the page parses (the escaping must survive a hostile title).++ assert.ok(blocks.length >= 1);++ });++++ await check('a hostile title is escaped everywhere it appears, and never becomes markup or a URL', async () => {++ const job = (await first(t, '/api/v1/jobs?q=Frontend Engineer')).jobs[0];++ assert.ok(job.title.includes('<script>'), 'the fixture must carry the hostile title');++ for (const path of ['/jobs', `/jobs/${job.id}`, `/api/v1/jobs?q=Frontend Engineer`]) {++ const html = (await t.get(path)).text;++ assert.ok(!html.includes('<script>alert("xss")</script>'), `${path} rendered the title as markup`);++ assert.ok(!/window\.__pwned/.test(html), `${path} rendered the description's script`);++ if (path.startsWith('/jobs')) assert.ok(html.includes('<script>alert("xss")</script>'), `${path} did not escape the title`);++ }++ // The JSON-LD on its own page is still parseable JSON with the title as a string.++ const posting = ldBlocks((await t.get(`/jobs/${job.id}`)).text).find((b) => b['@type'] === 'JobPosting');++ assert.ok(posting.title.includes('<script>'));++ });++++ await check('a listing that is not there is a 404 page and a problem+json, the same for any unknown id', async () => {++ for (const id of ['job_00000000000000000000000000', 'nope', "1' OR 1=1--"]) {++ const page = await t.get(`/jobs/${encodeURIComponent(id)}`);++ assert.strictEqual(page.status, 404, id);++ assert.match(page.text, /Listing not found/);++ const api = await t.get(`/api/v1/jobs/${encodeURIComponent(id)}`);++ assert.strictEqual(api.status, 404);++ assert.strictEqual(api.json().code, 'work.jobs.not_found');++ }++ });++++ await check('every source is listed with its terms as applied and when it was last read', async () => {++ const api = await first(t, '/api/v1/sources');++ assert.deepStrictEqual(api.sources.map((s) => s.id).sort(), ['arbeitnow', 'remoteok', 'remotive']);++ for (const src of api.sources) {++ assert.ok(src.terms.length > 40, `${src.id} has no terms`);++ assert.match(src.terms, /link|attribute|mention|name/i, `${src.id}'s terms do not state the attribution rule`);++ assert.strictEqual(src.last_fetch.ok, true);++ assert.ok(src.last_fetch.finished_at);++ assert.ok(src.listings > 0);++ }++ const html = (await t.get('/sources')).text;++ assert.ok(html.includes('id="remotive"') && html.includes('id="remoteok"') && html.includes('id="arbeitnow"'));++ assert.ok(html.includes('Listing from Remotive') && html.includes('Listing from RemoteOK'));++ assert.match(html, /four a day/, 'Remotive\'s call limit is not stated');++ });++++ await check('the sitemap carries the newest listings and the site\'s own pages, and no private path', async () => {++ const xml = (await t.get('/sitemap.xml')).text;++ const locs = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);++ const jobs = (await first(t, '/api/v1/jobs?limit=100')).jobs;++ for (const job of jobs) assert.ok(locs.includes(`https://openvibe.work/jobs/${job.id}`), `${job.id} is not in the sitemap`);++ for (const p of ['/', '/jobs', '/sources', '/docs', '/updates']) assert.ok(locs.includes(`https://openvibe.work${p}`), `${p} is not in the sitemap`);++ for (const p of ['/saved', '/auth/login', '/api/v1/jobs']) assert.ok(!xml.includes(`https://openvibe.work${p}`), `${p} must not be in the sitemap`);++ assert.ok(locs.length <= 1000 + 5);++ });++++ await check('llms.txt sends a crawler to the listings and the sources, and robots keeps /saved out', async () => {++ const llms = (await t.get('/llms.txt')).text;++ assert.ok(llms.includes('(https://openvibe.work/jobs)'));++ assert.ok(llms.includes('(https://openvibe.work/sources)'));++ assert.ok(llms.includes('Arbeitnow, Remotive, RemoteOK'), 'the sources are not named for a crawler');++ assert.ok(!llms.includes('/saved'), 'llms.txt names a private page');++ const robots = (await t.get('/robots.txt')).text;++ assert.ok(robots.includes('Disallow: /saved'));++ assert.ok(!robots.includes('Disallow: /jobs'));++ });++++ await check('the home page is the search box, the latest listings and the sources with their counts', async () => {++ const r = await t.get('/');++ assert.strictEqual(r.status, 200);++ assert.ok(r.text.includes('Find work.'), 'the hero');++ assert.match(r.text, /action="\/jobs"/, 'the search box');++ assert.ok(r.text.includes('Latest remote jobs'));++ assert.ok(r.text.includes('DevOps Engineer'), 'no listing on the home page');++ for (const [name, url] of [['Arbeitnow', 'https://www.arbeitnow.com/'], ['Remotive', 'https://remotive.com/'], ['RemoteOK', 'https://remoteok.com/']]) {++ assert.ok(r.text.includes(`href="${url}"`), `the source ${name} is not linked on the home page`);++ }++ assert.match(r.text, /no paywall/i, '"how it works" does not say what is behind a paywall (nothing)');++ });++++ await check('search is limited per caller, and past it the API says so with Retry-After', async () => {++ const busy = await boot({ callerLimits: true, limitsNow: () => Date.now(), env: { WORK_INGEST_PAGES: '1' } });++ try {++ await busy.ctx.ingest.runAll();++ let limited = null;++ for (let i = 0; i < 70 && !limited; i++) {++ const r = await busy.get('/api/v1/jobs?q=engineer', { headers: { 'x-forwarded-for': '10.9.9.9' } });++ if (r.status === 429) limited = r;++ }++ assert.ok(limited, 'the search budget never engaged');++ assert.strictEqual(limited.json().code, 'rate_limited');++ assert.ok(Number(limited.headers.get('retry-after')) > 0);++ // Reading one listing is a plain read, not the search budget, and still works.++ assert.strictEqual((await busy.get('/api/v1/sources')).status, 200);++ } finally { await busy.close(); }++ });++ } finally { await t.close(); }++ done();++})().catch((e) => { console.error(e); process.exit(1); });@@@ -12,7 +12,10 @@ const { spawn } = require('child_process');const { measure, check, format } = require('openvibe-shared/perf-budget');// Budgets are set from the numbers measured on 2026-10-08 (the real ones, fresh database), rounded up with about 10%--// headroom; file counts are the measured counts and external files stay at 0.++// headroom; file counts are the measured counts and external files stay at 0. The CSS budget was raised from 16.5/4.1++// when the listings arrived (the search form, a listing list and the source blocks in app.css): the measured page is++// 18.2 KB raw / 4.3 KB brotli, still two stylesheets and no external file. The home page itself stayed inside its own++// budget — the listings replace the skeleton's placeholder sections rather than adding to them.const BUDGETS = {htmlRawKB: 27.5, // measured 24.9 (the home page: hero, features, steps, cta, JSON-LD)htmlBrotliKB: 6.9, // 6.2@@@ -20,8 +23,8 @@ const BUDGETS = {jsRawKB: 264, // 239.8jsBrotliKB: 62.5, // 56.5cssFiles: 2, // 2 (app.css + the cached /shared/showcase.css)-- cssRawKB: 16.5, // 15.0-- cssBrotliKB: 4.1, // 3.7++ cssRawKB: 20.5, // 15.0 before the listings; 18.2 after (the search form, the listing list, the source blocks)++ cssBrotliKB: 4.8, // 3.7 before; 4.3 afterexternalFiles: 0, // 0};nnew file mode 100644@@@ -0,0 +1,137 @@++'use strict';++/**++ * Saved searches: saving the search you are looking at, seeing each one with the count of listings that appeared++ * since you last looked, the count resetting when you open /saved, signing in being required, a signed-in write from++ * another site being refused, and nobody seeing anybody else's.++ *++ * The clock is a test's own, because "new since you last looked" is a claim about time.++ */++const assert = require('assert');++const { boot, check, done } = require('./helpers/boot');++++(async () => {++ // The clock starts at the real now: the session is a Network token, and this service checks it against its own++ // clock, so a test that moves time must move only a little or the token stops verifying.++ const clock = { t: Date.now() };++ const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });++ const kim = t.network.addUser('kim');++ const lee = t.network.addUser('lee');++ const SAME = { 'sec-fetch-site': 'same-origin' };++++ try {++ await t.ctx.ingest.runAll();++++ await check('a saved search is a person\'s: anonymous is 401, an app token is 403, the other person sees nothing', async () => {++ assert.strictEqual((await t.get('/api/v1/saved-searches')).status, 401);++ const anon = await t.get('/api/v1/saved-searches');++ assert.strictEqual(anon.json().code, 'token.required');++++ const save = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'engineer' }, headers: SAME });++ assert.strictEqual(save.status, 201, save.text);++ const mine = save.json();++ assert.match(mine.id, /^ssc_[0-9A-HJKMNP-TV-Z]{26}$/);++ assert.strictEqual(mine.query, 'engineer');++ assert.strictEqual(mine.new_count, 0);++++ const other = await t.get('/api/v1/saved-searches', { as: lee });++ assert.deepStrictEqual(other.json().saved_searches, [], 'another person saw my saved search');++ assert.strictEqual((await t.get(`/api/v1/saved-searches/${mine.id}`, { as: lee, method: 'DELETE', headers: SAME })).status, 404);++ assert.strictEqual((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.length, 1, 'the other person deleted mine');++ });++++ await check('a saved search needs at least one filter, and its filters are the search\'s own', async () => {++ const empty = await t.get('/api/v1/saved-searches', { as: kim, json: { q: ' ' }, headers: SAME });++ assert.strictEqual(empty.status, 422);++ assert.strictEqual(empty.json().code, 'work.saved_searches.empty');++ const remote = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'devops', remote: 'yes', location: 'Worldwide' }, headers: SAME });++ assert.strictEqual(remote.status, 201, remote.text);++ const saved = remote.json();++ assert.deepStrictEqual(saved.filters, { remote: 'true', type: '', location: 'worldwide' });++ assert.strictEqual(saved.search_path, '/jobs?q=devops&remote=true&location=worldwide');++ // Running it again gives the same listings the search would.++ const run = (await t.get(saved.search_path)).text;++ assert.ok(run.includes('DevOps Engineer'));++ });++++ await check('a signed-in write from another site is refused; a Bearer token is not (it is that person\'s own)', async () => {++ const cross = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'x' }, headers: { 'sec-fetch-site': 'cross-site' } });++ assert.strictEqual(cross.status, 403);++ assert.strictEqual(cross.json().code, 'request.cross_site');++ const bearer = await t.get('/api/v1/saved-searches', { bearer: t.network.userToken(kim), json: { q: 'designer' } });++ assert.strictEqual(bearer.status, 201, bearer.text);++ });++++ await check('a listing that appears later counts as new, and opening /saved is what clears it', async () => {++ const saved = (await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'render wrangler' }, headers: SAME })).json();++ assert.strictEqual(saved.new_count, 0);++ // Nothing new yet: the board shows nothing matching.++ assert.strictEqual((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.find((x) => x.id === saved.id).new_count, 0);++++ // A little later the board starts showing one that matches.++ clock.t += 60_000;++ t.boards.add('remotive', {++ id: 1900999, url: 'https://remotive.com/remote-jobs/other/render-wrangler-1900999',++ title: 'Render Wrangler', company_name: 'Raytrace', category: 'Other', tags: ['gpu'],++ job_type: 'full_time', publication_date: '2026-10-02T10:00:00',++ candidate_required_location: 'Worldwide', salary: '', description: '<p>Keep the farm rendering.</p>',++ });++ await t.ctx.ingest.runAll();++++ const apiList = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;++ assert.strictEqual(apiList.find((x) => x.id === saved.id).new_count, 1, JSON.stringify(apiList));++++ // Opening /saved shows the count, and is itself the act of looking.++ const page = await t.get('/saved', { as: kim });++ assert.strictEqual(page.status, 200);++ assert.match(page.text, /1 new/);++ assert.strictEqual((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.find((x) => x.id === saved.id).new_count, 0, 'opening /saved did not clear the count');++ assert.strictEqual((await t.get('/saved', { as: kim })).text.includes('nothing new'), true);++ });++++ await check('saving from the page needs a sign-in, must come from this site, and lands on /saved', async () => {++ const anon = await t.get('/saved');++ assert.strictEqual(anon.status, 200);++ assert.match(anon.text, /Sign in with OpenVibe/);++ assert.match(anon.text, /href="\/auth\/login\?next=%2Fsaved"/);++++ const cross = await t.get('/saved', { as: kim, form: { q: 'engineer' }, headers: { 'sec-fetch-site': 'cross-site' } });++ assert.strictEqual(cross.status, 403);++++ const ok = await t.get('/saved', { as: kim, form: { q: 'support engineer', remote: 'true' }, headers: SAME });++ assert.strictEqual(ok.status, 303);++ assert.strictEqual(ok.headers.get('location'), '/saved');++ const list = await t.get('/saved', { as: kim });++ assert.ok(list.text.includes('“support engineer”'), 'the saved search is not on the page');++ assert.match(list.text, /href="\/jobs\?q=support\+engineer&remote=true"/, 'no link to run it again');++ assert.match(list.text, /Forget it/);++ });++++ await check('forgetting a saved search removes it, and only it', async () => {++ const before = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;++ assert.ok(before.length >= 2);++ const target = before.find((x) => x.query === 'engineer');++ const gone = await t.get(`/api/v1/saved-searches/${target.id}`, { as: kim, method: 'DELETE', headers: SAME });++ assert.strictEqual(gone.status, 204);++ const after = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;++ assert.deepStrictEqual(after.map((x) => x.id), before.filter((x) => x.id !== target.id).map((x) => x.id));++ assert.strictEqual((await t.get(`/api/v1/saved-searches/${target.id}`, { as: kim, method: 'DELETE', headers: SAME })).status, 404);++ });++++ await check('the page\'s delete button forgets exactly the one it names', async () => {++ const list = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;++ const keep = list[0];++ const drop = list[1];++ const form = await t.get(`/saved/${drop.id}/delete`, { as: kim, method: 'POST', headers: SAME });++ assert.strictEqual(form.status, 303);++ assert.strictEqual(form.headers.get('location'), '/saved');++ const after = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;++ assert.ok(after.some((x) => x.id === keep.id), 'the wrong saved search was deleted');++ assert.ok(!after.some((x) => x.id === drop.id));++ // Nobody else's, even with the right id.++ const other = await t.get(`/saved/${keep.id}/delete`, { as: lee, method: 'POST', headers: SAME });++ assert.strictEqual(other.status, 303);++ assert.ok((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.some((x) => x.id === keep.id), 'another person deleted my saved search');++ });++ } finally { await t.close(); }++ done();++})().catch((e) => { console.error(e); process.exit(1); });@@@ -46,6 +46,17 @@ const rel = (p, dir) => path.relative(dir, p).split(path.sep).join('/');const text = (p) => fs.readFileSync(p, 'utf8');(async () => {++ // This is the skeleton's own test. A service generated from the skeleton (this repository, once its tokens were++ // replaced) carries no placeholders left to rewrite, so the generator can only copy it under its own name and++ // there is nothing here to test. Say so and stop, rather than assert against a rewrite that cannot happen.++ const pkgText = text(path.join(ROOT, 'package.json'));++ if (!TOKEN_RE.test(pkgText)) {++ // The label first, as the shared runner reads it: this file is listed with ○ and is not counted as passed.++ process.stdout.write('the generator: skipped (this tree carries no skeleton placeholders left to rewrite: it is a generated service, not the skeleton itself)\n');++ done();++ return;++ }++const base = tmpDir('ov-new-service-');const dir = path.join(base, 'openvibe-sample');const printed = runGenerator(dir);
@@ -22,7 +22,31 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
@@ -22,7 +22,31 @@ OV_NETWORK_INTERNAL_URL=http://127.0.0.1:4000
OV_OAUTH_CLIENT_ID=work
OV_OAUTH_CLIENT_ID=work
OV_OAUTH_CLIENT_SECRET=
OV_OAUTH_CLIENT_SECRET=
# OV_OAUTH_REDIRECT_URI=https://openvibe.work/auth/callback
# OV_OAUTH_REDIRECT_URI=https://openvibe.work/auth/callback
+# OV_NETWORK_ISSUER=https://openvibe.network
+# The audience a person's own session token carries.
+# OV_SESSION_AUDIENCE=openvibe.network
+# The audience app, agent and service tokens for this service's API carry.
# WORK_AUDIENCE=openvibe.work
# WORK_AUDIENCE=openvibe.work
# optional: defaults to true in production
# optional: defaults to true in production
# COOKIE_SECURE=true
# COOKIE_SECURE=true
+
+# The job-board ingest (server/ingest/). Off unless WORK_INGEST=on: a development run and the test suite read no board.
+# Set it on the host, or no listing ever arrives. Each source is read at most every WORK_INGEST_INTERVAL_HOURS (6),
+# which is exactly the four calls a day Remotive's terms allow.
+#WORK_INGEST=on
+#WORK_INGEST_INTERVAL_HOURS=6
+#WORK_INGEST_TIMEOUT_MS=15000
+#WORK_INGEST_PAGES=3
+#WORK_INGEST_LIMIT=100
+#WORK_RETENTION_DAYS=30
+# The boards' public APIs. Point these at a stand-in server to test the ingest; nothing else in the service fetches
+# anything, and never a URL a caller typed.
+#WORK_ARBEITNOW_URL=https://www.arbeitnow.com/api/job-board-api
+#WORK_REMOTIVE_URL=https://remotive.com/api/remote-jobs
+#WORK_REMOTEOK_URL=https://remoteok.com/api
+# The User-Agent the boards see. Defaults to OpenVibeWork/0.1 (+<BASE_URL>).
+#WORK_USER_AGENT=
+
+# An embedded PGlite directory in development, when DATABASE_URL is unset.
+#WORK_PGLITE_DIR=
@@ -2,6 +2,33 @@
@@ -2,6 +2,33 @@
What changed in OpenVibe.Work, newest first. Each site also publishes its patch notes at /updates.
What changed in OpenVibe.Work, newest first. Each site also publishes its patch notes at /updates.
+## 0.2.0 — 2026-10-08
+
+**Job listings, searchable, with their provenance — and saved searches.**
+
+- **Listings, gathered from open boards.** A background ingest reads Arbeitnow, Remotive and RemoteOK through their
+ public APIs (server/jobs/sources.js, server/ingest/). Each board's terms are applied as written: the source is named
+ and the original listing is linked on every page that shows it, and only a short plain-text excerpt is stored — never
+ the full description. Every listing keeps the board's own id and URL, and a listing without both a title, a company,
+ an id and an http(s) link back is refused rather than published. The ingest is off unless `WORK_INGEST=on`, calls a
+ source at most every six hours (exactly the four a day Remotive's terms allow), upserts on `(source, source_id)`, and
+ expires a listing no board has shown for 30 days.
+- **Search** at `/jobs` (`q`, `remote`, `type`, `location`), newest first, 30 a page with a cursor, with facets for
+ remote and job type. One listing at `/jobs/:id`, with a schema.org `JobPosting` whose `url` is the original.
+- **Saved searches** for a signed-in person: save the search you are looking at, and `/saved` shows each one with the
+ count of listings that appeared since you last looked (opening the page is what clears it).
+- **The API** at `/api/v1`: `GET /jobs`, `GET /jobs/:id`, `GET /sources` (each board's terms as applied and its last
+ fetch), and `POST`/`GET`/`DELETE /saved-searches`. Reading is public; a saved search is a person's.
+- **Pages**: a home page with the search box, the latest remote listings and each source with its count; `/sources`,
+ which states each board's terms as applied; and `/docs` for the API. A sitemap carrying the newest listings, and
+ `llms.txt` pointing a crawler at the listings and the sources.
+- **Limits**: the search and the saved-search writes have per-caller budgets of their own
+ (`work.jobs.search`, `work.saved_searches.write`).
+- **New tables**: `work_listings`, `work_saved_searches`, `work_source_fetches` (`migrations/0002_listings.sql`).
+- **New tools**: `npm run ingest` reads every board once and exits.
+- Also: `test/skeleton.test.js` now skips itself in a generated service (this repository, whose skeleton placeholders
+ were already rewritten), saying so, rather than asserting against a rewrite that can no longer happen.
+
## 0.1.0 — 2026-10-08
## 0.1.0 — 2026-10-08
- **First release:** the service starts from the OpenVibe skeleton — sign-in with OpenVibe.Network (OAuth 2 + PKCE), server-rendered pages through the OpenVibe Frame, the `/api/v1` mount with per-caller limits, crawl artifacts, PostgreSQL migrations, the deploy files and the test suite.
- **First release:** the service starts from the OpenVibe skeleton — sign-in with OpenVibe.Network (OAuth 2 + PKCE), server-rendered pages through the OpenVibe Frame, the `/api/v1` mount with per-caller limits, crawl artifacts, PostgreSQL migrations, the deploy files and the test suite.
@@ -2,42 +2,79 @@
@@ -2,42 +2,79 @@
> Find work, with an agent on your side.
> Find work, with an agent on your side.
-**Status:** skeleton. It runs, it is tested, and its pages and API are live — but the product itself is not written yet.
+**Status:** the listings and the search are live (listings gathered from three public job boards, search, saved
+searches, an API and the pages around them). The agent pillar — help preparing and tracking an application — is not
+written yet.
**Domain:** `openvibe.work` · **Port:** 4960 · **Service id:** `work` · **Env prefix:** `WORK`
**Domain:** `openvibe.work` · **Port:** 4960 · **Service id:** `work` · **Env prefix:** `WORK`
**License:** AGPL-3.0 (same as every OpenVibe service).
**License:** AGPL-3.0 (same as every OpenVibe service).
-## What is already here
+## What it does
+
+Job listings gathered from public job boards, each one carrying its provenance — the board it came from and a link
+back to the original listing — searchable, filterable, and readable without an account. A signed-in person can keep a
+search and see how many listings have appeared since they last looked. Nothing is republished: Work stores a short
+plain-text excerpt and sends the reader to the board for the full listing and to apply.
+
+## What is here
| Piece | Where | What it does |
| Piece | Where | What it does |
|---|---|---|
|---|---|---|
| Config | [server/config.js](server/config.js) | Every value from the environment; `load(env)` is pure so tests build a config without touching `process.env` |
| Config | [server/config.js](server/config.js) | Every value from the environment; `load(env)` is pure so tests build a config without touching `process.env` |
| App | [server/app.js](server/app.js) | helmet (CSP, frame-ancestors none), the Network session middleware, `/auth`, the legal pages, static assets, the `/api/v1` mount, the 404 and the error handler |
| App | [server/app.js](server/app.js) | helmet (CSP, frame-ancestors none), the Network session middleware, `/auth`, the legal pages, static assets, the `/api/v1` mount, the 404 and the error handler |
| Sign-in | [server/auth/sso.js](server/auth/sso.js) | OAuth 2 authorization code with PKCE (S256) against OpenVibe.Network; httpOnly `work_at` / `work_rt` cookies; `/auth/me` for the shared navbar |
| Sign-in | [server/auth/sso.js](server/auth/sso.js) | OAuth 2 authorization code with PKCE (S256) against OpenVibe.Network; httpOnly `work_at` / `work_rt` cookies; `/auth/me` for the shared navbar |
-| Signing key | [server/auth/keys.js](server/auth/keys.js) | The Network JWKS through `openvibe-sdk/auth`, kept fresh, verified offline |
-| Who is calling | [server/http/principal.js](server/http/principal.js) | `req.principal`: a person, an app/agent/service with a capability, or anonymous |
-| API | [server/http/api.js](server/http/api.js) | `GET /api/v1/ping`; the router, the problem+json errors and the guards are wired for the product's routes |
-| Pages | [server/http/pages.js](server/http/pages.js), [server/render/](server/render/) | The home page and `/updates`, server-rendered through `openvibe-shared/shell`, readable without JavaScript |
-| Discovery | [server/http/discovery.js](server/http/discovery.js) | `robots.txt`, `sitemap.xml`, `llms.txt`, `llms-full.txt` and the home page's JSON-LD |
-| Limits | [server/http/caller-limits.js](server/http/caller-limits.js), [deploy/nginx/](deploy/nginx/) | Per-caller limits at the API and per-address limits at nginx |
+| Sources | [server/jobs/sources.js](server/jobs/sources.js) | The three boards: where each is fetched, its terms, how its payload becomes a listing, and what is refused |
+| Ingest | [server/ingest/](server/ingest/) | The background read (timer, User-Agent, 15 s timeout, upsert by source + the board's id, expiry) |
+| Listings | [server/jobs/store.js](server/jobs/store.js), [migrations/0002_listings.sql](migrations/0002_listings.sql) | Search, facets, the cursor, saved searches, the fetch record |
+| Text | [server/jobs/text.js](server/jobs/text.js) | HTML → plain text, and the ~300-character excerpt that is all we keep |
+| API | [server/http/api.js](server/http/api.js) | `/api/v1`: jobs, one job, the sources, saved searches; problem+json errors |
+| Pages | [server/http/pages.js](server/http/pages.js), [server/render/](server/render/) | `/`, `/jobs`, `/jobs/:id`, `/saved`, `/sources`, `/docs`, `/updates` — server-rendered, no JavaScript needed |
+| Discovery | [server/http/discovery.js](server/http/discovery.js) | `robots.txt`, `sitemap.xml` (the newest listings too), `llms.txt`, `llms-full.txt`, JSON-LD |
+| Limits | [server/http/caller-limits.js](server/http/caller-limits.js), [deploy/nginx/](deploy/nginx/) | Per-caller limits at the API (the search and the saved-search writes have their own) and per-address limits at nginx |
| Health | [server/observability.js](server/observability.js) | `/api/health`, a truthful `/api/ready` (the database is required) and Prometheus metrics on loopback only |
| Health | [server/observability.js](server/observability.js) | `/api/health`, a truthful `/api/ready` (the database is required) and Prometheus metrics on loopback only |
| Database | [server/db.js](server/db.js), [migrations/](migrations/) | PostgreSQL through `openvibe-sdk/db`; `NNNN_*.sql` applied at boot; PGlite in development |
| Database | [server/db.js](server/db.js), [migrations/](migrations/) | PostgreSQL through `openvibe-sdk/db`; `NNNN_*.sql` applied at boot; PGlite in development |
-| Process | [server/index.js](server/index.js) | Listens on `PORT`, and stops gracefully through `openvibe-sdk/service` |
+| Process | [server/index.js](server/index.js) | Listens on `PORT`, starts the ingest when it is on, and stops gracefully through `openvibe-sdk/service` |
| Deploy | [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), [deploy/systemd/openvibe-work.service](deploy/systemd/openvibe-work.service) | nginx vhost and systemd unit (port 4960, `/opt/openvibe.work`, `/etc/openvibe/work.env`) |
| Deploy | [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), [deploy/systemd/openvibe-work.service](deploy/systemd/openvibe-work.service) | nginx vhost and systemd unit (port 4960, `/opt/openvibe.work`, `/etc/openvibe/work.env`) |
-| Tests | [test/](test/) | `npm test`: every `test/*.test.js` in its own process, on a temp PGlite database with an in-process mock of OpenVibe.Network |
+| Tests | [test/](test/) | `npm test`: every `test/*.test.js` in its own process, on a temp PGlite database, with a mock OpenVibe.Network and a stand-in for each job board |
## API
## API
| Route | Who | |
| Route | Who | |
|---|---|---|
|---|---|---|
| `GET /api/v1/ping` | anyone | `{ ok: true, service: "work" }` |
| `GET /api/v1/ping` | anyone | `{ ok: true, service: "work" }` |
+| `GET /api/v1/jobs` | anyone | Search: `q`, `remote`, `type`, `location`, `limit` (to 100), `cursor` (the `next` of the previous page). Newest first, 30 a page, with facets |
+| `GET /api/v1/jobs/:id` | anyone | One listing: the board's fields, the excerpt, the source, and `url` — the original to apply on |
+| `GET /api/v1/sources` | anyone | The boards, each one's terms as applied, its listing count and its last fetch |
+| `GET /api/v1/saved-searches` | a person | Your saved searches, each with `new_count` — new since you last looked |
+| `POST /api/v1/saved-searches` | a person | `{ q?, remote?, type?, location? }` (at least one). 201 with the saved search |
+| `DELETE /api/v1/saved-searches/:id` | a person | Forget one. 204 |
+
+**Who can call it:** reading the listings is public. A saved search belongs to a person, so those three routes need a
+signed-in person — their Network token as a Bearer, or this site's session — and answer 401 otherwise. An app or agent
+token cannot hold one (403): a saved search is not a project's. A write made with the session cookie must come from
+`openvibe.work` itself (`request.cross_site` otherwise).
-Everything else the site serves is a page. The product adds its routes in [server/http/api.js](server/http/api.js), naming
-each route's capability in [server/http/principal.js](server/http/principal.js) (`CAPABILITIES`) and its numbers in
-[server/http/caller-limits.js](server/http/caller-limits.js) (`BUDGETS`).
+**Errors** are RFC 9457 `application/problem+json` with a stable `code` (`work.jobs.not_found`,
+`work.saved_searches.empty`, `work.saved_searches.limit`, `rate_limited`, …).
-**Who can call it:** a person, with their Network token as a Bearer or this site's session; or an app, agent or service
-with a Network token for audience `openvibe.work` that holds the route's capability. A write made with the session cookie
-must come from `openvibe.work` itself. **Errors** are RFC 9457 `application/problem+json` with a stable `code`.
+## Data sources and their terms
+
+Each board is read through its own public API, and its terms are applied on every page that shows one of its listings
+— they are also served by `GET /api/v1/sources` and written out on [/sources](server/http/pages.js).
+
+| Source | API | Terms, as applied here |
+|---|---|---|
+| **Arbeitnow** | `GET https://www.arbeitnow.com/api/job-board-api?page=N` | Free to use. Arbeitnow is named on every listing and the listing links to its page on arbeitnow.com. Its first pages (configurable) are read. |
+| **Remotive** | `GET https://remotive.com/api/remote-jobs?limit=100` | Remotive must be named as the source and the listing linked. The API is called about four times a day — the six-hour interval is exactly that. |
+| **RemoteOK** | `GET https://remoteok.com/api` | RemoteOK must be named as the source and the listing linked. The response's first element is a legal notice, not a listing, and is skipped. |
+
+What is stored per listing: `source`, the board's own id, the original `url`, title, company, location, the remote
+flag, tags, job type, salary text when the board gives one, when it was posted, when we fetched it, when it was first
+seen and last seen, and a **short plain-text excerpt** — never the full description. The listing page says *"Read the
+full listing on <source>"* and links there; that is where a reader applies.
+
+Listings are upserted on `(source, source_id)` and **expire** — hidden from the search, the pages and the sitemap —
+when no board has shown one for 30 days. An ingest never invents a listing: one without a title, a company, an id or
+an http(s) URL to link back to is refused.
## Configuration
## Configuration
@@ -45,12 +82,18 @@ See [.env.example](.env.example). Required in production: `OV_OAUTH_CLIENT_SECRE
@@ -45,12 +82,18 @@ See [.env.example](.env.example). Required in production: `OV_OAUTH_CLIENT_SECRE
Network), `BASE_URL`, `DATABASE_URL` and `DATABASE_DIRECT_URL`. The database is the only required readiness check; the
Network), `BASE_URL`, `DATABASE_URL` and `DATABASE_DIRECT_URL`. The database is the only required readiness check; the
Network signing key, the OAuth client and Valkey are optional (the service says so, per check, on `/api/ready`).
Network signing key, the OAuth client and Valkey are optional (the service says so, per check, on `/api/ready`).
+The ingest is **off** unless `WORK_INGEST=on`, so a development run and the test suite never call a job board. When it
+is on, `WORK_INGEST_INTERVAL_HOURS` (6), `WORK_INGEST_PAGES` (3), `WORK_INGEST_LIMIT` (100), `WORK_INGEST_TIMEOUT_MS`
+(15000), `WORK_RETENTION_DAYS` (30) and the per-source base URLs (`WORK_ARBEITNOW_URL`, `WORK_REMOTIVE_URL`,
+`WORK_REMOTEOK_URL`) decide what it does.
+
## Development
## Development
```bash
```bash
npm install
npm install
-fnm exec --using=22 npm test # every test/*.test.js, on temp PGlite databases with a mock Network
-fnm exec --using=22 npm run dev # http://localhost:4960
+fnm exec --using=22 npm test # every test/*.test.js, on temp PGlite databases with a mock Network and stand-in boards
+fnm exec --using=22 npm run dev # http://localhost:4960 (no ingest unless WORK_INGEST=on)
+fnm exec --using=22 npm run ingest # read every board once, then exit
```
```
Without `DATABASE_URL` development uses an embedded PGlite database in `data/pglite` (one process only). `npm run
Without `DATABASE_URL` development uses an embedded PGlite database in `data/pglite` (one process only). `npm run
@@ -60,6 +103,7 @@ test:pg` runs the same suite through PostgreSQL and PgBouncer (see [.github/work
@@ -60,6 +103,7 @@ test:pg` runs the same suite through PostgreSQL and PgBouncer (see [.github/work
- **Deploy:** `sudo ovhost deploy work` on the host (git checkout at `/opt/openvibe.work`, unit
- **Deploy:** `sudo ovhost deploy work` on the host (git checkout at `/opt/openvibe.work`, unit
`openvibe-work.service` on 127.0.0.1:4960, env `/etc/openvibe/work.env`, database `ov_work` on the data role).
`openvibe-work.service` on 127.0.0.1:4960, env `/etc/openvibe/work.env`, database `ov_work` on the data role).
+ Set `WORK_INGEST=on` in `/etc/openvibe/work.env` or no listing will ever arrive.
- **nginx:** [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), installed with `ov-vhost-install`.
- **nginx:** [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), installed with `ov-vhost-install`.
- **Rollback:** ovhost puts the previous sha back by itself when `/api/ready` does not answer after the restart.
- **Rollback:** ovhost puts the previous sha back by itself when `/api/ready` does not answer after the restart.
- Register the service and its capabilities in **OpenVibe.Contracts** (`contracts-service: work` in CI) and with
- Register the service and its capabilities in **OpenVibe.Contracts** (`contracts-service: work` in CI) and with
@@ -72,8 +116,17 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md).
@@ -72,8 +116,17 @@ Reporting a vulnerability: [SECURITY.md](SECURITY.md).
- Session tokens are httpOnly cookies; a FedCM assertion or an app or service token is never a session.
- Session tokens are httpOnly cookies; a FedCM assertion or an app or service token is never a session.
- Secrets live only in the env file; only environment variable names appear in code and docs, and no secret is logged.
- Secrets live only in the env file; only environment variable names appear in code and docs, and no secret is logged.
- Request bodies are never logged.
- Request bodies are never logged.
-- Nothing in a request may decide a URL this service fetches: a caller's URL goes to OpenVibe.Tools, whose own guard
- decides what may be fetched.
+- Nothing in a request may decide a URL this service fetches: the ingest fetches only the three base URLs from the
+ configuration, with this service's User-Agent, a timeout and a size cap, and stores only http(s) listing URLs.
+- A board's text is data, never markup: every value goes through the `html` tagged template, and a listing URL that
+ is not http(s) is refused rather than linked.
+
+## Not yet
+
+- **The agent**: drafting, tailoring and tracking an application, with the person's approval at each step.
+- **Alerts**: being told when a listing that fits appears — that comes through OpenVibe.Watch with the agent pillar.
+ Today a saved search is a search you do not have to type again, plus a count of what is new.
+- More boards, and a per-source schedule finer than "the same six hours for everyone".
---
---
@@ -1,8 +1,8 @@
@@ -1,8 +1,8 @@
{
{
- "stage": "skeleton",
+ "stage": "beta",
"code": true,
"code": true,
"wave": "W20",
"wave": "W20",
- "plan": "OpenVibe.Work starts from the OpenVibe service skeleton; the product's own plan is written here when it is",
+ "plan": "Job listings gathered from open boards with their provenance, search and saved searches; the agent pillar comes next.",
"runtime": "node 22, express 4, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"runtime": "node 22, express 4, PostgreSQL 18 (openvibe-sdk/db, async) + Valkey 9",
"port": "4960",
"port": "4960",
"domain": "openvibe.work",
"domain": "openvibe.work",
@@ -14,21 +14,23 @@
@@ -14,21 +14,23 @@
"openvibe-sdk v0.35.2",
"openvibe-sdk v0.35.2",
"openvibe-shared v2.14.0"
"openvibe-shared v2.14.0"
],
],
- "summary": "OpenVibe.Work — Find work, with an agent on your side. A new OpenVibe service on the shared skeleton, not deployed yet: sign-in with OpenVibe.Network (OAuth 2 + PKCE, httpOnly session cookies), server-rendered pages through the OpenVibe Frame, the /api/v1 mount with per-caller limits, crawl artifacts, PostgreSQL migrations applied at boot, the nginx and systemd deploy files and a test suite on PGlite with a mock Network.",
+ "summary": "OpenVibe.Work gathers job listings from three public job boards (Arbeitnow, Remotive, RemoteOK) under their terms — the source is named and the original listing is linked on every page, and only a short excerpt is kept. It searches them with facets, shows one listing with a schema.org JobPosting whose url is the original, and lets a signed-in person keep a search and see what is new since they last looked. Public reads need no token.",
"works": [
"works": [
- "Sign-in with OpenVibe.Network (PKCE S256, server-side code exchange, httpOnly session cookies)",
- "Server-rendered pages: the home page and /updates, through the OpenVibe Frame, complete without JavaScript",
- "Crawl artifacts: robots.txt, sitemap.xml, llms.txt, llms-full.txt and the home page's JSON-LD",
- "The API mount at /api/v1 with the problem+json error shape, the per-address limits and the per-caller limits",
- "Capability verification for app, agent and service tokens (audience openvibe.work) and for a person's own token",
- "Truthful readiness (/api/ready: the database is required) and Prometheus metrics on loopback",
- "PostgreSQL with migrations/NNNN_*.sql (openvibe-sdk/db); PGlite in development",
- "Deploy files: deploy/nginx/openvibe.work.conf and deploy/systemd/openvibe-work.service",
- "Tests on a temp PGlite database with an in-process mock of OpenVibe.Network"
+ "The ingest: three boards read on a timer (WORK_INGEST=on, off in tests and development), each with its own User-Agent, 15-second timeout, upsert by (source, source_id) and expiry after 30 days unseen",
+ "The listings: provenance kept and shown (source name, link to the original, when it was fetched), a short plain-text excerpt, never the full description",
+ "Search: /jobs?q=&remote=&type=&location= newest first, 30 a page with a cursor, facets for remote and job type",
+ "One listing: /jobs/:id with the board's fields, the excerpt, a 'Read the full listing on <source>' link and a JobPosting JSON-LD whose url is the original",
+ "Saved searches for a signed-in person, each with the count of listings new since they last looked; /saved is where that count is cleared",
+ "The API (/api/v1): jobs, one job, sources with their terms and last fetch, and saved searches",
+ "The pages: home, /jobs, /jobs/:id, /saved, /sources, /docs and /updates — server-rendered, complete without JavaScript",
+ "Discovery: a sitemap carrying the newest listings, llms.txt, robots.txt (saved searches excluded) and JSON-LD",
+ "Per-caller limits on the search and on saved-search writes; problem+json errors with stable codes",
+ "Tests: the ingest, the search, the pages and the saved searches against stand-in boards, on a temp PGlite database with a mock Network"
],
],
"notYet": [
"notYet": [
- "The product itself: its tables and migrations, its capabilities, its routes in server/http/api.js and its pages in server/http/pages.js",
- "Its service and capability manifests in openvibe-contracts (contracts-service is work)",
+ "The agent pillar: drafting, tailoring and tracking an application with the person's approval at each step",
+ "Alerts: being told when a listing that fits appears — that comes through OpenVibe.Watch with the agent pillar; today a saved search is a repeatable search plus a count of what is new",
+ "Its service and capability manifests in openvibe-contracts (contracts-service is work): the API is live, but no <work>.jobs.* capability is registered yet, so app and agent tokens cannot call it",
"A deployment: the service is not registered with OpenVibe.Services nor deployed by OpenVibe.Host"
"A deployment: the service is not registered with OpenVibe.Services nor deployed by OpenVibe.Host"
],
],
"capabilitiesRegistered": false,
"capabilitiesRegistered": false,
@@ -37,9 +39,15 @@
@@ -37,9 +39,15 @@
"placeholderInSites": false,
"placeholderInSites": false,
"features": [
"features": [
"Network sign-in",
"Network sign-in",
+ "job listings with provenance",
+ "search and facets",
+ "saved searches",
+ "sources and their terms",
+ "api",
"server-rendered pages",
"server-rendered pages",
- "api mount",
+ "discovery files",
"limits and readiness",
"limits and readiness",
+ "background ingest",
"deploy files"
"deploy files"
],
],
"updated": "2026-10-08"
"updated": "2026-10-08"
@@ -96,6 +96,16 @@ server {
@@ -96,6 +96,16 @@ server {
proxy_pass http://127.0.0.1:4960;
proxy_pass http://127.0.0.1:4960;
}
}
+ # Saving or forgetting a saved search is a write made with a session cookie: the strict form zone, but reading
+ # /saved is an ordinary page and keeps the page budget (the browser reloads it freely).
+ location ~ ^/saved(/[A-Za-z0-9_-]+/delete)?$ {
+ limit_except GET HEAD {
+ limit_req zone=work_forms burst=10 nodelay;
+ limit_req_status 429;
+ }
+ proxy_pass http://127.0.0.1:4960;
+ }
+
# ── Prometheus metrics: host-local scrapers only (http://127.0.0.1:4960/metrics) ──
# ── Prometheus metrics: host-local scrapers only (http://127.0.0.1:4960/metrics) ──
location = /metrics { return 404; }
location = /metrics { return 404; }
new file mode 100644
new file mode 100644
@@ -0,0 +1,56 @@
@@ -0,0 +1,56 @@
+-- phase: expand
+-- OpenVibe.Work: the job listings gathered from the open job boards (server/jobs/sources.js), the searches a person
+-- saved and when they last looked at each, and what each source's last fetch did.
+--
+-- A listing keeps its provenance: the source it came from and the original URL it must always link back to. Only a
+-- short plain-text excerpt is stored — never the board's full description, which stays on the board. Nothing here is
+-- ever a credential, a key or a token.
+
+CREATE TABLE work_listings (
+ id text COLLATE "C" PRIMARY KEY, -- job_<base32>, derived from (source, source_id): stable
+ source text COLLATE "C" NOT NULL, -- arbeitnow | remotive | remoteok
+ source_id text COLLATE "C" NOT NULL, -- the board's own id for it
+ url text NOT NULL, -- the original listing; always linked, never rehosted
+ title text NOT NULL,
+ company text NOT NULL,
+ location text,
+ remote boolean NOT NULL DEFAULT false,
+ tags text NOT NULL DEFAULT '', -- comma-joined, for display
+ job_type text COLLATE "C", -- the board's own words, lowercased where it repeats one
+ salary text, -- the board's own salary text, when it gives one
+ excerpt text NOT NULL, -- first ~300 characters of the description, plain text
+ search_text text NOT NULL, -- lower(title, company, tags, location, excerpt)
+ posted_at text COLLATE "C", -- ISO 8601 UTC, when the board says so
+ sort_at text COLLATE "C" NOT NULL, -- posted_at, or fetched_at when the board gave none
+ fetched_at text COLLATE "C" NOT NULL, -- when the ingest read the board
+ first_seen_at text COLLATE "C" NOT NULL,
+ last_seen_at text COLLATE "C" NOT NULL, -- touched on every fetch that still lists it
+ expired boolean NOT NULL DEFAULT false -- not seen for the retention window: hidden, kept
+);
+-- One row per (source, its id): the ingest upserts on this.
+CREATE UNIQUE INDEX work_listings_by_source ON work_listings (source, source_id);
+CREATE INDEX work_listings_latest ON work_listings (expired, sort_at DESC, id DESC);
+CREATE INDEX work_listings_by_last_seen ON work_listings (last_seen_at);
+
+-- A saved search belongs to the person who saved it (user:usr_…). The filters are the same four the search takes.
+CREATE TABLE work_saved_searches (
+ id text COLLATE "C" PRIMARY KEY, -- ssc_<ULID>
+ subject text COLLATE "C" NOT NULL,
+ query text NOT NULL DEFAULT '',
+ remote text COLLATE "C" NOT NULL DEFAULT '', -- '' (any) | 'true' | 'false'
+ type text COLLATE "C" NOT NULL DEFAULT '',
+ location text NOT NULL DEFAULT '',
+ created_at text COLLATE "C" NOT NULL,
+ last_seen_at text COLLATE "C" NOT NULL -- listings first seen after this are "new"
+);
+CREATE INDEX work_saved_searches_by_subject ON work_saved_searches (subject, created_at DESC);
+
+-- What each source's last fetch did, so /sources can say when it last ran and how it went.
+CREATE TABLE work_source_fetches (
+ source text COLLATE "C" PRIMARY KEY,
+ started_at text COLLATE "C" NOT NULL,
+ finished_at text COLLATE "C",
+ ok boolean NOT NULL DEFAULT false,
+ listings integer NOT NULL DEFAULT 0,
+ error text
+);
@@ -1,12 +1,13 @@
@@ -1,12 +1,13 @@
{
{
"name": "openvibe-work",
"name": "openvibe-work",
- "version": "0.1.0",
+ "version": "0.2.0",
"private": true,
"private": true,
"description": "OpenVibe.Work — Find work, with an agent on your side.",
"description": "OpenVibe.Work — Find work, with an agent on your side.",
"main": "server/index.js",
"main": "server/index.js",
"scripts": {
"scripts": {
"start": "node server/index.js",
"start": "node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
"dev": "NODE_ENV=development node server/index.js",
+ "ingest": "node scripts/ingest.js",
"test": "node test/run.js",
"test": "node test/run.js",
"test:pg": "WORK_TEST_STORE=pg node test/run.js"
"test:pg": "WORK_TEST_STORE=pg node test/run.js"
},
},
@@ -42,3 +42,43 @@ form.inline { display: inline; }
@@ -42,3 +42,43 @@ form.inline { display: inline; }
.page .sc-ic .ov-icon { --ovi-size: 28px !important; }
.page .sc-ic .ov-icon { --ovi-size: 28px !important; }
button.sc-btn { background: transparent; }
button.sc-btn { background: transparent; }
button.sc-btn.sc-primary { border: 0; }
button.sc-btn.sc-primary { border: 0; }
+
+/* ── The listings ───────────────────────────────────────── */
+.job-search { display: grid; grid-template-columns: minmax(0, 2fr) minmax(0, 1fr) auto minmax(0, 1fr) auto; gap: 8px; align-items: center; margin: 12px 0; }
+.job-search input, .job-search select { font: inherit; width: 100%; padding: 7px 9px; border-radius: 8px; border: 1px solid var(--border, #1f2d47); background: var(--bg-secondary, #0f1729); color: inherit; }
+.job-clear { font-size: .9rem; }
+@media (max-width: 720px) { .job-search { grid-template-columns: 1fr 1fr; } }
+.job-facets { display: flex; flex-wrap: wrap; gap: 8px 24px; align-items: baseline; margin: 8px 0 4px; font-size: .92rem; }
+.facets { list-style: none; display: flex; flex-wrap: wrap; gap: 6px; margin: 0; padding: 0; align-items: baseline; }
+.facets .facet-label { color: var(--text-muted, #7386a3); font-size: .85rem; }
+.facets a { text-decoration: none; padding: 1px 8px; border: 1px solid var(--border, #1f2d47); border-radius: 999px; font-size: .85rem; }
+.facets li.on a { border-color: var(--accent, #3b82f6); color: var(--text-primary, #e6edf7); }
+.facet-n { color: var(--text-muted, #7386a3); }
+
+.job-list { list-style: none; margin: 8px 0 0; padding: 0; }
+.job-item { border-top: 1px solid var(--border, #1f2d47); padding: 14px 0; }
+.job-item:first-child { border-top: 0; }
+.job-title { margin: 0 0 4px; }
+.job-meta { margin: 0 0 6px; color: var(--text-secondary, #96a7c2); font-size: .92rem; }
+.job-company { font-weight: 600; color: var(--text-primary, #e6edf7); }
+.job-excerpt { margin: 0 0 8px; max-width: 78ch; }
+.job-tags { list-style: none; display: flex; flex-wrap: wrap; gap: 6px; margin: 0 0 8px; padding: 0; }
+.job-tags li { font-size: .78rem; border: 1px solid var(--border-light, #2c3d5c); border-radius: 999px; padding: 0 8px; color: var(--text-muted, #7386a3); }
+.job-provenance { margin: 0; font-size: .88rem; }
+.job-source { font-weight: 600; }
+.job-pager { display: flex; gap: 12px; align-items: center; margin: 18px 0 8px; }
+.job-facts { display: grid; grid-template-columns: max-content minmax(0, 1fr); gap: 4px 16px; margin: 16px 0; font-size: .92rem; }
+.job-facts dt { color: var(--text-secondary, #96a7c2); }
+.job-facts dd { margin: 0; overflow-wrap: anywhere; }
+.job-apply { margin: 16px 0; }
+.job-excerpt-block { max-width: 78ch; }
+
+.saved-list { list-style: none; margin: 12px 0; padding: 0; }
+.saved-item { border-top: 1px solid var(--border, #1f2d47); padding: 12px 0; }
+.saved-query { margin: 0 0 4px; }
+.saved-form fieldset { border: 1px solid var(--border, #1f2d47); border-radius: 10px; padding: 12px; }
+.saved-form legend { padding: 0 6px; color: var(--text-secondary, #96a7c2); }
+.saved-form label { display: inline-flex; flex-direction: column; gap: 3px; margin: 0 12px 10px 0; font-size: .88rem; color: var(--text-secondary, #96a7c2); }
+.saved-form input, .saved-form select { font: inherit; padding: 6px 8px; border-radius: 8px; border: 1px solid var(--border, #1f2d47); background: var(--bg-secondary, #0f1729); color: inherit; }
+.source-block { border-top: 1px solid var(--border, #1f2d47); padding-top: 12px; margin-top: 18px; }
+.source-links { margin: 6px 0 12px; }
new file mode 100644
new file mode 100644
@@ -0,0 +1,33 @@
@@ -0,0 +1,33 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Read every job board once, right now, and exit — the same work the timer does (server/ingest/), for an operator who
+ * wants the listings without waiting for the next interval, or wants to see what a board answers.
+ *
+ * node scripts/ingest.js every source
+ * node scripts/ingest.js arbeitnow one source
+ *
+ * It is not gated on WORK_INGEST: running it by hand is the point. It still only fetches the configured base URLs.
+ */
+const configLib = require('../server/config');
+const { openStore } = require('../server/db');
+const { createIngest } = require('../server/ingest');
+
+(async () => {
+ const config = configLib.load();
+ const s = await openStore(config, { log: console });
+ const ingest = createIngest({ config, s, log: console });
+ try {
+ const want = process.argv.slice(2);
+ const out = want.length
+ ? await Promise.all(want.map((id) => ingest.runSource(id)))
+ : await ingest.runAll();
+ for (const r of out) {
+ process.stdout.write(`${r.ok ? 'ok ' : 'FAIL'} ${r.source || ''}: ${r.listings} listing(s)${r.error ? ` — ${r.error}` : ''}\n`);
+ }
+ process.exitCode = out.every((r) => r.ok) ? 0 : 1;
+ } finally {
+ await s.close();
+ }
+})().catch((err) => { console.error('[OpenVibe.Work] ingest failed:', err && err.message ? err.message : err); process.exit(1); });
@@ -30,6 +30,7 @@ const { createApi } = require('./http/api');
@@ -30,6 +30,7 @@ const { createApi } = require('./http/api');
const { createPageRoutes } = require('./http/pages');
const { createPageRoutes } = require('./http/pages');
const { createServiceReadiness } = require('./observability');
const { createServiceReadiness } = require('./observability');
const { createCallerLimits } = require('./http/caller-limits');
const { createCallerLimits } = require('./http/caller-limits');
+const { createIngest } = require('./ingest');
const { assetVersion, send } = require('./render/layout');
const { assetVersion, send } = require('./render/layout');
const { html } = require('./render/html');
const { html } = require('./render/html');
@@ -62,6 +63,9 @@ async function createApp(opts = {}) {
@@ -62,6 +63,9 @@ async function createApp(opts = {}) {
const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);
const valkey = opts.valkey !== undefined ? opts.valkey : (config.valkey.url ? require('openvibe-sdk/valkey').createValkey({ url: config.valkey.url, prefix: config.valkey.prefix, log }) : null);
ctx.valkey = valkey;
ctx.valkey = valkey;
ctx.limits = createCallerLimits({ config, now: opts.limitsNow || (() => Date.now()), registry: metrics.registry, log, enabled: opts.callerLimits !== false, valkey });
ctx.limits = createCallerLimits({ config, now: opts.limitsNow || (() => Date.now()), registry: metrics.registry, log, enabled: opts.callerLimits !== false, valkey });
+ // The job-board ingest. Built here so a test can run one source against its stand-in server, but started only by
+ // server/index.js, and only when WORK_INGEST=on: nothing fetches a board in a test or a development run.
+ ctx.ingest = createIngest({ config, s, fetchImpl, log });
app.use(contracts.http.middleware());
app.use(contracts.http.middleware());
app.use(helmet({
app.use(helmet({
@@ -9,6 +9,7 @@
@@ -9,6 +9,7 @@
require('dotenv').config();
require('dotenv').config();
const trim = (s) => String(s || '').replace(/\/+$/, '');
const trim = (s) => String(s || '').replace(/\/+$/, '');
const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);
const int = (v, def) => (Number.isFinite(parseInt(v, 10)) ? parseInt(v, 10) : def);
+const num = (v, def) => (v != null && v !== '' && Number.isFinite(Number(v)) ? Number(v) : def);
function load(env = process.env) {
function load(env = process.env) {
const nodeEnv = env.NODE_ENV || 'development';
const nodeEnv = env.NODE_ENV || 'development';
@@ -52,6 +53,29 @@ function load(env = process.env) {
@@ -52,6 +53,29 @@ function load(env = process.env) {
sessionAudience: env.OV_SESSION_AUDIENCE || 'openvibe.network',
sessionAudience: env.OV_SESSION_AUDIENCE || 'openvibe.network',
},
},
cookies: { secure: env.COOKIE_SECURE ? env.COOKIE_SECURE === 'true' : isProduction },
cookies: { secure: env.COOKIE_SECURE ? env.COOKIE_SECURE === 'true' : isProduction },
+
+ // The public job boards the ingest reads (server/jobs/sources.js). The base URLs are configurable so a
+ // test points them at a stand-in server; nothing else in the service ever fetches a URL, and never one a
+ // caller typed. Their terms are quoted in the manifest that describes each source, and applied on every
+ // page that shows a listing: the source is named and the original listing is linked.
+ sources: {
+ arbeitnow: { url: trim(env.WORK_ARBEITNOW_URL || 'https://www.arbeitnow.com/api/job-board-api') },
+ remotive: { url: trim(env.WORK_REMOTIVE_URL || 'https://remotive.com/api/remote-jobs') },
+ remoteok: { url: trim(env.WORK_REMOTEOK_URL || 'https://remoteok.com/api') },
+ },
+ // The background ingest (server/ingest/): off unless WORK_INGEST=on, so a test and a development run
+ // never call a job board. Once on, each source is fetched at most every intervalMs (Remotive's terms
+ // allow about four calls a day: six hours is exactly four), and listings not seen for retentionDays go.
+ ingest: {
+ enabled: String(env.WORK_INGEST || '').toLowerCase() === 'on',
+ intervalMs: Math.max(1, num(env.WORK_INGEST_INTERVAL_HOURS, 6)) * 3_600_000,
+ timeoutMs: Math.max(1000, int(env.WORK_INGEST_TIMEOUT_MS, 15_000)),
+ // Arbeitnow's board is paginated; the first N pages are read. Remotive takes a limit.
+ pages: Math.max(1, int(env.WORK_INGEST_PAGES, 3)),
+ limit: Math.max(1, int(env.WORK_INGEST_LIMIT, 100)),
+ retentionDays: Math.max(1, int(env.WORK_RETENTION_DAYS, 30)),
+ userAgent: env.WORK_USER_AGENT || `OpenVibeWork/0.1 (+${baseUrl})`,
+ },
};
};
}
}
@@ -3,19 +3,31 @@
@@ -3,19 +3,31 @@
/**
/**
* /api/v1 — OpenVibe.Work's API.
* /api/v1 — OpenVibe.Work's API.
*
*
- * GET /ping public liveness: { ok: true, service }
+ * GET /ping public liveness: { ok: true, service }
+ * GET /jobs public search the listings (q, remote, type, location, limit, cursor) with facets
+ * GET /jobs/:id public one listing, with its source and the original URL to apply on
+ * GET /sources public the boards we gather from, their terms as applied, and the last fetch
+ * GET /saved-searches person your saved searches, each with the count new since you last looked
+ * POST /saved-searches person save the search you are looking at
+ * DELETE /saved-searches/:id person forget one
+ *
+ * Reads are public: a listing is public data with its provenance. A saved search is a person's, so those three routes
+ * need a signed-in person (their token as a Bearer, or this site's session) and answer 401 otherwise — a saved search
+ * belongs to nobody else. Every route carries its per-caller numbers (../http/caller-limits.js).
*
*
- * The product adds its own routes here. Every route that reads or changes a person's data names the capability it
- * needs (`principal.requireCapability('<work>.thing.read')`, listed in CAPABILITIES in ./principal.js) and takes
- * its per-caller numbers (`limits.reads('<work>.thing.read')` or `limits.budget('<name>')`, ./caller-limits.js).
* A refusal is an RFC 9457 problem+json with a stable code.
* A refusal is an RFC 9457 problem+json with a stable code.
*/
*/
const express = require('express');
const express = require('express');
+const contracts = require('openvibe-contracts');
const { asyncRouter } = require('./router');
const { asyncRouter } = require('./router');
+const { sameOrigin } = require('./principal');
+const store = require('../jobs/store');
+const sources = require('../jobs/sources');
function createApi(ctx) {
function createApi(ctx) {
- const { config, principal, limits } = ctx;
+ const { config, s, principal, limits } = ctx;
const r = asyncRouter();
const r = asyncRouter();
+ const problem = (req, res, status, code, detail, extra) => contracts.http.sendProblem(res, status, code, { detail, ctx: req.ov, extra });
r.use(express.json({ limit: '64kb' }));
r.use(express.json({ limit: '64kb' }));
r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });
r.use((req, res, next) => { res.set('Cache-Control', 'no-store'); next(); });
@@ -24,6 +36,102 @@ function createApi(ctx) {
@@ -24,6 +36,102 @@ function createApi(ctx) {
// The one route the skeleton ships. It is public and counted per caller with the default read numbers.
// The one route the skeleton ships. It is public and counted per caller with the default read numbers.
r.get('/ping', limits.reads('work.api.read'), (_req, res) => res.json({ ok: true, service: config.service }));
r.get('/ping', limits.reads('work.api.read'), (_req, res) => res.json({ ok: true, service: config.service }));
+ const wire = (row) => store.toWire(row, { source: sources.byId(row.source) });
+ const person = (req, res) => {
+ if (req.principal.kind === 'user') {
+ // A write made with the session cookie must come from this site: another site cannot make a signed-in
+ // visitor save or forget a search. (A Bearer token is the person's own and is not tied to an origin.)
+ if (req.principal.viaSession && req.method !== 'GET' && req.method !== 'HEAD' && !sameOrigin(req, config.baseUrl)) {
+ problem(req, res, 403, 'request.cross_site', 'A signed-in request that changes something must come from openvibe.work itself.');
+ return null;
+ }
+ return req.principal.requester;
+ }
+ // A person's data is theirs: an app or agent token is not a person, and anonymous is not signed in.
+ problem(req, res, req.principal.kind === 'anonymous' ? 401 : 403, req.principal.kind === 'anonymous' ? 'token.required' : 'work.saved_searches.person_only',
+ req.principal.kind === 'anonymous'
+ ? 'Sign in at openvibe.work to keep saved searches, or send a person\'s Network token as a Bearer.'
+ : 'Saved searches belong to a person; an app or agent token cannot hold one. A person\'s token can.');
+ return null;
+ };
+
+ // ── The listings ────────────────────────────────────────
+ r.get('/jobs', limits.budget('work.jobs.search'), async (req, res) => {
+ const filters = store.filtersOf(req.query);
+ const cursor = store.decodeCursor(req.query.cursor);
+ if (req.query.cursor && !cursor) return problem(req, res, 422, 'work.jobs.cursor_invalid', 'cursor: the value the previous page returned as `next`, or absent');
+ const limit = Math.min(store.MAX_PAGE, Math.max(1, parseInt(req.query.limit, 10) || store.PAGE_SIZE));
+ const page = await store.search(s, filters, { limit, cursor });
+ res.json({
+ jobs: page.rows.map(wire),
+ next: store.encodeCursor(page.next),
+ count: page.rows.length,
+ filters,
+ facets: page.facets,
+ sources: sources.SOURCES.map((src) => ({ id: src.id, name: src.name, url: src.homepage })),
+ });
+ });
+
+ r.get('/jobs/:id', limits.reads('work.api.read'), async (req, res) => {
+ const row = await store.getListing(s, req.params.id);
+ if (!row) return problem(req, res, 404, 'work.jobs.not_found', 'No such listing. It may have expired from the board it came from.');
+ res.json(wire(row));
+ });
+
+ // ── The sources ─────────────────────────────────────────
+ r.get('/sources', limits.reads('work.api.read'), async (_req, res) => {
+ const fetches = await store.sourceFetches(s);
+ const counts = await store.countsBySource(s);
+ res.json({
+ sources: sources.SOURCES.map((src) => {
+ const f = fetches.get(src.id) || null;
+ return {
+ id: src.id,
+ name: src.name,
+ url: src.homepage,
+ terms: src.terms,
+ attribution: src.attribution,
+ listings: (counts.get(src.id) || {}).count || 0,
+ last_fetch: f ? { started_at: f.started_at, finished_at: f.finished_at || null, ok: !!f.ok, listings: Number(f.listings) || 0, error: f.error || null } : null,
+ };
+ }),
+ cadence_hours: Math.round(config.ingest.intervalMs / 3_600_000),
+ retention_days: config.ingest.retentionDays,
+ });
+ });
+
+ // ── Saved searches ──────────────────────────────────────
+ const savedWire = async (row) => store.toSavedWire(row, { newCount: await store.newCount(s, row) });
+
+ r.get('/saved-searches', limits.reads('work.api.read'), async (req, res) => {
+ const subject = person(req, res);
+ if (!subject) return undefined;
+ const rows = await store.listSaved(s, subject);
+ return res.json({ saved_searches: await Promise.all(rows.map(savedWire)) });
+ });
+
+ r.post('/saved-searches', limits.budget('work.saved_searches.write'), async (req, res) => {
+ const subject = person(req, res);
+ if (!subject) return undefined;
+ const b = req.body || {};
+ const filters = store.filtersOf({ q: b.q != null ? b.q : b.query, remote: b.remote, type: b.type, location: b.location });
+ if (!filters.q && !filters.remote && !filters.type && !filters.location) {
+ return problem(req, res, 422, 'work.saved_searches.empty', 'A saved search needs at least one of q, remote, type or location — otherwise it is the whole board.');
+ }
+ const out = await store.createSaved(s, subject, filters, { id: s.newId('ssc'), now: s.iso() });
+ if (out.error === 'too_many') return problem(req, res, 429, 'work.saved_searches.limit', `You can keep ${store.MAX_SAVED} saved searches; delete one first.`);
+ res.status(201).set('Location', `/api/v1/saved-searches/${out.row.id}`).json(await savedWire(out.row));
+ return undefined;
+ });
+
+ r.delete('/saved-searches/:id', limits.budget('work.saved_searches.write'), async (req, res) => {
+ const subject = person(req, res);
+ if (!subject) return undefined;
+ const gone = await store.deleteSaved(s, subject, req.params.id);
+ if (!gone) return problem(req, res, 404, 'work.saved_searches.not_found', 'You have no saved search with that id.');
+ return res.status(204).end();
+ });
+
return r;
return r;
}
}
@@ -31,8 +31,14 @@ function caller(req) {
@@ -31,8 +31,14 @@ function caller(req) {
* not declared here. The empty object is the skeleton's starting point:
* not declared here. The empty object is the skeleton's starting point:
*
*
* 'work.thing.create': { minute: 6, hour: 60 },
* 'work.thing.create': { minute: 6, hour: 60 },
+ *
+ * Searching the listings reads a local index, but it is the route a scraper would sit on and the one worth a
+ * number of its own; saving a search writes a row, so it gets a tighter one.
*/
*/
-const BUDGETS = {};
+const BUDGETS = {
+ 'work.jobs.search': { minute: 60, hour: 600 },
+ 'work.saved_searches.write': { minute: 20, hour: 200 },
+};
function createCallerLimits({ config, now = () => Date.now(), registry = null, log = console, enabled = true, valkey = null }) {
function createCallerLimits({ config, now = () => Date.now(), registry = null, log = console, enabled = true, valkey = null }) {
const refused = registry
const refused = registry
@@ -3,22 +3,30 @@
@@ -3,22 +3,30 @@
/**
/**
* Crawl artifacts for openvibe.work, built with openvibe-shared/seo: robots.txt, sitemap.xml, llms.txt and
* Crawl artifacts for openvibe.work, built with openvibe-shared/seo: robots.txt, sitemap.xml, llms.txt and
* llms-full.txt, and the home page's JSON-LD. The public pages are for search engines and AI crawlers; sign-in and
* llms-full.txt, and the home page's JSON-LD. The public pages are for search engines and AI crawlers; sign-in and
- * the API are not.
+ * the API are not, and neither are a person's saved searches.
*
*
- * The product extends PAGE_TEXT and publicPages() with its own pages; the routes and headers here stay.
+ * The sitemap lists the site's own pages plus the newest listings (their links back to the board are what a crawler
+ * wants to follow), capped at SITEMAP_JOBS. llms-full.txt describes the site's own pages, not every listing: a
+ * crawler wanting the listings has /jobs and the sitemap.
*/
*/
const fs = require('fs');
const fs = require('fs');
const path = require('path');
const path = require('path');
const seo = require('openvibe-shared/seo');
const seo = require('openvibe-shared/seo');
const cache = require('openvibe-shared/cache-policy');
const cache = require('openvibe-shared/cache-policy');
const { asyncRouter } = require('./router');
const { asyncRouter } = require('./router');
+const store = require('../jobs/store');
+const sources = require('../jobs/sources');
const SITE_NAME = 'OpenVibe.Work';
const SITE_NAME = 'OpenVibe.Work';
-const DESCRIPTION = 'OpenVibe.Work — Find work, with an agent on your side.';
-const DISALLOW = ['/auth/', '/api/'];
+const DESCRIPTION = 'OpenVibe.Work — Find work, with an agent on your side. Job listings from open boards with their source, searchable and free to read.';
+const DISALLOW = ['/auth/', '/api/', '/saved'];
+const SITEMAP_JOBS = 1000;
const PAGE_TEXT = {
const PAGE_TEXT = {
- '/': ['OpenVibe.Work home', 'OpenVibe.Work: Find work, with an agent on your side.'],
+ '/': ['OpenVibe.Work home', 'OpenVibe.Work gathers job listings from public job boards, names the source of every one and links back to the original listing. Search is free and needs no account.'],
+ '/jobs': ['Jobs', 'Search job listings gathered from Arbeitnow, Remotive and RemoteOK, newest first, filterable by words, workplace, job type and location.'],
+ '/sources': ['Sources', 'The public job boards OpenVibe.Work gathers from, the terms each is used under as applied here, and when each was last read.'],
+ '/docs': ['API', 'The OpenVibe.Work API: search the listings, read one with its provenance, list the sources and their terms, keep saved searches.'],
'/updates': ['What shipped on OpenVibe.Work', 'This site\'s update log, from the network changelog feed.'],
'/updates': ['What shipped on OpenVibe.Work', 'This site\'s update log, from the network changelog feed.'],
};
};
@@ -33,19 +41,22 @@ function siteUpdated() {
@@ -33,19 +41,22 @@ function siteUpdated() {
function homeJsonLd(config) {
function homeJsonLd(config) {
const site = String(config.baseUrl).replace(/\/+$/, '');
const site = String(config.baseUrl).replace(/\/+$/, '');
return [
return [
- seo.jsonLd.website({ name: SITE_NAME, url: site, description: DESCRIPTION }),
- seo.jsonLd.softwareApp({ name: SITE_NAME, url: site, description: DESCRIPTION, category: 'BusinessApplication', keywords: 'openvibe' }),
+ seo.jsonLd.website({ name: SITE_NAME, url: site, description: DESCRIPTION, searchUrl: `${site}/jobs?q={q}` }),
+ seo.jsonLd.softwareApp({ name: SITE_NAME, url: site, description: DESCRIPTION, category: 'BusinessApplication', keywords: 'openvibe, jobs, job search, remote jobs' }),
seo.jsonLd.webPage({ name: SITE_NAME, url: `${site}/`, description: DESCRIPTION, siteUrl: site }),
seo.jsonLd.webPage({ name: SITE_NAME, url: `${site}/`, description: DESCRIPTION, siteUrl: site }),
];
];
}
}
const publicPages = () => [
const publicPages = () => [
- { path: '/', changefreq: 'weekly', priority: 1.0 },
+ { path: '/', changefreq: 'daily', priority: 1.0 },
+ { path: '/jobs', changefreq: 'hourly', priority: 0.9 },
+ { path: '/sources', changefreq: 'weekly', priority: 0.6 },
+ { path: '/docs', changefreq: 'monthly', priority: 0.5 },
{ path: '/updates', changefreq: 'daily', priority: 0.5 },
{ path: '/updates', changefreq: 'daily', priority: 0.5 },
];
];
function createDiscoveryRoutes(ctx) {
function createDiscoveryRoutes(ctx) {
- const { config } = ctx;
+ const { config, s } = ctx;
const r = asyncRouter();
const r = asyncRouter();
const site = String(config.baseUrl).replace(/\/+$/, '');
const site = String(config.baseUrl).replace(/\/+$/, '');
const abs = (p) => `${site}${p}`;
const abs = (p) => `${site}${p}`;
@@ -53,21 +64,25 @@ function createDiscoveryRoutes(ctx) {
@@ -53,21 +64,25 @@ function createDiscoveryRoutes(ctx) {
r.get('/robots.txt', (_req, res) => {
r.get('/robots.txt', (_req, res) => {
res.type('text/plain').set('Cache-Control', TEXT).send(
res.type('text/plain').set('Cache-Control', TEXT).send(
- '# openvibe.work: the public pages are for search and AI crawlers; sign-in and the API are not.\n'
+ '# openvibe.work: the public pages are for search and AI crawlers; sign-in, the API and saved searches are not.\n'
+ seo.robotsTxt({ sitemaps: [abs('/sitemap.xml')], disallow: DISALLOW }));
+ seo.robotsTxt({ sitemaps: [abs('/sitemap.xml')], disallow: DISALLOW }));
});
});
r.get('/llms.txt', (_req, res) => {
r.get('/llms.txt', (_req, res) => {
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsTxt({
name: SITE_NAME,
name: SITE_NAME,
- summary: 'OpenVibe.Work: Find work, with an agent on your side.',
- details: 'Every page is server-rendered and readable without JavaScript.',
+ summary: DESCRIPTION,
+ details: 'Every page is server-rendered and readable without JavaScript. Listings come from public job boards under their terms: the source is named and the original listing is linked; only a short excerpt is kept here.',
sections: [
sections: [
{ title: 'Start here', links: [
{ title: 'Start here', links: [
{ title: 'OpenVibe.Work', url: abs('/'), note: 'Find work, with an agent on your side.' },
{ title: 'OpenVibe.Work', url: abs('/'), note: 'Find work, with an agent on your side.' },
+ { title: 'Search the listings', url: abs('/jobs'), note: 'Filter by words, workplace, job type and location.' },
+ { title: 'Where the listings come from', url: abs('/sources'), note: `${sources.SOURCES.map((x) => x.name).join(', ')}, with the terms each is used under.` },
{ title: 'What shipped on OpenVibe.Work', url: abs('/updates') },
{ title: 'What shipped on OpenVibe.Work', url: abs('/updates') },
] },
] },
- { title: 'Machine-readable', links: [
+ { title: 'For machines', links: [
+ { title: 'The API', url: abs('/docs'), note: 'GET /api/v1/jobs, /jobs/:id, /sources — public.' },
+ { title: 'The API, live', url: abs('/api/v1/jobs'), note: 'JSON listings with their source and the original URL.' },
{ title: 'Sitemap', url: abs('/sitemap.xml') },
{ title: 'Sitemap', url: abs('/sitemap.xml') },
{ title: 'Full text for language models', url: abs('/llms-full.txt') },
{ title: 'Full text for language models', url: abs('/llms-full.txt') },
{ title: 'Release metadata (JSON)', url: abs('/release.json') },
{ title: 'Release metadata (JSON)', url: abs('/release.json') },
@@ -84,20 +99,23 @@ function createDiscoveryRoutes(ctx) {
@@ -84,20 +99,23 @@ function createDiscoveryRoutes(ctx) {
const pages = publicPages().map((p) => ({ url: p.path, title: PAGE_TEXT[p.path][0], text: PAGE_TEXT[p.path][1] }));
const pages = publicPages().map((p) => ({ url: p.path, title: PAGE_TEXT[p.path][0], text: PAGE_TEXT[p.path][1] }));
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsFull({
res.type('text/plain').set('Cache-Control', TEXT).send(seo.llmsFull({
site: SITE_NAME,
site: SITE_NAME,
- summary: 'Every public page of OpenVibe.Work, one line each.',
+ summary: 'Every public page of OpenVibe.Work, one paragraph each; the listings themselves are at /jobs.',
base: site,
base: site,
maxBytes: 64 * 1024,
maxBytes: 64 * 1024,
sections: [{ title: 'Pages', pages }],
sections: [{ title: 'Pages', pages }],
}));
}));
});
});
- r.get('/sitemap.xml', (_req, res) => {
+ r.get('/sitemap.xml', async (_req, res) => {
const lastmod = siteUpdated();
const lastmod = siteUpdated();
const urls = publicPages().map((e) => ({ loc: abs(e.path), ...(lastmod ? { lastmod } : {}), changefreq: e.changefreq, priority: e.priority }));
const urls = publicPages().map((e) => ({ loc: abs(e.path), ...(lastmod ? { lastmod } : {}), changefreq: e.changefreq, priority: e.priority }));
+ // The newest listings, so a crawler reaches each board's own page through ours. No lastmod: these entries turn
+ // over with every ingest, and the site's own freshness is the pages above.
+ for (const row of await store.recentIds(s, SITEMAP_JOBS)) urls.push({ loc: abs(`/jobs/${row.id}`), changefreq: 'daily', priority: 0.4 });
res.type('application/xml').set('Cache-Control', TEXT).send(seo.sitemapXml(urls));
res.type('application/xml').set('Cache-Control', TEXT).send(seo.sitemapXml(urls));
});
});
return r;
return r;
}
}
-module.exports = { createDiscoveryRoutes, homeJsonLd, publicPages, DESCRIPTION, SITE_NAME };
+module.exports = { createDiscoveryRoutes, homeJsonLd, publicPages, DESCRIPTION, SITE_NAME, DISALLOW, PAGE_TEXT };
@@ -1,11 +1,13 @@
@@ -1,11 +1,13 @@
'use strict';
'use strict';
/**
/**
- * Public pages: the home page and the update log. Crawl artifacts (robots.txt, sitemap.xml, llms.txt,
- * llms-full.txt, JSON-LD) are http/discovery.js.
+ * Public pages: the home page, the search (/jobs), one listing (/jobs/:id), the saved searches (/saved), the sources
+ * (/sources) and the update log (/updates). Crawl artifacts (robots.txt, sitemap.xml, llms.txt, llms-full.txt,
+ * JSON-LD) are http/discovery.js.
*
*
- * Every page works without JavaScript and is server-rendered through openvibe-shared/shell (render/layout.js).
- * The product replaces the home page's sections and adds its own routes here.
+ * Every page is server-rendered through openvibe-shared/shell and works without JavaScript: the search is a GET form,
+ * saving and deleting a saved search are plain POSTs. What is shown is read, never restated — the listings come from
+ * the database, the sources and their terms from server/jobs/sources.js.
*/
*/
const ovServe = require('openvibe-shared/serve');
const ovServe = require('openvibe-shared/serve');
const frame = require('openvibe-shared/frame');
const frame = require('openvibe-shared/frame');
@@ -13,58 +15,343 @@ const showcase = require('openvibe-shared/showcase');
@@ -13,58 +15,343 @@ const showcase = require('openvibe-shared/showcase');
const cache = require('openvibe-shared/cache-policy');
const cache = require('openvibe-shared/cache-policy');
const { asyncRouter } = require('./router');
const { asyncRouter } = require('./router');
const { createDiscoveryRoutes, homeJsonLd } = require('./discovery');
const { createDiscoveryRoutes, homeJsonLd } = require('./discovery');
-const { html, raw } = require('../render/html');
+const { sameOrigin } = require('./principal');
+const { html, raw, table, notice, time } = require('../render/html');
const { send } = require('../render/layout');
const { send } = require('../render/layout');
+const listing = require('../render/listing');
+const store = require('../jobs/store');
+const sources = require('../jobs/sources');
const SITE_NAME = 'OpenVibe.Work';
const SITE_NAME = 'OpenVibe.Work';
const TAGLINE = 'Find work, with an agent on your side.';
const TAGLINE = 'Find work, with an agent on your side.';
function createPageRoutes(ctx) {
function createPageRoutes(ctx) {
- const { config } = ctx;
+ const { config, s } = ctx;
const r = asyncRouter();
const r = asyncRouter();
const PUBLIC_CACHE = cache.htmlHeaders({ maxAge: 300 });
const PUBLIC_CACHE = cache.htmlHeaders({ maxAge: 300 });
const page = (req, res, o, status = 200) => send(res, status, { viewer: req.viewer, config, path: req.originalUrl, ...o });
const page = (req, res, o, status = 200) => send(res, status, { viewer: req.viewer, config, path: req.originalUrl, ...o });
const signedIn = (req) => req.viewer && req.viewer.kind === 'user' && req.viewer.subject;
const signedIn = (req) => req.viewer && req.viewer.kind === 'user' && req.viewer.subject;
+ const subjectOf = (req) => `user:${req.viewer.subject}`;
+ const sourceOf = (row) => sources.byId(row.source);
+
+ // ── Search: one form, one set of filters, one place they are read ──
+ const filterForm = (filters, { facets = null, action = '/jobs' } = {}) => html`<form class="job-search" method="get" action="${action}" role="search">
+<label class="sr-only" for="q">Search listings</label>
+<input type="search" id="q" name="q" value="${filters.q}" placeholder="Job title, skill, company" maxlength="120">
+<label class="sr-only" for="location">Location</label>
+<input type="text" id="location" name="location" value="${filters.location}" placeholder="Location" maxlength="120">
+<label class="sr-only" for="remote">Remote</label>
+<select id="remote" name="remote">
+<option value=""${filters.remote === '' ? raw(' selected') : ''}>Remote or on-site</option>
+<option value="true"${filters.remote === 'true' ? raw(' selected') : ''}>Remote only</option>
+<option value="false"${filters.remote === 'false' ? raw(' selected') : ''}>On-site only</option>
+</select>
+<label class="sr-only" for="type">Job type</label>
+<input type="text" id="type" name="type" value="${filters.type}" placeholder="Job type" maxlength="60">
+<button class="sc-btn sc-primary" type="submit">Search</button>
+${(filters.q || filters.remote || filters.type || filters.location) ? html`<a class="job-clear" href="${action}">Clear</a>` : ''}
+</form>
+${facets ? facetBar(filters, facets, action) : ''}`;
+
+ /** The facets: what is actually in the result set, as links that set one filter and keep the rest. */
+ const facetBar = (filters, facets, action) => {
+ const link = (patch, label, active, count) => html`<li${active ? raw(' class="on"') : ''}><a href="${withFilters(filters, { ...patch, cursor: null }, action)}">${label}${count == null ? '' : html` <span class="facet-n">${count}</span>`}</a></li>`;
+ return html`<div class="job-facets">
+<p class="facet-head"><strong>${facets.matches}</strong> listing${facets.matches === 1 ? '' : 's'} match${facets.matches === 1 ? 'es' : ''} this search</p>
+<ul class="facets" aria-label="Remote"><li class="facet-label">Workplace</li>
+${link({ remote: null }, 'Any', filters.remote === '', null)}
+${link({ remote: 'true' }, 'Remote', filters.remote === 'true', facets.remote.true)}
+${link({ remote: 'false' }, 'On-site', filters.remote === 'false', facets.remote.false)}
+</ul>
+${facets.type.length ? html`<ul class="facets" aria-label="Job type"><li class="facet-label">Type</li>
+${facets.type.map((t) => link({ type: t.type }, t.type, filters.type === t.type, t.count))}
+</ul>` : ''}
+</div>`;
+ };
+
+ const passFilters = (f) => {
+ const out = {};
+ if (f.q) out.q = f.q;
+ if (f.remote) out.remote = f.remote;
+ if (f.type) out.type = f.type;
+ if (f.location) out.location = f.location;
+ return out;
+ };
+ /** A URL for one filter patch, over the current query string. Null removes a filter. */
+ function withFilters(filters, patch, path) {
+ const merged = { ...passFilters(filters), ...patch };
+ const params = new URLSearchParams();
+ for (const [k, v] of Object.entries(merged)) if (v != null && v !== '') params.set(k, String(v));
+ const q = params.toString();
+ return q ? `${path}?${q}` : path;
+ }
+
+ /** Which of the three sources a listing came from, named; the ingest keeps to these and nothing else. */
+ const sourceLine = (row) => html`${listing.sourceLink(row, sourceOf(row))}`;
// ── Home ─────────────────────────────────────────────────
// ── Home ─────────────────────────────────────────────────
- r.get('/', (req, res) => {
+ r.get('/', async (req, res) => {
+ const [latest, counts, rows] = await Promise.all([store.listRecent(s, 6, { remote: true }), store.countsBySource(s), store.search(s, {}, { limit: 1 })]);
+ const total = rows.facets.matches;
+ const fetches = await store.sourceFetches(s);
const hero = showcase.hero({
const hero = showcase.hero({
- eyebrow: `${SITE_NAME} · ${TAGLINE}`,
- title: SITE_NAME,
- accent: TAGLINE,
- lede: `${SITE_NAME} is a new OpenVibe service. This is its skeleton: the platform plumbing below is in place, and the product is what goes on top of it.`,
- actions: signedIn(req)
- ? [{ label: 'Your account', href: '/auth/me' }, { label: 'What shipped', href: '/updates' }]
- : [{ label: 'Sign in with OpenVibe', href: '/auth/login?next=%2F', primary: true }, { label: 'What shipped', href: '/updates' }],
- note: 'Open source (AGPL-3.0). Sign-in is OpenVibe.Network; a person\'s token stays in an httpOnly cookie.',
+ eyebrow: `${SITE_NAME} · job listings with their provenance`,
+ title: 'Find work.',
+ accent: 'Every listing shows where it came from.',
+ lede: `${SITE_NAME} gathers job listings from public job boards, keeps the board's own link on every one, and lets you search them. No paywall, no account needed to look: sign in only to keep a search. An agent that helps you apply comes next.`,
+ actions: [{ label: 'Search the listings', href: '#search', primary: true }, { label: 'Where they come from', href: '/sources' }],
+ note: 'Open source (AGPL-3.0). Listings are shown with the source named and a link to the original — read the full listing there.',
});
});
page(req, res, {
page(req, res, {
index: true, cache: signedIn(req) ? null : PUBLIC_CACHE,
index: true, cache: signedIn(req) ? null : PUBLIC_CACHE,
jsonLd: homeJsonLd(config),
jsonLd: homeJsonLd(config),
styles: [showcase.STYLESHEET],
styles: [showcase.STYLESHEET],
body: html`${raw(hero)}
body: html`${raw(hero)}
-${raw(showcase.features({
- title: 'What is already here',
- lede: 'The parts every OpenVibe service needs, so a new product starts from them instead of rebuilding them.',
- items: [
- { icon: 'ov:account', title: 'Sign-in with OpenVibe.Network', text: 'OAuth 2 with PKCE (S256), server-side code exchange, httpOnly session cookies, and offline token verification against the Network signing key.' },
- { icon: 'ov:page', title: 'Server-rendered pages', text: 'The OpenVibe Frame (navbar and footer), the shared theme, one stylesheet, crawl artifacts and JSON-LD. Every page is complete without JavaScript.' },
- { icon: 'ov:tools', title: 'An API mount and capabilities', text: 'GET /api/v1/ping today; the router, the problem+json errors and the app/person capability guard are wired and waiting for the product\'s routes.' },
- { icon: 'ov:gauge', title: 'Limits and readiness', text: 'Per-address and per-caller rate limits, /api/health, a truthful /api/ready and Prometheus metrics on loopback.' },
- { icon: 'ov:db', title: 'PostgreSQL and migrations', text: 'openvibe-sdk/db, migrations/NNNN_*.sql applied at boot, PGlite in development, and a store with an injectable clock.' },
- { icon: 'ov:deploy', title: 'Deploy files and tests', text: 'nginx and systemd units, an environment template, and a test suite on PGlite with a mock Network, all through npm test.' },
- ],
- }))}
+<section class="sc-sec" id="search" aria-labelledby="h-search"><h2 id="h-search">Search the listings</h2>
+${filterForm({ q: '', remote: '', type: '', location: '' }, { action: '/jobs' })}
+<p class="muted">${total} listing${total === 1 ? '' : 's'} from ${sources.SOURCES.length} sources, newest first.</p></section>
+<section class="sc-sec" aria-labelledby="h-latest"><h2 id="h-latest">Latest remote jobs</h2>
+${listing.listingList(latest, { sourceOf, empty: 'No listings yet — the ingest has not run.' })}
+<p><a class="sc-btn" href="/jobs?remote=true">All remote listings</a></p></section>
+<section class="sc-sec" aria-labelledby="h-sources"><h2 id="h-sources">Where the listings come from</h2>
+<p>Each source below is a public job board read through its own API, under its own terms. Every listing on this site names its source and links to the original; we keep a short excerpt and send you to the board for the rest.</p>
+${table(['Source', 'Listings', 'Last read', 'Terms'], sources.SOURCES.map((src) => {
+ const f = fetches.get(src.id);
+ return [
+ html`<a href="${src.homepage}" rel="noopener">${src.name}</a>`,
+ String((counts.get(src.id) || {}).count || 0),
+ f && f.finished_at ? html`${time(f.finished_at)}${f.ok ? '' : html` <span class="muted small">(the last read failed)</span>`}` : html`<span class="muted">not read yet</span>`,
+ html`${src.attribution} — <a href="/sources#${src.id}">terms as applied</a>`,
+ ];
+ }))}</section>
${raw(showcase.steps({
${raw(showcase.steps({
- title: 'How a new service starts',
+ title: 'How it works',
items: [
items: [
- { title: 'Generate it', text: 'node scripts/new-service.js replaces the skeleton\'s tokens (service id, name, domain, port, tagline, env prefix) in every file and file name.' },
- { title: 'Install and run', text: 'npm install, then npm run dev: the service listens on its port with an embedded PGlite database, if no DATABASE_URL is set.' },
- { title: 'Make it a product', text: 'Add the product\'s tables and migrations, its capabilities, its routes in server/http/api.js and its pages in server/http/pages.js.' },
- { title: 'Grant and deploy', text: 'Register the service and its capabilities with OpenVibe.Contracts and OpenVibe.Services, then install the unit and the vhost.' },
+ { title: 'Gathered from open boards', text: 'A background ingest reads each board\'s public API every few hours, keeps the listing\'s own id and URL, and stores a short excerpt — never the full description.' },
+ { title: 'Search it, with the source on it', text: 'Filter by text, workplace, job type and location. Every result names the board it came from and links back to it.' },
+ { title: 'Read it on the board', text: 'The listing page shows the excerpt and sends you to the original listing to read it in full and apply. Nothing is republished.' },
+ { title: 'Keep a search', text: 'Sign in to save a search; /saved shows each one with the count of listings that appeared since you last looked.' },
+ { title: 'Nothing behind a paywall', text: 'Reading and searching is free and needs no account. There is no premium tier on the listings, and listings expire when a board stops showing them.' },
+ { title: 'An agent, next', text: 'The agent that drafts and tracks applications is the next pillar; alerts through OpenVibe.Watch come with it.' },
],
],
}))}
}))}
-${raw(showcase.cta({ title: 'Read the API', text: 'GET /api/v1/ping is the whole API today. The rest is what the product adds.', actions: [{ label: 'The API', href: '/api/v1/ping' }, { label: 'What shipped', href: '/updates' }] }))}`,
+${raw(showcase.cta({ title: 'Prefer the API?', text: 'The same search, the sources and their terms, and your saved searches over HTTP. Reading is public.', actions: [{ label: 'The API', href: '/docs' }, { label: 'The sources', href: '/sources' }] }))}`,
+ });
+ });
+
+ // ── Search ───────────────────────────────────────────────
+ r.get('/jobs', async (req, res) => {
+ const filters = store.filtersOf(req.query);
+ const cursor = store.decodeCursor(req.query.cursor);
+ // A cursor we did not write is not a position in the result set; start from the top rather than guess.
+ const p = await store.search(s, filters, { cursor });
+ const nextUrl = p.next ? withFilters(filters, { cursor: store.encodeCursor(p.next) }, '/jobs') : null;
+ const described = [filters.q ? `“${filters.q}”` : '', filters.remote === 'true' ? 'remote' : filters.remote === 'false' ? 'on-site' : '', filters.type ? filters.type : '', filters.location ? `in ${filters.location}` : ''].filter(Boolean).join(' ');
+ page(req, res, {
+ index: true, cache: signedIn(req) ? null : PUBLIC_CACHE,
+ title: described ? `Jobs: ${described}` : 'Jobs',
+ description: `Job listings from open boards${described ? ` matching ${described}` : ''}, newest first, each with the source it came from and a link to the original listing.`,
+ crumbs: [{ label: 'Jobs' }],
+ styles: [showcase.STYLESHEET],
+ body: html`<h1>Jobs</h1>
+${filterForm(filters, { facets: p.facets })}
+${listing.listingList(p.rows, { sourceOf, empty: 'No listings match this search. Try fewer words, or clear a filter and search again.' })}
+<nav class="job-pager" aria-label="Pages">
+${cursor ? html`<a class="sc-btn" href="${withFilters(filters, { cursor: null }, '/jobs')}">First page</a>` : ''}
+${nextUrl ? html`<a class="sc-btn sc-primary" href="${nextUrl}">Older listings</a>` : html`<span class="muted">That is every listing matching this search.</span>`}
+</nav>
+<p class="muted small">Read the full listing on its source: each entry links back to the board it came from.</p>`,
+ });
+ });
+
+ // ── One listing ──────────────────────────────────────────
+ r.get('/jobs/:id', async (req, res) => {
+ const row = await store.getListing(s, req.params.id);
+ if (!row) {
+ return page(req, res, {
+ title: 'Listing not found',
+ body: html`<h1>Listing not found</h1><p>No listing here with that id. A listing is removed once the board it came from stops showing it, and old links stop working.</p><p><a href="/jobs">Search the listings</a></p>`,
+ }, 404);
+ }
+ const src = sourceOf(row);
+ const l = listing.meta(row);
+ page(req, res, {
+ index: true, cache: PUBLIC_CACHE,
+ title: `${l.title} at ${l.company}`,
+ description: listing.plainExcerpt(row).slice(0, 300) || `${l.title} at ${l.company}, from ${src.name}.`,
+ crumbs: [{ label: 'Jobs', href: '/jobs' }, { label: l.title }],
+ styles: [showcase.STYLESHEET],
+ jsonLd: [listing.jobPosting(row, { siteUrl: config.baseUrl })],
+ body: html`<article class="job">
+<h1>${l.title}</h1>
+<p class="job-meta"><span class="job-company">${l.company}</span> · <span class="job-location">${l.location || 'Location not given'}</span>${l.remote ? ' · Remote' : ''}${l.job_type ? html` · <span class="job-type">${l.job_type}</span>` : ''}</p>
+${l.salary ? html`<p class="job-salary"><strong>Salary:</strong> ${l.salary} <span class="muted small">(as the board gives it)</span></p>` : ''}
+${l.tags.length ? html`<ul class="job-tags">${l.tags.map((t) => html`<li>${t}</li>`)}</ul>` : ''}
+<section class="job-excerpt-block" aria-labelledby="h-excerpt"><h2 id="h-excerpt">Excerpt</h2>
+<p>${l.excerpt}</p>
+<p class="muted small">This is a short excerpt only. The full listing stays on the board.</p></section>
+<p class="job-apply"><a class="sc-btn sc-primary" href="${l.url}" rel="nofollow noopener external" target="_blank">Read the full listing on ${src.name}</a></p>
+<dl class="job-facts">
+<dt>Source</dt><dd>${sourceLine(row)} — <a href="/sources#${src.id}">its terms as applied</a></dd>
+${l.posted_at ? html`<dt>Posted</dt><dd>${time(l.posted_at)}</dd>` : ''}
+<dt>Read from the board</dt><dd>${time(row.fetched_at)}</dd>
+<dt>Board's id</dt><dd><code>${row.source_id}</code></dd>
+</dl>
+<p class="muted small">Found a problem with this listing, or want it gone? It is the board's listing, not ours — the source above is where it lives. In code: <code>GET /api/v1/jobs/${row.id}</code>.</p>
+</article>`,
+ });
+ });
+
+ // ── Saved searches ───────────────────────────────────────
+ function savedForm(req, filters, { problem = null } = {}) {
+ return html`${problem ? notice(problem, 'warn') : ''}
+<form class="saved-form" method="post" action="/saved">
+<fieldset><legend>Keep this search</legend>
+<label>Words <input type="text" name="q" value="${filters.q}" maxlength="120" placeholder="Job title, skill, company"></label>
+<label>Location <input type="text" name="location" value="${filters.location}" maxlength="120"></label>
+<label>Job type <input type="text" name="type" value="${filters.type}" maxlength="60"></label>
+<label>Workplace <select name="remote">
+<option value=""${filters.remote === '' ? raw(' selected') : ''}>Any</option>
+<option value="true"${filters.remote === 'true' ? raw(' selected') : ''}>Remote only</option>
+<option value="false"${filters.remote === 'false' ? raw(' selected') : ''}>On-site only</option>
+</select></label>
+<button class="sc-btn sc-primary" type="submit">Save this search</button>
+</fieldset>
+</form>`;
+ }
+
+ const describeSaved = (row) => {
+ const bits = [row.query ? `“${row.query}”` : '', row.remote === 'true' ? 'remote' : row.remote === 'false' ? 'on-site' : '', row.type ? row.type : '', row.location ? `in ${row.location}` : ''].filter(Boolean);
+ return bits.length ? bits.join(' ') : 'everything';
+ };
+
+ r.get('/saved', async (req, res) => {
+ if (!signedIn(req)) {
+ return page(req, res, {
+ title: 'Saved searches',
+ body: html`<h1>Saved searches</h1>
+<p>Sign in to keep a search and see what has appeared since you last looked. Searching itself needs no account.</p>
+<p><a class="sc-btn sc-primary" href="/auth/login?next=%2Fsaved">Sign in with OpenVibe</a> <a class="sc-btn" href="/jobs">Search the listings</a></p>`,
+ });
+ }
+ const subject = subjectOf(req);
+ const rows = await store.listSaved(s, subject);
+ // The counts are read before last_seen_at moves: opening this page is what "looking" means.
+ const withCounts = await Promise.all(rows.map(async (row) => ({ row, count: await store.newCount(s, row) })));
+ const now = s.iso();
+ for (const row of rows) await store.touchSaved(s, subject, row.id, now);
+ const initial = store.filtersOf(req.query);
+ page(req, res, {
+ title: 'Saved searches', crumbs: [{ label: 'Saved searches' }],
+ body: html`<h1>Saved searches</h1>
+<p>${rows.length ? html`You are keeping ${rows.length} search${rows.length === 1 ? '' : 'es'}. The count is how many listings appeared since you last opened this page.` : 'You are not keeping any search yet.'}</p>
+${withCounts.length ? html`<ul class="saved-list">${withCounts.map(({ row, count }) => html`<li class="saved-item">
+<p class="saved-query"><a href="${store.toSavedWire(row).search_path}">${describeSaved(row)}</a> <span class="badge${count ? ' ok' : ''}">${count ? `${count} new` : 'nothing new'}</span></p>
+<p class="muted small">Saved ${time(row.created_at)} · last looked ${row.last_seen_at ? time(row.last_seen_at) : 'never'}</p>
+<form method="post" action="/saved/${row.id}/delete" class="inline"><button class="sc-btn" type="submit">Forget it</button></form>
+</li>`)}</ul>` : ''}
+${savedForm(req, initial)}
+<p class="muted small">Alerts — being told when a listing that fits appears — come later, through OpenVibe.Watch. For now a saved search is a search you do not have to type again, and a count of what is new.</p>
+<p class="muted small">In code: <code>GET /api/v1/saved-searches</code>.</p>`,
+ });
+ });
+
+ r.post('/saved', require('express').urlencoded({ extended: false, limit: '8kb' }), async (req, res) => {
+ res.set('Cache-Control', cache.htmlHeaders({ private: true }));
+ if (!signedIn(req)) return res.redirect(303, '/auth/login?next=%2Fsaved');
+ // A signed-in write must come from this site, as everywhere else in this service.
+ if (!sameOrigin(req, config.baseUrl)) return res.status(403).type('text/plain').send('Saving a search must come from openvibe.work itself.');
+ const filters = store.filtersOf(req.body || {});
+ // A search with nothing in it is the whole board, which is not something worth keeping (the API refuses it
+ // too). Say so on the page rather than saving it.
+ if (!filters.q && !filters.remote && !filters.type && !filters.location) {
+ return page(req, res, {
+ title: 'Saved searches', crumbs: [{ label: 'Saved searches' }],
+ body: html`<h1>Saved searches</h1>
+${notice('Give the search at least one thing to look for — words, a location, a job type or a workplace — or there is nothing to save.', 'warn')}
+<p><a href="/saved">Back to your saved searches</a></p>`,
+ }, 422);
+ }
+ const out = await store.createSaved(s, subjectOf(req), filters, { id: s.newId('ssc'), now: s.iso() });
+ if (out.error === 'too_many') {
+ const rows = await store.listSaved(s, subjectOf(req));
+ return page(req, res, { title: 'Saved searches', body: html`<h1>Saved searches</h1>${notice(`You can keep ${store.MAX_SAVED} saved searches. Forget one below, then save this again.`, 'warn')}<p><a href="/saved">Back to your saved searches</a> (${rows.length})</p>` }, 429);
+ }
+ return res.redirect(303, '/saved');
+ });
+
+ r.post('/saved/:id/delete', async (req, res) => {
+ res.set('Cache-Control', cache.htmlHeaders({ private: true }));
+ if (!signedIn(req)) return res.redirect(303, '/auth/login?next=%2Fsaved');
+ if (!sameOrigin(req, config.baseUrl)) return res.status(403).type('text/plain').send('Forgetting a search must come from openvibe.work itself.');
+ await store.deleteSaved(s, subjectOf(req), req.params.id);
+ res.redirect(303, '/saved');
+ });
+
+ // ── The sources and their terms ──────────────────────────
+ r.get('/sources', async (req, res) => {
+ const [fetches, counts] = await Promise.all([store.sourceFetches(s), store.countsBySource(s)]);
+ page(req, res, {
+ index: true, cache: PUBLIC_CACHE,
+ title: 'Sources',
+ description: 'The public job boards OpenVibe.Work gathers listings from, the terms each one is used under as applied here, and when each was last read.',
+ crumbs: [{ label: 'Sources' }],
+ body: html`<h1>Sources</h1>
+<p>Every listing here comes from one of these boards, read through its public API. We keep the board's own id and URL for each listing, show a short excerpt, and link to the original on every page that shows it. Nothing is republished, and no board is read more often than its terms allow.</p>
+${sources.SOURCES.map((src) => {
+ const f = fetches.get(src.id);
+ return html`<section class="source-block" id="${src.id}" aria-labelledby="h-${src.id}">
+<h2 id="h-${src.id}"><a href="${src.homepage}" rel="noopener">${src.name}</a></h2>
+<p class="muted small">${(counts.get(src.id) || {}).count || 0} live listing(s) from this board right now.</p>
+<h3>Its terms, as applied here</h3>
+<p>${src.terms}</p>
+<p class="source-attribution">Every listing from this board is shown as: <strong>${src.attribution}</strong>, linked back to the original. On this site that means:</p>
+<ul class="source-links"><li>List and detail pages name ${src.name} and link the listing to the original URL on the board.</li>
+<li>The listing page carries a <em>“Read the full listing on ${src.name}”</em> link, and the full description is not stored or shown.</li>
+<li>${src.name} listings also carry their source in the JSON-LD on the listing page.</li></ul>
+<h3>What each read does</h3>
+<dl class="job-facts">
+<dt>API</dt><dd><code>${src.requests(config)[0]}</code></dd>
+<dt>Last read</dt><dd>${f && f.started_at ? html`${time(f.started_at)}${f.ok ? ' — succeeded' : ' — failed'}${f.error ? html` <span class="muted small">(${f.error})</span>` : ''}` : html`<span class="muted">not read yet</span>`}</dd>
+<dt>Reads every</dt><dd>${Math.round(config.ingest.intervalMs / 3_600_000)} hours ${src.id === 'remotive' ? '(its terms allow about four a day)' : ''}</dd>
+<dt>Listings kept</dt><dd>${(counts.get(src.id) || {}).count || 0}, retired when the board stops showing one for ${config.ingest.retentionDays} days</dd>
+</dl>
+</section>`;
+ })}
+<p class="muted">Machine-readable, with each source's terms and last fetch: <a href="/api/v1/sources"><code>GET /api/v1/sources</code></a>.</p>`,
+ });
+ });
+
+ // ── The API reference ────────────────────────────────────
+ r.get('/docs', (req, res) => {
+ const b = config.baseUrl;
+ const search = `curl -s '${b}/api/v1/jobs?q=engineer&remote=true'
+# → { "jobs": [ { "id": "job_…", "title": "…", "company": "…", "url": "https://…", "source": "remotive", "source_name": "Remotive", "posted_at": "…", "excerpt": "…" } ], "next": null, "facets": { … } }`;
+ page(req, res, {
+ index: true, cache: PUBLIC_CACHE,
+ title: 'API',
+ description: 'The OpenVibe.Work API: search the listings, read one with its provenance, list the sources and their terms, and keep saved searches. Reading is public.',
+ crumbs: [{ label: 'API' }],
+ body: html`<h1>The API</h1>
+<p>Everything the site does is this API. Reading is public; a saved search is a person's.</p>
+<h2 id="who">Who can call it</h2>
+<ul>
+<li><strong>Anyone</strong>: <code>GET /api/v1/jobs</code>, <code>GET /api/v1/jobs/:id</code> and <code>GET /api/v1/sources</code> need no token.</li>
+<li><strong>A person</strong>: the saved searches need a signed-in person — their Network token as <code>Authorization: Bearer</code>, or this site's sign-in. They belong to that person alone.</li>
+</ul>
+${table(['Route', 'Who', 'What it does'], [
+ [html`<code>GET /api/v1/jobs</code>`, 'public', html`Search: <code>?q=</code>, <code>?remote=true|false</code>, <code>?type=</code>, <code>?location=</code>, <code>?limit=</code> (to ${store.MAX_PAGE}), <code>?cursor=</code> (the <code>next</code> the previous page returned). Newest first, ${store.PAGE_SIZE} a page, with facets for remote and job type.`],
+ [html`<code>GET /api/v1/jobs/:id</code>`, 'public', 'One listing: the board\'s fields, the excerpt, the source, and <code>url</code> — the original listing to apply on.'],
+ [html`<code>GET /api/v1/sources</code>`, 'public', 'The boards, each one\'s terms as applied, its listing count and when it was last read.'],
+ [html`<code>GET /api/v1/saved-searches</code>`, 'person', 'Your saved searches, each with <code>new_count</code> — listings that appeared since you last looked.'],
+ [html`<code>POST /api/v1/saved-searches</code>`, 'person', html`Save one: <code>{ q?, remote?, type?, location? }</code>. At least one filter. 201 with the saved search.`],
+ [html`<code>DELETE /api/v1/saved-searches/:id</code>`, 'person', 'Forget one. 204.'],
+ ])}
+${raw(showcase.code({ title: 'Search, then read one', samples: [{ label: 'bash', lang: 'bash', code: search }] }))}
+<h2 id="terms">The terms we apply</h2>
+<p>Each source is read under its own terms, which are stated on <a href="/sources">/sources</a> and in <code>GET /api/v1/sources</code>: the source is named, the original listing is linked, and only a short excerpt is stored. If you build on this API, carry the attribution and the link back with the listing.</p>
+<h2 id="limits">Limits</h2>
+<p>Search is limited per caller (per person, per app, else per address); past it the API answers 429 <code>rate_limited</code> with <code>Retry-After</code>. Errors are RFC 9457 <code>application/problem+json</code> with a stable <code>code</code>.</p>`,
});
});
});
});
@@ -80,4 +367,4 @@ ${raw(showcase.cta({ title: 'Read the API', text: 'GET /api/v1/ping is the whole
@@ -80,4 +367,4 @@ ${raw(showcase.cta({ title: 'Read the API', text: 'GET /api/v1/ping is the whole
return r;
return r;
}
}
-module.exports = { createPageRoutes };
+module.exports = { createPageRoutes, SITE_NAME, TAGLINE };
@@ -8,7 +8,7 @@ const express = require('express');
@@ -8,7 +8,7 @@ const express = require('express');
function asyncRouter() {
function asyncRouter() {
const r = express.Router();
const r = express.Router();
- for (const method of ['get', 'post']) {
+ for (const method of ['get', 'post', 'delete', 'put', 'patch']) {
const orig = r[method].bind(r);
const orig = r[method].bind(r);
r[method] = (path, ...handlers) => orig(path, ...handlers.map((h) => (h.length === 4 ? h : (req, res, next) => {
r[method] = (path, ...handlers) => orig(path, ...handlers.map((h) => (h.length === 4 ? h : (req, res, next) => {
try {
try {
@@ -14,7 +14,7 @@ const { gracefulStop } = require('openvibe-sdk/service');
@@ -14,7 +14,7 @@ const { gracefulStop } = require('openvibe-sdk/service');
function createLifecycle({ server, ctx, exit, signals, timers = [] }) {
function createLifecycle({ server, ctx, exit, signals, timers = [] }) {
return gracefulStop({
return gracefulStop({
name: 'OpenVibe.Work', server, deadlineExitCode: 0, exit, signals, deadlineMs: 10_000,
name: 'OpenVibe.Work', server, deadlineExitCode: 0, exit, signals, deadlineMs: 10_000,
- close: [() => { for (const t of timers) clearInterval(t); }, () => ctx.keys.client.stop(), () => ctx.s.close()],
+ close: [() => { for (const t of timers) clearInterval(t); }, () => ctx.ingest.stop(), () => ctx.keys.client.stop(), () => ctx.s.close()],
});
});
}
}
@@ -27,6 +27,7 @@ async function start() {
@@ -27,6 +27,7 @@ async function start() {
});
});
server.keepAliveTimeout = 65_000;
server.keepAliveTimeout = 65_000;
ctx.keys.client.start();
ctx.keys.client.start();
+ ctx.ingest.start();
createLifecycle({ server, ctx });
createLifecycle({ server, ctx });
return { server, ctx };
return { server, ctx };
new file mode 100644
new file mode 100644
@@ -0,0 +1,117 @@
@@ -0,0 +1,117 @@
+'use strict';
+
+/**
+ * The background ingest: every six hours, read each job board's public API (server/jobs/sources.js), keep what it
+ * shows and retire what it stopped showing.
+ *
+ * It is off unless WORK_INGEST=on, so neither a test nor a development run ever calls a job board; when it is on,
+ * each source keeps its own timer, its own last-fetch row and its own failures — one board being down never stops the
+ * others, and the failure is recorded and shown on /sources rather than hidden.
+ *
+ * Every outbound request goes to a base URL from the configuration (never to a URL anyone typed), with this
+ * service's User-Agent, a 15-second timeout and a size cap, and only http(s).
+ */
+const sources = require('../jobs/sources');
+const store = require('../jobs/store');
+
+const MAX_BYTES = 8 * 1024 * 1024;
+const START_DELAY_MS = 5_000;
+
+async function fetchJson(url, { fetchImpl, timeoutMs, userAgent }) {
+ let res;
+ try {
+ res = await fetchImpl(url, {
+ headers: { 'user-agent': userAgent, accept: 'application/json' },
+ redirect: 'follow',
+ signal: AbortSignal.timeout(timeoutMs),
+ });
+ } catch (err) {
+ throw new Error(err && err.name === 'TimeoutError' ? `no answer in ${Math.round(timeoutMs / 1000)}s` : 'could not be reached');
+ }
+ if (!res.ok) throw new Error(`HTTP ${res.status} from the board`);
+ const text = await res.text();
+ if (text.length > MAX_BYTES) throw new Error('the board answered with more than 8 MB');
+ try { return JSON.parse(text); } catch { throw new Error('the board answered with something that is not JSON'); }
+}
+
+function createIngest({ config, s, fetchImpl = globalThis.fetch, log = console }) {
+ const timers = new Map();
+ let running = false;
+
+ /** Read one source end to end and record what happened. Never throws: the caller is a timer. */
+ async function runSource(sourceId) {
+ const source = sources.byId(sourceId);
+ if (!source) throw new Error(`no source ${sourceId}`);
+ const startedAt = s.iso();
+ const fetchedAt = startedAt;
+ await store.beginFetch(s, source.id, startedAt);
+ let listings = 0;
+ try {
+ const seen = new Map();
+ for (const url of source.requests(config)) {
+ const json = await fetchJson(url, { fetchImpl, timeoutMs: config.ingest.timeoutMs, userAgent: config.ingest.userAgent });
+ for (const raw of source.parse(json)) {
+ const row = sources.toRow(raw, { sourceId: source.id, fetchedAt, idOf: store.idOf });
+ // Two pages can carry the same listing; the later one wins, exactly as the board would show it.
+ if (row) seen.set(row.source_id, row);
+ }
+ }
+ const rows = [...seen.values()];
+ const out = await store.upsertListings(s, rows, { sourceId: source.id, fetchedAt });
+ listings = out.seen;
+ await store.finishFetch(s, source.id, { ok: true, listings, error: null, finishedAt: s.iso() });
+ log.log(`[Ingest] ${source.name}: ${listings} listings (${out.added} new, ${out.updated} updated)`);
+ return { ok: true, listings, added: out.added, updated: out.updated, rows };
+ } catch (err) {
+ const detail = (err && err.message) || String(err);
+ await store.finishFetch(s, source.id, { ok: false, listings, error: detail, finishedAt: s.iso() });
+ log.warn(`[Ingest] ${source.name} failed: ${detail}`);
+ return { ok: false, error: detail, listings };
+ }
+ }
+
+ /** Every source, then the expiry sweep: a listing no board has shown for the retention window goes. */
+ async function runAll() {
+ if (running) return [];
+ running = true;
+ try {
+ const out = [];
+ for (const source of sources.SOURCES) out.push({ source: source.id, ...(await runSource(source.id)) });
+ try {
+ const gone = await store.expire(s, config.ingest.retentionDays, s.now());
+ if (gone) log.log(`[Ingest] expired ${gone} listings not seen for ${config.ingest.retentionDays} days`);
+ } catch (err) {
+ log.warn(`[Ingest] expiry failed: ${(err && err.message) || err}`);
+ }
+ return out;
+ } finally { running = false; }
+ }
+
+ /**
+ * Start the timers: one per source, so a slow board delays only itself. The first run waits a few seconds so it
+ * never competes with the process starting up or with a deploy's readiness check.
+ */
+ function start() {
+ if (!config.ingest.enabled || timers.size) return false;
+ for (const source of sources.SOURCES) {
+ const timer = setInterval(() => { runSource(source.id).catch(() => {}); }, config.ingest.intervalMs);
+ timer.unref();
+ timers.set(source.id, timer);
+ }
+ const kick = setTimeout(() => { runAll().catch(() => {}); }, START_DELAY_MS);
+ kick.unref();
+ timers.set('start', kick);
+ log.log(`[Ingest] on: ${sources.ids().join(', ')} every ${Math.round(config.ingest.intervalMs / 3_600_000)}h`);
+ return true;
+ }
+
+ function stop() {
+ for (const timer of timers.values()) clearInterval(timer);
+ for (const timer of timers.values()) clearTimeout(timer);
+ timers.clear();
+ }
+
+ return { start, stop, runSource, runAll, timers };
+}
+
+module.exports = { createIngest };
new file mode 100644
new file mode 100644
@@ -0,0 +1,191 @@
@@ -0,0 +1,191 @@
+'use strict';
+
+/**
+ * The job boards OpenVibe.Work gathers from, one entry each: where it fetches, the terms it must apply, and how its
+ * payload becomes a listing. Nothing here is invented: a field a board does not give stays empty, and every listing
+ * keeps the board's own id and the original URL to link back to.
+ *
+ * Adding a source is adding an entry here and nothing else — the ingest, the search and the pages read this list.
+ * Each entry is:
+ *
+ * id the short name stored on every listing, and shown to the reader
+ * name the board's name, as it wants to be attributed
+ * homepage the board itself
+ * terms its terms of use for the API, in the board's own terms, as applied here
+ * attribution the line every page that shows one of its listings must carry
+ * requests(config) the URLs to fetch, in order
+ * parse(json) the payload → normalized listings (see normalize below)
+ * remoteDefault what a listing from this board is when the board says nothing
+ *
+ * A normalized listing is:
+ * { source_id, url, title, company, location, remote, tags[], job_type, salary, posted_at, description_html }
+ */
+
+const { excerpt } = require('./text');
+
+const clip = (s, n) => String(s == null ? '' : s).replace(/\s+/g, ' ').trim().slice(0, n);
+const list = (v, n = 12) => (Array.isArray(v) ? v : [v]).filter((x) => x != null && String(x).trim()).map((x) => clip(x, 60)).slice(0, n);
+
+/** An ISO 8601 UTC instant, or null. Boards mix Unix seconds, Unix milliseconds and date strings. */
+function instant(v) {
+ if (v == null || v === '') return null;
+ if (typeof v === 'number' || /^\d+$/.test(String(v))) {
+ const n = Number(v);
+ // Seconds, milliseconds or microseconds since the epoch — whichever lands in a sane range.
+ const ms = n > 1e14 ? n / 1000 : n > 1e11 ? n : n * 1000;
+ const d = new Date(ms);
+ return Number.isNaN(d.getTime()) ? null : d.toISOString();
+ }
+ const d = new Date(String(v));
+ return Number.isNaN(d.getTime()) ? null : d.toISOString();
+}
+
+/**
+ * The listing's URL, or null. Only http(s) is kept: anything else (a javascript: URL, a data: URL) is not a link
+ * back to a listing and is dropped with the listing rather than shown.
+ */
+function httpUrl(v) {
+ try {
+ const u = new URL(String(v));
+ return u.protocol === 'http:' || u.protocol === 'https:' ? u.toString().slice(0, 2000) : null;
+ } catch { return null; }
+}
+
+const money = (n) => (n == null || n === '' || !Number.isFinite(Number(n)) || Number(n) <= 0 ? null : Math.round(Number(n)));
+
+const SOURCES = [
+ {
+ id: 'arbeitnow',
+ name: 'Arbeitnow',
+ homepage: 'https://www.arbeitnow.com/',
+ terms: 'Arbeitnow publishes its job board API free of charge. We attribute Arbeitnow by name and link every listing back to its page on arbeitnow.com.',
+ attribution: 'Listing from Arbeitnow',
+ remoteDefault: false,
+ requests(config) {
+ const pages = [];
+ for (let page = 1; page <= config.ingest.pages; page++) {
+ const u = new URL(config.sources.arbeitnow.url);
+ u.searchParams.set('page', String(page));
+ pages.push(u.toString());
+ }
+ return pages;
+ },
+ // { data: [ { slug, company_name, title, description (HTML), remote, url, tags[], job_types[], location, created_at } ] }
+ parse(json) {
+ const rows = (json && Array.isArray(json.data)) ? json.data : [];
+ return rows.map((r) => ({
+ source_id: clip(r.slug || r.url, 200),
+ url: httpUrl(r.url),
+ title: clip(r.title, 300),
+ company: clip(r.company_name, 200),
+ location: clip(r.location, 200),
+ remote: r.remote === true,
+ tags: list(r.tags),
+ job_type: list(r.job_types).join(', ') || null,
+ salary: null, // this board gives no salary
+ posted_at: instant(r.created_at),
+ description_html: r.description,
+ }));
+ },
+ },
+ {
+ id: 'remotive',
+ name: 'Remotive',
+ homepage: 'https://remotive.com/',
+ terms: 'To use the Remotive API you must mention Remotive as the source and link to the listing. The API is read at most about four times a day, which is what the six-hour ingest interval is.',
+ attribution: 'Listing from Remotive',
+ remoteDefault: true, // every Remotive listing is remote by definition
+ requests(config) {
+ const u = new URL(config.sources.remotive.url);
+ u.searchParams.set('limit', String(config.ingest.limit));
+ return [u.toString()];
+ },
+ // { jobs: [ { id, url, title, company_name, category, tags[], job_type, publication_date, candidate_required_location, salary, description (HTML) } ] }
+ parse(json) {
+ const rows = (json && Array.isArray(json.jobs)) ? json.jobs : [];
+ return rows.map((j) => ({
+ source_id: clip(j.id, 200),
+ url: httpUrl(j.url),
+ title: clip(j.title, 300),
+ company: clip(j.company_name, 200),
+ location: clip(j.candidate_required_location, 200),
+ remote: true,
+ // Remotive labels a job with both tags and one category; a listing row has one labels column, so both
+ // are kept there, in the board's own words rather than our reading of them.
+ tags: list([].concat(j.tags || [], j.category || [])),
+ job_type: clip(j.job_type, 80) || null,
+ salary: clip(j.salary, 200) || null,
+ posted_at: instant(j.publication_date),
+ description_html: j.description,
+ }));
+ },
+ },
+ {
+ id: 'remoteok',
+ name: 'RemoteOK',
+ homepage: 'https://remoteok.com/',
+ terms: 'RemoteOK asks that a use of its API names RemoteOK as the source and links back to the listing. The response\'s first element is a legal notice, not a listing, and is skipped.',
+ attribution: 'Listing from RemoteOK',
+ remoteDefault: true,
+ requests(config) { return [config.sources.remoteok.url]; },
+ // The response is an array whose first element is the legal notice; every later element is a listing.
+ parse(json) {
+ const all = Array.isArray(json) ? json : [];
+ const rows = all.filter((r) => r && (r.id != null || r.slug) && r.position);
+ return rows.map((r) => {
+ const min = money(r.salary_min);
+ const max = money(r.salary_max);
+ return {
+ source_id: clip(r.id != null ? r.id : r.slug, 200),
+ url: httpUrl(r.url || (r.slug ? `https://remoteok.com/remote-jobs/${r.slug}` : null)),
+ title: clip(r.position, 300),
+ company: clip(r.company, 200),
+ location: clip(r.location, 200),
+ remote: true,
+ tags: list(r.tags),
+ job_type: clip(r.job_type || r.type, 80) || null,
+ salary: min && max ? `$${min.toLocaleString('en-US')}–$${max.toLocaleString('en-US')} a year` : null,
+ posted_at: instant(r.date),
+ description_html: r.description,
+ };
+ });
+ },
+ },
+];
+
+const byId = (id) => SOURCES.find((s) => s.id === id) || null;
+const ids = () => SOURCES.map((s) => s.id);
+
+/**
+ * A parsed row → the columns of work_listings, or null when it is not a listing we may keep. What we refuse: a row
+ * with no id, no title, no company or no http(s) URL to link back to — without those a listing has no provenance and
+ * must not be published. The excerpt is what is stored; the board keeps the full description.
+ */
+function toRow(raw, { sourceId, fetchedAt, idOf }) {
+ // The URL is checked again here even though each parse does it: this is the last gate before anything is stored,
+ // and a listing without an http(s) link back to its board has no provenance and is not published.
+ const url = raw && httpUrl(raw.url);
+ if (!raw || !raw.source_id || !raw.title || !raw.company || !url) return null;
+ const tags = list(raw.tags);
+ const row = {
+ id: idOf(sourceId, raw.source_id),
+ source: sourceId,
+ source_id: String(raw.source_id),
+ url,
+ title: String(raw.title),
+ company: String(raw.company),
+ location: raw.location || null,
+ remote: !!raw.remote,
+ tags: tags.join(', '),
+ job_type: raw.job_type || null,
+ salary: raw.salary || null,
+ excerpt: excerpt(raw.description_html).slice(0, 400),
+ posted_at: raw.posted_at || null,
+ sort_at: raw.posted_at || fetchedAt,
+ fetched_at: fetchedAt,
+ };
+ row.search_text = [row.title, row.company, row.tags, row.location, row.excerpt, row.salary].filter(Boolean).join(' ').toLowerCase().slice(0, 4000);
+ return row;
+}
+
+module.exports = { SOURCES, byId, ids, toRow };
new file mode 100644
new file mode 100644
@@ -0,0 +1,290 @@
@@ -0,0 +1,290 @@
+'use strict';
+
+/**
+ * The listings, the searches a person saved and what each source's last fetch did (migrations/0002_listings.sql).
+ * Every function takes the store (server/db.js createStore) first, as everywhere else in this service.
+ *
+ * A listing is never restated: the row holds the board's own fields, a short excerpt and the original URL. Search is
+ * a plain case-insensitive substring over one lowercased column — a job board of this size does not need a text
+ * index, and the LIKE pattern is a bound parameter, never built from what a caller typed.
+ */
+const crypto = require('crypto');
+
+const PAGE_SIZE = 30;
+const MAX_PAGE = 100;
+const TYPE_FACETS = 12;
+
+// ── ids ─────────────────────────────────────────────────────
+// A listing's id is derived from (source, source_id), so the same listing keeps the same URL on this site however
+// often the board is re-read, and two sources can never collide. 26 base32 characters, the shape of a ULID.
+const B32 = '0123456789abcdefghjkmnpqrstvwxyz';
+function idOf(source, sourceId) {
+ const digest = crypto.createHash('sha256').update(`${source}:${sourceId}`).digest();
+ let bits = 0;
+ let value = 0;
+ let out = '';
+ for (const byte of digest) {
+ value = (value << 8) | byte;
+ bits += 8;
+ while (bits >= 5 && out.length < 26) {
+ out += B32[(value >>> (bits - 5)) & 31];
+ bits -= 5;
+ }
+ if (out.length >= 26) break;
+ }
+ return `job_${out}`;
+}
+const LISTING_ID = /^job_[0-9abcdefghjkmnpqrstvwxyz]{26}$/;
+
+// ── search ──────────────────────────────────────────────────
+const lower = (s) => String(s == null ? '' : s).trim().toLowerCase().slice(0, 120);
+/** A LIKE pattern that matches the term literally: % and _ a caller typed are characters, not wildcards. */
+const like = (term) => `%${term.replace(/[\\%_]/g, '\\$&')}%`;
+
+/** The four filters a search takes, from anything (a query string, an API call, a saved search row). */
+function filtersOf(input = {}) {
+ const remote = String((input.remote == null ? '' : input.remote)).toLowerCase();
+ return {
+ // A search comes from a query string (`q`), an API body or a saved search row (`query`).
+ q: lower(input.q == null ? input.query : input.q),
+ remote: remote === 'true' || remote === 'yes' || remote === '1' ? 'true' : remote === 'false' || remote === 'no' || remote === '0' ? 'false' : '',
+ type: lower(input.type),
+ location: lower(input.location),
+ };
+}
+
+const num = (v) => (v == null ? 0 : Number(v));
+const parseTags = (s) => String(s || '').split(',').map((t) => t.trim()).filter(Boolean);
+
+/** A row → the listing as the pages and the API see it. `url` is always the board's own. */
+function toWire(row, { source } = {}) {
+ return {
+ id: row.id,
+ source: row.source,
+ source_name: (source && source.name) || row.source,
+ source_url: (source && source.homepage) || null,
+ url: row.url,
+ title: row.title,
+ company: row.company,
+ location: row.location || null,
+ remote: !!row.remote,
+ tags: parseTags(row.tags),
+ job_type: row.job_type || null,
+ salary: row.salary || null,
+ excerpt: row.excerpt,
+ posted_at: row.posted_at || null,
+ fetched_at: row.fetched_at,
+ first_seen_at: row.first_seen_at,
+ attribution: (source && source.attribution) || null,
+ };
+}
+
+// The filters a caller set, as bound parameters — never string-built. Everything is a LIKE on a lowercased column,
+// and the term's own % and _ are literal characters.
+const PARAM = {
+ q: (n) => `($${n} = '' OR search_text LIKE $${n} ESCAPE '\\')`,
+ remote: (n) => `($${n} = '' OR remote = ($${n} = 'true'))`,
+ type: (n) => `($${n} = '' OR lower(coalesce(job_type, '')) LIKE $${n} ESCAPE '\\')`,
+ location: (n) => `($${n} = '' OR lower(coalesce(location, '')) LIKE $${n} ESCAPE '\\')`,
+};
+const valueOf = { q: (f) => (f.q ? like(f.q) : ''), remote: (f) => f.remote, type: (f) => (f.type ? like(f.type) : ''), location: (f) => (f.location ? like(f.location) : '') };
+
+/**
+ * One page of listings, newest first, plus the facets for the current scope. The cursor is the position of the last
+ * row (sort_at, id); a page is the rows strictly after it, so a listing that arrives mid-scroll does not shift it.
+ */
+async function search(s, filters, { limit = PAGE_SIZE, cursor = null } = {}) {
+ const f = filtersOf(filters);
+ const n = Math.min(MAX_PAGE, Math.max(1, Number(limit) || PAGE_SIZE));
+ const args = [];
+ const where = ['expired = false'];
+ for (const name of ['q', 'remote', 'type', 'location']) {
+ args.push(valueOf[name](f));
+ where.push(PARAM[name](args.length));
+ }
+ const scope = where.join(' AND ');
+ // How many listings the reader's filters actually match, counted exactly rather than inferred from the page.
+ const matches = Number(await s.db.value(`SELECT count(*) AS n FROM work_listings WHERE ${scope}`, args));
+ const at = cursor && cursor.sort_at && cursor.id ? cursor : null;
+ if (at) {
+ args.push(at.sort_at, at.id);
+ where.push(`(sort_at, id) < ($${args.length - 1}, $${args.length})`);
+ }
+ const rows = await s.db.many(
+ `SELECT * FROM work_listings WHERE ${where.join(' AND ')} ORDER BY sort_at DESC, id DESC LIMIT ${n + 1}`, args);
+ const pageRows = rows.slice(0, n);
+ const last = pageRows[pageRows.length - 1];
+ return {
+ rows: pageRows,
+ next: rows.length > n && last ? { sort_at: last.sort_at, id: last.id } : null,
+ facets: { matches, ...(await facets(s, f)) },
+ };
+}
+
+/**
+ * The counts the filter bar shows: how many are remote and not, and the job types present, for the text and location
+ * in force. The remote and type filters are deliberately left out of their own facet — applied, they would show only
+ * the answer the reader already chose.
+ */
+async function facets(s, f) {
+ const args = [valueOf.q(f), valueOf.location(f)];
+ const scope = `expired = false AND ${PARAM.q(1)} AND ${PARAM.location(2)}`;
+ const remoteRows = await s.db.many(`SELECT remote, count(*) AS n FROM work_listings WHERE ${scope} GROUP BY remote`, args);
+ const typeRows = await s.db.many(
+ `SELECT lower(job_type) AS t, count(*) AS n FROM work_listings
+ WHERE ${scope} AND coalesce(job_type, '') <> ''
+ GROUP BY lower(job_type) ORDER BY n DESC, t ASC LIMIT ${TYPE_FACETS}`, args);
+ const remote = { true: 0, false: 0 };
+ for (const r of remoteRows) remote[r.remote ? 'true' : 'false'] = num(r.n);
+ // `scope` counts the words and the location only — the population each facet below divides up.
+ return { remote, type: typeRows.map((r) => ({ type: r.t, count: num(r.n) })), scope: remote.true + remote.false };
+}
+
+const getListing = (s, id) => (LISTING_ID.test(String(id)) ? s.db.maybe('SELECT * FROM work_listings WHERE id = $1 AND expired = false', [id]) : Promise.resolve(null));
+
+/** How many live listings each source has, and when one was last seen — the home page's provenance line. */
+async function countsBySource(s) {
+ const rows = await s.db.many(
+ `SELECT source, count(*) AS n, max(last_seen_at) AS last_seen_at
+ FROM work_listings WHERE expired = false GROUP BY source`);
+ const out = new Map(rows.map((r) => [r.source, { count: num(r.n), last_seen_at: r.last_seen_at }]));
+ return out;
+}
+
+/** The newest live listings; `remote: true` is the remote-only selection the home page shows. */
+const listRecent = (s, limit = 5, { remote = false } = {}) => s.db.many(
+ `SELECT * FROM work_listings WHERE expired = false${remote ? ' AND remote = true' : ''} ORDER BY sort_at DESC, id DESC LIMIT $1`,
+ [Math.min(50, Math.max(1, limit))]);
+
+/** The ids of the newest live listings, for the sitemap: its own cap, so a big board cannot bloat the file. */
+const recentIds = (s, limit = 1000) => s.db.many(
+ `SELECT id FROM work_listings WHERE expired = false ORDER BY sort_at DESC, id DESC LIMIT $1`, [Math.min(1000, Math.max(1, limit))]);
+
+/** The listings first seen after a saved search's last look — what "new since you last looked" counts. */
+function newCountSql() {
+ return `SELECT count(*) AS n FROM work_listings
+ WHERE expired = false AND first_seen_at > $1
+ AND ${PARAM.q(2)} AND ${PARAM.remote(3)} AND ${PARAM.type(4)} AND ${PARAM.location(5)}`;
+}
+
+// ── ingest writes ───────────────────────────────────────────
+/**
+ * Upsert one source's listings. A listing the board still shows is updated in place and its last_seen_at moves; one
+ * that reappears after expiring comes back (expired = false). Returns { seen, added, updated }.
+ */
+async function upsertListings(s, rows, { sourceId, fetchedAt }) {
+ let added = 0;
+ let updated = 0;
+ await s.tx(async () => {
+ for (const row of rows) {
+ const out = await s.db.maybe(
+ `INSERT INTO work_listings (id, source, source_id, url, title, company, location, remote, tags, job_type, salary, excerpt, search_text, posted_at, sort_at, fetched_at, first_seen_at, last_seen_at, expired)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $16, $16, false)
+ ON CONFLICT (source, source_id) DO UPDATE SET
+ url = EXCLUDED.url, title = EXCLUDED.title, company = EXCLUDED.company, location = EXCLUDED.location,
+ remote = EXCLUDED.remote, tags = EXCLUDED.tags, job_type = EXCLUDED.job_type, salary = EXCLUDED.salary,
+ excerpt = EXCLUDED.excerpt, search_text = EXCLUDED.search_text, posted_at = EXCLUDED.posted_at,
+ sort_at = EXCLUDED.sort_at, fetched_at = EXCLUDED.fetched_at, last_seen_at = EXCLUDED.last_seen_at,
+ expired = false
+ RETURNING (xmax = 0) AS inserted`,
+ [row.id, row.source, row.source_id, row.url, row.title, row.company, row.location, row.remote, row.tags, row.job_type, row.salary, row.excerpt, row.search_text, row.posted_at, row.sort_at, fetchedAt]);
+ if (out && out.inserted) added++; else updated++;
+ }
+ });
+ return { seen: rows.length, added, updated };
+}
+
+/** A source's fetch record: started now, finished when it finished, with what happened. */
+const beginFetch = (s, sourceId, startedAt) => s.db.query(
+ `INSERT INTO work_source_fetches (source, started_at, finished_at, ok, listings, error) VALUES ($1, $2, NULL, false, 0, NULL)
+ ON CONFLICT (source) DO UPDATE SET started_at = EXCLUDED.started_at, finished_at = NULL, ok = false, listings = 0, error = NULL`,
+ [sourceId, startedAt]);
+
+const finishFetch = (s, sourceId, { ok, listings, error, finishedAt }) => s.db.query(
+ 'UPDATE work_source_fetches SET finished_at = $2, ok = $3, listings = $4, error = $5 WHERE source = $1',
+ [sourceId, finishedAt, ok, listings, error ? String(error).slice(0, 300) : null]);
+
+async function sourceFetches(s) {
+ const rows = await s.db.many('SELECT * FROM work_source_fetches');
+ return new Map(rows.map((r) => [r.source, r]));
+}
+
+/**
+ * Listings the boards stopped showing go. A listing not seen for `days` is expired rather than deleted: the row keeps
+ * its history, the search and the pages stop offering it, and a board that shows it again brings it back.
+ */
+async function expire(s, days, now) {
+ const cutoff = new Date(now - days * 86_400_000).toISOString();
+ return s.db.exec('UPDATE work_listings SET expired = true WHERE expired = false AND last_seen_at < $1', [cutoff]);
+}
+
+// ── saved searches ──────────────────────────────────────────
+const SAVED_ID = /^ssc_[0-9A-HJKMNP-TV-Z]{26}$/;
+const MAX_SAVED = 50;
+
+const listSaved = (s, subject) => s.db.many('SELECT * FROM work_saved_searches WHERE subject = $1 ORDER BY created_at DESC, id DESC', [subject]);
+
+function toSavedWire(row, { newCount = 0 } = {}) {
+ const query = queryOf(row);
+ return {
+ id: row.id,
+ query: row.query || '',
+ filters: { remote: row.remote || '', type: row.type || '', location: row.location || '' },
+ created_at: row.created_at,
+ last_seen_at: row.last_seen_at,
+ new_count: Number(newCount) || 0,
+ // Where to run it again: the person's own search, reproducible from the saved row alone.
+ search_path: `/jobs${query ? `?${query}` : ''}`,
+ };
+}
+
+async function createSaved(s, subject, filters, { id, now }) {
+ const f = filtersOf(filters);
+ const count = await s.db.value('SELECT count(*) AS n FROM work_saved_searches WHERE subject = $1', [subject]);
+ if (Number(count) >= MAX_SAVED) return { error: 'too_many' };
+ const row = await s.db.maybe(
+ `INSERT INTO work_saved_searches (id, subject, query, remote, type, location, created_at, last_seen_at)
+ VALUES ($1, $2, $3, $4, $5, $6, $7, $7) RETURNING *`,
+ [id, subject, f.q, f.remote, f.type, f.location, now]);
+ return { row };
+}
+
+const getSaved = (s, subject, id) => (SAVED_ID.test(String(id)) ? s.db.maybe('SELECT * FROM work_saved_searches WHERE id = $1 AND subject = $2', [id, subject]) : Promise.resolve(null));
+const deleteSaved = (s, subject, id) => (SAVED_ID.test(String(id)) ? s.db.exec('DELETE FROM work_saved_searches WHERE id = $1 AND subject = $2', [id, subject]) : Promise.resolve(0));
+
+/** The count of listings that first appeared after the saved search was last looked at. */
+async function newCount(s, row) {
+ const f = filtersOf(row);
+ return Number(await s.db.value(newCountSql(), [row.last_seen_at, valueOf.q(f), valueOf.remote(f), valueOf.type(f), valueOf.location(f)]));
+}
+
+/** Mark a saved search as looked at (the moment its count stops being "new"). */
+const touchSaved = (s, subject, id, at) => s.db.query('UPDATE work_saved_searches SET last_seen_at = $3 WHERE id = $1 AND subject = $2', [id, subject, at]);
+
+/** The query string a saved search runs, for the "run it again" link and the API's search_url. */
+function queryOf(row) {
+ const params = new URLSearchParams();
+ if (row.query) params.set('q', row.query);
+ if (row.remote) params.set('remote', row.remote);
+ if (row.type) params.set('type', row.type);
+ if (row.location) params.set('location', row.location);
+ return params.toString();
+}
+
+// ── the cursor ──────────────────────────────────────────────
+const encodeCursor = (c) => (c ? Buffer.from(JSON.stringify([c.sort_at, c.id]), 'utf8').toString('base64url') : null);
+function decodeCursor(v) {
+ if (typeof v !== 'string' || !v || v.length > 200) return null;
+ try {
+ const [at, id] = JSON.parse(Buffer.from(v, 'base64url').toString('utf8'));
+ // Both parts go into a SQL comparison; only the two shapes this service writes are accepted.
+ if (typeof at !== 'string' || !/^\d{4}-\d{2}-\d{2}T[\d:.]+Z$/.test(at) || !LISTING_ID.test(String(id))) return null;
+ return { sort_at: at, id: String(id) };
+ } catch { return null; }
+}
+
+module.exports = {
+ search, facets, getListing, countsBySource, listRecent, recentIds, upsertListings, beginFetch, finishFetch, sourceFetches,
+ expire, listSaved, createSaved, getSaved, deleteSaved, newCount, touchSaved, toWire, toSavedWire, filtersOf,
+ idOf, encodeCursor, decodeCursor, PAGE_SIZE, MAX_PAGE, MAX_SAVED, LISTING_ID,
+};
new file mode 100644
new file mode 100644
@@ -0,0 +1,53 @@
@@ -0,0 +1,53 @@
+'use strict';
+
+/**
+ * The boards send a listing's description as HTML. We never republish it: we store a short plain-text excerpt and
+ * link back to the listing for the rest. These are the two steps that make that safe — strip the markup, then cut on
+ * a word boundary — and they are deliberately conservative: script and style content goes with the tags, and anything
+ * that still looks like markup after the strip is removed rather than shown.
+ */
+
+const EXCERPT_CHARS = 300;
+
+const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', '#39': "'", '#34': '"', '#38': '&', '#60': '<', '#62': '>' };
+
+function decode(text) {
+ return String(text).replace(/&(#x?[0-9a-f]+|[a-z]+);/gi, (m, body) => {
+ const key = body.toLowerCase();
+ if (Object.prototype.hasOwnProperty.call(ENTITIES, key)) return ENTITIES[key];
+ if (key[0] === '#') {
+ const code = key[1] === 'x' ? parseInt(key.slice(2), 16) : parseInt(key.slice(1), 10);
+ // Only real, printable code points: a control character or a surrogate is not text anyone wrote.
+ if (Number.isFinite(code) && code >= 32 && code <= 0x10ffff && !(code >= 0xd800 && code <= 0xdfff)) {
+ try { return String.fromCodePoint(code); } catch { return ' '; }
+ }
+ }
+ return ' ';
+ });
+}
+
+/** The description as plain text: no tags, no script or style content, whitespace collapsed. */
+function stripHtml(html) {
+ let s = String(html == null ? '' : html);
+ s = s.replace(/<!--[\s\S]*?-->/g, ' ');
+ // A script or style element's content is code, not prose; drop it with its tags.
+ s = s.replace(/<(script|style|noscript|template|svg)\b[^>]*>[\s\S]*?<\/\1\s*>/gi, ' ');
+ // Block boundaries become a space so words either side do not run together.
+ s = s.replace(/<\/?(br|p|div|li|ul|ol|tr|td|th|h[1-6]|section|article|header|footer|blockquote|pre|figure|hr)\b[^>]*>/gi, ' ');
+ s = s.replace(/<[^>]*>/g, ' ');
+ s = decode(s);
+ // An unmatched '<' left by broken markup is not something a listing should show.
+ s = s.replace(/<[^>]*$/g, ' ').replace(/[<>]/g, ' ');
+ return s.replace(/\s+/g, ' ').trim();
+}
+
+/** The excerpt: the first ~300 characters of the plain text, cut on a word, with a trailing ellipsis. */
+function excerpt(html, max = EXCERPT_CHARS) {
+ const text = stripHtml(html);
+ if (text.length <= max) return text;
+ const cut = text.slice(0, max);
+ const at = cut.lastIndexOf(' ');
+ return `${(at > max * 0.6 ? cut.slice(0, at) : cut).replace(/[\s,;:.\-–—]+$/, '')}…`;
+}
+
+module.exports = { stripHtml, excerpt, EXCERPT_CHARS };
@@ -24,6 +24,9 @@ const TAGLINE = 'Find work, with an agent on your side.';
@@ -24,6 +24,9 @@ const TAGLINE = 'Find work, with an agent on your side.';
const PUBLIC_DIR = path.join(__dirname, '..', '..', 'public');
const PUBLIC_DIR = path.join(__dirname, '..', '..', 'public');
// The product's own navigation: one entry per public page it serves.
// The product's own navigation: one entry per public page it serves.
const NAV = [
const NAV = [
+ { label: 'Jobs', href: '/jobs' },
+ { label: 'Sources', href: '/sources' },
+ { label: 'API', href: '/docs' },
{ label: 'What shipped', href: '/updates' },
{ label: 'What shipped', href: '/updates' },
];
];
new file mode 100644
new file mode 100644
@@ -0,0 +1,96 @@
@@ -0,0 +1,96 @@
+'use strict';
+
+/**
+ * How a listing is shown, in one place so the list and the detail page cannot drift apart: the fields the board
+ * gave, the short excerpt, and the provenance every listing carries — the source named and the original linked.
+ *
+ * Everything dynamic goes through the `html` tagged template, so a board's text (a title with <script> in it, a
+ * company name with quotes) is escaped where it is shown, and a URL is only ever a link when it is the http(s) URL
+ * the ingest kept.
+ */
+const { html, badge } = require('./html');
+const { stripHtml } = require('../jobs/text');
+
+const MONTHS = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'];
+const DAYS = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];
+
+/** A date out of an ISO instant, in plain words; '' when there is none. */
+function dateWords(iso) {
+ const d = iso ? new Date(iso) : null;
+ if (!d || Number.isNaN(d.getTime())) return '';
+ return `${d.getUTCDate()} ${MONTHS[d.getUTCMonth()]} ${d.getUTCFullYear()}`;
+}
+
+const meta = (row) => ({
+ id: row.id,
+ url: row.url,
+ title: row.title,
+ company: row.company,
+ location: row.location || '',
+ remote: !!row.remote,
+ tags: String(row.tags || '').split(',').map((t) => t.trim()).filter(Boolean),
+ job_type: row.job_type || '',
+ salary: row.salary || '',
+ excerpt: row.excerpt,
+ posted_at: row.posted_at || null,
+ source: row.source,
+});
+
+/** The source's name, linked to the original listing: the attribution its terms require, in one element. */
+function sourceLink(row, source) {
+ const name = (source && source.name) || row.source;
+ return html`<a class="job-source" href="${row.url}" rel="nofollow noopener external" target="_blank">${name}</a>`;
+}
+
+/** The one-line provenance: where it came from, linked back, and when we read that. */
+function provenance(row, source) {
+ return html`<p class="job-provenance"><span class="muted">Listing from</span> ${sourceLink(row, source)}${row.posted_at ? html` <span class="muted">· posted ${dateWords(row.posted_at)}</span>` : ''}</p>`;
+}
+
+/** One listing in a list: title, who and where, the excerpt, the tags, and the link back to the board. */
+function listingItem(row, { source }) {
+ const l = meta(row);
+ return html`<li class="job-item" id="${l.id}">
+<h3 class="job-title"><a href="/jobs/${l.id}">${l.title}</a></h3>
+<p class="job-meta"><span class="job-company">${l.company}</span> · <span class="job-location">${l.location || 'Location not given'}</span>${l.remote ? html` · ${badge('Remote', 'ok')}` : ''}${l.job_type ? html` · <span class="job-type">${l.job_type}</span>` : ''}${l.salary ? html` · <span class="job-salary">${l.salary}</span>` : ''}</p>
+<p class="job-excerpt">${l.excerpt}</p>
+${l.tags.length ? html`<ul class="job-tags">${l.tags.slice(0, 8).map((t) => html`<li>${t}</li>`)}</ul>` : ''}
+${provenance(row, source)}
+</li>`;
+}
+
+/** A list of listings, or a line saying there is nothing to show. */
+function listingList(rows, { sourceOf, empty = 'No listings match this search.' } = {}) {
+ if (!rows.length) return html`<p class="muted">${empty}</p>`;
+ return html`<ul class="job-list">${rows.map((row) => listingItem(row, { source: sourceOf(row) }))}</ul>`;
+}
+
+/**
+ * schema.org JobPosting for one listing, from the fields the board gave and nothing else: the url is the original
+ * listing (never this page), and the description is the excerpt we hold — the full text stays on the board.
+ */
+function jobPosting(row, { siteUrl }) {
+ const l = meta(row);
+ const node = {
+ '@context': 'https://schema.org',
+ '@type': 'JobPosting',
+ title: l.title,
+ description: l.excerpt,
+ url: l.url,
+ identifier: { '@type': 'PropertyValue', name: l.source, value: row.source_id || l.id },
+ hiringOrganization: { '@type': 'Organization', name: l.company },
+ directApply: false,
+ publisher: { '@type': 'Organization', name: 'OpenVibe.Work', url: siteUrl },
+ };
+ if (l.posted_at) node.datePosted = l.posted_at;
+ if (l.job_type) node.employmentType = l.job_type;
+ if (l.remote) node.jobLocationType = 'TELECOMMUTE';
+ if (l.location) node.jobLocation = { '@type': 'Place', address: { '@type': 'PostalAddress', addressLocality: l.location } };
+ if (l.salary) node.baseSalary = { '@type': 'MonetaryAmount', description: l.salary };
+ return node;
+}
+
+/** The excerpt as plain text, for the meta description and the API. */
+const plainExcerpt = (row) => stripHtml(row.excerpt || '');
+
+module.exports = { listingItem, listingList, sourceLink, provenance, jobPosting, dateWords, plainExcerpt, meta };
@@ -20,8 +20,14 @@ const READ = 'work.api.read';
@@ -20,8 +20,14 @@ const READ = 'work.api.read';
const rosa = t.network.addUser('rosa');
const rosa = t.network.addUser('rosa');
const sam = t.network.addUser('sam');
const sam = t.network.addUser('sam');
- await check('BUDGETS starts empty: the product declares its own expensive routes', () => {
- assert.deepStrictEqual(BUDGETS, {});
+ await check('the product declares its own expensive routes in BUDGETS, and only those', () => {
+ // Searching the listings and saving a search: the two routes the product gives numbers of their own.
+ assert.deepStrictEqual(Object.keys(BUDGETS).sort(), ['work.jobs.search', 'work.saved_searches.write']);
+ for (const [name, own] of Object.entries(BUDGETS)) {
+ assert.ok(name.startsWith('work.'), name);
+ assert.ok(own.minute > 0 && own.hour >= own.minute, name);
+ }
+ assert.throws(() => t.ctx.limits.budget('work.thing.create'), /no budget named/);
});
});
await check('a read: 3 a minute per address, then 429 rate_limited with Retry-After; another address passes', async () => {
await check('a read: 3 a minute per address, then 429 rate_limited with Retry-After; another address passes', async () => {
@@ -55,7 +55,7 @@ const PRIVATE = ['/auth/', '/api/'];
@@ -55,7 +55,7 @@ const PRIVATE = ['/auth/', '/api/'];
for (const p of ['/auth/']) assert.ok(!r.text.includes(`https://openvibe.work${p}`), `private path in llms.txt: ${p}`);
for (const p of ['/auth/']) assert.ok(!r.text.includes(`https://openvibe.work${p}`), `private path in llms.txt: ${p}`);
});
});
- await check('llms-full.txt: 200 text/plain, the site title, and every page the sitemap lists', async () => {
+ await check('llms-full.txt: 200 text/plain, the site title, and every page the sitemap lists that is the site\'s own', async () => {
const r = await t.get('/llms-full.txt');
const r = await t.get('/llms-full.txt');
assert.strictEqual(r.status, 200);
assert.strictEqual(r.status, 200);
assert.match(r.headers.get('content-type'), /^text\/plain/);
assert.match(r.headers.get('content-type'), /^text\/plain/);
@@ -64,8 +64,12 @@ const PRIVATE = ['/auth/', '/api/'];
@@ -64,8 +64,12 @@ const PRIVATE = ['/auth/', '/api/'];
const sitemap = await t.get('/sitemap.xml');
const sitemap = await t.get('/sitemap.xml');
const locs = [...sitemap.text.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
const locs = [...sitemap.text.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
assert.ok(locs.length >= 2, `sitemap too small: ${locs.length}`);
assert.ok(locs.length >= 2, `sitemap too small: ${locs.length}`);
- for (const loc of locs) assert.ok(r.text.includes(loc), `llms-full.txt is missing ${loc}`);
+ // The site's own pages are described in full. A listing URL is in the sitemap (so a crawler follows it to
+ // the board) but deliberately not in llms-full.txt: there is no full text here to give, only an excerpt.
+ const own = locs.filter((l) => !/^https:\/\/openvibe\.work\/jobs\/job_/.test(l));
+ for (const loc of own) assert.ok(r.text.includes(loc), `llms-full.txt is missing ${loc}`);
assert.ok(r.text.includes('URL: https://openvibe.work/updates\n'), 'no /updates entry');
assert.ok(r.text.includes('URL: https://openvibe.work/updates\n'), 'no /updates entry');
+ assert.ok(r.text.includes('https://openvibe.work/jobs'), 'the listings are not named');
for (const p of PRIVATE) assert.ok(!r.text.includes(`https://openvibe.work${p}`), `private path in llms-full.txt: ${p}`);
for (const p of PRIVATE) assert.ok(!r.text.includes(`https://openvibe.work${p}`), `private path in llms-full.txt: ${p}`);
});
});
new file mode 100644
new file mode 100644
@@ -0,0 +1,179 @@
@@ -0,0 +1,179 @@
+'use strict';
+/**
+ * Stand-in job boards for the ingest tests: one HTTP server with a path per board, answering the shape each real API
+ * documents, with small fixtures written to look like the real thing (including RemoteOK's legal notice as the first
+ * array element, and an Arbeitnow listing whose title carries a <script> tag, so the pages' escaping is provable).
+ *
+ * const b = await startBoards();
+ * b.env WORK_ARBEITNOW_URL / WORK_REMOTIVE_URL / WORK_REMOTEOK_URL, for boot({ env })
+ * b.state.arbeitnow = 'fail' one board answering HTTP 500 (default 'ok')
+ * b.state.remotive = 'json' one board answering something that is not JSON
+ * b.add('arbeitnow', row) a listing the board starts showing (see b.fixtures for the shape)
+ * b.requests [{ path, query, userAgent }] for every call
+ *
+ * The fixtures are ours, not a board's: no real listing is copied here.
+ */
+const http = require('http');
+
+/** One listing per board, in that board's own payload shape. */
+function fixtures() {
+ return {
+ arbeitnow: [
+ {
+ slug: 'senior-backend-engineer-berlin',
+ company_name: 'Nordlicht Systems',
+ title: 'Senior Backend Engineer (Node.js)',
+ description: '<p>We run a fleet of Node.js services.</p><ul><li>PostgreSQL</li><li>Valkey</li></ul><p>You will own the billing pipeline end to end.</p>',
+ remote: false,
+ url: 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345',
+ tags: ['nodejs', 'postgresql'],
+ job_types: ['permanent'],
+ location: 'Berlin, Germany',
+ created_at: 1791500000,
+ },
+ {
+ slug: 'frontend-engineer-remote-eu',
+ company_name: 'Helles Werk',
+ title: 'Frontend Engineer <script>alert("xss")</script>',
+ description: '<div>Build the design system.<script>window.__pwned = true</script></div><div>TypeScript, CSS.</div>',
+ remote: true,
+ url: 'https://www.arbeitnow.com/view/frontend-engineer-remote-eu-23456',
+ tags: ['typescript', 'css'],
+ job_types: ['permanent', 'full_time'],
+ location: 'Remote (EU)',
+ created_at: 1791600000,
+ },
+ ],
+ arbeitnowPage2: [
+ {
+ slug: 'data-analyst-munich',
+ company_name: 'Alpen Daten',
+ title: 'Data Analyst',
+ description: '<p>SQL, dashboards and a lot of questions.</p>',
+ remote: false,
+ url: 'https://www.arbeitnow.com/view/data-analyst-munich-34567',
+ tags: ['sql'],
+ job_types: ['permanent'],
+ location: 'Munich, Germany',
+ created_at: 1791700000,
+ },
+ ],
+ remotive: [
+ {
+ id: 1900123,
+ url: 'https://remotive.com/remote-jobs/software-dev/support-engineer-1900123',
+ title: 'Support Engineer',
+ company_name: 'Cassiopeia',
+ category: 'Software Development',
+ tags: ['customer support', 'saas'],
+ job_type: 'full_time',
+ publication_date: '2026-09-30T09:12:00',
+ candidate_required_location: 'Europe',
+ salary: '€45,000 – €55,000',
+ description: '<p>Answer customers, write runbooks.</p><p>Postgres and a sharp eye.</p>',
+ },
+ {
+ id: 1900789,
+ url: 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789',
+ title: 'Platform Engineer',
+ company_name: 'Meridian Rail',
+ category: 'Software Development',
+ tags: ['kubernetes', 'terraform'],
+ job_type: 'full_time',
+ publication_date: '2026-09-29T07:45:00',
+ candidate_required_location: 'Remote (EU)',
+ salary: '€70,000 – €85,000',
+ // Longer than the excerpt we keep, so a test can prove the tail is never stored or shown.
+ description: `<p>You will run the platform other engineers build on.</p>
+<p>That means Kubernetes clusters, Terraform modules, a queue nobody has to think about, and an on-call rotation that
+people actually join voluntarily. We look after the boring parts of shipping software so product teams do not have to.</p>
+<p>You will work with people across four time zones, mostly in writing. We keep our runbooks honest and our postmortems
+blameless. The office dog is called Waffle.</p>`,
+ },
+ {
+ id: 1900456,
+ url: 'https://remotive.com/remote-jobs/marketing/content-marketer-1900456',
+ title: 'Content Marketer',
+ company_name: 'Longform Ltd',
+ category: 'Marketing',
+ tags: ['writing'],
+ job_type: 'contract',
+ publication_date: '2026-09-28T14:00:00',
+ candidate_required_location: 'Worldwide',
+ salary: '',
+ description: '<p>Write about developer tools.</p>',
+ },
+ ],
+ remoteok: [
+ { legal: "RemoteOK's API is offered for personal use; please link back to the listing and name RemoteOK as the source." },
+ {
+ id: '1049001',
+ slug: 'devops-engineer-skyward',
+ company: 'Skyward Cloud',
+ position: 'DevOps Engineer',
+ tags: ['devops', 'kubernetes'],
+ location: 'Worldwide',
+ salary_min: 90000,
+ salary_max: 130000,
+ date: '2026-10-01T08:00:00+00:00',
+ url: 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001',
+ description: '<p>Kubernetes, Terraform, on-call.</p>',
+ },
+ {
+ id: '1049002',
+ slug: 'technical-writer-atlas',
+ company: 'Atlas Docs',
+ position: 'Technical Writer',
+ tags: ['docs'],
+ location: 'EU',
+ salary_min: null,
+ salary_max: null,
+ date: '2026-09-25T08:00:00+00:00',
+ url: 'https://remoteok.com/remote-jobs/remote-technical-writer-atlas-1049002',
+ description: '<p>Document an API.</p>',
+ },
+ ],
+ };
+}
+
+async function startBoards() {
+ const data = fixtures();
+ const requests = [];
+ const state = { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' };
+ const added = { arbeitnow: [], remotive: [], remoteok: [] };
+
+ const send = (res, status, body, type = 'application/json') => { res.writeHead(status, { 'content-type': type }); res.end(typeof body === 'string' ? body : JSON.stringify(body)); };
+
+ const server = http.createServer((req, res) => {
+ const url = new URL(req.url, 'http://stand-in');
+ requests.push({ path: url.pathname, query: url.search, userAgent: req.headers['user-agent'] || '', accept: req.headers.accept || '' });
+ const board = url.pathname.replace(/^\//, '');
+ if (!['arbeitnow', 'remotive', 'remoteok'].includes(board)) return send(res, 404, { error: 'not a board' });
+ if (state[board] === 'fail') return send(res, 500, { error: 'the board is having a bad day' }, 'text/plain');
+ if (state[board] === 'html') return send(res, 200, '<html>not json</html>', 'text/html');
+ if (board === 'arbeitnow') {
+ const page = Number(url.searchParams.get('page') || 1);
+ const rows = page === 1 ? data.arbeitnow : page === 2 ? data.arbeitnowPage2 : [];
+ return send(res, 200, { data: [...rows, ...added.arbeitnow], links: { next: page < 2 ? `https://www.arbeitnow.com/api/job-board-api?page=${page + 1}` : null } });
+ }
+ if (board === 'remotive') return send(res, 200, { 'job-count': data.remotive.length + added.remotive.length, jobs: [...data.remotive, ...added.remotive] });
+ // RemoteOK: the legal notice is element 0, exactly as the real API sends it.
+ return send(res, 200, [...data.remoteok.slice(0, 1), ...data.remoteok.slice(1), ...added.remoteok]);
+ });
+ await new Promise((r) => server.listen(0, '127.0.0.1', r));
+ const base = `http://127.0.0.1:${server.address().port}`;
+
+ return {
+ url: base,
+ fixtures: data,
+ state,
+ requests,
+ env: { WORK_ARBEITNOW_URL: `${base}/arbeitnow`, WORK_REMOTIVE_URL: `${base}/remotive`, WORK_REMOTEOK_URL: `${base}/remoteok` },
+ /** Make a board start showing one more listing (the shape of that board's own payload). */
+ add(board, row) { added[board].push(row); },
+ reset() { Object.assign(state, { arbeitnow: 'ok', remotive: 'ok', remoteok: 'ok' }); requests.length = 0; },
+ close: () => new Promise((r) => server.close(r)),
+ };
+}
+
+module.exports = { startBoards, fixtures };
@@ -10,10 +10,15 @@ process.env.WORK_API_RATE_LIMIT_PER_MIN = process.env.WORK_API_RATE_LIMIT_PER_MI
@@ -10,10 +10,15 @@ process.env.WORK_API_RATE_LIMIT_PER_MIN = process.env.WORK_API_RATE_LIMIT_PER_MI
* const t = await boot();
* const t = await boot();
* t.get(path, { as: user, bearer, json, form, headers, method, cookie })
* t.get(path, { as: user, bearer, json, form, headers, method, cookie })
* t.signIn(user) → cookie header value (work_at = a Network user token, as after /auth/callback)
* t.signIn(user) → cookie header value (work_at = a Network user token, as after /auth/callback)
- * t.logs(), t.dbDump()
+ * t.boards (the stand-in job boards: state, add, requests), t.logs(), t.dbDump()
+ *
+ * The three job-board base URLs always point at the stand-in (test/helpers/boards.js), so an ingest run in a test
+ * never touches the internet. The ingest itself is off unless a test asks for it: it runs when a test calls
+ * t.ctx.ingest.runSource(…) or t.ctx.ingest.runAll().
*/
*/
const http = require('http');
const http = require('http');
const { startNetwork } = require('./mocks');
const { startNetwork } = require('./mocks');
+const { startBoards } = require('./boards');
const captured = [];
const captured = [];
for (const m of ['log', 'info', 'warn', 'error']) {
for (const m of ['log', 'info', 'warn', 'error']) {
@@ -23,10 +28,12 @@ for (const m of ['log', 'info', 'warn', 'error']) {
@@ -23,10 +28,12 @@ for (const m of ['log', 'info', 'warn', 'error']) {
async function boot(opts = {}) {
async function boot(opts = {}) {
const network = await startNetwork(opts.network || {});
const network = await startNetwork(opts.network || {});
+ const boards = await startBoards();
const env = {
const env = {
NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.work', TRUST_PROXY: '1',
NODE_ENV: 'test', PORT: '0', BASE_URL: 'https://openvibe.work', TRUST_PROXY: '1',
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
OV_NETWORK_URL: network.url, OV_NETWORK_INTERNAL_URL: network.url,
OV_OAUTH_CLIENT_ID: 'work', OV_OAUTH_CLIENT_SECRET: 'work-secret', COOKIE_SECURE: 'false',
OV_OAUTH_CLIENT_ID: 'work', OV_OAUTH_CLIENT_SECRET: 'work-secret', COOKIE_SECURE: 'false',
+ ...boards.env,
...(opts.env || {}),
...(opts.env || {}),
};
};
for (const [k, v] of Object.entries(opts.env || {})) if (v === null) delete env[k];
for (const [k, v] of Object.entries(opts.env || {})) if (v === null) delete env[k];
@@ -66,12 +73,14 @@ async function boot(opts = {}) {
@@ -66,12 +73,14 @@ async function boot(opts = {}) {
}
}
const t = {
const t = {
- base, network, config, ctx: built.ctx, get, signIn, dbDump,
+ base, network, boards, config, ctx: built.ctx, get, signIn, dbDump,
logs: () => captured.join('\n'),
logs: () => captured.join('\n'),
async close() {
async close() {
+ built.ctx.ingest.stop();
await new Promise((r) => server.close(r));
await new Promise((r) => server.close(r));
built.ctx.keys.client.stop();
built.ctx.keys.client.stop();
await testdb.close();
await testdb.close();
+ await boards.close();
await network.close();
await network.close();
},
},
};
};
new file mode 100644
new file mode 100644
@@ -0,0 +1,173 @@
@@ -0,0 +1,173 @@
+'use strict';
+/**
+ * The background ingest against the stand-in boards (test/helpers/boards.js): what it keeps from each board and what
+ * it refuses, that reading a board twice updates rather than duplicates, that a board which fails is recorded and
+ * does not stop the others, that only a short plain-text excerpt is stored, and that a listing no board has shown for
+ * the retention window expires out of the search and the pages.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+const store = require('../server/jobs/store');
+
+const DAY = 86_400_000;
+
+(async () => {
+ const clock = { t: Date.parse('2026-10-01T06:00:00Z') };
+ const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });
+ const ingest = t.ctx.ingest;
+ const s = t.ctx.s;
+
+ try {
+ await check('reading the three boards keeps every listing with its source, its id and the original URL', async () => {
+ const out = await ingest.runAll();
+ assert.deepStrictEqual(out.map((o) => o.source), ['arbeitnow', 'remotive', 'remoteok']);
+ assert.ok(out.every((o) => o.ok), JSON.stringify(out.map((o) => o.error)));
+ // Arbeitnow's two pages (2 + 1), Remotive's two jobs, RemoteOK's two listings.
+ const counts = await store.countsBySource(s);
+ assert.strictEqual(counts.get('arbeitnow').count, 3);
+ assert.strictEqual(counts.get("remotive").count, 3);
+ assert.strictEqual(counts.get('remoteok').count, 2);
+
+ const page = (await t.get('/api/v1/jobs?q=DevOps')).json();
+ assert.strictEqual(page.jobs.length, 1);
+ const job = page.jobs[0];
+ assert.strictEqual(job.source, 'remoteok');
+ assert.strictEqual(job.source_name, 'RemoteOK');
+ assert.strictEqual(job.url, 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001');
+ assert.strictEqual(job.company, 'Skyward Cloud');
+ assert.strictEqual(job.salary, '$90,000–$130,000 a year');
+ assert.strictEqual(job.remote, true);
+ assert.deepStrictEqual(job.tags, ['devops', 'kubernetes']);
+ assert.ok(job.posted_at.startsWith('2026-10-01'));
+
+ // RemoteOK's first array element is a legal notice, not a listing: nothing was made of it.
+ const all = (await t.get('/api/v1/jobs?limit=100')).json();
+ assert.strictEqual(all.count, 8);
+ assert.ok(!JSON.stringify(all.jobs).includes('personal use'), 'the legal notice became a listing');
+ });
+
+ await check('every request used this service\'s User-Agent, a JSON Accept, and only the configured hosts', async () => {
+ assert.ok(t.boards.requests.length >= 4);
+ for (const r of t.boards.requests) {
+ assert.strictEqual(r.userAgent, 'OpenVibeWork/0.1 (+https://openvibe.work)', r.path);
+ assert.match(r.accept, /application\/json/);
+ }
+ assert.deepStrictEqual([...new Set(t.boards.requests.map((r) => r.path))].sort(), ['/arbeitnow', '/remoteok', '/remotive']);
+ // Arbeitnow is paginated as configured (WORK_INGEST_PAGES=2); Remotive is asked for a limit.
+ assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=1'));
+ assert.ok(t.boards.requests.some((r) => r.path === '/arbeitnow' && r.query === '?page=2'));
+ assert.ok(t.boards.requests.some((r) => r.path === '/remotive' && r.query === '?limit=100'));
+ });
+
+ await check('an HTML description becomes a short plain-text excerpt: no tags, no script code, under 320 characters', async () => {
+ const job = (await t.get('/api/v1/jobs?q=Frontend')).json().jobs[0];
+ assert.ok(!/[<>]/.test(job.excerpt), job.excerpt);
+ assert.ok(!/window\.__pwned/.test(job.excerpt), 'script content was kept');
+ assert.ok(job.excerpt.includes('Build the design system'), job.excerpt);
+ assert.ok(job.excerpt.length <= 320, String(job.excerpt.length));
+ const page = (await t.get('/api/v1/jobs?q=Senior Backend')).json().jobs[0];
+ assert.ok(page.excerpt.includes('PostgreSQL'), page.excerpt);
+ assert.ok(page.excerpt.includes('fleet of Node.js services'), page.excerpt);
+ // The full description is nowhere in the database: only the excerpt is stored.
+ const dump = await t.dbDump();
+ assert.ok(!dump.includes('billing pipeline end to end</p>'), 'the full description was stored');
+ });
+
+ await check('the text helper strips markup and entities, and cuts a long excerpt on a word', () => {
+ const { stripHtml, excerpt, EXCERPT_CHARS } = require('../server/jobs/text');
+ assert.strictEqual(stripHtml('<p>Hello <b>world</b> — it's 5 & 6 <ok></p>'), "Hello world — it's 5 & 6 ok");
+ assert.strictEqual(stripHtml('<div>a</div><div>b</div>'), 'a b');
+ assert.strictEqual(stripHtml('<style>p{color:red}</style><p>real</p>'), 'real');
+ assert.strictEqual(stripHtml('<script>alert(1)</script>'), '');
+ assert.strictEqual(stripHtml(null), '');
+ // A doubly-encoded tag decodes to text and is then dropped, never re-formed into markup.
+ assert.ok(!stripHtml('<script>alert(1)</script>').includes('<'));
+ assert.strictEqual(excerpt('<p>short</p>'), 'short');
+ const long = excerpt(`<p>${'word '.repeat(120)}</p>`);
+ assert.ok(long.length <= EXCERPT_CHARS + 1, String(long.length));
+ assert.ok(long.endsWith('…'));
+ assert.ok(!long.includes(' ') && /^\S/.test(long), 'the excerpt is cut mid-word or padded');
+ });
+
+ await check('reading a board again updates the listing in place: no duplicate, same id, newer fetch', async () => {
+ const before = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];
+ const seenBefore = before.fetched_at;
+ clock.t += 2 * 3600_000;
+ // The board's own text changed under the same id, as a board edits a listing.
+ t.boards.fixtures.remoteok[1].position = 'DevOps Engineer (Platform)';
+ const remoteok = (await ingest.runAll()).find((o) => o.source === 'remoteok');
+ assert.deepStrictEqual([remoteok.ok, remoteok.listings, remoteok.added, remoteok.updated], [true, 2, 0, 2]);
+ const after = (await t.get('/api/v1/jobs?q=DevOps')).json().jobs[0];
+ assert.strictEqual(after.id, before.id, 'the id must not change when a board is read again');
+ assert.strictEqual(after.title, 'DevOps Engineer (Platform)');
+ assert.ok(after.fetched_at > seenBefore, 'fetched_at did not move');
+ assert.strictEqual((await t.get("/api/v1/jobs?limit=100")).json().count, 8);
+ t.boards.fixtures.remoteok[1].position = 'DevOps Engineer';
+ });
+
+ await check('a listing the board stopped showing expires out, leaving the others', async () => {
+ // Arbeitnow's page 2 goes quiet: one listing was there, is not any more.
+ const gone = t.boards.fixtures.arbeitnowPage2.shift();
+ clock.t += 31 * DAY;
+ await ingest.runAll();
+ assert.strictEqual((await store.countsBySource(s)).get('arbeitnow').count, 2);
+ assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 0);
+ const row = await s.db.maybe("SELECT * FROM work_listings WHERE source_id = $1 AND expired", [gone.slug]);
+ assert.ok(row, 'the expired listing was deleted rather than kept');
+ assert.strictEqual((await t.get(`/jobs/${row.id}`)).status, 404);
+ // And it comes back if the board shows it again.
+ t.boards.fixtures.arbeitnowPage2.push(gone);
+ await ingest.runAll();
+ assert.strictEqual((await t.get('/api/v1/jobs?q=Data Analyst')).json().count, 1);
+ });
+
+ await check('a board that fails is recorded, says why, and does not stop the others', async () => {
+ clock.t += 3600_000;
+ t.boards.state.remotive = 'fail';
+ t.boards.state.remoteok = 'html';
+ const out = await ingest.runAll();
+ assert.strictEqual(out.find((o) => o.source === 'remotive').ok, false);
+ assert.strictEqual(out.find((o) => o.source === 'arbeitnow').ok, true, 'one board failing stopped another');
+ const src = (await t.get('/api/v1/sources')).json();
+ const remotive = src.sources.find((x) => x.id === 'remotive');
+ assert.strictEqual(remotive.last_fetch.ok, false);
+ assert.match(remotive.last_fetch.error, /500/);
+ assert.ok(remotive.last_fetch.finished_at, 'the failure was not timestamped');
+ assert.match(src.sources.find((x) => x.id === 'remoteok').last_fetch.error, /not JSON/);
+ // What was already gathered stays: a failed read never empties the board.
+ assert.strictEqual(remotive.listings, 3);
+ t.boards.reset();
+ });
+
+ await check('the ingest is off unless WORK_INGEST=on, so a test or a development run reads no board', async () => {
+ assert.strictEqual(t.config.ingest.enabled, false);
+ assert.strictEqual(ingest.start(), false);
+ const other = await boot({ env: { WORK_INGEST: 'on', WORK_INGEST_INTERVAL_HOURS: '6' } });
+ try {
+ assert.strictEqual(other.config.ingest.enabled, true);
+ assert.strictEqual(other.config.ingest.intervalMs, 6 * 3600_000);
+ assert.strictEqual(other.ctx.ingest.start(), true);
+ assert.strictEqual(other.ctx.ingest.timers.size, 4, 'one timer per source plus the first run');
+ other.ctx.ingest.stop();
+ } finally { await other.close(); }
+ });
+
+ await check('a page that is not a listing is refused rather than published', async () => {
+ // A row with no URL to link back to, or a URL that is not http(s), has no provenance: it is not kept.
+ const { toRow } = require('../server/jobs/sources');
+ const base = { source_id: 'x', title: 't', company: 'c', url: 'https://example.org/x', tags: [], description_html: '<p>hi</p>' };
+ assert.strictEqual(toRow({ ...base, url: 'javascript:alert(1)' }, { sourceId: 'remotive', fetchedAt: 'x', idOf: store.idOf }), null);
+ assert.strictEqual(toRow({ ...base, url: null }, { sourceId: 'remotive', fetchedAt: 'x', idOf: store.idOf }), null);
+ assert.strictEqual(toRow({ ...base, title: '' }, { sourceId: 'remotive', fetchedAt: 'x', idOf: store.idOf }), null);
+ const row = toRow(base, { sourceId: 'remotive', fetchedAt: '2026-10-01T00:00:00.000Z', idOf: store.idOf });
+ assert.strictEqual(row.sort_at, '2026-10-01T00:00:00.000Z', 'a listing with no date sorts by when we read it');
+ });
+
+ await check('a listing\'s id is derived from its source and the board\'s id, so it is stable', () => {
+ assert.strictEqual(store.idOf('remotive', '1900123'), store.idOf('remotive', '1900123'));
+ assert.notStrictEqual(store.idOf('remotive', '1900123'), store.idOf('remoteok', '1900123'));
+ assert.match(store.idOf('arbeitnow', 'x'), store.LISTING_ID);
+ });
+ } finally { await t.close(); }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
new file mode 100644
new file mode 100644
@@ -0,0 +1,226 @@
@@ -0,0 +1,226 @@
+'use strict';
+/**
+ * The listings as a reader and as a crawler meet them: the search and its filters and facets, the cursor, one listing
+ * with its provenance and its JSON-LD, the escaping of a hostile title, and the discovery files that carry the
+ * listings (sitemap) and the sources (llms.txt).
+ *
+ * The listings come from the stand-in boards through the real ingest, so what is searched is what was really stored.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+
+const ldBlocks = (html) => [...html.matchAll(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/g)].map((m) => JSON.parse(m[1]));
+const first = async (t, path) => (await t.get(path)).json();
+
+(async () => {
+ const t = await boot({ env: { WORK_INGEST_PAGES: '2' } });
+ try {
+ await t.ctx.ingest.runAll();
+
+ await check('search: text matches the title, the company and the tags; results are newest first', async () => {
+ const byTitle = await first(t, '/api/v1/jobs?q=devops');
+ assert.deepStrictEqual(byTitle.jobs.map((j) => j.title), ['DevOps Engineer']);
+ const byCompany = await first(t, '/api/v1/jobs?q=Skyward');
+ assert.deepStrictEqual(byCompany.jobs.map((j) => j.company), ['Skyward Cloud']);
+ // The term is found in a tag and in the excerpt, so both listings answer.
+ const byTag = await first(t, '/api/v1/jobs?q=terraform');
+ assert.deepStrictEqual(byTag.jobs.map((j) => j.company).sort(), ['Meridian Rail', 'Skyward Cloud']);
+ // Case does not matter, and a % a caller typed is a character, not a wildcard.
+ assert.strictEqual((await first(t, '/api/v1/jobs?q=DEVOPS')).count, 1);
+ assert.strictEqual((await first(t, '/api/v1/jobs?q=%25')).count, 0);
+ const all = await first(t, '/api/v1/jobs?limit=100');
+ const dates = all.jobs.map((j) => j.posted_at);
+ assert.deepStrictEqual(dates, [...dates].sort().reverse(), 'newest first');
+ });
+
+ await check('filters: remote, job type and location, with the facets counting the same scope', async () => {
+ const remote = await first(t, '/api/v1/jobs?remote=true');
+ assert.ok(remote.count > 0);
+ assert.ok(remote.jobs.every((j) => j.remote === true));
+ const onSite = await first(t, '/api/v1/jobs?remote=false');
+ assert.ok(onSite.count > 0);
+ assert.ok(onSite.jobs.every((j) => j.remote === false));
+ assert.strictEqual(remote.count + onSite.count, (await first(t, '/api/v1/jobs?limit=100')).count);
+
+ const contract = await first(t, '/api/v1/jobs?type=contract');
+ assert.deepStrictEqual(contract.jobs.map((j) => j.title), ['Content Marketer']);
+ assert.strictEqual(contract.facets.matches, 1, 'the count must be what the filters match');
+ assert.strictEqual(contract.facets.scope, 8, 'the type facet must not narrow its own scope');
+
+ const germany = await first(t, '/api/v1/jobs?location=Germany');
+ assert.deepStrictEqual(germany.jobs.map((j) => j.title).sort(), ['Data Analyst', 'Senior Backend Engineer (Node.js)']);
+
+ const combined = await first(t, '/api/v1/jobs?q=engineer&remote=true');
+ assert.ok(combined.count >= 1 && combined.jobs.every((j) => /engineer/i.test(j.title)));
+ const facetTypes = (await first(t, '/api/v1/jobs')).facets.type;
+ assert.ok(facetTypes.length > 0 && facetTypes.every((x) => x.type && x.count > 0), JSON.stringify(facetTypes));
+ });
+
+ await check('the cursor pages through without repeating or losing a listing', async () => {
+ const seen = [];
+ let cursor = null;
+ for (let i = 0; i < 5; i++) {
+ const page = await first(t, `/api/v1/jobs?limit=3${cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''}`);
+ seen.push(...page.jobs.map((j) => j.id));
+ cursor = page.next;
+ if (!cursor) break;
+ }
+ const all = (await first(t, '/api/v1/jobs?limit=100')).jobs.map((j) => j.id);
+ assert.strictEqual(seen.length, all.length);
+ assert.deepStrictEqual([...new Set(seen)].length, all.length, 'a listing was repeated');
+ assert.deepStrictEqual(seen, all);
+ // A cursor we did not write is refused rather than guessed at.
+ assert.strictEqual((await t.get('/api/v1/jobs?cursor=nonsense')).status, 422);
+ });
+
+ await check('the list page shows every listing with its source named and linked back to the board', async () => {
+ const html = (await t.get('/jobs')).text;
+ assert.strictEqual((await t.get('/jobs')).status, 200);
+ for (const [name, url] of [
+ ['Arbeitnow', 'https://www.arbeitnow.com/view/senior-backend-engineer-berlin-12345'],
+ ['Remotive', 'https://remotive.com/remote-jobs/software-dev/platform-engineer-1900789'],
+ ['RemoteOK', 'https://remoteok.com/remote-jobs/remote-devops-engineer-skyward-1049001'],
+ ]) {
+ assert.ok(html.includes(name), `the source ${name} is not named on the list`);
+ assert.ok(html.includes(`href="${url}"`), `the original listing ${url} is not linked on the list`);
+ }
+ assert.match(html, /rel="nofollow noopener external"/, 'the link back must be marked as external');
+ // Filters and facets are on the page, and paging is a plain link (no JavaScript).
+ assert.ok(html.includes('name="remote"') && html.includes('name="location"'), 'the filter form is missing');
+ assert.match(html, /class="facets"/);
+ assert.match(html, /href="\/jobs\?[^"]*remote=true"/, 'a facet link is missing');
+ });
+
+ await check('one listing: the board\'s fields, the excerpt, the source and "read the full listing on" link', async () => {
+ const job = (await first(t, '/api/v1/jobs?q=Platform Engineer')).jobs[0];
+ const r = await t.get(`/jobs/${job.id}`);
+ assert.strictEqual(r.status, 200);
+ const html = r.text;
+ assert.ok(html.includes('Platform Engineer'));
+ assert.ok(html.includes('Meridian Rail'));
+ assert.ok(html.includes('Remote (EU)'));
+ assert.ok(html.includes('€70,000 – €85,000'), 'the salary the board gave');
+ assert.ok(html.includes('<li>kubernetes</li>'), 'the tags');
+ assert.ok(html.includes('Read the full listing on Remotive'), 'no apply link');
+ assert.ok(html.includes(`href="${job.url}"`), 'the apply link must be the original URL');
+ assert.match(html, /<dt>Source<\/dt>/, 'the source is not stated');
+ // The excerpt is shown; the tail of the description was never stored, so it cannot be shown.
+ assert.ok(html.includes('You will run the platform other engineers build on'), 'the excerpt is missing');
+ assert.ok(!html.includes('Waffle'), 'the full description was republished');
+ assert.ok(html.length < 60_000, `the page is ${html.length} bytes: too much of the description is on it`);
+ assert.ok(html.includes('/sources#remotive'), 'no link to the source\'s terms as applied');
+ });
+
+ await check('the listing page carries a valid JobPosting whose url is the original, not this page', async () => {
+ const job = (await first(t, '/api/v1/jobs?q=DevOps')).jobs[0];
+ const html = (await t.get(`/jobs/${job.id}`)).text;
+ const blocks = ldBlocks(html);
+ const posting = blocks.find((b) => b['@type'] === 'JobPosting');
+ assert.ok(posting, `no JobPosting node: ${JSON.stringify(blocks.map((b) => b['@type']))}`);
+ assert.strictEqual(posting.title, 'DevOps Engineer');
+ assert.strictEqual(posting.url, job.url);
+ assert.notStrictEqual(posting.url, `https://openvibe.work/jobs/${job.id}`);
+ assert.strictEqual(posting.hiringOrganization.name, 'Skyward Cloud');
+ assert.strictEqual(posting.jobLocationType, 'TELECOMMUTE');
+ assert.match(posting.datePosted, /^2026-10-01T/);
+ assert.strictEqual(posting.identifier.name, 'remoteok');
+ assert.ok(posting.description.length <= 320);
+ assert.ok(posting['@context'] === 'https://schema.org');
+ // Every JSON-LD block on the page parses (the escaping must survive a hostile title).
+ assert.ok(blocks.length >= 1);
+ });
+
+ await check('a hostile title is escaped everywhere it appears, and never becomes markup or a URL', async () => {
+ const job = (await first(t, '/api/v1/jobs?q=Frontend Engineer')).jobs[0];
+ assert.ok(job.title.includes('<script>'), 'the fixture must carry the hostile title');
+ for (const path of ['/jobs', `/jobs/${job.id}`, `/api/v1/jobs?q=Frontend Engineer`]) {
+ const html = (await t.get(path)).text;
+ assert.ok(!html.includes('<script>alert("xss")</script>'), `${path} rendered the title as markup`);
+ assert.ok(!/window\.__pwned/.test(html), `${path} rendered the description's script`);
+ if (path.startsWith('/jobs')) assert.ok(html.includes('<script>alert("xss")</script>'), `${path} did not escape the title`);
+ }
+ // The JSON-LD on its own page is still parseable JSON with the title as a string.
+ const posting = ldBlocks((await t.get(`/jobs/${job.id}`)).text).find((b) => b['@type'] === 'JobPosting');
+ assert.ok(posting.title.includes('<script>'));
+ });
+
+ await check('a listing that is not there is a 404 page and a problem+json, the same for any unknown id', async () => {
+ for (const id of ['job_00000000000000000000000000', 'nope', "1' OR 1=1--"]) {
+ const page = await t.get(`/jobs/${encodeURIComponent(id)}`);
+ assert.strictEqual(page.status, 404, id);
+ assert.match(page.text, /Listing not found/);
+ const api = await t.get(`/api/v1/jobs/${encodeURIComponent(id)}`);
+ assert.strictEqual(api.status, 404);
+ assert.strictEqual(api.json().code, 'work.jobs.not_found');
+ }
+ });
+
+ await check('every source is listed with its terms as applied and when it was last read', async () => {
+ const api = await first(t, '/api/v1/sources');
+ assert.deepStrictEqual(api.sources.map((s) => s.id).sort(), ['arbeitnow', 'remoteok', 'remotive']);
+ for (const src of api.sources) {
+ assert.ok(src.terms.length > 40, `${src.id} has no terms`);
+ assert.match(src.terms, /link|attribute|mention|name/i, `${src.id}'s terms do not state the attribution rule`);
+ assert.strictEqual(src.last_fetch.ok, true);
+ assert.ok(src.last_fetch.finished_at);
+ assert.ok(src.listings > 0);
+ }
+ const html = (await t.get('/sources')).text;
+ assert.ok(html.includes('id="remotive"') && html.includes('id="remoteok"') && html.includes('id="arbeitnow"'));
+ assert.ok(html.includes('Listing from Remotive') && html.includes('Listing from RemoteOK'));
+ assert.match(html, /four a day/, 'Remotive\'s call limit is not stated');
+ });
+
+ await check('the sitemap carries the newest listings and the site\'s own pages, and no private path', async () => {
+ const xml = (await t.get('/sitemap.xml')).text;
+ const locs = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map((m) => m[1]);
+ const jobs = (await first(t, '/api/v1/jobs?limit=100')).jobs;
+ for (const job of jobs) assert.ok(locs.includes(`https://openvibe.work/jobs/${job.id}`), `${job.id} is not in the sitemap`);
+ for (const p of ['/', '/jobs', '/sources', '/docs', '/updates']) assert.ok(locs.includes(`https://openvibe.work${p}`), `${p} is not in the sitemap`);
+ for (const p of ['/saved', '/auth/login', '/api/v1/jobs']) assert.ok(!xml.includes(`https://openvibe.work${p}`), `${p} must not be in the sitemap`);
+ assert.ok(locs.length <= 1000 + 5);
+ });
+
+ await check('llms.txt sends a crawler to the listings and the sources, and robots keeps /saved out', async () => {
+ const llms = (await t.get('/llms.txt')).text;
+ assert.ok(llms.includes('(https://openvibe.work/jobs)'));
+ assert.ok(llms.includes('(https://openvibe.work/sources)'));
+ assert.ok(llms.includes('Arbeitnow, Remotive, RemoteOK'), 'the sources are not named for a crawler');
+ assert.ok(!llms.includes('/saved'), 'llms.txt names a private page');
+ const robots = (await t.get('/robots.txt')).text;
+ assert.ok(robots.includes('Disallow: /saved'));
+ assert.ok(!robots.includes('Disallow: /jobs'));
+ });
+
+ await check('the home page is the search box, the latest listings and the sources with their counts', async () => {
+ const r = await t.get('/');
+ assert.strictEqual(r.status, 200);
+ assert.ok(r.text.includes('Find work.'), 'the hero');
+ assert.match(r.text, /action="\/jobs"/, 'the search box');
+ assert.ok(r.text.includes('Latest remote jobs'));
+ assert.ok(r.text.includes('DevOps Engineer'), 'no listing on the home page');
+ for (const [name, url] of [['Arbeitnow', 'https://www.arbeitnow.com/'], ['Remotive', 'https://remotive.com/'], ['RemoteOK', 'https://remoteok.com/']]) {
+ assert.ok(r.text.includes(`href="${url}"`), `the source ${name} is not linked on the home page`);
+ }
+ assert.match(r.text, /no paywall/i, '"how it works" does not say what is behind a paywall (nothing)');
+ });
+
+ await check('search is limited per caller, and past it the API says so with Retry-After', async () => {
+ const busy = await boot({ callerLimits: true, limitsNow: () => Date.now(), env: { WORK_INGEST_PAGES: '1' } });
+ try {
+ await busy.ctx.ingest.runAll();
+ let limited = null;
+ for (let i = 0; i < 70 && !limited; i++) {
+ const r = await busy.get('/api/v1/jobs?q=engineer', { headers: { 'x-forwarded-for': '10.9.9.9' } });
+ if (r.status === 429) limited = r;
+ }
+ assert.ok(limited, 'the search budget never engaged');
+ assert.strictEqual(limited.json().code, 'rate_limited');
+ assert.ok(Number(limited.headers.get('retry-after')) > 0);
+ // Reading one listing is a plain read, not the search budget, and still works.
+ assert.strictEqual((await busy.get('/api/v1/sources')).status, 200);
+ } finally { await busy.close(); }
+ });
+ } finally { await t.close(); }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
@@ -12,7 +12,10 @@ const { spawn } = require('child_process');
@@ -12,7 +12,10 @@ const { spawn } = require('child_process');
const { measure, check, format } = require('openvibe-shared/perf-budget');
const { measure, check, format } = require('openvibe-shared/perf-budget');
// Budgets are set from the numbers measured on 2026-10-08 (the real ones, fresh database), rounded up with about 10%
// Budgets are set from the numbers measured on 2026-10-08 (the real ones, fresh database), rounded up with about 10%
-// headroom; file counts are the measured counts and external files stay at 0.
+// headroom; file counts are the measured counts and external files stay at 0. The CSS budget was raised from 16.5/4.1
+// when the listings arrived (the search form, a listing list and the source blocks in app.css): the measured page is
+// 18.2 KB raw / 4.3 KB brotli, still two stylesheets and no external file. The home page itself stayed inside its own
+// budget — the listings replace the skeleton's placeholder sections rather than adding to them.
const BUDGETS = {
const BUDGETS = {
htmlRawKB: 27.5, // measured 24.9 (the home page: hero, features, steps, cta, JSON-LD)
htmlRawKB: 27.5, // measured 24.9 (the home page: hero, features, steps, cta, JSON-LD)
htmlBrotliKB: 6.9, // 6.2
htmlBrotliKB: 6.9, // 6.2
@@ -20,8 +23,8 @@ const BUDGETS = {
@@ -20,8 +23,8 @@ const BUDGETS = {
jsRawKB: 264, // 239.8
jsRawKB: 264, // 239.8
jsBrotliKB: 62.5, // 56.5
jsBrotliKB: 62.5, // 56.5
cssFiles: 2, // 2 (app.css + the cached /shared/showcase.css)
cssFiles: 2, // 2 (app.css + the cached /shared/showcase.css)
- cssRawKB: 16.5, // 15.0
- cssBrotliKB: 4.1, // 3.7
+ cssRawKB: 20.5, // 15.0 before the listings; 18.2 after (the search form, the listing list, the source blocks)
+ cssBrotliKB: 4.8, // 3.7 before; 4.3 after
externalFiles: 0, // 0
externalFiles: 0, // 0
};
};
new file mode 100644
new file mode 100644
@@ -0,0 +1,137 @@
@@ -0,0 +1,137 @@
+'use strict';
+/**
+ * Saved searches: saving the search you are looking at, seeing each one with the count of listings that appeared
+ * since you last looked, the count resetting when you open /saved, signing in being required, a signed-in write from
+ * another site being refused, and nobody seeing anybody else's.
+ *
+ * The clock is a test's own, because "new since you last looked" is a claim about time.
+ */
+const assert = require('assert');
+const { boot, check, done } = require('./helpers/boot');
+
+(async () => {
+ // The clock starts at the real now: the session is a Network token, and this service checks it against its own
+ // clock, so a test that moves time must move only a little or the token stops verifying.
+ const clock = { t: Date.now() };
+ const t = await boot({ now: () => clock.t, env: { WORK_INGEST_PAGES: '2' } });
+ const kim = t.network.addUser('kim');
+ const lee = t.network.addUser('lee');
+ const SAME = { 'sec-fetch-site': 'same-origin' };
+
+ try {
+ await t.ctx.ingest.runAll();
+
+ await check('a saved search is a person\'s: anonymous is 401, an app token is 403, the other person sees nothing', async () => {
+ assert.strictEqual((await t.get('/api/v1/saved-searches')).status, 401);
+ const anon = await t.get('/api/v1/saved-searches');
+ assert.strictEqual(anon.json().code, 'token.required');
+
+ const save = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'engineer' }, headers: SAME });
+ assert.strictEqual(save.status, 201, save.text);
+ const mine = save.json();
+ assert.match(mine.id, /^ssc_[0-9A-HJKMNP-TV-Z]{26}$/);
+ assert.strictEqual(mine.query, 'engineer');
+ assert.strictEqual(mine.new_count, 0);
+
+ const other = await t.get('/api/v1/saved-searches', { as: lee });
+ assert.deepStrictEqual(other.json().saved_searches, [], 'another person saw my saved search');
+ assert.strictEqual((await t.get(`/api/v1/saved-searches/${mine.id}`, { as: lee, method: 'DELETE', headers: SAME })).status, 404);
+ assert.strictEqual((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.length, 1, 'the other person deleted mine');
+ });
+
+ await check('a saved search needs at least one filter, and its filters are the search\'s own', async () => {
+ const empty = await t.get('/api/v1/saved-searches', { as: kim, json: { q: ' ' }, headers: SAME });
+ assert.strictEqual(empty.status, 422);
+ assert.strictEqual(empty.json().code, 'work.saved_searches.empty');
+ const remote = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'devops', remote: 'yes', location: 'Worldwide' }, headers: SAME });
+ assert.strictEqual(remote.status, 201, remote.text);
+ const saved = remote.json();
+ assert.deepStrictEqual(saved.filters, { remote: 'true', type: '', location: 'worldwide' });
+ assert.strictEqual(saved.search_path, '/jobs?q=devops&remote=true&location=worldwide');
+ // Running it again gives the same listings the search would.
+ const run = (await t.get(saved.search_path)).text;
+ assert.ok(run.includes('DevOps Engineer'));
+ });
+
+ await check('a signed-in write from another site is refused; a Bearer token is not (it is that person\'s own)', async () => {
+ const cross = await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'x' }, headers: { 'sec-fetch-site': 'cross-site' } });
+ assert.strictEqual(cross.status, 403);
+ assert.strictEqual(cross.json().code, 'request.cross_site');
+ const bearer = await t.get('/api/v1/saved-searches', { bearer: t.network.userToken(kim), json: { q: 'designer' } });
+ assert.strictEqual(bearer.status, 201, bearer.text);
+ });
+
+ await check('a listing that appears later counts as new, and opening /saved is what clears it', async () => {
+ const saved = (await t.get('/api/v1/saved-searches', { as: kim, json: { q: 'render wrangler' }, headers: SAME })).json();
+ assert.strictEqual(saved.new_count, 0);
+ // Nothing new yet: the board shows nothing matching.
+ assert.strictEqual((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.find((x) => x.id === saved.id).new_count, 0);
+
+ // A little later the board starts showing one that matches.
+ clock.t += 60_000;
+ t.boards.add('remotive', {
+ id: 1900999, url: 'https://remotive.com/remote-jobs/other/render-wrangler-1900999',
+ title: 'Render Wrangler', company_name: 'Raytrace', category: 'Other', tags: ['gpu'],
+ job_type: 'full_time', publication_date: '2026-10-02T10:00:00',
+ candidate_required_location: 'Worldwide', salary: '', description: '<p>Keep the farm rendering.</p>',
+ });
+ await t.ctx.ingest.runAll();
+
+ const apiList = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;
+ assert.strictEqual(apiList.find((x) => x.id === saved.id).new_count, 1, JSON.stringify(apiList));
+
+ // Opening /saved shows the count, and is itself the act of looking.
+ const page = await t.get('/saved', { as: kim });
+ assert.strictEqual(page.status, 200);
+ assert.match(page.text, /1 new/);
+ assert.strictEqual((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.find((x) => x.id === saved.id).new_count, 0, 'opening /saved did not clear the count');
+ assert.strictEqual((await t.get('/saved', { as: kim })).text.includes('nothing new'), true);
+ });
+
+ await check('saving from the page needs a sign-in, must come from this site, and lands on /saved', async () => {
+ const anon = await t.get('/saved');
+ assert.strictEqual(anon.status, 200);
+ assert.match(anon.text, /Sign in with OpenVibe/);
+ assert.match(anon.text, /href="\/auth\/login\?next=%2Fsaved"/);
+
+ const cross = await t.get('/saved', { as: kim, form: { q: 'engineer' }, headers: { 'sec-fetch-site': 'cross-site' } });
+ assert.strictEqual(cross.status, 403);
+
+ const ok = await t.get('/saved', { as: kim, form: { q: 'support engineer', remote: 'true' }, headers: SAME });
+ assert.strictEqual(ok.status, 303);
+ assert.strictEqual(ok.headers.get('location'), '/saved');
+ const list = await t.get('/saved', { as: kim });
+ assert.ok(list.text.includes('“support engineer”'), 'the saved search is not on the page');
+ assert.match(list.text, /href="\/jobs\?q=support\+engineer&remote=true"/, 'no link to run it again');
+ assert.match(list.text, /Forget it/);
+ });
+
+ await check('forgetting a saved search removes it, and only it', async () => {
+ const before = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;
+ assert.ok(before.length >= 2);
+ const target = before.find((x) => x.query === 'engineer');
+ const gone = await t.get(`/api/v1/saved-searches/${target.id}`, { as: kim, method: 'DELETE', headers: SAME });
+ assert.strictEqual(gone.status, 204);
+ const after = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;
+ assert.deepStrictEqual(after.map((x) => x.id), before.filter((x) => x.id !== target.id).map((x) => x.id));
+ assert.strictEqual((await t.get(`/api/v1/saved-searches/${target.id}`, { as: kim, method: 'DELETE', headers: SAME })).status, 404);
+ });
+
+ await check('the page\'s delete button forgets exactly the one it names', async () => {
+ const list = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;
+ const keep = list[0];
+ const drop = list[1];
+ const form = await t.get(`/saved/${drop.id}/delete`, { as: kim, method: 'POST', headers: SAME });
+ assert.strictEqual(form.status, 303);
+ assert.strictEqual(form.headers.get('location'), '/saved');
+ const after = (await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches;
+ assert.ok(after.some((x) => x.id === keep.id), 'the wrong saved search was deleted');
+ assert.ok(!after.some((x) => x.id === drop.id));
+ // Nobody else's, even with the right id.
+ const other = await t.get(`/saved/${keep.id}/delete`, { as: lee, method: 'POST', headers: SAME });
+ assert.strictEqual(other.status, 303);
+ assert.ok((await t.get('/api/v1/saved-searches', { as: kim })).json().saved_searches.some((x) => x.id === keep.id), 'another person deleted my saved search');
+ });
+ } finally { await t.close(); }
+ done();
+})().catch((e) => { console.error(e); process.exit(1); });
@@ -46,6 +46,17 @@ const rel = (p, dir) => path.relative(dir, p).split(path.sep).join('/');
@@ -46,6 +46,17 @@ const rel = (p, dir) => path.relative(dir, p).split(path.sep).join('/');
const text = (p) => fs.readFileSync(p, 'utf8');
const text = (p) => fs.readFileSync(p, 'utf8');
(async () => {
(async () => {
+ // This is the skeleton's own test. A service generated from the skeleton (this repository, once its tokens were
+ // replaced) carries no placeholders left to rewrite, so the generator can only copy it under its own name and
+ // there is nothing here to test. Say so and stop, rather than assert against a rewrite that cannot happen.
+ const pkgText = text(path.join(ROOT, 'package.json'));
+ if (!TOKEN_RE.test(pkgText)) {
+ // The label first, as the shared runner reads it: this file is listed with ○ and is not counted as passed.
+ process.stdout.write('the generator: skipped (this tree carries no skeleton placeholders left to rewrite: it is a generated service, not the skeleton itself)\n');
+ done();
+ return;
+ }
+
const base = tmpDir('ov-new-service-');
const base = tmpDir('ov-new-service-');
const dir = path.join(base, 'openvibe-sample');
const dir = path.join(base, 'openvibe-sample');
const printed = runGenerator(dir);
const printed = runGenerator(dir);