README.md

main 23.5 KB · 284 lines · markdown

1	# OpenVibe.Work
2
3	> Find work on open job boards.
4
5	Live site: https://openvibe.work · API docs: https://openvibe.services · For AI: https://openvibe.work/llms.txt
6	Run it: `npm ci && npm run dev` (Node 22 or newer) · Tests: `npm test`
7
8	Status: the listings, the search and the agent pillar are live (listings gathered from four public job boards (two of them read through OpenVibe.Sources),
9	search, saved searches, job applications with their status history, help preparing one from the person's OpenVibe.Actor
10	agent, an API and the pages around them).
11	Domain: `openvibe.work` · Port: 4960 · Service id: `work` · Env prefix: `WORK`
12	License: AGPL-3.0 (same as every OpenVibe service).
13
14	## Purpose
15
16	OpenVibe.Work is the network's job-listings service: it gathers listings from public job boards, keeps each one's
17	provenance and a short plain-text excerpt, and makes them searchable without an account. A signed-in person can keep a
18	search, see how many listings have appeared since they last looked, track the jobs they apply to and ask their
19	OpenVibe.Actor agent to help prepare one. Nothing is republished — every listing links back to the board, where the
20	reader reads the full listing and applies.
21
22	## What it does
23
24	Job listings gathered from public job boards, each one carrying its provenance — the board it came from and a link
25	back to the original listing — searchable, filterable, and readable without an account. A signed-in person can keep a
26	search, see how many listings have appeared since they last looked, track the jobs they apply to (with notes, dates,
27	next steps and a status history), and ask their OpenVibe.Actor agent to help prepare an application. Nothing is
28	republished: Work stores a short plain-text excerpt and sends the reader to the board for the full listing and to apply.
29
30	## What is here
31
32	\| Piece \| Where \| What it does \|
33	\|---\|---\|---\|
34	\| Config \| [server/config.js](server/config.js) \| Every value from the environment; `load(env)` is pure so tests build a config without touching `process.env` \|
35	\| App \| [server/app.js](server/app.js) \| helmet (CSP, frame-ancestors none), the Network session middleware, `/auth`, the legal pages, static assets, the `/api/v1` mount, the 404 and the error handler \|
36	\| Sign-in \| [server/auth/sso.js](server/auth/sso.js) \| OAuth 2 authorization code with PKCE (S256) against OpenVibe.Network; httpOnly `work_at` / `work_rt` cookies; `/auth/me` for the shared navbar \|
37	\| Sources \| [server/jobs/sources.js](server/jobs/sources.js) \| The boards: where each is fetched, its terms, how its payload becomes a listing, and what is refused \|
38	\| Ingest \| [server/ingest/](server/ingest/) \| The background read (timer, User-Agent, 15 s timeout, upsert by source + the board's id, expiry) \|
39	\| Listings \| [server/jobs/store.js](server/jobs/store.js), [migrations/0002_listings.sql](migrations/0002_listings.sql) \| Search, facets, the cursor, saved searches, the fetch record \|
40	\| Text \| [server/jobs/text.js](server/jobs/text.js) \| HTML → plain text, and the ~300-character excerpt that is all we keep \|
41	\| Applications \| [server/applications.js](server/applications.js), [migrations/0005_applications.sql](migrations/0005_applications.sql) \| The jobs a person tracks: statuses, notes, dates, next steps and the append-only status history \|
42	\| Actor \| [server/actor.js](server/actor.js) \| `POST /api/v1/tasks` on the person's OpenVibe.Actor agent with Work's own service token \|
43	\| API \| [server/http/api.js](server/http/api.js) \| `/api/v1`: jobs, one job, the sources, saved searches, applications; problem+json errors \|
44	\| Pages \| [server/http/pages.js](server/http/pages.js), [server/render/](server/render/) \| `/`, `/jobs`, `/jobs/:id`, `/saved`, `/applications`, `/applications/:id`, `/sources`, `/docs`, `/updates` — server-rendered, no JavaScript needed \|
45	\| Discovery \| [server/http/discovery.js](server/http/discovery.js) \| `robots.txt`, `sitemap.xml` (the newest listings too), `llms.txt`, `llms-full.txt`, JSON-LD; with `INDEXNOW_KEY`, each ingest run's new listings are sent to IndexNow \|
46	\| Limits \| [server/http/caller-limits.js](server/http/caller-limits.js), [deploy/nginx/](deploy/nginx/) \| Per-caller limits at the API (the search and the saved-search writes have their own) and per-address limits at nginx \|
47	\| Health \| [server/observability.js](server/observability.js) \| `/api/health`, a truthful `/api/ready` (the database is required) and Prometheus metrics on loopback only \|
48	\| Telemetry \| [server/app.js](server/app.js), [server/index.js](server/index.js), [migrations/0006_telemetry.sql](migrations/0006_telemetry.sql) \| Every request aggregated by route template, method and status class, never by raw URL; once a minute one `http.request` sample per key, plus the active-requests, p95-latency and event-loop-lag gauges, written to this service's own `telemetry_samples` table (openvibe-sdk 0.46 telemetry, the `platform.telemetry-sample@1` shape); rows older than 14 days are deleted daily; `/shared` files, static assets, health, readiness and `/metrics` are not counted. `app.js` builds the collector (`createHttpTelemetry`) and mounts its middleware; `index.js` flushes it every minute and on stop and runs the retention \|
49	\| Database \| [server/db.js](server/db.js), [migrations/](migrations/) \| PostgreSQL through `openvibe-sdk/db`; `NNNN_*.sql` applied at boot; PGlite in development \|
50	\| Process \| [server/index.js](server/index.js) \| Listens on `PORT`, starts the ingest when it is on, and stops gracefully through `openvibe-sdk/service` \|
51	\| Deploy \| [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), [deploy/systemd/openvibe-work.service](deploy/systemd/openvibe-work.service) \| nginx vhost and systemd unit (port 4960, `/opt/openvibe.work`, `/etc/openvibe/work.env`) \|
52	\| Tests \| [test/](test/) \| `npm test`: every `test/*.test.js` in its own process, on a temp PGlite database, with a mock OpenVibe.Network and a stand-in for each job board \|
53
54	## Owns
55
56	Its own PostgreSQL tables, created by [migrations/](migrations/) and written by nothing else:
57
58	- `work_listings` (0002) — one row per listing, upserted on `(source, source_id)`, with the board's own id, the
59	original `url` to link back to, the fields the board gives, and a short plain-text excerpt. Expired listings are
60	kept but hidden.
61	- `work_saved_searches` (0002) — a search a person saved, keyed `subject = user:usr_…`, with when they last looked.
62	- `work_source_fetches` (0002) — what each board's last fetch did, so `/sources` can say when it last ran and how.
63	- `work_applications` (0005) — a job application a person tracks, keyed `subject_id = user:usr_…`, with its status,
64	notes, dates, next step and the last Actor task it asked for. One per person per listing.
65	- `work_application_events` (0005) — the append-only history of an application's status changes.
66	- `account_data_events` (0003) — the receipts of the ADR-033 export and deletion deliveries this service applied, so a
67	redelivered event changes nothing.
68
69	0001 creates no tables. The service is the authority for these rows and for nothing else.
70
71	## Does not own
72
73	- Identity and accounts — OpenVibe.Network: sign-in (OAuth client `work`, PKCE S256), the signing keys (JWKS) and
74	the canonical subject. Work holds only `user:usr_…` on its own rows.
75	- Event delivery — OpenVibe.Events: Work receives `network.account.export_requested` and `network.account.deleted`
76	and creates its two subscriptions at boot; it does not own the topics or the delivery.
77	- The listings themselves — the boards (Arbeitnow, We Work Remotely, RemoteOK, Hacker News "Who is hiring" and Remotive): Work keeps a short excerpt and a link back,
78	never the full description, and reads each board only on the terms it publishes. A board that writes a listing as
79	one pipe-separated line (Hacker News) has that line split into company, role(s), place, terms and pay
80	(`server/jobs/hiring-line.js`); the line itself stays at the front of the excerpt.
81
82	## Depends on
83
84	- OpenVibe.Network — SSO sign-in and JWKS verification, and the internal routes an export part or a deletion
85	confirmation is pushed to (`OV_NETWORK_URL`, `OV_NETWORK_INTERNAL_URL`, `OV_OAUTH_CLIENT_ID`,
86	`OV_OAUTH_CLIENT_SECRET`, `OV_NETWORK_ISSUER`, `OV_SESSION_AUDIENCE`, `WORK_AUDIENCE`).
87	- OpenVibe.Events — the two account subscriptions created at boot and the delivery posted to this service's
88	loopback `/internal/events` (`WORK_EVENTS_URL` or `EVENTS_URL`, `WORK_EVENTS_SECRET`, `WORK_EVENTS_ENDPOINT`,
89	`WORK_EVENTS_SUBSCRIBE`).
90	- OpenVibe.Actor — the person's agent: Work posts `POST /api/v1/tasks` with its own service token (capability
91	`actor.task.create`, audience `openvibe.actor`) when a person asks for help preparing an application
92	(`WORK_ACTOR_URL`, `WORK_ACTOR_PUBLIC_URL`, `WORK_ACTOR_TIMEOUT_MS`).
93	- The job boards — Remotive is the only host Work itself fetches (`WORK_REMOTIVE_URL`, `WORK_USER_AGENT`), because
94	its API terms grant a named, linked use while its robots.txt disallows `/api/*`. Arbeitnow, RemoteOK, We Work Remotely
95	and Hacker News "Who is hiring" arrive through OpenVibe.Sources (`server/ingest/from-sources.js`).
96	- PostgreSQL (`DATABASE_URL`, `DATABASE_DIRECT_URL`; an embedded PGlite database in development via
97	`WORK_PGLITE_DIR`) and Valkey for shared limit counters (`VALKEY_URL`, `VALKEY_PREFIX`).
98	- Packages: `openvibe-contracts` v0.289.1, `openvibe-sdk` v0.161.0 (`db`, `auth`, `account-data`, `limits`,
99	`valkey`, `service`) and `openvibe-shared` v4.179.0 (`frame`, `legal`, `serve`, `release`, `metrics`, `ready`,
100	`seo`, `shell`, `cache-policy`, `showcase`, `app-icon`).
101
102	## Capabilities
103
104	[server/http/principal.js](server/http/principal.js) declares one capability, `work.application.manage`: an app or
105	agent token may manage job applications with it for the person it acts for (a person needs no capability for their
106	own). The service manifest (`work`, openvibe-contracts) has listed it since 0.147.0, from the proposal in
107	[docs/contracts-proposal/work.application.manage.md](docs/contracts-proposal/work.application.manage.md). The
108	person-only saved-search routes still name no capability.
109
110	Work calls one capability on another service: `actor.task.create` (audience `openvibe.actor`) when a person asks their
111	agent to help prepare an application ([server/actor.js](server/actor.js)). It also uses two machine-to-machine
112	surfaces: OpenVibe.Events' subscription API (scope `events.subscription.manage`) to create its two account-event
113	subscriptions at boot ([server/events-consumer.js](server/events-consumer.js)), and OpenVibe.Network's internal account
114	export/deletion routes, pushed with this service's own client-credentials token
115	([server/identity/account-data.js](server/identity/account-data.js)).
116
117	## API
118
119	\| Route \| Who \| \|
120	\|---\|---\|---\|
121	\| `GET /api/v1/ping` \| anyone \| `{ ok: true, service: "work" }` \|
122	\| `GET /api/v1/jobs` \| anyone \| Search: `q`, `remote`, `type`, `location`, `limit` (to 100), `cursor` (the `next` of the previous page). Newest first, 30 a page, with facets \|
123	\| `GET /api/v1/jobs/:id` \| anyone \| One listing: the board's fields, the excerpt, the source, and `url` — the original to apply on \|
124	\| `GET /jobs.atom` \| anyone \| The same search as an Atom feed (`q`, `remote`, `type`, `location`): the newest 25 listings, each linking its page here, cached 5 minutes. The search page links it in its head, and Get alerts (on the search page and on each saved search) opens OpenVibe.Watch's feed form with it filled in, so Watch tells you, or runs your agent, when a listing that fits appears \|
125	\| `GET /api/v1/sources` \| anyone \| The boards, each one's terms as applied, its listing count and its last fetch \|
126	\| `GET /api/v1/saved-searches` \| a person \| Your saved searches, each with `new_count` — new since you last looked \|
127	\| `POST /api/v1/saved-searches` \| a person \| `{ q?, remote?, type?, location? }` (at least one). 201 with the saved search \|
128	\| `DELETE /api/v1/saved-searches/:id` \| a person \| Forget one. 204 \|
129	\| `GET /api/v1/me/applications` \| a person (or app with `work.application.manage`) \| Your applications, filtered by `status`, newest first, cursor-paginated \|
130	\| `POST /api/v1/me/applications` \| a person (or app with `work.application.manage`) \| `{ listing_id }` or `{ title, company, url }`. 201 with the application; a repeat listing answers 200 with the existing one \|
131	\| `GET /api/v1/me/applications/:id` \| a person (or app with `work.application.manage`) \| One application with its status-change history \|
132	\| `PATCH /api/v1/me/applications/:id` \| a person (or app with `work.application.manage`) \| `{ status?, notes?, applied_at?, next_step?, next_step_at? }`; a status change appends an event \|
133	\| `DELETE /api/v1/me/applications/:id` \| a person (or app with `work.application.manage`) \| Forget one. 204 \|
134	\| `POST /api/v1/me/applications/:id/assist` \| a person \| `{ ask? }` — ask the person's OpenVibe.Actor agent; `{ task_id, task_url }` \|
135
136	Who can call it: reading the listings is public. A saved search belongs to a person, so those three routes need a
137	signed-in person — their Network token as a Bearer, or this site's session — and answer 401 otherwise. An app or agent
138	token cannot hold one (403): a saved search is not a project's. Applications are a person's too, but an app or agent
139	token with `work.application.manage` may manage them; asking the person's agent (`…/assist`) is always the person
140	themselves. A write made with the session cookie must come from `openvibe.work` itself (`request.cross_site`
141	otherwise).
142
143	Errors are RFC 9457 `application/problem+json` with a stable `code` (`work.jobs.not_found`,
144	`work.saved_searches.empty`, `work.saved_searches.limit`, `work.applications_full`, `work.actor_unavailable`,
145	`rate_limited`, …).
146
147	## Data sources and their terms
148
149	Each board is read through its own public API, and its terms are applied on every page that shows one of its listings
150	— they are also served by `GET /api/v1/sources` and written out on [/sources](server/http/pages.js).
151
152	\| Source \| API \| Terms, as applied here \|
153	\|---\|---\|---\|
154	\| Arbeitnow \| `GET https://www.arbeitnow.com/api/job-board-api?page=N` \| Read by OpenVibe.Sources (source `arbeitnow-job-board`), never by Work: its first three pages are fetched under the board's robots and terms, and Work follows that change feed (`server/ingest/from-sources.js`). Arbeitnow is named on every listing and the listing links to its page on arbeitnow.com; a job the board stops showing is retired by Sources after a week. \|
155	\| Remotive \| `GET https://remotive.com/api/remote-jobs?limit=100` \| Remotive must be named as the source and the listing linked. The API is called about four times a day — the six-hour interval is exactly that. It stays Work's own fetch because remotive.com/robots.txt disallows `/api/*`, so OpenVibe.Sources may not read it, while Remotive's API terms grant the named, linked use. \|
156	\| RemoteOK \| `GET https://remoteok.com/api` \| Read by OpenVibe.Sources (source `remoteok-job-board`), never by Work: the API is fetched under remoteok.com's robots.txt and its terms. RemoteOK must be named as the source and the listing linked; the response's first element is a legal notice, not a listing, and Sources' mapping skips it. \|
157
158	What is stored per listing: `source`, the board's own id, the original `url`, title, company, location, the remote
159	flag, tags, job type, salary text when the board gives one, when it was posted, when we fetched it, when it was first
160	seen and last seen, and a short plain-text excerpt — never the full description. The listing page says *"Read the
161	full listing on <source>"* and links there; that is where a reader applies.
162
163	Listings are upserted on `(source, source_id)` and expire — hidden from the search, the pages and the sitemap —
164	when no board has shown one for 30 days. An ingest never invents a listing: one without a title, a company, an id or
165	an http(s) URL to link back to is refused.
166
167	## Configuration
168
169	See [.env.example](.env.example). Required in production: `OV_OAUTH_CLIENT_SECRET` (the `work` OAuth client on the
170	Network), `BASE_URL`, `DATABASE_URL` and `DATABASE_DIRECT_URL`. The database is the only required readiness check; the
171	Network signing key, the OAuth client and Valkey are optional (the service says so, per check, on `/api/ready`).
172
173	The ingest is off unless `WORK_INGEST=on`, so a development run and the test suite never call a job board. When it
174	is on, `WORK_INGEST_INTERVAL_HOURS` (6), `WORK_INGEST_LIMIT` (100), `WORK_INGEST_TIMEOUT_MS` (15000),
175	`WORK_RETENTION_DAYS` (30), `WORK_REMOTIVE_URL` and `OV_SOURCES_INTERNAL_URL` decide what it does.
176
177	The agent pillar is driven by `WORK_ACTOR_URL` (default `http://127.0.0.1:4950`) — unset it (an empty value) and
178	`POST …/assist` answers 503 `work.actor_unavailable` — and `WORK_ACTOR_PUBLIC_URL` (default `https://openvibe.actor`),
179	the origin a `task_url` points at. `WORK_APPLICATIONS_MAX` (500) caps how many applications a person may track, and
180	`WORK_APPLICATIONS_ASSISTS_PER_DAY` (10) caps how often a person may ask their agent.
181
182	## Development
183
184	```bash
185	npm install
186	fnm exec --using=22 npm test # every test/*.test.js, on temp PGlite databases with a mock Network and stand-in boards
187	fnm exec --using=22 npm run dev # http://localhost:4960 (no ingest unless WORK_INGEST=on)
188	fnm exec --using=22 npm run ingest # read every board once, then exit
189	```
190
191	Without `DATABASE_URL` development uses an embedded PGlite database in `data/pglite` (one process only). `npm run
192	test:pg` runs the same suite through PostgreSQL and PgBouncer (see [.github/workflows/ci.yml](.github/workflows/ci.yml)).
193
194	## Acceptance
195
196	`npm test` runs every `test/*.test.js` in its own process on a temp PGlite database with a mock OpenVibe.Network and a
197	stand-in for each job board ([test/helpers/boards.js](test/helpers/boards.js)); `npm run test:pg` runs the same suite
198	through PostgreSQL and PgBouncer. No test reaches the internet. The main files:
199
200	- [test/ingest.test.js](test/ingest.test.js) — the three stand-in boards end to end: what is kept and what is refused,
201	a re-read updating rather than duplicating, a failing board recorded without stopping the others, the
202	~300-character excerpt only, and expiry after the retention window.
203	- [test/jobs.test.js](test/jobs.test.js) — search, filters, facets and the cursor, one listing with its provenance and
204	JSON-LD, escaping of a hostile title, and the discovery files that carry the listings and the sources.
205	- [test/saved.test.js](test/saved.test.js) — a saved search belongs to a person: anonymous is 401, an app token is
206	403, a signed-in write from another site is refused, and nobody sees anybody else's.
207	- [test/account-data.test.js](test/account-data.test.js) — ADR-033 export and deletion through `/internal/events` with
208	a stand-in Network: only that person's rows, applied once, a bad signature and a forwarded request refused.
209	- [test/auth-ops.test.js](test/auth-ops.test.js) — PKCE sign-in, truthful readiness, `/release.json`, loopback-only
210	`/metrics` and pages useful without JavaScript.
211	- [test/auth-jwks.test.js](test/auth-jwks.test.js) — the Network signing key fetched and verified, an outage survived
212	on cached keys, and a rotation honoured.
213	- [test/caller-limits.test.js](test/caller-limits.test.js) — per-caller limits, the 429 `rate_limited` before any
214	work, and the product's own route budgets.
215	- and the rest: [test/security-session.test.js](test/security-session.test.js),
216	[test/security-secrets.test.js](test/security-secrets.test.js), [test/discovery.test.js](test/discovery.test.js),
217	[test/layout.test.js](test/layout.test.js), [test/open-redirect.test.js](test/open-redirect.test.js),
218	[test/no-internal-key.test.js](test/no-internal-key.test.js),
219	[test/nginx-auth-limit.test.js](test/nginx-auth-limit.test.js),
220	[test/asset-cache.test.js](test/asset-cache.test.js), [test/perf-budget.test.js](test/perf-budget.test.js),
221	[test/service-kit.test.js](test/service-kit.test.js).
222
223	## Deploy (for the lead)
224
225	- Deploy: `sudo ovhost deploy work` on the host (git checkout at `/opt/openvibe.work`, unit
226	`openvibe-work.service` on 127.0.0.1:4960, env `/etc/openvibe/work.env`, database `ov_work` on the data role).
227	Set `WORK_INGEST=on` in `/etc/openvibe/work.env` or no listing will ever arrive.
228	- nginx: [deploy/nginx/openvibe.work.conf](deploy/nginx/openvibe.work.conf), installed with `ov-vhost-install`.
229	- Rollback: ovhost puts the previous sha back by itself when `/api/ready` does not answer after the restart.
230	- Register the service and its capabilities in OpenVibe.Contracts (`contracts-service: work` in CI) and with
231	OpenVibe.Services before the first deploy.
232
233	## Account export and deletion
234
235	A person's account at OpenVibe.Network can be exported and deleted, and every service holding their rows answers its
236	part (ADR-033). Work receives `network.account.export_requested` and `network.account.deleted` at `POST /internal/events`
237	(loopback only) — the person's tables are mapped in [server/identity/account-data.js](server/identity/account-data.js),
238	and the boot-time subscriptions are created by [server/events-consumer.js](server/events-consumer.js):
239
240	- Exported: the searches a person saved (`saved_searches.json`), the applications they tracked (`applications.json`)
241	and each application's status history (`application_events.json`), pushed to
242	`POST /internal/account-exports/:id/parts` with this service's own token. Nothing here is a secret — Work stores no
243	token, key or credential.
244	- Erased: a saved search and an application are the person's own and nothing anyone else's page hangs under them,
245	so they are deleted whole (an application's events go with it) and nothing is kept. Work then confirms with
246	`POST /internal/account-deletions/:id/confirmations` and the counts.
247	- Anonymized: nothing. There is no row Work keeps that was written by this person for another person to read.
248
249	The listings (`work_listings`) and the per-source fetch record (`work_source_fetches`) hold no person's rows: neither
250	has a subject column, so neither is exported nor erased.
251
252	Environment: `WORK_EVENTS_SECRET` (comma-separated for rotation, 32+ characters each; unset makes the route answer
253	503), `WORK_EVENTS_URL` (or `EVENTS_URL`) is where the two subscriptions are created at boot (off when unset), and
254	`WORK_EVENTS_ENDPOINT` overrides the loopback endpoint; `WORK_EVENTS_SUBSCRIBE=0` turns the boot-time subscription off.
255
256	## Security (threat notes)
257
258	Reporting a vulnerability: [SECURITY.md](SECURITY.md).
259
260	- Session tokens are httpOnly cookies; a FedCM assertion or an app or service token is never a session.
261	- Secrets live only in the env file; only environment variable names appear in code and docs, and no secret is logged.
262	- Request bodies are never logged.
263	- Nothing in a request may decide a URL this service fetches: the ingest fetches only the three base URLs from the
264	configuration, with this service's User-Agent, a timeout and a size cap, and stores only http(s) listing URLs.
265	- A board's text is data, never markup: every value goes through the `html` tagged template, and a listing URL that
266	is not http(s) is refused rather than linked.
267
268	## Not yet
269
270	- Approval at each step: the agent help is one-shot today — Work posts a task and hands back its link. An approval
271	loop (the agent proposing an action and the person approving it) comes with the rest of the agent pillar.
272	- More boards, and a per-source schedule finer than "the same six hours for everyone".
273
274	---
275
276	Part of the [OpenVibe network](https://openvibe.network). Built in the open by [OpenVibers](https://github.com/OpenVibers).
277
278	<!-- versions:start -->
279	- openvibe-contracts: v0.419.0
280	- openvibe-sdk: v0.161.0
281	- openvibe-shared: v4.241.0
282	- openvibe-publishing: v1.7.0
283	<!-- versions:end -->
284