openvibers/openvibe.link Commits
Plan T14, follow-up 3 of the Node gap audit… (#12)
Changes
@@@ -24,7 +24,12 @@ jobs:- run: pip install av numpy- run: test -z "$(gofmt -l .)" || (gofmt -l . && exit 1)- run: go vet ./...++ # The job worker isolates jobs in unprivileged user and network namespaces, which Ubuntu's AppArmor restricts by++ # default; its tests skip where they cannot run, and OPENVIBE_WORKER_TESTS=require makes them fail here instead.++ - run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true- run: go test -race -count=1 ./...++ env:++ OPENVIBE_WORKER_TESTS: requirebuild:name: cross-compile@@@ -180,9 +180,10 @@ Commands are never queued while offline and never replayed after a reconnect.### Jobs--The job frames of OpenVibe.Contracts `platform.job-frame@1` (plan T14 Run) ride the same link and envelope.--**Running jobs is not implemented yet:** the Node advertises no runtime class, so it refuses every well-formed job--with `class not available`, and nothing executes.++The job frames of OpenVibe.Contracts `platform.job-frame@1` (plan T14 Run) ride the same link and envelope. The Node++runs class `function` only, and only when the owner turned the worker on in the local config, the OS is Linux and a++boot-time probe could start a process in its own user, network and PID namespaces ([worker.md](worker.md)).++Otherwise it advertises no class and refuses every well-formed job with `class not available`, and nothing executes.| direction | type | fields ||------------------|----------------|-------------------------------------------------------------------------------------------|@@@ -205,11 +206,34 @@ checked in this order:| `function job needs an artifact with an exact version` | `bad_value` | a `function` job without `artifact`, or a name or version outside the contract's pattern (a range) || `net policy not supported` | `unsupported` | `net` is present and not `deny` || `missing or invalid args, ttl_ms or limits` | `bad_value` | `args` is not an object, or `ttl_ms` or a limit is missing or below 1 |--| `class not available` | `unsupported` | the class is valid but this Node does not run it (today: every class) |----A resent `job` with an id already answered gets the same answer again, never a second acceptance; the Node--remembers the last 1024 well-formed ids. `job_cancel` for an id that is not running (unknown, refused or ended) and--`job_exit_ack` for an unknown id are ignored, with no answer: an `ack` keyed by a job id means the job was accepted.++| `class not available` | `unsupported` | the class is valid but this Node does not run it (every class but `function`; every class while the worker is off) |++| `the e-stop or local stop is latched` | `estopped` or `local_stop` | the stop latch is set: no job starts until it clears |++| `class not available` | `shutting_down` | the Node is stopping |++| `unknown artifact` | `unsupported` | `artifact` is not a function the owner declared, at that exact version |++| `worker busy` | `not_ready` | `worker.caps.max_jobs` jobs are already running |++++A resent `job` with an id already answered gets the same answer again, never a second acceptance and never a second++process: `ack` while it runs, its `job_exit` once it ended; the Node remembers the last 1024 well-formed ids.++++An accepted job runs once the `ack` is out: `job_started` (`started_ms` anchors every usage second), its stdout as++`job_stdout` chunks (at most 16 KiB each, UTF-8, `chunk_seq` from 1; best effort, not resent), one `job_usage` for++every wall-clock second once it has fully elapsed (`second` from 0, `cpu_ms` that second's CPU time), then++`job_exit`. Its `usage` is authoritative for metering (Bot writes `usage-sample` readings `service:"run"`,++`operation:"function.invoke"`, unit `s`, idempotency key `run:<job id>:<n>`; none of those fields changes on a++resend). The Node resends `job_started` for running jobs and `job_exit` for ended ones after every reconnect, and++`job_exit` again on a `job` or `job_cancel` for that id, until `job_exit_ack`. `job_exit.reason`:++++| `reason` | `code` | when |++|-------------|---------|-----------------------------------------------------------------------------------------------------|++| `exited` | status | the process ended by itself (`code` is null if a signal it did not get from the worker ended it); `result` is what it wrote to fd 3 when it exited 0 |++| `cancelled` | null | `job_cancel`: the process group is killed; a job cancelled before it started never starts |++| `ttl` | null | `ttl_ms` (from receipt, clamped to `worker.caps.max_ttl_ms`) ran out, before or while it ran |++| `limit` | null | `wall_ms`, `cpu_ms` or `mem_bytes` (each clamped to the local cap), more stdout than `max_output_bytes`, or more than 256 KiB of result |++| `stopped` | null | the e-stop or the local stop latched, or the Node shut down |++| `failed` | null | the worker could not run it: the namespaces could not be created, the spawn failed, or its CPU limit could not be set |++++`job_cancel` for an id that is not running (unknown, refused or ended) and `job_exit_ack` for an unknown id are++ignored, with no answer: an `ack` keyed by a job id means the job was accepted.A Node that runs jobs lists its classes in `status.capabilities.worker` as `{"runtime_classes": ["function"]}`; thekey is absent while it runs none, and `worker` is reserved as a plugin name.nnew file mode 100644@@@ -0,0 +1,123 @@++# The job worker (`function` jobs)++++The Node can run **function jobs** (OpenVibe.Contracts `platform.job@1`, class `function`) sent over the control link++([protocol.md](protocol.md#jobs)). A job is untrusted input: it names a function, and the Node runs it only if the++owner declared that function, at that exact version, in the local config. Nothing is downloaded and no path comes from++the server. The worker is **off by default**; off, the Node refuses every job `class not available`, as before.++++## Config++++`config.json`, section `worker` (defaults shown; a zero cap means the default):++++```json++"worker": {++ "enabled": false,++ "functions": [++ {"name": "thumbnail", "version": "1.2.0", "command": ["/opt/fn/thumbnail/bin/run"], "env": {"MODE": "fast"}}++ ],++ "run_as": {"uid": 2001, "gid": 2001},++ "allow_same_user": false,++ "caps": {++ "max_ttl_ms": 600000,++ "max_wall_ms": 300000,++ "max_cpu_ms": 300000,++ "max_mem_bytes": 536870912,++ "max_output_bytes": 1048576,++ "max_jobs": 1++ }++}++```++++- `functions`: `name` and `version` match a job's `artifact` exactly (the contract's patterns; listed once each);++ `command[0]` is an absolute path. `env` is added to the job's environment.++- `run_as`: the unprivileged uid and gid jobs run as (not 0; no supplementary groups). It needs the Node to run as++ root, which the system install does. Create a dedicated user for it with no login and no groups.++- `allow_same_user`: run jobs as the Node's own (non-root) user instead. A job can then read every file the Node can,++ `credential.json` included, and use its groups (`gpio`, `dialout`, `video`): **for development only**. With neither++ `run_as` nor `allow_same_user` the worker stays off.++- `caps`: the owner's local caps. A job's `ttl_ms`, `limits.wall_ms`, `limits.cpu_ms` and `limits.mem_bytes` are++ each clamped to them (the stricter value wins, as with `max_command_ms`). `max_output_bytes` caps one job's stdout;++ `max_jobs` how many run at once.++++The class `function` is advertised in `status.capabilities.worker` only when `enabled` is true, the OS is Linux and a++**boot-time probe** passes: it starts `/bin/sh` the way a job is started and checks that the process got user,++network and PID namespaces of its own, that its CPU limit can be set and that its CPU and memory can be read. If the++probe fails the Node logs why (`worker off: …`) and refuses every job, exactly as with the worker off. On Ubuntu 24.04++and later, unprivileged user namespaces are restricted by AppArmor; a Node running as root with `run_as` is not++affected.++++## How a job runs++++1. **Admit** (before the `ack`): refused unless the stop latch is clear, the artifact is declared and fewer than++ `max_jobs` jobs run. An id the worker already holds is answered from it (`ack` while it runs, its `job_exit` once++ it ended) and never starts a second process.++2. **Start**: a fresh working directory (`$TMPDIR/openvibe-job-*`, owned by `run_as`, removed afterwards); an++ environment built from scratch (`PATH=/usr/local/bin:/usr/bin:/bin`, `HOME` and `TMPDIR` the working directory,++ `LANG=C.UTF-8`, `OPENVIBE_JOB_ID`, and the function's `env`; nothing inherited from the Node); `args` as one line of++ JSON on stdin; stderr discarded. The process starts in its own process group and in new user, network and PID++ namespaces, with `PR_SET_PDEATHSIG` SIGKILL, then gets `RLIMIT_CPU` (at `cpu_ms`, rounded up to a second) and++ `RLIMIT_CORE` 0. `job_started` follows.++3. **Run**: stdout streams as `job_stdout`; a watchdog samples CPU time and resident memory of every process in the++ job's PID namespace every 200 ms and sends `job_usage` for every second once it fully elapsed.++4. **End**: the function's result is what it writes to **fd 3** (JSON, at most 256 KiB), reported only when it exits++ 0. `job_exit` carries the reason, the exit code (null when it was killed) and the usage, and is resent until++ `job_exit_ack`.++++Every kill is SIGKILL to the whole process group and to the leader, which is PID 1 of the job's PID namespace, so its++death also kills every process the job left behind. The watchdog kills at `ttl_ms` (from receipt), `wall_ms` (from++start), `cpu_ms`, `mem_bytes`, at `max_output_bytes` of stdout, on `job_cancel`, and the moment the e-stop or the local++stop latches (`openvibe-node stop`); while the latch is set every job is refused. A Node shutting down kills its jobs++(`stopped`).++++## Isolation model and its limits++++In five lines:++++1. Only owner-declared executables run, matched by exact name and version; the server chooses which and with what JSON.++2. Each job is a separate unprivileged process (`run_as`) in its own process group and user, network and PID namespaces.++3. The network namespace has only a loopback that is down: no IP network, not even to the Node or `localhost`.++4. Nothing of the Node's environment reaches it; its working directory is fresh and removed after the job.++5. ttl, wall, CPU, memory and output are enforced by a per-job watchdog that SIGKILLs the whole group; `RLIMIT_CPU` backs the CPU cap in the kernel.++++What it does **not** protect against (no seccomp, no cgroups, no mount namespace yet):++++- **Files**: there is no filesystem isolation. A job reads and writes whatever its uid may: with `run_as`, every++ world-readable file and world-writable directory; with `allow_same_user`, everything the Node can, including++ `credential.json` and the local control socket (so it could clear the local stop).++- **Unix sockets by path** (Docker's, D-Bus's, the Node's control socket) are reachable: a network namespace does not++ cover filesystem sockets; only their file permissions do.++- **The kernel**: no seccomp filter, so the whole syscall surface is open, including what user namespaces expose to++ unprivileged code (a common source of local privilege escalations). Keep the kernel patched.++- **Memory and CPU are sampled**, not enforced by cgroups: a burst between two 200 ms samples can exceed `mem_bytes`++ (and meet the kernel's OOM killer first); the `RLIMIT_CPU` backstop is set just after start and covers each process++ on its own, not their sum.++- **No limit on processes, threads, disk or I/O**: a fork bomb is bounded only by the uid's `RLIMIT_NPROC`, disk use++ in the working directory or `/tmp` only by free space; `/tmp` and `/dev/shm` are shared with the machine.++- **`/proc` is the host's**: a job can list the machine's processes and their command lines (not signal them: its PID++ namespace has no host PIDs).++- **The `args` and the result are trusted to the function**: the worker passes the server's JSON unchanged; validating++ it is the function's job.++++## Refusals and exit reasons++++Refused with `nack` (keyed by the job id; the first matching row wins, after the checks every Node does, in++[protocol.md](protocol.md#jobs)):++++| `message` | `fault_code` | when |++|---------------------------------------|----------------------------|---------------------------------------------------------|++| `class not available` | `unsupported` | the worker is off (disabled, not Linux, probe failed), or the class is not `function` |++| `the e-stop or local stop is latched` | `estopped` or `local_stop` | the stop latch is set |++| `class not available` | `shutting_down` | the Node is stopping |++| `unknown artifact` | `unsupported` | no declared function has that name and exact version |++| `worker busy` | `not_ready` | `max_jobs` jobs are running |++++Accepted jobs end with `job_exit.reason`:++++| `reason` | when |++|-------------|-----------------------------------------------------------------------------------------------------------|++| `exited` | the process ended by itself; `code` is its status (null if a signal the worker did not send ended it) |++| `cancelled` | `job_cancel`; a job cancelled before it started never starts |++| `ttl` | `ttl_ms` (clamped) ran out, before or while it ran |++| `limit` | `wall_ms`, `cpu_ms` (sampled, or `RLIMIT_CPU`'s SIGXCPU) or `mem_bytes` exceeded, stdout past `max_output_bytes`, or a result over 256 KiB |++| `stopped` | the e-stop or local stop latched, or the Node shut down |++| `failed` | the job could not run isolated: namespaces not created, spawn error, working directory not made, CPU limit not set; never run with less isolation |@@@ -17,6 +17,7 @@ import ("io/fs""os""path/filepath"++ "regexp""runtime""strings")@@@ -106,6 +107,66 @@ type Config struct {// Python is the interpreter for plugins without an explicit command (default: <state>/venv/bin/python, then python3).Python string `json:"python,omitempty"`LogLevel string `json:"log_level,omitempty"`++ // Worker runs `function` jobs from the control link (docs/worker.md). Off unless enabled.++ Worker WorkerConfig `json:"worker,omitzero"`++}++++// WorkerConfig is the job worker. Disabled (the default), every job is refused "class not available". Only the++// functions listed here ever run: a job names one by name and exact version, and nothing is downloaded.++type WorkerConfig struct {++ Enabled bool `json:"enabled,omitempty"`++ Functions []FunctionConfig `json:"functions,omitempty"`++ // RunAs is the dedicated unprivileged uid/gid jobs run as; it needs the Node to run as root. Without it the Node++ // refuses to run jobs unless AllowSameUser is set.++ RunAs *RunAs `json:"run_as,omitempty"`++ // AllowSameUser runs jobs as the Node's own user (no run_as). A job can then read every file the Node can,++ // including credential.json, and use the Node user's groups (gpio, dialout, video): for development only.++ AllowSameUser bool `json:"allow_same_user,omitempty"`++ Caps WorkerCaps `json:"caps,omitzero"`++}++++// FunctionConfig is one function a job may run: Command is started with the job's args as JSON on stdin.++type FunctionConfig struct {++ Name string `json:"name"`++ Version string `json:"version"`++ Command []string `json:"command"` // Command[0] is an absolute path++ Env map[string]string `json:"env,omitempty"`++}++++// RunAs is a uid and gid on this machine.++type RunAs struct {++ UID uint32 `json:"uid"`++ GID uint32 `json:"gid"`++}++++// WorkerCaps are the owner's local caps on every job; a job's own ttl_ms and limits are clamped to them (the++// stricter value wins). Zero means the default (DefaultWorkerCaps).++type WorkerCaps struct {++ MaxTTLMS int64 `json:"max_ttl_ms,omitempty"`++ MaxWallMS int64 `json:"max_wall_ms,omitempty"`++ MaxCPUMS int64 `json:"max_cpu_ms,omitempty"`++ MaxMemBytes int64 `json:"max_mem_bytes,omitempty"`++ MaxOutputBytes int64 `json:"max_output_bytes,omitempty"` // stdout of one job, in bytes++ MaxJobs int `json:"max_jobs,omitempty"` // jobs running at once++}++++// DefaultWorkerCaps are the caps for every field the config leaves at zero.++var DefaultWorkerCaps = WorkerCaps{MaxTTLMS: 600000, MaxWallMS: 300000, MaxCPUMS: 300000, MaxMemBytes: 512 << 20,++ MaxOutputBytes: 1 << 20, MaxJobs: 1}++++// WithDefaults fills the zero fields of c from DefaultWorkerCaps.++func (c WorkerCaps) WithDefaults() WorkerCaps {++ d := DefaultWorkerCaps++ for _, f := range []struct{ v, def *int64 }{{&c.MaxTTLMS, &d.MaxTTLMS}, {&c.MaxWallMS, &d.MaxWallMS},++ {&c.MaxCPUMS, &d.MaxCPUMS}, {&c.MaxMemBytes, &d.MaxMemBytes}, {&c.MaxOutputBytes, &d.MaxOutputBytes}} {++ if *f.v <= 0 {++ *f.v = *f.def++ }++ }++ if c.MaxJobs <= 0 {++ c.MaxJobs = d.MaxJobs++ }++ return c}type LocalLimits struct {@@@ -239,6 +300,35 @@ func (c *Config) Validate() error {return errors.New("config: limits must be between 0 and 1")}}++ return c.Worker.validate()++}++++var (++ functionNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,127}$`)++ functionVersionRe = regexp.MustCompile(`^[0-9A-Za-z][0-9A-Za-z.+-]{0,63}$`)++)++++func (w WorkerConfig) validate() error {++ seen := map[string]bool{}++ for _, f := range w.Functions {++ if !functionNameRe.MatchString(f.Name) || !functionVersionRe.MatchString(f.Version) {++ return fmt.Errorf("config: worker function %q version %q: name or version is not a valid artifact name or exact version", f.Name, f.Version)++ }++ if seen[f.Name+"@"+f.Version] {++ return fmt.Errorf("config: worker function %s@%s listed twice", f.Name, f.Version)++ }++ seen[f.Name+"@"+f.Version] = true++ if len(f.Command) == 0 || !filepath.IsAbs(f.Command[0]) {++ return fmt.Errorf("config: worker function %s@%s needs a command starting with an absolute path", f.Name, f.Version)++ }++ }++ if w.RunAs != nil && (w.RunAs.UID == 0 || w.RunAs.GID == 0) {++ return errors.New("config: worker.run_as must not be root (uid or gid 0)")++ }++ c := w.Caps++ if c.MaxTTLMS < 0 || c.MaxWallMS < 0 || c.MaxCPUMS < 0 || c.MaxMemBytes < 0 || c.MaxOutputBytes < 0 || c.MaxJobs < 0 {++ return errors.New("config: worker.caps must not be negative")++ }return nil}@@@ -4,18 +4,21 @@ import ("sync""github.com/OpenVibers/OpenVibe.Node/internal/protocol"++ "github.com/OpenVibers/OpenVibe.Node/internal/safety")// jobSeenMax bounds how many job ids the Node remembers answers for.const jobSeenMax = 1024--// jobs answers job frames. Nothing runs yet: a job is refused unless its class is advertised, and an accepted job is--// only remembered (the worker that runs it fills classes and executes accepted jobs).++// jobs answers job frames: a job is refused unless its class is advertised and admit (the worker) takes it. Running++// jobs live in the worker's own table, not here.type jobs struct {mu sync.Mutexclasses []string // runtime classes this Node runs and advertises; empty: none-- answers map[string]protocol.Message // job id → the ack or nack it got, resent unchanged for a repeated job++ answers map[string]protocol.Message // job id → the ack or nack it got, or its job_exit once it endedorder []string // ids in answers, oldest first++ // admit reserves an accepted job in the worker or returns the fault and reason to nack it with; nil accepts.++ admit func(protocol.Job) (fault, reason string)}func newJobs(classes []string) *jobs {@@@ -38,6 +41,9 @@ func (j *jobs) request(r protocol.JobRequest) (answer protocol.Message, acceptedreturn a, false}fault, reason := r.Check(j.classes)++ if fault == "" && j.admit != nil {++ fault, reason = j.admit(r.Job)++ }if fault != "" {answer = protocol.Nack{ID: id, FaultCode: fault, Message: reason}} else {@@@ -55,6 +61,15 @@ func (j *jobs) request(r protocol.JobRequest) (answer protocol.Message, acceptedreturn answer, accepted}++// ended makes a job's job_exit the answer to a resent job frame for it.++func (j *jobs) ended(e protocol.JobExit) {++ j.mu.Lock()++ defer j.mu.Unlock()++ if _, ok := j.answers[e.ID]; ok {++ j.answers[e.ID] = e++ }++}++// accepted reports whether id was accepted (and so would be running).func (j *jobs) accepted(id string) bool {j.mu.Lock()@@@ -63,8 +78,15 @@ func (j *jobs) accepted(id string) bool {return ok}--// job answers a job frame with ack or nack, keyed by the job id.++// job answers a job frame with ack or nack, keyed by the job id, and runs an accepted job. A job the worker holds is++// answered from it (an ack while it runs, its job_exit once it ended) and never started twice.func (n *Node) job(r protocol.JobRequest) {++ if n.worker != nil {++ if m, ok := n.worker.Known(r.Job.ID); ok {++ n.send(m)++ return++ }++ }answer, accepted := n.jobs.request(r)if nk, ok := answer.(protocol.Nack); ok {n.log.Info("job refused", "id", r.Job.ID, "class", r.Job.Class, "reason", nk.Message, "err", r.Err)@@@ -72,14 +94,42 @@ func (n *Node) job(r protocol.JobRequest) {n.log.Info("job accepted", "id", r.Job.ID, "class", r.Job.Class)}n.send(answer)++ if accepted && n.worker != nil {++ n.worker.Launch(r.Job.ID)++ }++}++++// sendJobFrame sends a worker frame; a job_exit also becomes the answer to a resent job frame.++func (n *Node) sendJobFrame(m protocol.Message) {++ if e, ok := m.(protocol.JobExit); ok {++ n.jobs.ended(e)++ }++ n.send(m)}// jobCancel stops an accepted job. An unknown, refused or ended id is ignored, as platform.job-frame@1 says, and no--// frame answers it: an ack keyed by a job id means the job was accepted.++// frame answers it: an ack keyed by a job id means the job was accepted. An ended job's unacked job_exit is resent.func (n *Node) jobCancel(c protocol.JobCancel) {++ if n.worker != nil {++ if !n.worker.Cancel(c.ID) {++ n.log.Debug("job_cancel for a job that is not running ignored", "id", c.ID)++ }++ return++ }if !n.jobs.accepted(c.ID) {n.log.Debug("job_cancel for a job that is not running ignored", "id", c.ID)return}-- n.log.Warn("job_cancel: job execution is not implemented, nothing to stop", "id", c.ID)++ n.log.Warn("job_cancel: no worker runs jobs on this Node, nothing to stop", "id", c.ID)++}++++// stopFault is the fault a job is refused with while the stop latch is set; "" when it is clear.++func stopFault(st safety.LatchState) string {++ switch {++ case st.Local:++ return protocol.FaultLocalStop++ case st.Remote:++ return protocol.FaultEstopped++ }++ return ""}@@@ -19,6 +19,7 @@ import ("os""os/exec""runtime"++ "slices""sort""strings""sync"@@@ -32,6 +33,7 @@ import ("github.com/OpenVibers/OpenVibe.Node/internal/protocol""github.com/OpenVibers/OpenVibe.Node/internal/safety""github.com/OpenVibers/OpenVibe.Node/internal/video"++ "github.com/OpenVibers/OpenVibe.Node/internal/worker")// Options configure a Node.@@@ -42,8 +44,9 @@ type Options struct {Log *slog.LoggerVersion string-- // RuntimeClasses are the job classes this Node runs and advertises in status.capabilities.worker. None yet:-- // every job is refused "class not available" until the worker that executes jobs fills this in.++ // RuntimeClasses are the job classes this Node runs and advertises in status.capabilities.worker. New adds++ // `function` when config.worker is enabled and its isolation probe passes; with none, every job is refused++ // "class not available".RuntimeClasses []string// For tests.@@@ -60,9 +63,11 @@ type Node struct {latch *safety.Latchdedup *safety.Dedupjobs *jobs-- mgr *plugins.Manager-- link *link.Link-- pub *video.Publisher++ // worker runs function jobs; nil unless config.worker is enabled on Linux and its boot-time probe passed.++ worker *worker.Worker++ mgr *plugins.Manager++ link *link.Link++ pub *video.Publisherctx context.Context@@@ -149,6 +154,20 @@ func New(opt Options) (*Node, error) {ICEServers: opt.Creds.ICEServers, Log: opt.Log, IncludeLoopback: opt.VideoLoopback})}}++ if opt.Config.Worker.Enabled {++ w := worker.New(opt.Config.Worker, n.sendJobFrame, opt.Log)++ if err := w.Probe(); err != nil {++ opt.Log.Error("worker off: jobs cannot run isolated here, every job is refused", "err", err)++ } else {++ n.worker = w++ w.SetStopped(stopFault(n.latch.State()))++ n.jobs.admit = w.Admit++ if !slices.Contains(n.jobs.classes, protocol.ClassFunction) {++ n.jobs.classes = append(n.jobs.classes, protocol.ClassFunction)++ }++ opt.Log.Info("worker on", "functions", len(opt.Config.Worker.Functions))++ }++ }n.latch.OnChange(n.onLatch)return n, nil}@@@ -188,6 +207,9 @@ func (n *Node) Run(ctx context.Context) error {<-ctx.Done()n.log.Info("shutting down: stopping every actuator")n.mgr.StopAll()++ if n.worker != nil {++ n.worker.Close()++ }cancelPlugins()n.mgr.Wait()wg.Wait()@@@ -236,8 +258,11 @@ func (n *Node) Frame(f protocol.Frame) {case protocol.JobCancel:n.jobCancel(m)case protocol.JobExitAck:-- // No job runs yet, so no job_exit is ever waiting for this.-- n.log.Debug("job_exit_ack ignored", "id", m.ID)++ if n.worker != nil {++ n.worker.ExitAck(m.ID)++ } else {++ n.log.Debug("job_exit_ack ignored: no worker", "id", m.ID)++ }}}@@@ -278,6 +303,9 @@ func (n *Node) applyConfig(c protocol.Config) {n.mu.Unlock()if first {n.sendStatus()++ if n.worker != nil {++ n.worker.Resend()++ }}n.sendEstopState()if n.link != nil {@@@ -428,6 +456,9 @@ func (n *Node) send(m protocol.Message) {// ---- latch ----func (n *Node) onLatch(st safety.LatchState) {++ if n.worker != nil {++ n.worker.SetStopped(stopFault(st)) // kills every job at once while stopped++ }if st.Stopped() {n.log.Warn("stop latched", "remote", st.Remote, "local", st.Local)n.mgr.EstopAll()nnew file mode 100644@@@ -0,0 +1,195 @@++//go:build linux++++package node++++import (++ "context"++ "encoding/json"++ "fmt"++ "io"++ "log/slog"++ "os"++ "path/filepath"++ "slices"++ "strings"++ "testing"++ "time"++++ "github.com/OpenVibers/OpenVibe.Node/internal/config"++ "github.com/OpenVibers/OpenVibe.Node/internal/fakebot"++ "github.com/OpenVibers/OpenVibe.Node/internal/link"++ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"++)++++// TestWorkerHelperProcess is the job's process when this test binary runs as the `hello` function (`-- job`): it++// appends a line to the marker file named in its args, prints, holds for 1.5 s and writes its result to fd 3.++func TestWorkerHelperProcess(t *testing.T) {++ i := slices.Index(os.Args, "--")++ if i < 0 || i+1 >= len(os.Args) || os.Args[i+1] != "job" {++ return++ }++ var args struct {++ Marker string `json:"marker"`++ }++ _ = json.NewDecoder(os.Stdin).Decode(&args)++ if f, err := os.OpenFile(args.Marker, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600); err == nil {++ fmt.Fprintln(f, "started")++ f.Close()++ }++ fmt.Println("hello from the job")++ time.Sleep(1500 * time.Millisecond)++ fmt.Fprint(os.NewFile(3, "result"), `{"ok":true}`)++ os.Exit(0)++}++++// startWorkerNode pairs a Node with no plugin and no video whose worker runs the `hello` function, and waits for the++// status that advertises it. It is skipped where the kernel does not let this user create the job namespaces.++func startWorkerNode(t *testing.T) *env {++ t.Helper()++ exe, err := os.Executable()++ if err != nil {++ t.Fatal(err)++ }++ home, err := os.MkdirTemp("", "ovn")++ if err != nil {++ t.Fatal(err)++ }++ t.Cleanup(func() { os.RemoveAll(home) })++ paths := config.HomePaths(home)++ srv := fakebot.New()++ t.Cleanup(srv.Close)++ srv.Config = protocol.Config{HeartbeatMS: 100, Limits: protocol.Limits{MaxCommandMS: 1000}}++ srv.AddCode("TEST2345")++ creds, err := link.Pair(context.Background(), nil, srv.URL(), protocol.PairRequest{Code: "TEST2345", AgentVersion: "test",++ DeviceKind: "onboard"})++ if err != nil {++ t.Fatal(err)++ }++ cfg := config.Default()++ cfg.Plugins = nil++ cfg.Video = config.VideoConfig{Source: "off"}++ cfg.Worker = config.WorkerConfig{Enabled: true, AllowSameUser: true, Caps: config.WorkerCaps{MaxMemBytes: 8 << 30},++ Functions: []config.FunctionConfig{{Name: "hello", Version: "1.0.0", Command: []string{exe, "-test.run=^TestWorkerHelperProcess$", "--", "job"}}}}++ if err := cfg.Validate(); err != nil {++ t.Fatal(err)++ }++ n, err := New(Options{Config: cfg, Paths: paths, Creds: creds, Version: "test", HeartbeatInterval: 100 * time.Millisecond,++ LinkBackoffMin: 50 * time.Millisecond, LatchPoll: 50 * time.Millisecond, Log: slog.New(slog.NewTextHandler(io.Discard, nil))})++ if err != nil {++ t.Fatal(err)++ }++ if !slices.Contains(n.jobs.runtimeClasses(), protocol.ClassFunction) {++ if os.Getenv("OPENVIBE_WORKER_TESTS") == "require" {++ t.Fatal("job isolation unavailable: the worker probe failed")++ }++ t.Skip("job isolation unavailable here (OPENVIBE_WORKER_TESTS=require fails instead)")++ }++ ctx, cancel := context.WithCancel(context.Background())++ done := make(chan struct{})++ go func() { n.Run(ctx); close(done) }()++ e := &env{t: t, srv: srv, node: n, paths: paths, cancel: cancel, done: done}++ t.Cleanup(e.stop)++ e.conn = e.nextConn()++ _, err = e.conn.Expect(protocol.TypeStatus, 10*time.Second, func(m protocol.Message) bool {++ _, ok := m.(protocol.Status).Capabilities[protocol.CapWorker]++ return ok++ })++ if err != nil {++ t.Fatal(err)++ }++ return e++}++++// TestJobRunsOverTheLink: a job sent by the fake Bot is acked, then job_started, job_stdout, job_usage (second 0)++// and job_exit arrive in that order; a job frame resent while it runs is acked again and starts no second process;++// after the end a resent job is answered with the same job_exit, before and after job_exit_ack.++func TestJobRunsOverTheLink(t *testing.T) {++ e := startWorkerNode(t)++ marker := filepath.Join(e.paths.StateDir, "starts")++ j := testJob(jobID(1))++ j.Artifact = &protocol.Artifact{Name: "hello", Version: "1.0.0"}++ j.Args = json.RawMessage(fmt.Sprintf(`{"marker":%q}`, marker))++ j.Limits.MemBytes = 4 << 30++ isJob := func(m protocol.Message) bool {++ switch f := m.(type) {++ case protocol.JobStarted:++ return f.ID == j.ID++ case protocol.JobExit:++ return f.ID == j.ID++ }++ return false++ }++++ if r := e.jobReply(j); r != (protocol.Ack{ID: j.ID}) {++ t.Fatalf("%+v", r)++ }++ if _, err := e.conn.Expect(protocol.TypeJobStarted, 10*time.Second, isJob); err != nil {++ t.Fatal(err)++ }++ if r := e.jobReply(j); r != (protocol.Ack{ID: j.ID}) {++ t.Fatalf("resent while running: %+v", r)++ }++ f, err := e.conn.Expect(protocol.TypeJobExit, 15*time.Second, isJob)++ if err != nil {++ t.Fatal(err)++ }++ ex := f.Msg.(protocol.JobExit)++ if ex.Reason != protocol.ExitExited || ex.Code == nil || *ex.Code != 0 || string(ex.Result) != `{"ok":true}` || ex.Usage.StartedMS == nil {++ t.Fatalf("%+v %s", ex, ex.Result)++ }++++ // Order on the wire: ack, job_started, job_stdout, job_usage, job_exit; one job_started only.++ var kinds []string++ var out strings.Builder++ started := 0++ for _, f := range e.conn.Frames() {++ switch m := f.Msg.(type) {++ case protocol.JobStarted:++ if m.ID == j.ID {++ started++++ kinds = append(kinds, f.Type)++ }++ case protocol.JobStdout:++ if m.ID == j.ID {++ out.WriteString(m.Chunk)++ kinds = append(kinds, f.Type)++ }++ case protocol.JobUsage:++ if m.ID == j.ID && m.Second == 0 && m.StartedMS == *ex.Usage.StartedMS {++ kinds = append(kinds, f.Type)++ }++ case protocol.JobExit:++ if m.ID == j.ID {++ kinds = append(kinds, f.Type)++ }++ }++ }++ kinds = slices.Compact(kinds)++ want := []string{protocol.TypeJobStarted, protocol.TypeJobStdout, protocol.TypeJobUsage, protocol.TypeJobExit}++ if started != 1 || !slices.Equal(kinds, want) || out.String() != "hello from the job\n" {++ t.Fatalf("%d job_started, frames %v, stdout %q", started, kinds, out.String())++ }++++ // Ended: a resent job gets the same job_exit, before and after job_exit_ack, and never runs again.++ for _, ack := range []bool{false, true} {++ if ack {++ if err := e.conn.JobExitAck(j.ID); err != nil {++ t.Fatal(err)++ }++ }++ if err := e.conn.Job(j); err != nil {++ t.Fatal(err)++ }++ f, err := e.conn.Expect(protocol.TypeJobExit, 5*time.Second, isJob)++ if err != nil {++ t.Fatal(err)++ }++ if again := f.Msg.(protocol.JobExit); again.Reason != ex.Reason || *again.Usage.StartedMS != *ex.Usage.StartedMS || again.Usage.WallMS != ex.Usage.WallMS {++ t.Fatalf("resent job answered %+v, first job_exit %+v", again, ex)++ }++ }++ b, err := os.ReadFile(marker)++ if err != nil || string(b) != "started\n" {++ t.Fatalf("the function ran %q (%v), want once", b, err)++ }++}@@@ -97,6 +97,11 @@ const (JobNetUnsupported = "net policy not supported"JobBadLimits = "missing or invalid args, ttl_ms or limits"JobNotAvailable = "class not available"++++ // Refused by a running worker (platform.job@1: a missing artifact, the local kill switch, limits it cannot honor).++ JobUnknownArtifact = "unknown artifact" // not a function declared in the Node's local config++ JobStopped = "the e-stop or local stop is latched" // fault estopped or local_stop++ JobBusy = "worker busy" // worker.caps.max_jobs are running; fault not_ready)// job_exit reasons.nnew file mode 100644@@@ -0,0 +1,304 @@++//go:build linux++++package worker++++import (++ "bytes"++ "encoding/json"++ "errors"++ "fmt"++ "io"++ "io/fs"++ "net"++ "os"++ "path/filepath"++ "slices"++ "strings"++ "testing"++ "time"++++ "github.com/OpenVibers/OpenVibe.Node/internal/config"++ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"++)++++// TestHelperProcess is the job's process when this test binary runs as a function (`-- <mode>`); otherwise it does++// nothing.++func TestHelperProcess(t *testing.T) {++ i := slices.Index(os.Args, "--")++ if i < 0 || i+1 >= len(os.Args) {++ return++ }++ result := os.NewFile(3, "result")++ switch os.Args[i+1] {++ case "stream":++ in, _ := io.ReadAll(os.Stdin)++ fmt.Println("line 1")++ time.Sleep(300 * time.Millisecond)++ fmt.Println("line 2 é")++ time.Sleep(1900 * time.Millisecond)++ fmt.Fprintf(result, `{"got":%s}`, bytes.TrimSpace(in))++ case "sleep":++ fmt.Println("sleeping")++ time.Sleep(time.Hour)++ case "flood":++ line := strings.Repeat("x", 1023) + "\n"++ for {++ os.Stdout.WriteString(line)++ }++ case "env":++ cwd, _ := os.Getwd()++ // A down loopback still lets a socket bind 127.0.0.1; connecting to it is what fails.++ lerr := errors.New("no listener")++ if l, err := net.Listen("tcp", "127.0.0.1:0"); err == nil {++ _, lerr = net.DialTimeout("tcp", l.Addr().String(), time.Second)++ }++ _, derr := net.DialTimeout("tcp", "192.0.2.1:9", time.Second)++ _ = json.NewEncoder(result).Encode(map[string]any{"env": os.Environ(), "cwd": cwd, "pid": os.Getpid(),++ "uid": os.Getuid(), "loopback": fmt.Sprint(lerr), "dial": fmt.Sprint(derr)})++ }++ os.Exit(0)++}++++// newRunWorker is a worker running the helper modes, skipped where the kernel does not let this user create the++// namespaces (CI sets OPENVIBE_WORKER_TESTS=require to fail instead).++func newRunWorker(t *testing.T, caps config.WorkerCaps, modes ...string) (*Worker, *sink) {++ t.Helper()++ if caps.MaxMemBytes == 0 {++ caps.MaxMemBytes = 8 << 30++ }++ s := &sink{}++ w := New(helperConfig(t, caps, modes...), s.send, quiet())++ if err := w.Probe(); err != nil {++ if os.Getenv("OPENVIBE_WORKER_TESTS") == "require" {++ t.Fatalf("job isolation unavailable: %v", err)++ }++ t.Skipf("job isolation unavailable here (OPENVIBE_WORKER_TESTS=require fails instead): %v", err)++ }++ t.Cleanup(w.Close)++ return w, s++}++++func run(t *testing.T, w *Worker, j protocol.Job) {++ t.Helper()++ if f, r := w.Admit(j); f != "" {++ t.Fatalf("%s refused: %s %s", j.ID, f, r)++ }++ w.Launch(j.ID)++}++++// TestRunStreamsAndMeters: a function gets its args on stdin, its stdout arrives as job_stdout in order, one++// job_usage per elapsed wall-clock second anchored on job_started, and job_exit carries code 0, the result it wrote to++// fd 3 and usage covering every second sent. The job_exit is resent until job_exit_ack, then the job is forgotten.++func TestRunStreamsAndMeters(t *testing.T) {++ w, s := newRunWorker(t, config.WorkerCaps{}, "stream")++ j := testJob(1, "stream")++ run(t, w, j)++ ex := s.exit(t, j.ID)++ if ex.Reason != protocol.ExitExited || ex.Code == nil || *ex.Code != 0 || string(ex.Result) != `{"got":{"n":7}}` {++ t.Fatalf("%+v result %s", ex, ex.Result)++ }++ if ex.Usage.StartedMS == nil || ex.Usage.WallMS < 2000 || ex.Usage.CPUMS == nil || ex.Usage.MemPeakBytes == nil || *ex.Usage.MemPeakBytes == 0 {++ t.Fatalf("usage %+v", ex.Usage)++ }++ frames := s.all()++ started, ok := frames[0].(protocol.JobStarted)++ if !ok || started.StartedMS != *ex.Usage.StartedMS {++ t.Fatalf("first frame %+v, want job_started at %d", frames[0], *ex.Usage.StartedMS)++ }++ if _, ok := frames[len(frames)-1].(protocol.JobExit); !ok {++ t.Fatalf("last frame %+v, want job_exit", frames[len(frames)-1])++ }++ var out strings.Builder++ var chunk uint64++ var second int64++ for _, m := range frames {++ switch f := m.(type) {++ case protocol.JobStdout:++ if f.ChunkSeq != chunk+1 {++ t.Fatalf("chunk_seq %d after %d", f.ChunkSeq, chunk)++ }++ chunk = f.ChunkSeq++ out.WriteString(f.Chunk)++ case protocol.JobUsage:++ if f.StartedMS != started.StartedMS || f.Second != second || f.CPUMS == nil {++ t.Fatalf("usage %+v, want second %d at %d", f, second, started.StartedMS)++ }++ second++++ }++ }++ if out.String() != "line 1\nline 2 é\n" {++ t.Fatalf("stdout %q", out.String())++ }++ if second < 2 || second*1000 > ex.Usage.WallMS {++ t.Fatalf("%d usage seconds for wall_ms %d", second, ex.Usage.WallMS)++ }++++ if m, ok := w.Known(j.ID); !ok || m.(protocol.JobExit).Reason != protocol.ExitExited {++ t.Fatalf("a resent job is answered %+v, want its job_exit", m)++ }++ w.Resend()++ if again := s.wait(t, protocol.TypeJobExit, j.ID, 2, 5*time.Second).(protocol.JobExit); *again.Usage.StartedMS != *ex.Usage.StartedMS || again.Usage.WallMS != ex.Usage.WallMS {++ t.Fatalf("resent job_exit %+v differs from %+v", again.Usage, ex.Usage)++ }++ w.ExitAck(j.ID)++ if _, ok := w.Known(j.ID); ok {++ t.Fatal("job still held after job_exit_ack")++ }++}++++// TestIsolation: the job sees none of the Node's environment, has no network (not even loopback), is PID 1 of its own++// PID namespace, runs as the configured user, and its working directory is fresh and removed afterwards.++func TestIsolation(t *testing.T) {++ t.Setenv("OPENVIBE_NODE_SECRET", "leak")++ w, s := newRunWorker(t, config.WorkerCaps{}, "env")++ j := testJob(1, "env")++ run(t, w, j)++ ex := s.exit(t, j.ID)++ var r struct {++ Env []string++ Cwd string++ PID, UID int++ Loopback, Dial string++ }++ if err := json.Unmarshal(ex.Result, &r); err != nil || ex.Reason != protocol.ExitExited {++ t.Fatalf("%+v %s: %v", ex, ex.Result, err)++ }++ if r.PID != 1 || r.UID != os.Getuid() {++ t.Fatalf("pid %d uid %d, want 1 and %d", r.PID, r.UID, os.Getuid())++ }++ if r.Loopback == "<nil>" || r.Dial == "<nil>" {++ t.Fatalf("network reachable: loopback %s, dial %s", r.Loopback, r.Dial)++ }++ for _, kv := range r.Env {++ if strings.HasPrefix(kv, "OPENVIBE_NODE_SECRET=") {++ t.Fatalf("the Node's environment leaked: %v", r.Env)++ }++ }++ if !slices.Contains(r.Env, "OPENVIBE_JOB_ID="+j.ID) || !slices.Contains(r.Env, "HOME="+r.Cwd) {++ t.Fatalf("env %v", r.Env)++ }++ if !strings.HasPrefix(filepath.Base(r.Cwd), "openvibe-job-") {++ t.Fatalf("cwd %s", r.Cwd)++ }++ if _, err := os.Stat(r.Cwd); !errors.Is(err, fs.ErrNotExist) {++ t.Fatalf("working directory %s not removed: %v", r.Cwd, err)++ }++}++++// TestTTLAndLimitKill: ttl_ms (from receipt) and wall_ms (from start) kill the process group, each clamped by the++// local cap when that is stricter; job_exit says ttl or limit with no exit code.++func TestTTLAndLimitKill(t *testing.T) {++ for i, c := range []struct {++ name string++ caps config.WorkerCaps++ ttl, wall int64++ reason string++ }{++ {"ttl", config.WorkerCaps{}, 800, 60000, protocol.ExitTTL},++ {"ttl clamped by the local cap", config.WorkerCaps{MaxTTLMS: 800}, 60000, 60000, protocol.ExitTTL},++ {"wall_ms", config.WorkerCaps{}, 60000, 800, protocol.ExitLimit},++ {"wall_ms clamped by the local cap", config.WorkerCaps{MaxWallMS: 800}, 60000, 60000, protocol.ExitLimit},++ } {++ t.Run(c.name, func(t *testing.T) {++ w, s := newRunWorker(t, c.caps, "sleep")++ j := testJob(i+1, "sleep")++ j.TTLMS, j.Limits.WallMS = c.ttl, c.wall++ begin := time.Now()++ run(t, w, j)++ ex := s.exit(t, j.ID)++ if ex.Reason != c.reason || ex.Code != nil || ex.Usage.StartedMS == nil || s.count(protocol.TypeJobStarted, j.ID) != 1 {++ t.Fatalf("%+v", ex)++ }++ if d := time.Since(begin); d > 5*time.Second {++ t.Fatalf("killed after %s", d)++ }++ })++ }++}++++// TestCancelKill: job_cancel kills a running job (cancelled, no code); a cancel after the end resends the unacked++// job_exit and is ignored once acked; a job cancelled before it started never starts.++func TestCancelKill(t *testing.T) {++ w, s := newRunWorker(t, config.WorkerCaps{MaxJobs: 2}, "sleep")++ j := testJob(1, "sleep")++ run(t, w, j)++ s.wait(t, protocol.TypeJobStarted, j.ID, 1, 10*time.Second)++ if !w.Cancel(j.ID) {++ t.Fatal("running job unknown")++ }++ ex := s.exit(t, j.ID)++ if ex.Reason != protocol.ExitCancelled || ex.Code != nil || ex.Usage.StartedMS == nil {++ t.Fatalf("%+v", ex)++ }++ w.Cancel(j.ID)++ s.wait(t, protocol.TypeJobExit, j.ID, 2, 5*time.Second)++ w.ExitAck(j.ID)++ if w.Cancel(j.ID) {++ t.Fatal("cancel of an acked job was not ignored")++ }++++ p := testJob(2, "sleep")++ if f, r := w.Admit(p); f != "" {++ t.Fatalf("%s %s", f, r)++ }++ w.Cancel(p.ID)++ w.Launch(p.ID)++ ex = s.exit(t, p.ID)++ if ex.Reason != protocol.ExitCancelled || ex.Usage.StartedMS != nil || ex.Usage.WallMS != 0 || s.count(protocol.TypeJobStarted, p.ID) != 0 {++ t.Fatalf("%+v", ex)++ }++}++++// TestOutputCap: stdout past max_output_bytes is cut at exactly the cap and the job is killed (limit); chunks are++// numbered from 1 and never larger than chunkBytes.++func TestOutputCap(t *testing.T) {++ const capBytes = 64 << 10++ w, s := newRunWorker(t, config.WorkerCaps{MaxOutputBytes: capBytes}, "flood")++ j := testJob(1, "flood")++ run(t, w, j)++ ex := s.exit(t, j.ID)++ if ex.Reason != protocol.ExitLimit || ex.Code != nil {++ t.Fatalf("%+v", ex)++ }++ total, seq := 0, uint64(0)++ for _, m := range s.all() {++ if f, ok := m.(protocol.JobStdout); ok {++ if f.ChunkSeq != seq+1 || len(f.Chunk) > chunkBytes {++ t.Fatalf("chunk %d after %d, %d bytes", f.ChunkSeq, seq, len(f.Chunk))++ }++ seq = f.ChunkSeq++ total += len(f.Chunk)++ }++ }++ if total != capBytes {++ t.Fatalf("%d stdout bytes sent, cap %d", total, capBytes)++ }++}++++// TestLatchKillsAndRefuses: setting the stop latch kills every running job at once (stopped) and refuses new jobs with++// the latch's fault; once cleared, jobs run again.++func TestLatchKillsAndRefuses(t *testing.T) {++ w, s := newRunWorker(t, config.WorkerCaps{MaxJobs: 2}, "sleep")++ a, b := testJob(1, "sleep"), testJob(2, "sleep")++ run(t, w, a)++ run(t, w, b)++ s.wait(t, protocol.TypeJobStarted, a.ID, 1, 10*time.Second)++ s.wait(t, protocol.TypeJobStarted, b.ID, 1, 10*time.Second)++ w.SetStopped(protocol.FaultEstopped)++ for _, j := range []protocol.Job{a, b} {++ if ex := s.exit(t, j.ID); ex.Reason != protocol.ExitStopped || ex.Code != nil {++ t.Fatalf("%+v", ex)++ }++ }++ if f, r := w.Admit(testJob(3, "sleep")); f != protocol.FaultEstopped || r != protocol.JobStopped {++ t.Fatalf("while latched: %q %q", f, r)++ }++ w.SetStopped("")++ c := testJob(4, "sleep")++ run(t, w, c)++ s.wait(t, protocol.TypeJobStarted, c.ID, 1, 10*time.Second)++ w.Cancel(c.ID)++ if ex := s.exit(t, c.ID); ex.Reason != protocol.ExitCancelled {++ t.Fatalf("%+v", ex)++ }++}nnew file mode 100644@@@ -0,0 +1,517 @@++// Package worker runs `function` jobs (OpenVibe.Contracts platform.job@1) from the control link. It runs only the++// functions the owner declared in the local config, each job as one child process in its own process group and in new++// user, network and PID namespaces (no network at all), with a scrubbed environment and a fresh working directory, and++// kills it at its ttl, at its limits, on job_cancel and while the stop latch is set. It never runs a job with less++// isolation: a job whose namespaces cannot be created fails without running, and a Node whose boot-time Probe fails++// does not advertise the class. docs/worker.md has the model and its limits.++package worker++++import (++ "bytes"++ "encoding/json"++ "io"++ "log/slog"++ "os"++ "os/exec"++ "sort"++ "strings"++ "sync"++ "time"++ "unicode/utf8"++++ "github.com/OpenVibers/OpenVibe.Node/internal/config"++ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"++)++++const (++ chunkBytes = 16 << 10 // the most stdout bytes one job_stdout carries++ resultBytes = 256 << 10 // the most bytes of result a function may write to fd 3++ sampleEvery = 200 * time.Millisecond++ endedMax = 1024 // ended jobs kept for their job_exit until job_exit_ack++)++++// Worker runs function jobs and sends their frames with send.++type Worker struct {++ cfg config.WorkerConfig++ caps config.WorkerCaps++ send func(protocol.Message)++ log *slog.Logger++++ // isolate puts a job's command in its namespaces; tests replace it to stand for a kernel that cannot.++ isolate func(*exec.Cmd, config.WorkerConfig) error++++ mu sync.Mutex++ stopped string // the fault code while the stop latch is set; "" when clear++ closed bool // shutting down++ jobs map[string]*job // admitted, running, or ended with its job_exit not yet acked++ ended []string // ids of ended jobs in jobs, oldest first++ wg sync.WaitGroup++}++++type job struct {++ req protocol.Job++ fn config.FunctionConfig++ received time.Time++ ttl time.Duration++ limits protocol.JobLimits // the job's, clamped to the local caps++++ reason string // why it was killed (the first reason wins); "" while nothing killed it++ launched bool // Launch was called++ proc *os.Process // set while the process runs++ startedMS int64++ exit *protocol.JobExit // set once it ended++}++++// New returns a Worker for cfg. It runs nothing until Probe has passed and a job is admitted and launched.++func New(cfg config.WorkerConfig, send func(protocol.Message), log *slog.Logger) *Worker {++ if log == nil {++ log = slog.Default()++ }++ return &Worker{cfg: cfg, caps: cfg.Caps.WithDefaults(), send: send, log: log, isolate: isolate, jobs: map[string]*job{}}++}++++// Probe starts a process the way a job is started and checks that it got user, network and PID namespaces of its own++// and that its CPU limit can be set and its usage read. The class must not be advertised when Probe fails.++func (w *Worker) Probe() error { return probe(w.cfg, w.isolate) }++++// Admit reserves a job, or returns the fault code and reason to nack it with. It runs nothing (Launch does, once the++// ack is out). An id the worker already holds is admitted again without a second reservation.++func (w *Worker) Admit(j protocol.Job) (fault, reason string) {++ w.mu.Lock()++ defer w.mu.Unlock()++ if _, ok := w.jobs[j.ID]; ok {++ return "", ""++ }++ if w.stopped != "" {++ return w.stopped, protocol.JobStopped++ }++ if w.closed {++ return protocol.FaultShuttingDown, protocol.JobNotAvailable++ }++ if j.Class != protocol.ClassFunction {++ return protocol.FaultUnsupported, protocol.JobNotAvailable++ }++ fn, ok := w.function(j.Artifact)++ if !ok {++ return protocol.FaultUnsupported, protocol.JobUnknownArtifact++ }++ running := 0++ for _, jb := range w.jobs {++ if jb.exit == nil {++ running++++ }++ }++ if running >= w.caps.MaxJobs {++ return protocol.FaultNotReady, protocol.JobBusy++ }++ w.jobs[j.ID] = &job{req: j, fn: fn, received: time.Now(), ttl: ms(min(j.TTLMS, w.caps.MaxTTLMS)),++ limits: protocol.JobLimits{WallMS: min(j.Limits.WallMS, w.caps.MaxWallMS), CPUMS: min(j.Limits.CPUMS, w.caps.MaxCPUMS),++ MemBytes: min(j.Limits.MemBytes, w.caps.MaxMemBytes)}}++ return "", ""++}++++func (w *Worker) function(a *protocol.Artifact) (config.FunctionConfig, bool) {++ if a == nil {++ return config.FunctionConfig{}, false++ }++ for _, f := range w.cfg.Functions {++ if f.Name == a.Name && f.Version == a.Version {++ return f, true++ }++ }++ return config.FunctionConfig{}, false++}++++// Launch runs an admitted job on its own goroutine. A job already launched, or unknown, is left alone.++func (w *Worker) Launch(id string) {++ w.mu.Lock()++ defer w.mu.Unlock()++ jb, ok := w.jobs[id]++ if !ok || jb.launched || w.closed {++ return++ }++ jb.launched = true++ w.wg.Add(1)++ go w.run(jb)++}++++// Known returns what a resent job frame for id is answered with: an ack while the job waits or runs, its job_exit++// once it ended (until job_exit_ack).++func (w *Worker) Known(id string) (protocol.Message, bool) {++ w.mu.Lock()++ defer w.mu.Unlock()++ jb, ok := w.jobs[id]++ switch {++ case !ok:++ return nil, false++ case jb.exit != nil:++ return *jb.exit, true++ }++ return protocol.Ack{ID: id}, true++}++++// Cancel kills a job (its job_exit says cancelled; one not started yet never starts). For an ended job whose job_exit++// is not acked yet it resends that job_exit. An unknown id is ignored; Cancel reports whether id was known.++func (w *Worker) Cancel(id string) bool {++ w.mu.Lock()++ jb, ok := w.jobs[id]++ var resend *protocol.JobExit++ if ok {++ resend = jb.exit++ w.killLocked(jb, protocol.ExitCancelled)++ }++ w.mu.Unlock()++ if resend != nil {++ w.send(*resend)++ }++ return ok++}++++// ExitAck forgets an ended job: its usage is recorded, so its job_exit is not resent any more.++func (w *Worker) ExitAck(id string) {++ w.mu.Lock()++ defer w.mu.Unlock()++ if jb, ok := w.jobs[id]; !ok || jb.exit == nil {++ return++ }++ delete(w.jobs, id)++ for i, e := range w.ended {++ if e == id {++ w.ended = append(w.ended[:i], w.ended[i+1:]...)++ break++ }++ }++}++++// SetStopped follows the stop latch (the e-stop or the local stop). A fault code kills every job at once (job_exit++// says stopped) and refuses new ones with that fault until SetStopped("").++func (w *Worker) SetStopped(fault string) {++ w.mu.Lock()++ defer w.mu.Unlock()++ w.stopped = fault++ if fault == "" {++ return++ }++ for _, jb := range w.jobs {++ w.killLocked(jb, protocol.ExitStopped)++ }++}++++// Resend sends again, after a reconnect, job_started for every running job and job_exit for every ended job whose++// job_exit is not acked yet.++func (w *Worker) Resend() {++ w.mu.Lock()++ var msgs []protocol.Message++ for _, jb := range w.jobs {++ if jb.exit == nil && jb.proc != nil {++ msgs = append(msgs, protocol.JobStarted{ID: jb.req.ID, StartedMS: jb.startedMS})++ }++ }++ for _, id := range w.ended {++ msgs = append(msgs, *w.jobs[id].exit)++ }++ w.mu.Unlock()++ for _, m := range msgs {++ w.send(m)++ }++}++++// Close kills every job (job_exit says stopped), refuses new ones and waits up to 5 s for them to end.++func (w *Worker) Close() {++ w.mu.Lock()++ w.closed = true++ for _, jb := range w.jobs {++ w.killLocked(jb, protocol.ExitStopped)++ }++ w.mu.Unlock()++ done := make(chan struct{})++ go func() { w.wg.Wait(); close(done) }()++ select {++ case <-done:++ case <-time.After(5 * time.Second):++ w.log.Error("jobs did not end within 5 s of being killed")++ }++}++++func (w *Worker) kill(jb *job, reason string) {++ w.mu.Lock()++ defer w.mu.Unlock()++ w.killLocked(jb, reason)++}++++// killLocked records why jb is killed and SIGKILLs its process group; a job not started yet then never starts.++func (w *Worker) killLocked(jb *job, reason string) {++ if jb.exit != nil {++ return++ }++ if jb.reason == "" {++ jb.reason = reason++ }++ if jb.proc != nil {++ killGroup(jb.proc)++ }++}++++func (w *Worker) run(jb *job) {++ defer w.wg.Done()++ ex := w.execute(jb)++ w.mu.Lock()++ jb.exit = &ex++ w.ended = append(w.ended, ex.ID)++ for len(w.ended) > endedMax {++ delete(w.jobs, w.ended[0])++ w.ended = w.ended[1:]++ }++ w.mu.Unlock()++ w.log.Info("job ended", "id", ex.ID, "reason", ex.Reason, "wall_ms", ex.Usage.WallMS)++ w.send(ex)++}++++// execute runs jb's process to its end and returns its job_exit.++func (w *Worker) execute(jb *job) protocol.JobExit {++ id := jb.req.ID++ never := func(reason string, err error) protocol.JobExit {++ if err != nil {++ w.log.Error("job not run", "id", id, "err", err)++ }++ return protocol.JobExit{ID: id, Reason: reason, Result: json.RawMessage("null")}++ }++ w.mu.Lock()++ reason := jb.reason++ w.mu.Unlock()++ if reason != "" { // cancelled or stopped before it started++ return never(reason, nil)++ }++ dir, err := os.MkdirTemp("", "openvibe-job-")++ if err != nil {++ return never(protocol.ExitFailed, err)++ }++ defer os.RemoveAll(dir)++ if r := w.cfg.RunAs; r != nil {++ if err := os.Chown(dir, int(r.UID), int(r.GID)); err != nil {++ return never(protocol.ExitFailed, err)++ }++ }++ cmd := exec.Command(jb.fn.Command[0], jb.fn.Command[1:]...)++ cmd.Dir, cmd.Env = dir, jobEnv(jb.fn, dir, id)++ cmd.Stdin = bytes.NewReader(append(append([]byte(nil), jb.req.Args...), '\n'))++ if err := w.isolate(cmd, w.cfg); err != nil {++ return never(protocol.ExitFailed, err)++ }++ outR, outW, err := os.Pipe()++ if err != nil {++ return never(protocol.ExitFailed, err)++ }++ defer outR.Close()++ resR, resW, err := os.Pipe()++ if err != nil {++ outW.Close()++ return never(protocol.ExitFailed, err)++ }++ defer resR.Close()++ cmd.Stdout, cmd.ExtraFiles = outW, []*os.File{resW}++++ w.mu.Lock()++ if jb.reason == "" && time.Since(jb.received) >= jb.ttl {++ jb.reason = protocol.ExitTTL++ }++ reason = jb.reason++ var start time.Time++ if reason == "" {++ err = cmd.Start()++ start = time.Now()++ if err == nil {++ jb.proc, jb.startedMS = cmd.Process, start.UnixMilli()++ }++ }++ w.mu.Unlock()++ outW.Close()++ resW.Close()++ if reason != "" {++ return never(reason, nil)++ }++ if err != nil {++ return never(protocol.ExitFailed, err)++ }++ pid := cmd.Process.Pid++ w.send(protocol.JobStarted{ID: id, StartedMS: jb.startedMS})++ if err := limitCPU(pid, jb.limits.CPUMS); err != nil {++ w.log.Error("job killed: its CPU limit could not be set", "id", id, "err", err)++ w.kill(jb, protocol.ExitFailed)++ }++++ stop, watched := make(chan struct{}), make(chan accounting, 1)++ go func() { watched <- w.watch(jb, pid, start, stop) }()++ var result []byte++ var readers sync.WaitGroup++ readers.Add(2)++ go func() { defer readers.Done(); w.pump(jb, outR) }()++ go func() { defer readers.Done(); result = w.readResult(jb, resR) }()++ readers.Wait()++ _ = cmd.Wait()++ wall := time.Since(start)++ close(stop)++ acc := <-watched++++ w.mu.Lock()++ reason = jb.reason++ jb.proc = nil++ w.mu.Unlock()++ code, cpuLimit := exitStatus(cmd.ProcessState)++ if reason == "" {++ reason = protocol.ExitExited++ if cpuLimit {++ reason = protocol.ExitLimit++ }++ }++ ex := protocol.JobExit{ID: id, Reason: reason, Code: code, Result: json.RawMessage("null")}++ if r := bytes.TrimSpace(result); reason == protocol.ExitExited && code != nil && *code == 0 && len(r) > 0 && json.Valid(r) {++ ex.Result = json.RawMessage(r)++ }++ // wall_ms is at least (n+1) s for every usage second n already sent, as platform.job-frame@1 requires.++ wallMS := max(wall.Milliseconds(), acc.seconds*1000)++ cpuMS := max(acc.cpuMS, (cmd.ProcessState.UserTime() + cmd.ProcessState.SystemTime()).Milliseconds())++ mem := max(acc.memPeak, maxRSS(cmd.ProcessState))++ started := jb.startedMS++ ex.Usage = protocol.JobExitUsage{StartedMS: &started, WallMS: wallMS, CPUMS: &cpuMS, MemPeakBytes: &mem}++ return ex++}++++// accounting is what watch measured: usage seconds sent, and the CPU time and peak resident memory sampled.++type accounting struct{ seconds, cpuMS, memPeak int64 }++++// watch is a job's watchdog until stop closes: it kills the job at its ttl (from receipt), at wall_ms (from start)++// and when sampled CPU time or resident memory exceed the limits, and sends job_usage for every wall-clock second++// once it has fully elapsed.++func (w *Worker) watch(jb *job, pid int, start time.Time, stop <-chan struct{}) accounting {++ var acc accounting++ ttl := time.NewTimer(time.Until(jb.received.Add(jb.ttl)))++ defer ttl.Stop()++ wall := time.NewTimer(time.Until(start.Add(ms(jb.limits.WallMS))))++ defer wall.Stop()++ usage := time.NewTimer(time.Until(start.Add(time.Second)))++ defer usage.Stop()++ tick := time.NewTicker(sampleEvery)++ defer tick.Stop()++ var ns string++ var lastCPU int64++ sample := func() {++ if ns == "" {++ ns, _ = pidNS(pid)++ }++ cpu, rss, err := sampleProcs(pid, ns)++ if err != nil {++ return++ }++ acc.cpuMS, acc.memPeak = max(acc.cpuMS, cpu), max(acc.memPeak, rss)++ if acc.cpuMS > jb.limits.CPUMS || rss > jb.limits.MemBytes {++ w.kill(jb, protocol.ExitLimit)++ }++ }++ sample()++ for {++ select {++ case <-stop:++ return acc++ case <-ttl.C:++ w.kill(jb, protocol.ExitTTL)++ case <-wall.C:++ w.kill(jb, protocol.ExitLimit)++ case <-tick.C:++ sample()++ case <-usage.C:++ for time.Since(start) >= time.Duration(acc.seconds+1)*time.Second {++ cpu := acc.cpuMS - lastCPU++ lastCPU = acc.cpuMS++ w.send(protocol.JobUsage{ID: jb.req.ID, StartedMS: jb.startedMS, Second: acc.seconds, CPUMS: &cpu})++ acc.seconds++++ }++ usage.Reset(time.Until(start.Add(time.Duration(acc.seconds+1) * time.Second)))++ }++ }++}++++// pump sends the job's stdout as job_stdout chunks of at most chunkBytes, never splitting a UTF-8 sequence, and++// kills the job (limit) once it wrote more than max_output_bytes; the excess is read and dropped.++func (w *Worker) pump(jb *job, r io.Reader) {++ buf := make([]byte, chunkBytes-utf8.UTFMax) // room for a rune carried over from the last read++ var carry []byte++ var total int64++ var seq uint64++ over := false++ emit := func(b []byte) {++ seq++++ w.send(protocol.JobStdout{ID: jb.req.ID, ChunkSeq: seq, Chunk: strings.ToValidUTF8(string(b), "�")})++ }++ for {++ n, err := r.Read(buf)++ if n > 0 && !over {++ b := buf[:n]++ if left := w.caps.MaxOutputBytes - total; int64(n) > left {++ b, over = b[:left], true++ }++ total += int64(len(b))++ b = append(carry, b...)++ cut := completeRunes(b)++ if over {++ cut = len(b)++ }++ if cut > 0 {++ emit(b[:cut])++ }++ carry = append([]byte(nil), b[cut:]...)++ if over {++ w.kill(jb, protocol.ExitLimit)++ }++ }++ if err != nil {++ if len(carry) > 0 {++ emit(carry)++ }++ return++ }++ }++}++++// completeRunes is how many leading bytes of b end on a rune boundary: a rune cut by the end of a read waits for++// the next read. Invalid bytes count as complete (they become U+FFFD).++func completeRunes(b []byte) int {++ for i := len(b) - 1; i >= 0 && i >= len(b)-utf8.UTFMax; i-- {++ if utf8.RuneStart(b[i]) {++ if utf8.FullRune(b[i:]) {++ return len(b)++ }++ return i++ }++ }++ return len(b)++}++++// readResult reads what the function writes to fd 3, its result; more than resultBytes kills the job (limit).++func (w *Worker) readResult(jb *job, r io.Reader) []byte {++ b, _ := io.ReadAll(io.LimitReader(r, resultBytes+1))++ if len(b) > resultBytes {++ w.kill(jb, protocol.ExitLimit)++ _, _ = io.Copy(io.Discard, r)++ return nil++ }++ return b++}++++// jobEnv is the whole environment of a job: nothing is inherited from the Node (no credential, no secrets), only a++// fixed PATH, the working directory as HOME and TMPDIR, the job id, and the function's own env from the config.++func jobEnv(fn config.FunctionConfig, dir, id string) []string {++ env := map[string]string{"PATH": "/usr/local/bin:/usr/bin:/bin", "HOME": dir, "TMPDIR": dir, "LANG": "C.UTF-8",++ "OPENVIBE_JOB_ID": id}++ for k, v := range fn.Env {++ env[k] = v++ }++ out := make([]string, 0, len(env))++ for k, v := range env {++ out = append(out, k+"="+v)++ }++ sort.Strings(out)++ return out++}++++func ms(v int64) time.Duration { return time.Duration(v) * time.Millisecond }nnew file mode 100644@@@ -0,0 +1,189 @@++//go:build linux++++package worker++++import (++ "errors"++ "fmt"++ "os"++ "os/exec"++ "strconv"++ "strings"++ "syscall"++ "unsafe"++++ "github.com/OpenVibers/OpenVibe.Node/internal/config"++)++++// clkTck is USER_HZ, the unit of /proc/<pid>/stat times: 100 on every architecture Go runs Linux on.++const clkTck = 100++++var pageSize = int64(os.Getpagesize())++++// isolate starts cmd in its own process group and in new user, network and PID namespaces: no network but a loopback++// that is down, and when the process exits every process it left behind is killed with it. It runs as worker.run_as++// (with no supplementary groups; the Node must be root) or, with allow_same_user, as the Node's own non-root user.++func isolate(cmd *exec.Cmd, cfg config.WorkerConfig) error {++ uid, gid := os.Geteuid(), os.Getegid()++ attr := &syscall.SysProcAttr{Setpgid: true, Pdeathsig: syscall.SIGKILL,++ Cloneflags: syscall.CLONE_NEWUSER | syscall.CLONE_NEWNET | syscall.CLONE_NEWPID}++ switch {++ case cfg.RunAs != nil:++ if uid != 0 {++ return errors.New("worker.run_as needs the Node to run as root")++ }++ r := cfg.RunAs++ attr.UidMappings = []syscall.SysProcIDMap{{ContainerID: int(r.UID), HostID: int(r.UID), Size: 1}}++ attr.GidMappings = []syscall.SysProcIDMap{{ContainerID: int(r.GID), HostID: int(r.GID), Size: 1}}++ attr.GidMappingsEnableSetgroups = true++ attr.Credential = &syscall.Credential{Uid: r.UID, Gid: r.GID} // no Groups: setgroups(0) drops root's groups++ case cfg.AllowSameUser:++ if uid == 0 {++ return errors.New("the Node runs as root: set worker.run_as instead of allow_same_user")++ }++ attr.UidMappings = []syscall.SysProcIDMap{{ContainerID: uid, HostID: uid, Size: 1}}++ attr.GidMappings = []syscall.SysProcIDMap{{ContainerID: gid, HostID: gid, Size: 1}}++ default:++ return errors.New("set worker.run_as (or allow_same_user, for development) to run jobs")++ }++ cmd.SysProcAttr = attr++ return nil++}++++// probe starts /bin/sh the way a job is started and checks that it got user, network and PID namespaces of its own,++// that its CPU limit can be set and that its usage can be read.++func probe(cfg config.WorkerConfig, isolate func(*exec.Cmd, config.WorkerConfig) error) error {++ cmd := exec.Command("/bin/sh", "-c", "read x")++ cmd.Env, cmd.Dir = []string{"PATH=/usr/bin:/bin"}, "/"++ if err := isolate(cmd, cfg); err != nil {++ return err++ }++ in, err := cmd.StdinPipe()++ if err != nil {++ return err++ }++ if err := cmd.Start(); err != nil {++ in.Close()++ return fmt.Errorf("starting a process in new user, network and PID namespaces: %w", err)++ }++ defer func() { killGroup(cmd.Process); in.Close(); _ = cmd.Wait() }()++ pid := cmd.Process.Pid++ for _, ns := range []string{"user", "net", "pid"} {++ mine, err1 := os.Readlink("/proc/self/ns/" + ns)++ its, err2 := os.Readlink("/proc/" + strconv.Itoa(pid) + "/ns/" + ns)++ if err1 != nil || err2 != nil || mine == its {++ return fmt.Errorf("the probe process did not get a %s namespace of its own (%v, %v)", ns, err1, err2)++ }++ }++ if err := limitCPU(pid, 1000); err != nil {++ return err++ }++ if _, _, err := statOf(pid); err != nil {++ return fmt.Errorf("reading the probe's CPU and memory: %w", err)++ }++ return nil++}++++// killGroup SIGKILLs the job's process group while its leader lives, then the leader; the leader is PID 1 of the++// job's PID namespace, so its death also kills anything that left the group.++func killGroup(p *os.Process) {++ if p.Signal(syscall.Signal(0)) == nil {++ _ = syscall.Kill(-p.Pid, syscall.SIGKILL)++ }++ _ = p.Kill()++}++++// limitCPU sets the process's RLIMIT_CPU (SIGXCPU at cpuMS rounded up to a second, SIGKILL a second later; every++// process it starts inherits it) and RLIMIT_CORE 0: the kernel's backstop behind the sampled cpu_ms check.++func limitCPU(pid int, cpuMS int64) error {++ sec := uint64((cpuMS + 999) / 1000)++ limits := []struct {++ res int++ lim syscall.Rlimit++ }{{syscall.RLIMIT_CPU, syscall.Rlimit{Cur: sec, Max: sec + 1}}, {syscall.RLIMIT_CORE, syscall.Rlimit{}}}++ for _, l := range limits {++ lim := l.lim++ if _, _, e := syscall.RawSyscall6(syscall.SYS_PRLIMIT64, uintptr(pid), uintptr(l.res), uintptr(unsafe.Pointer(&lim)), 0, 0, 0); e != 0 {++ return fmt.Errorf("prlimit: %w", e)++ }++ }++ return nil++}++++func pidNS(pid int) (string, error) { return os.Readlink("/proc/" + strconv.Itoa(pid) + "/ns/pid") }++++// sampleProcs sums the CPU time and resident memory of every process in the job's PID namespace ns (the leader alone++// while ns is unknown), so a process that left the process group is still counted.++func sampleProcs(pid int, ns string) (cpuMS, rssBytes int64, err error) {++ if ns == "" {++ return statOf(pid)++ }++ ents, err := os.ReadDir("/proc")++ if err != nil {++ return 0, 0, err++ }++ for _, e := range ents {++ p, err := strconv.Atoi(e.Name())++ if err != nil {++ continue++ }++ if l, err := os.Readlink("/proc/" + e.Name() + "/ns/pid"); err != nil || l != ns {++ continue++ }++ if c, r, err := statOf(p); err == nil {++ cpuMS, rssBytes = cpuMS+c, rssBytes+r++ }++ }++ return cpuMS, rssBytes, nil++}++++// statOf reads one process's CPU time (its own and its reaped children's) and resident memory from /proc.++func statOf(pid int) (cpuMS, rssBytes int64, err error) {++ b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")++ if err != nil {++ return 0, 0, err++ }++ s := string(b)++ i := strings.LastIndexByte(s, ')')++ if i < 0 {++ return 0, 0, errors.New("malformed /proc stat")++ }++ f := strings.Fields(s[i+1:]) // f[0] is field 3, state++ if len(f) < 22 {++ return 0, 0, errors.New("short /proc stat")++ }++ var ticks int64++ for _, k := range []int{11, 12, 13, 14} { // utime, stime, cutime, cstime++ v, err := strconv.ParseInt(f[k], 10, 64)++ if err != nil {++ return 0, 0, err++ }++ ticks += v++ }++ rss, err := strconv.ParseInt(f[21], 10, 64)++ if err != nil {++ return 0, 0, err++ }++ return ticks * 1000 / clkTck, rss * pageSize, nil++}++++// exitStatus is the process's exit code (nil when a signal ended it) and whether RLIMIT_CPU ended it.++func exitStatus(ps *os.ProcessState) (code *int, cpuLimit bool) {++ ws, ok := ps.Sys().(syscall.WaitStatus)++ if !ok {++ return nil, false++ }++ if ws.Exited() {++ c := ws.ExitStatus()++ return &c, false++ }++ return nil, ws.Signaled() && ws.Signal() == syscall.SIGXCPU++}++++// maxRSS is the peak resident memory the kernel reports for the process and the children it waited for.++func maxRSS(ps *os.ProcessState) int64 {++ if r, ok := ps.SysUsage().(*syscall.Rusage); ok {++ return int64(r.Maxrss) * 1024++ }++ return 0++}nnew file mode 100644@@@ -0,0 +1,35 @@++//go:build !linux++++package worker++++import (++ "errors"++ "os"++ "os/exec"++++ "github.com/OpenVibers/OpenVibe.Node/internal/config"++)++++// errNotLinux: the namespaces the worker isolates jobs with exist on Linux only, so elsewhere no job ever runs.++var errNotLinux = errors.New("the job worker runs on Linux only")++++func isolate(*exec.Cmd, config.WorkerConfig) error { return errNotLinux }++++func probe(config.WorkerConfig, func(*exec.Cmd, config.WorkerConfig) error) error { return errNotLinux }++++func killGroup(p *os.Process) { _ = p.Kill() }++++func limitCPU(int, int64) error { return errNotLinux }++++func pidNS(int) (string, error) { return "", errNotLinux }++++func sampleProcs(int, string) (int64, int64, error) { return 0, 0, errNotLinux }++++func exitStatus(ps *os.ProcessState) (*int, bool) {++ if c := ps.ExitCode(); c >= 0 {++ return &c, false++ }++ return nil, false++}++++func maxRSS(*os.ProcessState) int64 { return 0 }nnew file mode 100644@@@ -0,0 +1,187 @@++package worker++++import (++ "encoding/json"++ "errors"++ "fmt"++ "io"++ "log/slog"++ "os"++ "os/exec"++ "sync"++ "testing"++ "time"++++ "github.com/OpenVibers/OpenVibe.Node/internal/config"++ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"++)++++// sink records every frame the worker sends.++type sink struct {++ mu sync.Mutex++ frames []protocol.Message++}++++func (s *sink) send(m protocol.Message) {++ s.mu.Lock()++ defer s.mu.Unlock()++ s.frames = append(s.frames, m)++}++++func (s *sink) all() []protocol.Message {++ s.mu.Lock()++ defer s.mu.Unlock()++ return append([]protocol.Message(nil), s.frames...)++}++++func frameID(m protocol.Message) string {++ switch f := m.(type) {++ case protocol.JobStarted:++ return f.ID++ case protocol.JobStdout:++ return f.ID++ case protocol.JobUsage:++ return f.ID++ case protocol.JobExit:++ return f.ID++ }++ return ""++}++++// count is how many frames of type typ for job id were sent.++func (s *sink) count(typ, id string) int {++ n := 0++ for _, m := range s.all() {++ if m.MessageType() == typ && frameID(m) == id {++ n++++ }++ }++ return n++}++++// wait returns the n-th (from 1) frame of type typ for job id, waiting up to timeout for it.++func (s *sink) wait(t *testing.T, typ, id string, n int, timeout time.Duration) protocol.Message {++ t.Helper()++ end := time.Now().Add(timeout)++ for {++ seen := 0++ for _, m := range s.all() {++ if m.MessageType() == typ && frameID(m) == id {++ if seen++; seen == n {++ return m++ }++ }++ }++ if time.Now().After(end) {++ t.Fatalf("no %s #%d for %s within %s", typ, n, id, timeout)++ }++ time.Sleep(10 * time.Millisecond)++ }++}++++func (s *sink) exit(t *testing.T, id string) protocol.JobExit {++ t.Helper()++ return s.wait(t, protocol.TypeJobExit, id, 1, 20*time.Second).(protocol.JobExit)++}++++func testJob(n int, name string) protocol.Job {++ return protocol.Job{ID: fmt.Sprintf("job_01JAB2C3D4E5F6G7H8J9K%05d", n), Class: protocol.ClassFunction,++ Artifact: &protocol.Artifact{Name: name, Version: "1.0.0"}, Args: json.RawMessage(`{"n":7}`), TTLMS: 60000,++ Limits: protocol.JobLimits{WallMS: 60000, CPUMS: 60000, MemBytes: 4 << 30}}++}++++// helperConfig declares one function per helper mode, each running this test binary's TestHelperProcess.++func helperConfig(t *testing.T, caps config.WorkerCaps, modes ...string) config.WorkerConfig {++ t.Helper()++ exe, err := os.Executable()++ if err != nil {++ t.Fatal(err)++ }++ cfg := config.WorkerConfig{Enabled: true, AllowSameUser: true, Caps: caps}++ for _, m := range modes {++ cfg.Functions = append(cfg.Functions, config.FunctionConfig{Name: m, Version: "1.0.0",++ Command: []string{exe, "-test.run=^TestHelperProcess$", "--", m}})++ }++ return cfg++}++++func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }++++// TestAdmitRefusals: only a declared function at its exact version is admitted, never another class; max_jobs bounds++// the jobs held at once; while the stop latch is set every job is refused with its fault; nothing ever starts.++func TestAdmitRefusals(t *testing.T) {++ s := &sink{}++ w := New(helperConfig(t, config.WorkerCaps{MaxJobs: 1}, "sleep"), s.send, quiet())++ w.isolate = func(*exec.Cmd, config.WorkerConfig) error { return errors.New("not in this test") }++++ wrongVersion := testJob(2, "sleep")++ wrongVersion.Artifact.Version = "1.0.1"++ code := testJob(3, "sleep")++ code.Class = protocol.ClassCode++ for _, c := range []struct {++ j protocol.Job++ fault, reason string++ }{++ {testJob(1, "nope"), protocol.FaultUnsupported, protocol.JobUnknownArtifact},++ {wrongVersion, protocol.FaultUnsupported, protocol.JobUnknownArtifact},++ {code, protocol.FaultUnsupported, protocol.JobNotAvailable},++ } {++ if f, r := w.Admit(c.j); f != c.fault || r != c.reason {++ t.Errorf("%s %+v: %q %q, want %q %q", c.j.Class, c.j.Artifact, f, r, c.fault, c.reason)++ }++ }++++ held := testJob(4, "sleep")++ if f, r := w.Admit(held); f != "" {++ t.Fatalf("%s %s", f, r)++ }++ if f, r := w.Admit(held); f != "" {++ t.Fatalf("a held id re-admitted: %s %s", f, r)++ }++ if f, r := w.Admit(testJob(5, "sleep")); f != protocol.FaultNotReady || r != protocol.JobBusy {++ t.Fatalf("over max_jobs: %q %q", f, r)++ }++ w.Cancel(held.ID)++ w.Launch(held.ID)++ if ex := s.exit(t, held.ID); ex.Reason != protocol.ExitCancelled {++ t.Fatalf("%+v", ex)++ }++++ w.SetStopped(protocol.FaultLocalStop)++ if f, r := w.Admit(testJob(6, "sleep")); f != protocol.FaultLocalStop || r != protocol.JobStopped {++ t.Fatalf("latched: %q %q", f, r)++ }++ w.SetStopped("")++ if f, r := w.Admit(testJob(6, "sleep")); f != "" {++ t.Fatalf("after the latch cleared: %q %q", f, r)++ }++ for _, m := range s.all() {++ if _, ok := m.(protocol.JobStarted); ok {++ t.Fatalf("a refused job started: %+v", m)++ }++ }++}++++// TestIsolationUnavailableRefused: when the namespaces cannot be created the probe fails (the class is not++// advertised), and a job whose isolation fails ends `failed` without its process ever starting.++func TestIsolationUnavailableRefused(t *testing.T) {++ s := &sink{}++ w := New(helperConfig(t, config.WorkerCaps{}, "sleep"), s.send, quiet())++ w.isolate = func(*exec.Cmd, config.WorkerConfig) error { return errors.New("no user namespaces") }++ t.Cleanup(w.Close)++ if err := w.Probe(); err == nil {++ t.Fatal("the probe passed without namespaces")++ }++ j := testJob(1, "sleep")++ if f, r := w.Admit(j); f != "" {++ t.Fatalf("%s %s", f, r)++ }++ w.Launch(j.ID)++ ex := s.exit(t, j.ID)++ if ex.Reason != protocol.ExitFailed || ex.Code != nil || ex.Usage.WallMS != 0 || ex.Usage.StartedMS != nil || string(ex.Result) != "null" {++ t.Fatalf("%+v", ex)++ }++ if n := s.count(protocol.TypeJobStarted, j.ID); n != 0 {++ t.Fatalf("job_started sent %d times for a job that could not be isolated", n)++ }++}
@@ -24,7 +24,12 @@ jobs:
@@ -24,7 +24,12 @@ jobs:
- run: pip install av numpy
- run: pip install av numpy
- run: test -z "$(gofmt -l .)" || (gofmt -l . && exit 1)
- run: test -z "$(gofmt -l .)" || (gofmt -l . && exit 1)
- run: go vet ./...
- run: go vet ./...
+ # The job worker isolates jobs in unprivileged user and network namespaces, which Ubuntu's AppArmor restricts by
+ # default; its tests skip where they cannot run, and OPENVIBE_WORKER_TESTS=require makes them fail here instead.
+ - run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
- run: go test -race -count=1 ./...
- run: go test -race -count=1 ./...
+ env:
+ OPENVIBE_WORKER_TESTS: require
build:
build:
name: cross-compile
name: cross-compile
@@ -180,9 +180,10 @@ Commands are never queued while offline and never replayed after a reconnect.
@@ -180,9 +180,10 @@ Commands are never queued while offline and never replayed after a reconnect.
### Jobs
### Jobs
-The job frames of OpenVibe.Contracts `platform.job-frame@1` (plan T14 Run) ride the same link and envelope.
-**Running jobs is not implemented yet:** the Node advertises no runtime class, so it refuses every well-formed job
-with `class not available`, and nothing executes.
+The job frames of OpenVibe.Contracts `platform.job-frame@1` (plan T14 Run) ride the same link and envelope. The Node
+runs class `function` only, and only when the owner turned the worker on in the local config, the OS is Linux and a
+boot-time probe could start a process in its own user, network and PID namespaces ([worker.md](worker.md)).
+Otherwise it advertises no class and refuses every well-formed job with `class not available`, and nothing executes.
| direction | type | fields |
| direction | type | fields |
|------------------|----------------|-------------------------------------------------------------------------------------------|
|------------------|----------------|-------------------------------------------------------------------------------------------|
@@ -205,11 +206,34 @@ checked in this order:
@@ -205,11 +206,34 @@ checked in this order:
| `function job needs an artifact with an exact version` | `bad_value` | a `function` job without `artifact`, or a name or version outside the contract's pattern (a range) |
| `function job needs an artifact with an exact version` | `bad_value` | a `function` job without `artifact`, or a name or version outside the contract's pattern (a range) |
| `net policy not supported` | `unsupported` | `net` is present and not `deny` |
| `net policy not supported` | `unsupported` | `net` is present and not `deny` |
| `missing or invalid args, ttl_ms or limits` | `bad_value` | `args` is not an object, or `ttl_ms` or a limit is missing or below 1 |
| `missing or invalid args, ttl_ms or limits` | `bad_value` | `args` is not an object, or `ttl_ms` or a limit is missing or below 1 |
-| `class not available` | `unsupported` | the class is valid but this Node does not run it (today: every class) |
-
-A resent `job` with an id already answered gets the same answer again, never a second acceptance; the Node
-remembers the last 1024 well-formed ids. `job_cancel` for an id that is not running (unknown, refused or ended) and
-`job_exit_ack` for an unknown id are ignored, with no answer: an `ack` keyed by a job id means the job was accepted.
+| `class not available` | `unsupported` | the class is valid but this Node does not run it (every class but `function`; every class while the worker is off) |
+| `the e-stop or local stop is latched` | `estopped` or `local_stop` | the stop latch is set: no job starts until it clears |
+| `class not available` | `shutting_down` | the Node is stopping |
+| `unknown artifact` | `unsupported` | `artifact` is not a function the owner declared, at that exact version |
+| `worker busy` | `not_ready` | `worker.caps.max_jobs` jobs are already running |
+
+A resent `job` with an id already answered gets the same answer again, never a second acceptance and never a second
+process: `ack` while it runs, its `job_exit` once it ended; the Node remembers the last 1024 well-formed ids.
+
+An accepted job runs once the `ack` is out: `job_started` (`started_ms` anchors every usage second), its stdout as
+`job_stdout` chunks (at most 16 KiB each, UTF-8, `chunk_seq` from 1; best effort, not resent), one `job_usage` for
+every wall-clock second once it has fully elapsed (`second` from 0, `cpu_ms` that second's CPU time), then
+`job_exit`. Its `usage` is authoritative for metering (Bot writes `usage-sample` readings `service:"run"`,
+`operation:"function.invoke"`, unit `s`, idempotency key `run:<job id>:<n>`; none of those fields changes on a
+resend). The Node resends `job_started` for running jobs and `job_exit` for ended ones after every reconnect, and
+`job_exit` again on a `job` or `job_cancel` for that id, until `job_exit_ack`. `job_exit.reason`:
+
+| `reason` | `code` | when |
+|-------------|---------|-----------------------------------------------------------------------------------------------------|
+| `exited` | status | the process ended by itself (`code` is null if a signal it did not get from the worker ended it); `result` is what it wrote to fd 3 when it exited 0 |
+| `cancelled` | null | `job_cancel`: the process group is killed; a job cancelled before it started never starts |
+| `ttl` | null | `ttl_ms` (from receipt, clamped to `worker.caps.max_ttl_ms`) ran out, before or while it ran |
+| `limit` | null | `wall_ms`, `cpu_ms` or `mem_bytes` (each clamped to the local cap), more stdout than `max_output_bytes`, or more than 256 KiB of result |
+| `stopped` | null | the e-stop or the local stop latched, or the Node shut down |
+| `failed` | null | the worker could not run it: the namespaces could not be created, the spawn failed, or its CPU limit could not be set |
+
+`job_cancel` for an id that is not running (unknown, refused or ended) and `job_exit_ack` for an unknown id are
+ignored, with no answer: an `ack` keyed by a job id means the job was accepted.
A Node that runs jobs lists its classes in `status.capabilities.worker` as `{"runtime_classes": ["function"]}`; the
A Node that runs jobs lists its classes in `status.capabilities.worker` as `{"runtime_classes": ["function"]}`; the
key is absent while it runs none, and `worker` is reserved as a plugin name.
key is absent while it runs none, and `worker` is reserved as a plugin name.
new file mode 100644
new file mode 100644
@@ -0,0 +1,123 @@
@@ -0,0 +1,123 @@
+# The job worker (`function` jobs)
+
+The Node can run **function jobs** (OpenVibe.Contracts `platform.job@1`, class `function`) sent over the control link
+([protocol.md](protocol.md#jobs)). A job is untrusted input: it names a function, and the Node runs it only if the
+owner declared that function, at that exact version, in the local config. Nothing is downloaded and no path comes from
+the server. The worker is **off by default**; off, the Node refuses every job `class not available`, as before.
+
+## Config
+
+`config.json`, section `worker` (defaults shown; a zero cap means the default):
+
+```json
+"worker": {
+ "enabled": false,
+ "functions": [
+ {"name": "thumbnail", "version": "1.2.0", "command": ["/opt/fn/thumbnail/bin/run"], "env": {"MODE": "fast"}}
+ ],
+ "run_as": {"uid": 2001, "gid": 2001},
+ "allow_same_user": false,
+ "caps": {
+ "max_ttl_ms": 600000,
+ "max_wall_ms": 300000,
+ "max_cpu_ms": 300000,
+ "max_mem_bytes": 536870912,
+ "max_output_bytes": 1048576,
+ "max_jobs": 1
+ }
+}
+```
+
+- `functions`: `name` and `version` match a job's `artifact` exactly (the contract's patterns; listed once each);
+ `command[0]` is an absolute path. `env` is added to the job's environment.
+- `run_as`: the unprivileged uid and gid jobs run as (not 0; no supplementary groups). It needs the Node to run as
+ root, which the system install does. Create a dedicated user for it with no login and no groups.
+- `allow_same_user`: run jobs as the Node's own (non-root) user instead. A job can then read every file the Node can,
+ `credential.json` included, and use its groups (`gpio`, `dialout`, `video`): **for development only**. With neither
+ `run_as` nor `allow_same_user` the worker stays off.
+- `caps`: the owner's local caps. A job's `ttl_ms`, `limits.wall_ms`, `limits.cpu_ms` and `limits.mem_bytes` are
+ each clamped to them (the stricter value wins, as with `max_command_ms`). `max_output_bytes` caps one job's stdout;
+ `max_jobs` how many run at once.
+
+The class `function` is advertised in `status.capabilities.worker` only when `enabled` is true, the OS is Linux and a
+**boot-time probe** passes: it starts `/bin/sh` the way a job is started and checks that the process got user,
+network and PID namespaces of its own, that its CPU limit can be set and that its CPU and memory can be read. If the
+probe fails the Node logs why (`worker off: …`) and refuses every job, exactly as with the worker off. On Ubuntu 24.04
+and later, unprivileged user namespaces are restricted by AppArmor; a Node running as root with `run_as` is not
+affected.
+
+## How a job runs
+
+1. **Admit** (before the `ack`): refused unless the stop latch is clear, the artifact is declared and fewer than
+ `max_jobs` jobs run. An id the worker already holds is answered from it (`ack` while it runs, its `job_exit` once
+ it ended) and never starts a second process.
+2. **Start**: a fresh working directory (`$TMPDIR/openvibe-job-*`, owned by `run_as`, removed afterwards); an
+ environment built from scratch (`PATH=/usr/local/bin:/usr/bin:/bin`, `HOME` and `TMPDIR` the working directory,
+ `LANG=C.UTF-8`, `OPENVIBE_JOB_ID`, and the function's `env`; nothing inherited from the Node); `args` as one line of
+ JSON on stdin; stderr discarded. The process starts in its own process group and in new user, network and PID
+ namespaces, with `PR_SET_PDEATHSIG` SIGKILL, then gets `RLIMIT_CPU` (at `cpu_ms`, rounded up to a second) and
+ `RLIMIT_CORE` 0. `job_started` follows.
+3. **Run**: stdout streams as `job_stdout`; a watchdog samples CPU time and resident memory of every process in the
+ job's PID namespace every 200 ms and sends `job_usage` for every second once it fully elapsed.
+4. **End**: the function's result is what it writes to **fd 3** (JSON, at most 256 KiB), reported only when it exits
+ 0. `job_exit` carries the reason, the exit code (null when it was killed) and the usage, and is resent until
+ `job_exit_ack`.
+
+Every kill is SIGKILL to the whole process group and to the leader, which is PID 1 of the job's PID namespace, so its
+death also kills every process the job left behind. The watchdog kills at `ttl_ms` (from receipt), `wall_ms` (from
+start), `cpu_ms`, `mem_bytes`, at `max_output_bytes` of stdout, on `job_cancel`, and the moment the e-stop or the local
+stop latches (`openvibe-node stop`); while the latch is set every job is refused. A Node shutting down kills its jobs
+(`stopped`).
+
+## Isolation model and its limits
+
+In five lines:
+
+1. Only owner-declared executables run, matched by exact name and version; the server chooses which and with what JSON.
+2. Each job is a separate unprivileged process (`run_as`) in its own process group and user, network and PID namespaces.
+3. The network namespace has only a loopback that is down: no IP network, not even to the Node or `localhost`.
+4. Nothing of the Node's environment reaches it; its working directory is fresh and removed after the job.
+5. ttl, wall, CPU, memory and output are enforced by a per-job watchdog that SIGKILLs the whole group; `RLIMIT_CPU` backs the CPU cap in the kernel.
+
+What it does **not** protect against (no seccomp, no cgroups, no mount namespace yet):
+
+- **Files**: there is no filesystem isolation. A job reads and writes whatever its uid may: with `run_as`, every
+ world-readable file and world-writable directory; with `allow_same_user`, everything the Node can, including
+ `credential.json` and the local control socket (so it could clear the local stop).
+- **Unix sockets by path** (Docker's, D-Bus's, the Node's control socket) are reachable: a network namespace does not
+ cover filesystem sockets; only their file permissions do.
+- **The kernel**: no seccomp filter, so the whole syscall surface is open, including what user namespaces expose to
+ unprivileged code (a common source of local privilege escalations). Keep the kernel patched.
+- **Memory and CPU are sampled**, not enforced by cgroups: a burst between two 200 ms samples can exceed `mem_bytes`
+ (and meet the kernel's OOM killer first); the `RLIMIT_CPU` backstop is set just after start and covers each process
+ on its own, not their sum.
+- **No limit on processes, threads, disk or I/O**: a fork bomb is bounded only by the uid's `RLIMIT_NPROC`, disk use
+ in the working directory or `/tmp` only by free space; `/tmp` and `/dev/shm` are shared with the machine.
+- **`/proc` is the host's**: a job can list the machine's processes and their command lines (not signal them: its PID
+ namespace has no host PIDs).
+- **The `args` and the result are trusted to the function**: the worker passes the server's JSON unchanged; validating
+ it is the function's job.
+
+## Refusals and exit reasons
+
+Refused with `nack` (keyed by the job id; the first matching row wins, after the checks every Node does, in
+[protocol.md](protocol.md#jobs)):
+
+| `message` | `fault_code` | when |
+|---------------------------------------|----------------------------|---------------------------------------------------------|
+| `class not available` | `unsupported` | the worker is off (disabled, not Linux, probe failed), or the class is not `function` |
+| `the e-stop or local stop is latched` | `estopped` or `local_stop` | the stop latch is set |
+| `class not available` | `shutting_down` | the Node is stopping |
+| `unknown artifact` | `unsupported` | no declared function has that name and exact version |
+| `worker busy` | `not_ready` | `max_jobs` jobs are running |
+
+Accepted jobs end with `job_exit.reason`:
+
+| `reason` | when |
+|-------------|-----------------------------------------------------------------------------------------------------------|
+| `exited` | the process ended by itself; `code` is its status (null if a signal the worker did not send ended it) |
+| `cancelled` | `job_cancel`; a job cancelled before it started never starts |
+| `ttl` | `ttl_ms` (clamped) ran out, before or while it ran |
+| `limit` | `wall_ms`, `cpu_ms` (sampled, or `RLIMIT_CPU`'s SIGXCPU) or `mem_bytes` exceeded, stdout past `max_output_bytes`, or a result over 256 KiB |
+| `stopped` | the e-stop or local stop latched, or the Node shut down |
+| `failed` | the job could not run isolated: namespaces not created, spawn error, working directory not made, CPU limit not set; never run with less isolation |
@@ -17,6 +17,7 @@ import (
@@ -17,6 +17,7 @@ import (
"io/fs"
"io/fs"
"os"
"os"
"path/filepath"
"path/filepath"
+ "regexp"
"runtime"
"runtime"
"strings"
"strings"
)
)
@@ -106,6 +107,66 @@ type Config struct {
@@ -106,6 +107,66 @@ type Config struct {
// Python is the interpreter for plugins without an explicit command (default: <state>/venv/bin/python, then python3).
// Python is the interpreter for plugins without an explicit command (default: <state>/venv/bin/python, then python3).
Python string `json:"python,omitempty"`
Python string `json:"python,omitempty"`
LogLevel string `json:"log_level,omitempty"`
LogLevel string `json:"log_level,omitempty"`
+ // Worker runs `function` jobs from the control link (docs/worker.md). Off unless enabled.
+ Worker WorkerConfig `json:"worker,omitzero"`
+}
+
+// WorkerConfig is the job worker. Disabled (the default), every job is refused "class not available". Only the
+// functions listed here ever run: a job names one by name and exact version, and nothing is downloaded.
+type WorkerConfig struct {
+ Enabled bool `json:"enabled,omitempty"`
+ Functions []FunctionConfig `json:"functions,omitempty"`
+ // RunAs is the dedicated unprivileged uid/gid jobs run as; it needs the Node to run as root. Without it the Node
+ // refuses to run jobs unless AllowSameUser is set.
+ RunAs *RunAs `json:"run_as,omitempty"`
+ // AllowSameUser runs jobs as the Node's own user (no run_as). A job can then read every file the Node can,
+ // including credential.json, and use the Node user's groups (gpio, dialout, video): for development only.
+ AllowSameUser bool `json:"allow_same_user,omitempty"`
+ Caps WorkerCaps `json:"caps,omitzero"`
+}
+
+// FunctionConfig is one function a job may run: Command is started with the job's args as JSON on stdin.
+type FunctionConfig struct {
+ Name string `json:"name"`
+ Version string `json:"version"`
+ Command []string `json:"command"` // Command[0] is an absolute path
+ Env map[string]string `json:"env,omitempty"`
+}
+
+// RunAs is a uid and gid on this machine.
+type RunAs struct {
+ UID uint32 `json:"uid"`
+ GID uint32 `json:"gid"`
+}
+
+// WorkerCaps are the owner's local caps on every job; a job's own ttl_ms and limits are clamped to them (the
+// stricter value wins). Zero means the default (DefaultWorkerCaps).
+type WorkerCaps struct {
+ MaxTTLMS int64 `json:"max_ttl_ms,omitempty"`
+ MaxWallMS int64 `json:"max_wall_ms,omitempty"`
+ MaxCPUMS int64 `json:"max_cpu_ms,omitempty"`
+ MaxMemBytes int64 `json:"max_mem_bytes,omitempty"`
+ MaxOutputBytes int64 `json:"max_output_bytes,omitempty"` // stdout of one job, in bytes
+ MaxJobs int `json:"max_jobs,omitempty"` // jobs running at once
+}
+
+// DefaultWorkerCaps are the caps for every field the config leaves at zero.
+var DefaultWorkerCaps = WorkerCaps{MaxTTLMS: 600000, MaxWallMS: 300000, MaxCPUMS: 300000, MaxMemBytes: 512 << 20,
+ MaxOutputBytes: 1 << 20, MaxJobs: 1}
+
+// WithDefaults fills the zero fields of c from DefaultWorkerCaps.
+func (c WorkerCaps) WithDefaults() WorkerCaps {
+ d := DefaultWorkerCaps
+ for _, f := range []struct{ v, def *int64 }{{&c.MaxTTLMS, &d.MaxTTLMS}, {&c.MaxWallMS, &d.MaxWallMS},
+ {&c.MaxCPUMS, &d.MaxCPUMS}, {&c.MaxMemBytes, &d.MaxMemBytes}, {&c.MaxOutputBytes, &d.MaxOutputBytes}} {
+ if *f.v <= 0 {
+ *f.v = *f.def
+ }
+ }
+ if c.MaxJobs <= 0 {
+ c.MaxJobs = d.MaxJobs
+ }
+ return c
}
}
type LocalLimits struct {
type LocalLimits struct {
@@ -239,6 +300,35 @@ func (c *Config) Validate() error {
@@ -239,6 +300,35 @@ func (c *Config) Validate() error {
return errors.New("config: limits must be between 0 and 1")
return errors.New("config: limits must be between 0 and 1")
}
}
}
}
+ return c.Worker.validate()
+}
+
+var (
+ functionNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9._-]{0,127}$`)
+ functionVersionRe = regexp.MustCompile(`^[0-9A-Za-z][0-9A-Za-z.+-]{0,63}$`)
+)
+
+func (w WorkerConfig) validate() error {
+ seen := map[string]bool{}
+ for _, f := range w.Functions {
+ if !functionNameRe.MatchString(f.Name) || !functionVersionRe.MatchString(f.Version) {
+ return fmt.Errorf("config: worker function %q version %q: name or version is not a valid artifact name or exact version", f.Name, f.Version)
+ }
+ if seen[f.Name+"@"+f.Version] {
+ return fmt.Errorf("config: worker function %s@%s listed twice", f.Name, f.Version)
+ }
+ seen[f.Name+"@"+f.Version] = true
+ if len(f.Command) == 0 || !filepath.IsAbs(f.Command[0]) {
+ return fmt.Errorf("config: worker function %s@%s needs a command starting with an absolute path", f.Name, f.Version)
+ }
+ }
+ if w.RunAs != nil && (w.RunAs.UID == 0 || w.RunAs.GID == 0) {
+ return errors.New("config: worker.run_as must not be root (uid or gid 0)")
+ }
+ c := w.Caps
+ if c.MaxTTLMS < 0 || c.MaxWallMS < 0 || c.MaxCPUMS < 0 || c.MaxMemBytes < 0 || c.MaxOutputBytes < 0 || c.MaxJobs < 0 {
+ return errors.New("config: worker.caps must not be negative")
+ }
return nil
return nil
}
}
@@ -4,18 +4,21 @@ import (
@@ -4,18 +4,21 @@ import (
"sync"
"sync"
"github.com/OpenVibers/OpenVibe.Node/internal/protocol"
"github.com/OpenVibers/OpenVibe.Node/internal/protocol"
+ "github.com/OpenVibers/OpenVibe.Node/internal/safety"
)
)
// jobSeenMax bounds how many job ids the Node remembers answers for.
// jobSeenMax bounds how many job ids the Node remembers answers for.
const jobSeenMax = 1024
const jobSeenMax = 1024
-// jobs answers job frames. Nothing runs yet: a job is refused unless its class is advertised, and an accepted job is
-// only remembered (the worker that runs it fills classes and executes accepted jobs).
+// jobs answers job frames: a job is refused unless its class is advertised and admit (the worker) takes it. Running
+// jobs live in the worker's own table, not here.
type jobs struct {
type jobs struct {
mu sync.Mutex
mu sync.Mutex
classes []string // runtime classes this Node runs and advertises; empty: none
classes []string // runtime classes this Node runs and advertises; empty: none
- answers map[string]protocol.Message // job id → the ack or nack it got, resent unchanged for a repeated job
+ answers map[string]protocol.Message // job id → the ack or nack it got, or its job_exit once it ended
order []string // ids in answers, oldest first
order []string // ids in answers, oldest first
+ // admit reserves an accepted job in the worker or returns the fault and reason to nack it with; nil accepts.
+ admit func(protocol.Job) (fault, reason string)
}
}
func newJobs(classes []string) *jobs {
func newJobs(classes []string) *jobs {
@@ -38,6 +41,9 @@ func (j *jobs) request(r protocol.JobRequest) (answer protocol.Message, accepted
@@ -38,6 +41,9 @@ func (j *jobs) request(r protocol.JobRequest) (answer protocol.Message, accepted
return a, false
return a, false
}
}
fault, reason := r.Check(j.classes)
fault, reason := r.Check(j.classes)
+ if fault == "" && j.admit != nil {
+ fault, reason = j.admit(r.Job)
+ }
if fault != "" {
if fault != "" {
answer = protocol.Nack{ID: id, FaultCode: fault, Message: reason}
answer = protocol.Nack{ID: id, FaultCode: fault, Message: reason}
} else {
} else {
@@ -55,6 +61,15 @@ func (j *jobs) request(r protocol.JobRequest) (answer protocol.Message, accepted
@@ -55,6 +61,15 @@ func (j *jobs) request(r protocol.JobRequest) (answer protocol.Message, accepted
return answer, accepted
return answer, accepted
}
}
+// ended makes a job's job_exit the answer to a resent job frame for it.
+func (j *jobs) ended(e protocol.JobExit) {
+ j.mu.Lock()
+ defer j.mu.Unlock()
+ if _, ok := j.answers[e.ID]; ok {
+ j.answers[e.ID] = e
+ }
+}
+
// accepted reports whether id was accepted (and so would be running).
// accepted reports whether id was accepted (and so would be running).
func (j *jobs) accepted(id string) bool {
func (j *jobs) accepted(id string) bool {
j.mu.Lock()
j.mu.Lock()
@@ -63,8 +78,15 @@ func (j *jobs) accepted(id string) bool {
@@ -63,8 +78,15 @@ func (j *jobs) accepted(id string) bool {
return ok
return ok
}
}
-// job answers a job frame with ack or nack, keyed by the job id.
+// job answers a job frame with ack or nack, keyed by the job id, and runs an accepted job. A job the worker holds is
+// answered from it (an ack while it runs, its job_exit once it ended) and never started twice.
func (n *Node) job(r protocol.JobRequest) {
func (n *Node) job(r protocol.JobRequest) {
+ if n.worker != nil {
+ if m, ok := n.worker.Known(r.Job.ID); ok {
+ n.send(m)
+ return
+ }
+ }
answer, accepted := n.jobs.request(r)
answer, accepted := n.jobs.request(r)
if nk, ok := answer.(protocol.Nack); ok {
if nk, ok := answer.(protocol.Nack); ok {
n.log.Info("job refused", "id", r.Job.ID, "class", r.Job.Class, "reason", nk.Message, "err", r.Err)
n.log.Info("job refused", "id", r.Job.ID, "class", r.Job.Class, "reason", nk.Message, "err", r.Err)
@@ -72,14 +94,42 @@ func (n *Node) job(r protocol.JobRequest) {
@@ -72,14 +94,42 @@ func (n *Node) job(r protocol.JobRequest) {
n.log.Info("job accepted", "id", r.Job.ID, "class", r.Job.Class)
n.log.Info("job accepted", "id", r.Job.ID, "class", r.Job.Class)
}
}
n.send(answer)
n.send(answer)
+ if accepted && n.worker != nil {
+ n.worker.Launch(r.Job.ID)
+ }
+}
+
+// sendJobFrame sends a worker frame; a job_exit also becomes the answer to a resent job frame.
+func (n *Node) sendJobFrame(m protocol.Message) {
+ if e, ok := m.(protocol.JobExit); ok {
+ n.jobs.ended(e)
+ }
+ n.send(m)
}
}
// jobCancel stops an accepted job. An unknown, refused or ended id is ignored, as platform.job-frame@1 says, and no
// jobCancel stops an accepted job. An unknown, refused or ended id is ignored, as platform.job-frame@1 says, and no
-// frame answers it: an ack keyed by a job id means the job was accepted.
+// frame answers it: an ack keyed by a job id means the job was accepted. An ended job's unacked job_exit is resent.
func (n *Node) jobCancel(c protocol.JobCancel) {
func (n *Node) jobCancel(c protocol.JobCancel) {
+ if n.worker != nil {
+ if !n.worker.Cancel(c.ID) {
+ n.log.Debug("job_cancel for a job that is not running ignored", "id", c.ID)
+ }
+ return
+ }
if !n.jobs.accepted(c.ID) {
if !n.jobs.accepted(c.ID) {
n.log.Debug("job_cancel for a job that is not running ignored", "id", c.ID)
n.log.Debug("job_cancel for a job that is not running ignored", "id", c.ID)
return
return
}
}
- n.log.Warn("job_cancel: job execution is not implemented, nothing to stop", "id", c.ID)
+ n.log.Warn("job_cancel: no worker runs jobs on this Node, nothing to stop", "id", c.ID)
+}
+
+// stopFault is the fault a job is refused with while the stop latch is set; "" when it is clear.
+func stopFault(st safety.LatchState) string {
+ switch {
+ case st.Local:
+ return protocol.FaultLocalStop
+ case st.Remote:
+ return protocol.FaultEstopped
+ }
+ return ""
}
}
@@ -19,6 +19,7 @@ import (
@@ -19,6 +19,7 @@ import (
"os"
"os"
"os/exec"
"os/exec"
"runtime"
"runtime"
+ "slices"
"sort"
"sort"
"strings"
"strings"
"sync"
"sync"
@@ -32,6 +33,7 @@ import (
@@ -32,6 +33,7 @@ import (
"github.com/OpenVibers/OpenVibe.Node/internal/protocol"
"github.com/OpenVibers/OpenVibe.Node/internal/protocol"
"github.com/OpenVibers/OpenVibe.Node/internal/safety"
"github.com/OpenVibers/OpenVibe.Node/internal/safety"
"github.com/OpenVibers/OpenVibe.Node/internal/video"
"github.com/OpenVibers/OpenVibe.Node/internal/video"
+ "github.com/OpenVibers/OpenVibe.Node/internal/worker"
)
)
// Options configure a Node.
// Options configure a Node.
@@ -42,8 +44,9 @@ type Options struct {
@@ -42,8 +44,9 @@ type Options struct {
Log *slog.Logger
Log *slog.Logger
Version string
Version string
- // RuntimeClasses are the job classes this Node runs and advertises in status.capabilities.worker. None yet:
- // every job is refused "class not available" until the worker that executes jobs fills this in.
+ // RuntimeClasses are the job classes this Node runs and advertises in status.capabilities.worker. New adds
+ // `function` when config.worker is enabled and its isolation probe passes; with none, every job is refused
+ // "class not available".
RuntimeClasses []string
RuntimeClasses []string
// For tests.
// For tests.
@@ -60,9 +63,11 @@ type Node struct {
@@ -60,9 +63,11 @@ type Node struct {
latch *safety.Latch
latch *safety.Latch
dedup *safety.Dedup
dedup *safety.Dedup
jobs *jobs
jobs *jobs
- mgr *plugins.Manager
- link *link.Link
- pub *video.Publisher
+ // worker runs function jobs; nil unless config.worker is enabled on Linux and its boot-time probe passed.
+ worker *worker.Worker
+ mgr *plugins.Manager
+ link *link.Link
+ pub *video.Publisher
ctx context.Context
ctx context.Context
@@ -149,6 +154,20 @@ func New(opt Options) (*Node, error) {
@@ -149,6 +154,20 @@ func New(opt Options) (*Node, error) {
ICEServers: opt.Creds.ICEServers, Log: opt.Log, IncludeLoopback: opt.VideoLoopback})
ICEServers: opt.Creds.ICEServers, Log: opt.Log, IncludeLoopback: opt.VideoLoopback})
}
}
}
}
+ if opt.Config.Worker.Enabled {
+ w := worker.New(opt.Config.Worker, n.sendJobFrame, opt.Log)
+ if err := w.Probe(); err != nil {
+ opt.Log.Error("worker off: jobs cannot run isolated here, every job is refused", "err", err)
+ } else {
+ n.worker = w
+ w.SetStopped(stopFault(n.latch.State()))
+ n.jobs.admit = w.Admit
+ if !slices.Contains(n.jobs.classes, protocol.ClassFunction) {
+ n.jobs.classes = append(n.jobs.classes, protocol.ClassFunction)
+ }
+ opt.Log.Info("worker on", "functions", len(opt.Config.Worker.Functions))
+ }
+ }
n.latch.OnChange(n.onLatch)
n.latch.OnChange(n.onLatch)
return n, nil
return n, nil
}
}
@@ -188,6 +207,9 @@ func (n *Node) Run(ctx context.Context) error {
@@ -188,6 +207,9 @@ func (n *Node) Run(ctx context.Context) error {
<-ctx.Done()
<-ctx.Done()
n.log.Info("shutting down: stopping every actuator")
n.log.Info("shutting down: stopping every actuator")
n.mgr.StopAll()
n.mgr.StopAll()
+ if n.worker != nil {
+ n.worker.Close()
+ }
cancelPlugins()
cancelPlugins()
n.mgr.Wait()
n.mgr.Wait()
wg.Wait()
wg.Wait()
@@ -236,8 +258,11 @@ func (n *Node) Frame(f protocol.Frame) {
@@ -236,8 +258,11 @@ func (n *Node) Frame(f protocol.Frame) {
case protocol.JobCancel:
case protocol.JobCancel:
n.jobCancel(m)
n.jobCancel(m)
case protocol.JobExitAck:
case protocol.JobExitAck:
- // No job runs yet, so no job_exit is ever waiting for this.
- n.log.Debug("job_exit_ack ignored", "id", m.ID)
+ if n.worker != nil {
+ n.worker.ExitAck(m.ID)
+ } else {
+ n.log.Debug("job_exit_ack ignored: no worker", "id", m.ID)
+ }
}
}
}
}
@@ -278,6 +303,9 @@ func (n *Node) applyConfig(c protocol.Config) {
@@ -278,6 +303,9 @@ func (n *Node) applyConfig(c protocol.Config) {
n.mu.Unlock()
n.mu.Unlock()
if first {
if first {
n.sendStatus()
n.sendStatus()
+ if n.worker != nil {
+ n.worker.Resend()
+ }
}
}
n.sendEstopState()
n.sendEstopState()
if n.link != nil {
if n.link != nil {
@@ -428,6 +456,9 @@ func (n *Node) send(m protocol.Message) {
@@ -428,6 +456,9 @@ func (n *Node) send(m protocol.Message) {
// ---- latch ----
// ---- latch ----
func (n *Node) onLatch(st safety.LatchState) {
func (n *Node) onLatch(st safety.LatchState) {
+ if n.worker != nil {
+ n.worker.SetStopped(stopFault(st)) // kills every job at once while stopped
+ }
if st.Stopped() {
if st.Stopped() {
n.log.Warn("stop latched", "remote", st.Remote, "local", st.Local)
n.log.Warn("stop latched", "remote", st.Remote, "local", st.Local)
n.mgr.EstopAll()
n.mgr.EstopAll()
new file mode 100644
new file mode 100644
@@ -0,0 +1,195 @@
@@ -0,0 +1,195 @@
+//go:build linux
+
+package node
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "log/slog"
+ "os"
+ "path/filepath"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/OpenVibers/OpenVibe.Node/internal/config"
+ "github.com/OpenVibers/OpenVibe.Node/internal/fakebot"
+ "github.com/OpenVibers/OpenVibe.Node/internal/link"
+ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"
+)
+
+// TestWorkerHelperProcess is the job's process when this test binary runs as the `hello` function (`-- job`): it
+// appends a line to the marker file named in its args, prints, holds for 1.5 s and writes its result to fd 3.
+func TestWorkerHelperProcess(t *testing.T) {
+ i := slices.Index(os.Args, "--")
+ if i < 0 || i+1 >= len(os.Args) || os.Args[i+1] != "job" {
+ return
+ }
+ var args struct {
+ Marker string `json:"marker"`
+ }
+ _ = json.NewDecoder(os.Stdin).Decode(&args)
+ if f, err := os.OpenFile(args.Marker, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600); err == nil {
+ fmt.Fprintln(f, "started")
+ f.Close()
+ }
+ fmt.Println("hello from the job")
+ time.Sleep(1500 * time.Millisecond)
+ fmt.Fprint(os.NewFile(3, "result"), `{"ok":true}`)
+ os.Exit(0)
+}
+
+// startWorkerNode pairs a Node with no plugin and no video whose worker runs the `hello` function, and waits for the
+// status that advertises it. It is skipped where the kernel does not let this user create the job namespaces.
+func startWorkerNode(t *testing.T) *env {
+ t.Helper()
+ exe, err := os.Executable()
+ if err != nil {
+ t.Fatal(err)
+ }
+ home, err := os.MkdirTemp("", "ovn")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { os.RemoveAll(home) })
+ paths := config.HomePaths(home)
+ srv := fakebot.New()
+ t.Cleanup(srv.Close)
+ srv.Config = protocol.Config{HeartbeatMS: 100, Limits: protocol.Limits{MaxCommandMS: 1000}}
+ srv.AddCode("TEST2345")
+ creds, err := link.Pair(context.Background(), nil, srv.URL(), protocol.PairRequest{Code: "TEST2345", AgentVersion: "test",
+ DeviceKind: "onboard"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ cfg := config.Default()
+ cfg.Plugins = nil
+ cfg.Video = config.VideoConfig{Source: "off"}
+ cfg.Worker = config.WorkerConfig{Enabled: true, AllowSameUser: true, Caps: config.WorkerCaps{MaxMemBytes: 8 << 30},
+ Functions: []config.FunctionConfig{{Name: "hello", Version: "1.0.0", Command: []string{exe, "-test.run=^TestWorkerHelperProcess$", "--", "job"}}}}
+ if err := cfg.Validate(); err != nil {
+ t.Fatal(err)
+ }
+ n, err := New(Options{Config: cfg, Paths: paths, Creds: creds, Version: "test", HeartbeatInterval: 100 * time.Millisecond,
+ LinkBackoffMin: 50 * time.Millisecond, LatchPoll: 50 * time.Millisecond, Log: slog.New(slog.NewTextHandler(io.Discard, nil))})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !slices.Contains(n.jobs.runtimeClasses(), protocol.ClassFunction) {
+ if os.Getenv("OPENVIBE_WORKER_TESTS") == "require" {
+ t.Fatal("job isolation unavailable: the worker probe failed")
+ }
+ t.Skip("job isolation unavailable here (OPENVIBE_WORKER_TESTS=require fails instead)")
+ }
+ ctx, cancel := context.WithCancel(context.Background())
+ done := make(chan struct{})
+ go func() { n.Run(ctx); close(done) }()
+ e := &env{t: t, srv: srv, node: n, paths: paths, cancel: cancel, done: done}
+ t.Cleanup(e.stop)
+ e.conn = e.nextConn()
+ _, err = e.conn.Expect(protocol.TypeStatus, 10*time.Second, func(m protocol.Message) bool {
+ _, ok := m.(protocol.Status).Capabilities[protocol.CapWorker]
+ return ok
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return e
+}
+
+// TestJobRunsOverTheLink: a job sent by the fake Bot is acked, then job_started, job_stdout, job_usage (second 0)
+// and job_exit arrive in that order; a job frame resent while it runs is acked again and starts no second process;
+// after the end a resent job is answered with the same job_exit, before and after job_exit_ack.
+func TestJobRunsOverTheLink(t *testing.T) {
+ e := startWorkerNode(t)
+ marker := filepath.Join(e.paths.StateDir, "starts")
+ j := testJob(jobID(1))
+ j.Artifact = &protocol.Artifact{Name: "hello", Version: "1.0.0"}
+ j.Args = json.RawMessage(fmt.Sprintf(`{"marker":%q}`, marker))
+ j.Limits.MemBytes = 4 << 30
+ isJob := func(m protocol.Message) bool {
+ switch f := m.(type) {
+ case protocol.JobStarted:
+ return f.ID == j.ID
+ case protocol.JobExit:
+ return f.ID == j.ID
+ }
+ return false
+ }
+
+ if r := e.jobReply(j); r != (protocol.Ack{ID: j.ID}) {
+ t.Fatalf("%+v", r)
+ }
+ if _, err := e.conn.Expect(protocol.TypeJobStarted, 10*time.Second, isJob); err != nil {
+ t.Fatal(err)
+ }
+ if r := e.jobReply(j); r != (protocol.Ack{ID: j.ID}) {
+ t.Fatalf("resent while running: %+v", r)
+ }
+ f, err := e.conn.Expect(protocol.TypeJobExit, 15*time.Second, isJob)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ex := f.Msg.(protocol.JobExit)
+ if ex.Reason != protocol.ExitExited || ex.Code == nil || *ex.Code != 0 || string(ex.Result) != `{"ok":true}` || ex.Usage.StartedMS == nil {
+ t.Fatalf("%+v %s", ex, ex.Result)
+ }
+
+ // Order on the wire: ack, job_started, job_stdout, job_usage, job_exit; one job_started only.
+ var kinds []string
+ var out strings.Builder
+ started := 0
+ for _, f := range e.conn.Frames() {
+ switch m := f.Msg.(type) {
+ case protocol.JobStarted:
+ if m.ID == j.ID {
+ started++
+ kinds = append(kinds, f.Type)
+ }
+ case protocol.JobStdout:
+ if m.ID == j.ID {
+ out.WriteString(m.Chunk)
+ kinds = append(kinds, f.Type)
+ }
+ case protocol.JobUsage:
+ if m.ID == j.ID && m.Second == 0 && m.StartedMS == *ex.Usage.StartedMS {
+ kinds = append(kinds, f.Type)
+ }
+ case protocol.JobExit:
+ if m.ID == j.ID {
+ kinds = append(kinds, f.Type)
+ }
+ }
+ }
+ kinds = slices.Compact(kinds)
+ want := []string{protocol.TypeJobStarted, protocol.TypeJobStdout, protocol.TypeJobUsage, protocol.TypeJobExit}
+ if started != 1 || !slices.Equal(kinds, want) || out.String() != "hello from the job\n" {
+ t.Fatalf("%d job_started, frames %v, stdout %q", started, kinds, out.String())
+ }
+
+ // Ended: a resent job gets the same job_exit, before and after job_exit_ack, and never runs again.
+ for _, ack := range []bool{false, true} {
+ if ack {
+ if err := e.conn.JobExitAck(j.ID); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := e.conn.Job(j); err != nil {
+ t.Fatal(err)
+ }
+ f, err := e.conn.Expect(protocol.TypeJobExit, 5*time.Second, isJob)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if again := f.Msg.(protocol.JobExit); again.Reason != ex.Reason || *again.Usage.StartedMS != *ex.Usage.StartedMS || again.Usage.WallMS != ex.Usage.WallMS {
+ t.Fatalf("resent job answered %+v, first job_exit %+v", again, ex)
+ }
+ }
+ b, err := os.ReadFile(marker)
+ if err != nil || string(b) != "started\n" {
+ t.Fatalf("the function ran %q (%v), want once", b, err)
+ }
+}
@@ -97,6 +97,11 @@ const (
@@ -97,6 +97,11 @@ const (
JobNetUnsupported = "net policy not supported"
JobNetUnsupported = "net policy not supported"
JobBadLimits = "missing or invalid args, ttl_ms or limits"
JobBadLimits = "missing or invalid args, ttl_ms or limits"
JobNotAvailable = "class not available"
JobNotAvailable = "class not available"
+
+ // Refused by a running worker (platform.job@1: a missing artifact, the local kill switch, limits it cannot honor).
+ JobUnknownArtifact = "unknown artifact" // not a function declared in the Node's local config
+ JobStopped = "the e-stop or local stop is latched" // fault estopped or local_stop
+ JobBusy = "worker busy" // worker.caps.max_jobs are running; fault not_ready
)
)
// job_exit reasons.
// job_exit reasons.
new file mode 100644
new file mode 100644
@@ -0,0 +1,304 @@
@@ -0,0 +1,304 @@
+//go:build linux
+
+package worker
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "io/fs"
+ "net"
+ "os"
+ "path/filepath"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/OpenVibers/OpenVibe.Node/internal/config"
+ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"
+)
+
+// TestHelperProcess is the job's process when this test binary runs as a function (`-- <mode>`); otherwise it does
+// nothing.
+func TestHelperProcess(t *testing.T) {
+ i := slices.Index(os.Args, "--")
+ if i < 0 || i+1 >= len(os.Args) {
+ return
+ }
+ result := os.NewFile(3, "result")
+ switch os.Args[i+1] {
+ case "stream":
+ in, _ := io.ReadAll(os.Stdin)
+ fmt.Println("line 1")
+ time.Sleep(300 * time.Millisecond)
+ fmt.Println("line 2 é")
+ time.Sleep(1900 * time.Millisecond)
+ fmt.Fprintf(result, `{"got":%s}`, bytes.TrimSpace(in))
+ case "sleep":
+ fmt.Println("sleeping")
+ time.Sleep(time.Hour)
+ case "flood":
+ line := strings.Repeat("x", 1023) + "\n"
+ for {
+ os.Stdout.WriteString(line)
+ }
+ case "env":
+ cwd, _ := os.Getwd()
+ // A down loopback still lets a socket bind 127.0.0.1; connecting to it is what fails.
+ lerr := errors.New("no listener")
+ if l, err := net.Listen("tcp", "127.0.0.1:0"); err == nil {
+ _, lerr = net.DialTimeout("tcp", l.Addr().String(), time.Second)
+ }
+ _, derr := net.DialTimeout("tcp", "192.0.2.1:9", time.Second)
+ _ = json.NewEncoder(result).Encode(map[string]any{"env": os.Environ(), "cwd": cwd, "pid": os.Getpid(),
+ "uid": os.Getuid(), "loopback": fmt.Sprint(lerr), "dial": fmt.Sprint(derr)})
+ }
+ os.Exit(0)
+}
+
+// newRunWorker is a worker running the helper modes, skipped where the kernel does not let this user create the
+// namespaces (CI sets OPENVIBE_WORKER_TESTS=require to fail instead).
+func newRunWorker(t *testing.T, caps config.WorkerCaps, modes ...string) (*Worker, *sink) {
+ t.Helper()
+ if caps.MaxMemBytes == 0 {
+ caps.MaxMemBytes = 8 << 30
+ }
+ s := &sink{}
+ w := New(helperConfig(t, caps, modes...), s.send, quiet())
+ if err := w.Probe(); err != nil {
+ if os.Getenv("OPENVIBE_WORKER_TESTS") == "require" {
+ t.Fatalf("job isolation unavailable: %v", err)
+ }
+ t.Skipf("job isolation unavailable here (OPENVIBE_WORKER_TESTS=require fails instead): %v", err)
+ }
+ t.Cleanup(w.Close)
+ return w, s
+}
+
+func run(t *testing.T, w *Worker, j protocol.Job) {
+ t.Helper()
+ if f, r := w.Admit(j); f != "" {
+ t.Fatalf("%s refused: %s %s", j.ID, f, r)
+ }
+ w.Launch(j.ID)
+}
+
+// TestRunStreamsAndMeters: a function gets its args on stdin, its stdout arrives as job_stdout in order, one
+// job_usage per elapsed wall-clock second anchored on job_started, and job_exit carries code 0, the result it wrote to
+// fd 3 and usage covering every second sent. The job_exit is resent until job_exit_ack, then the job is forgotten.
+func TestRunStreamsAndMeters(t *testing.T) {
+ w, s := newRunWorker(t, config.WorkerCaps{}, "stream")
+ j := testJob(1, "stream")
+ run(t, w, j)
+ ex := s.exit(t, j.ID)
+ if ex.Reason != protocol.ExitExited || ex.Code == nil || *ex.Code != 0 || string(ex.Result) != `{"got":{"n":7}}` {
+ t.Fatalf("%+v result %s", ex, ex.Result)
+ }
+ if ex.Usage.StartedMS == nil || ex.Usage.WallMS < 2000 || ex.Usage.CPUMS == nil || ex.Usage.MemPeakBytes == nil || *ex.Usage.MemPeakBytes == 0 {
+ t.Fatalf("usage %+v", ex.Usage)
+ }
+ frames := s.all()
+ started, ok := frames[0].(protocol.JobStarted)
+ if !ok || started.StartedMS != *ex.Usage.StartedMS {
+ t.Fatalf("first frame %+v, want job_started at %d", frames[0], *ex.Usage.StartedMS)
+ }
+ if _, ok := frames[len(frames)-1].(protocol.JobExit); !ok {
+ t.Fatalf("last frame %+v, want job_exit", frames[len(frames)-1])
+ }
+ var out strings.Builder
+ var chunk uint64
+ var second int64
+ for _, m := range frames {
+ switch f := m.(type) {
+ case protocol.JobStdout:
+ if f.ChunkSeq != chunk+1 {
+ t.Fatalf("chunk_seq %d after %d", f.ChunkSeq, chunk)
+ }
+ chunk = f.ChunkSeq
+ out.WriteString(f.Chunk)
+ case protocol.JobUsage:
+ if f.StartedMS != started.StartedMS || f.Second != second || f.CPUMS == nil {
+ t.Fatalf("usage %+v, want second %d at %d", f, second, started.StartedMS)
+ }
+ second++
+ }
+ }
+ if out.String() != "line 1\nline 2 é\n" {
+ t.Fatalf("stdout %q", out.String())
+ }
+ if second < 2 || second*1000 > ex.Usage.WallMS {
+ t.Fatalf("%d usage seconds for wall_ms %d", second, ex.Usage.WallMS)
+ }
+
+ if m, ok := w.Known(j.ID); !ok || m.(protocol.JobExit).Reason != protocol.ExitExited {
+ t.Fatalf("a resent job is answered %+v, want its job_exit", m)
+ }
+ w.Resend()
+ if again := s.wait(t, protocol.TypeJobExit, j.ID, 2, 5*time.Second).(protocol.JobExit); *again.Usage.StartedMS != *ex.Usage.StartedMS || again.Usage.WallMS != ex.Usage.WallMS {
+ t.Fatalf("resent job_exit %+v differs from %+v", again.Usage, ex.Usage)
+ }
+ w.ExitAck(j.ID)
+ if _, ok := w.Known(j.ID); ok {
+ t.Fatal("job still held after job_exit_ack")
+ }
+}
+
+// TestIsolation: the job sees none of the Node's environment, has no network (not even loopback), is PID 1 of its own
+// PID namespace, runs as the configured user, and its working directory is fresh and removed afterwards.
+func TestIsolation(t *testing.T) {
+ t.Setenv("OPENVIBE_NODE_SECRET", "leak")
+ w, s := newRunWorker(t, config.WorkerCaps{}, "env")
+ j := testJob(1, "env")
+ run(t, w, j)
+ ex := s.exit(t, j.ID)
+ var r struct {
+ Env []string
+ Cwd string
+ PID, UID int
+ Loopback, Dial string
+ }
+ if err := json.Unmarshal(ex.Result, &r); err != nil || ex.Reason != protocol.ExitExited {
+ t.Fatalf("%+v %s: %v", ex, ex.Result, err)
+ }
+ if r.PID != 1 || r.UID != os.Getuid() {
+ t.Fatalf("pid %d uid %d, want 1 and %d", r.PID, r.UID, os.Getuid())
+ }
+ if r.Loopback == "<nil>" || r.Dial == "<nil>" {
+ t.Fatalf("network reachable: loopback %s, dial %s", r.Loopback, r.Dial)
+ }
+ for _, kv := range r.Env {
+ if strings.HasPrefix(kv, "OPENVIBE_NODE_SECRET=") {
+ t.Fatalf("the Node's environment leaked: %v", r.Env)
+ }
+ }
+ if !slices.Contains(r.Env, "OPENVIBE_JOB_ID="+j.ID) || !slices.Contains(r.Env, "HOME="+r.Cwd) {
+ t.Fatalf("env %v", r.Env)
+ }
+ if !strings.HasPrefix(filepath.Base(r.Cwd), "openvibe-job-") {
+ t.Fatalf("cwd %s", r.Cwd)
+ }
+ if _, err := os.Stat(r.Cwd); !errors.Is(err, fs.ErrNotExist) {
+ t.Fatalf("working directory %s not removed: %v", r.Cwd, err)
+ }
+}
+
+// TestTTLAndLimitKill: ttl_ms (from receipt) and wall_ms (from start) kill the process group, each clamped by the
+// local cap when that is stricter; job_exit says ttl or limit with no exit code.
+func TestTTLAndLimitKill(t *testing.T) {
+ for i, c := range []struct {
+ name string
+ caps config.WorkerCaps
+ ttl, wall int64
+ reason string
+ }{
+ {"ttl", config.WorkerCaps{}, 800, 60000, protocol.ExitTTL},
+ {"ttl clamped by the local cap", config.WorkerCaps{MaxTTLMS: 800}, 60000, 60000, protocol.ExitTTL},
+ {"wall_ms", config.WorkerCaps{}, 60000, 800, protocol.ExitLimit},
+ {"wall_ms clamped by the local cap", config.WorkerCaps{MaxWallMS: 800}, 60000, 60000, protocol.ExitLimit},
+ } {
+ t.Run(c.name, func(t *testing.T) {
+ w, s := newRunWorker(t, c.caps, "sleep")
+ j := testJob(i+1, "sleep")
+ j.TTLMS, j.Limits.WallMS = c.ttl, c.wall
+ begin := time.Now()
+ run(t, w, j)
+ ex := s.exit(t, j.ID)
+ if ex.Reason != c.reason || ex.Code != nil || ex.Usage.StartedMS == nil || s.count(protocol.TypeJobStarted, j.ID) != 1 {
+ t.Fatalf("%+v", ex)
+ }
+ if d := time.Since(begin); d > 5*time.Second {
+ t.Fatalf("killed after %s", d)
+ }
+ })
+ }
+}
+
+// TestCancelKill: job_cancel kills a running job (cancelled, no code); a cancel after the end resends the unacked
+// job_exit and is ignored once acked; a job cancelled before it started never starts.
+func TestCancelKill(t *testing.T) {
+ w, s := newRunWorker(t, config.WorkerCaps{MaxJobs: 2}, "sleep")
+ j := testJob(1, "sleep")
+ run(t, w, j)
+ s.wait(t, protocol.TypeJobStarted, j.ID, 1, 10*time.Second)
+ if !w.Cancel(j.ID) {
+ t.Fatal("running job unknown")
+ }
+ ex := s.exit(t, j.ID)
+ if ex.Reason != protocol.ExitCancelled || ex.Code != nil || ex.Usage.StartedMS == nil {
+ t.Fatalf("%+v", ex)
+ }
+ w.Cancel(j.ID)
+ s.wait(t, protocol.TypeJobExit, j.ID, 2, 5*time.Second)
+ w.ExitAck(j.ID)
+ if w.Cancel(j.ID) {
+ t.Fatal("cancel of an acked job was not ignored")
+ }
+
+ p := testJob(2, "sleep")
+ if f, r := w.Admit(p); f != "" {
+ t.Fatalf("%s %s", f, r)
+ }
+ w.Cancel(p.ID)
+ w.Launch(p.ID)
+ ex = s.exit(t, p.ID)
+ if ex.Reason != protocol.ExitCancelled || ex.Usage.StartedMS != nil || ex.Usage.WallMS != 0 || s.count(protocol.TypeJobStarted, p.ID) != 0 {
+ t.Fatalf("%+v", ex)
+ }
+}
+
+// TestOutputCap: stdout past max_output_bytes is cut at exactly the cap and the job is killed (limit); chunks are
+// numbered from 1 and never larger than chunkBytes.
+func TestOutputCap(t *testing.T) {
+ const capBytes = 64 << 10
+ w, s := newRunWorker(t, config.WorkerCaps{MaxOutputBytes: capBytes}, "flood")
+ j := testJob(1, "flood")
+ run(t, w, j)
+ ex := s.exit(t, j.ID)
+ if ex.Reason != protocol.ExitLimit || ex.Code != nil {
+ t.Fatalf("%+v", ex)
+ }
+ total, seq := 0, uint64(0)
+ for _, m := range s.all() {
+ if f, ok := m.(protocol.JobStdout); ok {
+ if f.ChunkSeq != seq+1 || len(f.Chunk) > chunkBytes {
+ t.Fatalf("chunk %d after %d, %d bytes", f.ChunkSeq, seq, len(f.Chunk))
+ }
+ seq = f.ChunkSeq
+ total += len(f.Chunk)
+ }
+ }
+ if total != capBytes {
+ t.Fatalf("%d stdout bytes sent, cap %d", total, capBytes)
+ }
+}
+
+// TestLatchKillsAndRefuses: setting the stop latch kills every running job at once (stopped) and refuses new jobs with
+// the latch's fault; once cleared, jobs run again.
+func TestLatchKillsAndRefuses(t *testing.T) {
+ w, s := newRunWorker(t, config.WorkerCaps{MaxJobs: 2}, "sleep")
+ a, b := testJob(1, "sleep"), testJob(2, "sleep")
+ run(t, w, a)
+ run(t, w, b)
+ s.wait(t, protocol.TypeJobStarted, a.ID, 1, 10*time.Second)
+ s.wait(t, protocol.TypeJobStarted, b.ID, 1, 10*time.Second)
+ w.SetStopped(protocol.FaultEstopped)
+ for _, j := range []protocol.Job{a, b} {
+ if ex := s.exit(t, j.ID); ex.Reason != protocol.ExitStopped || ex.Code != nil {
+ t.Fatalf("%+v", ex)
+ }
+ }
+ if f, r := w.Admit(testJob(3, "sleep")); f != protocol.FaultEstopped || r != protocol.JobStopped {
+ t.Fatalf("while latched: %q %q", f, r)
+ }
+ w.SetStopped("")
+ c := testJob(4, "sleep")
+ run(t, w, c)
+ s.wait(t, protocol.TypeJobStarted, c.ID, 1, 10*time.Second)
+ w.Cancel(c.ID)
+ if ex := s.exit(t, c.ID); ex.Reason != protocol.ExitCancelled {
+ t.Fatalf("%+v", ex)
+ }
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,517 @@
@@ -0,0 +1,517 @@
+// Package worker runs `function` jobs (OpenVibe.Contracts platform.job@1) from the control link. It runs only the
+// functions the owner declared in the local config, each job as one child process in its own process group and in new
+// user, network and PID namespaces (no network at all), with a scrubbed environment and a fresh working directory, and
+// kills it at its ttl, at its limits, on job_cancel and while the stop latch is set. It never runs a job with less
+// isolation: a job whose namespaces cannot be created fails without running, and a Node whose boot-time Probe fails
+// does not advertise the class. docs/worker.md has the model and its limits.
+package worker
+
+import (
+ "bytes"
+ "encoding/json"
+ "io"
+ "log/slog"
+ "os"
+ "os/exec"
+ "sort"
+ "strings"
+ "sync"
+ "time"
+ "unicode/utf8"
+
+ "github.com/OpenVibers/OpenVibe.Node/internal/config"
+ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"
+)
+
+const (
+ chunkBytes = 16 << 10 // the most stdout bytes one job_stdout carries
+ resultBytes = 256 << 10 // the most bytes of result a function may write to fd 3
+ sampleEvery = 200 * time.Millisecond
+ endedMax = 1024 // ended jobs kept for their job_exit until job_exit_ack
+)
+
+// Worker runs function jobs and sends their frames with send.
+type Worker struct {
+ cfg config.WorkerConfig
+ caps config.WorkerCaps
+ send func(protocol.Message)
+ log *slog.Logger
+
+ // isolate puts a job's command in its namespaces; tests replace it to stand for a kernel that cannot.
+ isolate func(*exec.Cmd, config.WorkerConfig) error
+
+ mu sync.Mutex
+ stopped string // the fault code while the stop latch is set; "" when clear
+ closed bool // shutting down
+ jobs map[string]*job // admitted, running, or ended with its job_exit not yet acked
+ ended []string // ids of ended jobs in jobs, oldest first
+ wg sync.WaitGroup
+}
+
+type job struct {
+ req protocol.Job
+ fn config.FunctionConfig
+ received time.Time
+ ttl time.Duration
+ limits protocol.JobLimits // the job's, clamped to the local caps
+
+ reason string // why it was killed (the first reason wins); "" while nothing killed it
+ launched bool // Launch was called
+ proc *os.Process // set while the process runs
+ startedMS int64
+ exit *protocol.JobExit // set once it ended
+}
+
+// New returns a Worker for cfg. It runs nothing until Probe has passed and a job is admitted and launched.
+func New(cfg config.WorkerConfig, send func(protocol.Message), log *slog.Logger) *Worker {
+ if log == nil {
+ log = slog.Default()
+ }
+ return &Worker{cfg: cfg, caps: cfg.Caps.WithDefaults(), send: send, log: log, isolate: isolate, jobs: map[string]*job{}}
+}
+
+// Probe starts a process the way a job is started and checks that it got user, network and PID namespaces of its own
+// and that its CPU limit can be set and its usage read. The class must not be advertised when Probe fails.
+func (w *Worker) Probe() error { return probe(w.cfg, w.isolate) }
+
+// Admit reserves a job, or returns the fault code and reason to nack it with. It runs nothing (Launch does, once the
+// ack is out). An id the worker already holds is admitted again without a second reservation.
+func (w *Worker) Admit(j protocol.Job) (fault, reason string) {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ if _, ok := w.jobs[j.ID]; ok {
+ return "", ""
+ }
+ if w.stopped != "" {
+ return w.stopped, protocol.JobStopped
+ }
+ if w.closed {
+ return protocol.FaultShuttingDown, protocol.JobNotAvailable
+ }
+ if j.Class != protocol.ClassFunction {
+ return protocol.FaultUnsupported, protocol.JobNotAvailable
+ }
+ fn, ok := w.function(j.Artifact)
+ if !ok {
+ return protocol.FaultUnsupported, protocol.JobUnknownArtifact
+ }
+ running := 0
+ for _, jb := range w.jobs {
+ if jb.exit == nil {
+ running++
+ }
+ }
+ if running >= w.caps.MaxJobs {
+ return protocol.FaultNotReady, protocol.JobBusy
+ }
+ w.jobs[j.ID] = &job{req: j, fn: fn, received: time.Now(), ttl: ms(min(j.TTLMS, w.caps.MaxTTLMS)),
+ limits: protocol.JobLimits{WallMS: min(j.Limits.WallMS, w.caps.MaxWallMS), CPUMS: min(j.Limits.CPUMS, w.caps.MaxCPUMS),
+ MemBytes: min(j.Limits.MemBytes, w.caps.MaxMemBytes)}}
+ return "", ""
+}
+
+func (w *Worker) function(a *protocol.Artifact) (config.FunctionConfig, bool) {
+ if a == nil {
+ return config.FunctionConfig{}, false
+ }
+ for _, f := range w.cfg.Functions {
+ if f.Name == a.Name && f.Version == a.Version {
+ return f, true
+ }
+ }
+ return config.FunctionConfig{}, false
+}
+
+// Launch runs an admitted job on its own goroutine. A job already launched, or unknown, is left alone.
+func (w *Worker) Launch(id string) {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ jb, ok := w.jobs[id]
+ if !ok || jb.launched || w.closed {
+ return
+ }
+ jb.launched = true
+ w.wg.Add(1)
+ go w.run(jb)
+}
+
+// Known returns what a resent job frame for id is answered with: an ack while the job waits or runs, its job_exit
+// once it ended (until job_exit_ack).
+func (w *Worker) Known(id string) (protocol.Message, bool) {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ jb, ok := w.jobs[id]
+ switch {
+ case !ok:
+ return nil, false
+ case jb.exit != nil:
+ return *jb.exit, true
+ }
+ return protocol.Ack{ID: id}, true
+}
+
+// Cancel kills a job (its job_exit says cancelled; one not started yet never starts). For an ended job whose job_exit
+// is not acked yet it resends that job_exit. An unknown id is ignored; Cancel reports whether id was known.
+func (w *Worker) Cancel(id string) bool {
+ w.mu.Lock()
+ jb, ok := w.jobs[id]
+ var resend *protocol.JobExit
+ if ok {
+ resend = jb.exit
+ w.killLocked(jb, protocol.ExitCancelled)
+ }
+ w.mu.Unlock()
+ if resend != nil {
+ w.send(*resend)
+ }
+ return ok
+}
+
+// ExitAck forgets an ended job: its usage is recorded, so its job_exit is not resent any more.
+func (w *Worker) ExitAck(id string) {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ if jb, ok := w.jobs[id]; !ok || jb.exit == nil {
+ return
+ }
+ delete(w.jobs, id)
+ for i, e := range w.ended {
+ if e == id {
+ w.ended = append(w.ended[:i], w.ended[i+1:]...)
+ break
+ }
+ }
+}
+
+// SetStopped follows the stop latch (the e-stop or the local stop). A fault code kills every job at once (job_exit
+// says stopped) and refuses new ones with that fault until SetStopped("").
+func (w *Worker) SetStopped(fault string) {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ w.stopped = fault
+ if fault == "" {
+ return
+ }
+ for _, jb := range w.jobs {
+ w.killLocked(jb, protocol.ExitStopped)
+ }
+}
+
+// Resend sends again, after a reconnect, job_started for every running job and job_exit for every ended job whose
+// job_exit is not acked yet.
+func (w *Worker) Resend() {
+ w.mu.Lock()
+ var msgs []protocol.Message
+ for _, jb := range w.jobs {
+ if jb.exit == nil && jb.proc != nil {
+ msgs = append(msgs, protocol.JobStarted{ID: jb.req.ID, StartedMS: jb.startedMS})
+ }
+ }
+ for _, id := range w.ended {
+ msgs = append(msgs, *w.jobs[id].exit)
+ }
+ w.mu.Unlock()
+ for _, m := range msgs {
+ w.send(m)
+ }
+}
+
+// Close kills every job (job_exit says stopped), refuses new ones and waits up to 5 s for them to end.
+func (w *Worker) Close() {
+ w.mu.Lock()
+ w.closed = true
+ for _, jb := range w.jobs {
+ w.killLocked(jb, protocol.ExitStopped)
+ }
+ w.mu.Unlock()
+ done := make(chan struct{})
+ go func() { w.wg.Wait(); close(done) }()
+ select {
+ case <-done:
+ case <-time.After(5 * time.Second):
+ w.log.Error("jobs did not end within 5 s of being killed")
+ }
+}
+
+func (w *Worker) kill(jb *job, reason string) {
+ w.mu.Lock()
+ defer w.mu.Unlock()
+ w.killLocked(jb, reason)
+}
+
+// killLocked records why jb is killed and SIGKILLs its process group; a job not started yet then never starts.
+func (w *Worker) killLocked(jb *job, reason string) {
+ if jb.exit != nil {
+ return
+ }
+ if jb.reason == "" {
+ jb.reason = reason
+ }
+ if jb.proc != nil {
+ killGroup(jb.proc)
+ }
+}
+
+func (w *Worker) run(jb *job) {
+ defer w.wg.Done()
+ ex := w.execute(jb)
+ w.mu.Lock()
+ jb.exit = &ex
+ w.ended = append(w.ended, ex.ID)
+ for len(w.ended) > endedMax {
+ delete(w.jobs, w.ended[0])
+ w.ended = w.ended[1:]
+ }
+ w.mu.Unlock()
+ w.log.Info("job ended", "id", ex.ID, "reason", ex.Reason, "wall_ms", ex.Usage.WallMS)
+ w.send(ex)
+}
+
+// execute runs jb's process to its end and returns its job_exit.
+func (w *Worker) execute(jb *job) protocol.JobExit {
+ id := jb.req.ID
+ never := func(reason string, err error) protocol.JobExit {
+ if err != nil {
+ w.log.Error("job not run", "id", id, "err", err)
+ }
+ return protocol.JobExit{ID: id, Reason: reason, Result: json.RawMessage("null")}
+ }
+ w.mu.Lock()
+ reason := jb.reason
+ w.mu.Unlock()
+ if reason != "" { // cancelled or stopped before it started
+ return never(reason, nil)
+ }
+ dir, err := os.MkdirTemp("", "openvibe-job-")
+ if err != nil {
+ return never(protocol.ExitFailed, err)
+ }
+ defer os.RemoveAll(dir)
+ if r := w.cfg.RunAs; r != nil {
+ if err := os.Chown(dir, int(r.UID), int(r.GID)); err != nil {
+ return never(protocol.ExitFailed, err)
+ }
+ }
+ cmd := exec.Command(jb.fn.Command[0], jb.fn.Command[1:]...)
+ cmd.Dir, cmd.Env = dir, jobEnv(jb.fn, dir, id)
+ cmd.Stdin = bytes.NewReader(append(append([]byte(nil), jb.req.Args...), '\n'))
+ if err := w.isolate(cmd, w.cfg); err != nil {
+ return never(protocol.ExitFailed, err)
+ }
+ outR, outW, err := os.Pipe()
+ if err != nil {
+ return never(protocol.ExitFailed, err)
+ }
+ defer outR.Close()
+ resR, resW, err := os.Pipe()
+ if err != nil {
+ outW.Close()
+ return never(protocol.ExitFailed, err)
+ }
+ defer resR.Close()
+ cmd.Stdout, cmd.ExtraFiles = outW, []*os.File{resW}
+
+ w.mu.Lock()
+ if jb.reason == "" && time.Since(jb.received) >= jb.ttl {
+ jb.reason = protocol.ExitTTL
+ }
+ reason = jb.reason
+ var start time.Time
+ if reason == "" {
+ err = cmd.Start()
+ start = time.Now()
+ if err == nil {
+ jb.proc, jb.startedMS = cmd.Process, start.UnixMilli()
+ }
+ }
+ w.mu.Unlock()
+ outW.Close()
+ resW.Close()
+ if reason != "" {
+ return never(reason, nil)
+ }
+ if err != nil {
+ return never(protocol.ExitFailed, err)
+ }
+ pid := cmd.Process.Pid
+ w.send(protocol.JobStarted{ID: id, StartedMS: jb.startedMS})
+ if err := limitCPU(pid, jb.limits.CPUMS); err != nil {
+ w.log.Error("job killed: its CPU limit could not be set", "id", id, "err", err)
+ w.kill(jb, protocol.ExitFailed)
+ }
+
+ stop, watched := make(chan struct{}), make(chan accounting, 1)
+ go func() { watched <- w.watch(jb, pid, start, stop) }()
+ var result []byte
+ var readers sync.WaitGroup
+ readers.Add(2)
+ go func() { defer readers.Done(); w.pump(jb, outR) }()
+ go func() { defer readers.Done(); result = w.readResult(jb, resR) }()
+ readers.Wait()
+ _ = cmd.Wait()
+ wall := time.Since(start)
+ close(stop)
+ acc := <-watched
+
+ w.mu.Lock()
+ reason = jb.reason
+ jb.proc = nil
+ w.mu.Unlock()
+ code, cpuLimit := exitStatus(cmd.ProcessState)
+ if reason == "" {
+ reason = protocol.ExitExited
+ if cpuLimit {
+ reason = protocol.ExitLimit
+ }
+ }
+ ex := protocol.JobExit{ID: id, Reason: reason, Code: code, Result: json.RawMessage("null")}
+ if r := bytes.TrimSpace(result); reason == protocol.ExitExited && code != nil && *code == 0 && len(r) > 0 && json.Valid(r) {
+ ex.Result = json.RawMessage(r)
+ }
+ // wall_ms is at least (n+1) s for every usage second n already sent, as platform.job-frame@1 requires.
+ wallMS := max(wall.Milliseconds(), acc.seconds*1000)
+ cpuMS := max(acc.cpuMS, (cmd.ProcessState.UserTime() + cmd.ProcessState.SystemTime()).Milliseconds())
+ mem := max(acc.memPeak, maxRSS(cmd.ProcessState))
+ started := jb.startedMS
+ ex.Usage = protocol.JobExitUsage{StartedMS: &started, WallMS: wallMS, CPUMS: &cpuMS, MemPeakBytes: &mem}
+ return ex
+}
+
+// accounting is what watch measured: usage seconds sent, and the CPU time and peak resident memory sampled.
+type accounting struct{ seconds, cpuMS, memPeak int64 }
+
+// watch is a job's watchdog until stop closes: it kills the job at its ttl (from receipt), at wall_ms (from start)
+// and when sampled CPU time or resident memory exceed the limits, and sends job_usage for every wall-clock second
+// once it has fully elapsed.
+func (w *Worker) watch(jb *job, pid int, start time.Time, stop <-chan struct{}) accounting {
+ var acc accounting
+ ttl := time.NewTimer(time.Until(jb.received.Add(jb.ttl)))
+ defer ttl.Stop()
+ wall := time.NewTimer(time.Until(start.Add(ms(jb.limits.WallMS))))
+ defer wall.Stop()
+ usage := time.NewTimer(time.Until(start.Add(time.Second)))
+ defer usage.Stop()
+ tick := time.NewTicker(sampleEvery)
+ defer tick.Stop()
+ var ns string
+ var lastCPU int64
+ sample := func() {
+ if ns == "" {
+ ns, _ = pidNS(pid)
+ }
+ cpu, rss, err := sampleProcs(pid, ns)
+ if err != nil {
+ return
+ }
+ acc.cpuMS, acc.memPeak = max(acc.cpuMS, cpu), max(acc.memPeak, rss)
+ if acc.cpuMS > jb.limits.CPUMS || rss > jb.limits.MemBytes {
+ w.kill(jb, protocol.ExitLimit)
+ }
+ }
+ sample()
+ for {
+ select {
+ case <-stop:
+ return acc
+ case <-ttl.C:
+ w.kill(jb, protocol.ExitTTL)
+ case <-wall.C:
+ w.kill(jb, protocol.ExitLimit)
+ case <-tick.C:
+ sample()
+ case <-usage.C:
+ for time.Since(start) >= time.Duration(acc.seconds+1)*time.Second {
+ cpu := acc.cpuMS - lastCPU
+ lastCPU = acc.cpuMS
+ w.send(protocol.JobUsage{ID: jb.req.ID, StartedMS: jb.startedMS, Second: acc.seconds, CPUMS: &cpu})
+ acc.seconds++
+ }
+ usage.Reset(time.Until(start.Add(time.Duration(acc.seconds+1) * time.Second)))
+ }
+ }
+}
+
+// pump sends the job's stdout as job_stdout chunks of at most chunkBytes, never splitting a UTF-8 sequence, and
+// kills the job (limit) once it wrote more than max_output_bytes; the excess is read and dropped.
+func (w *Worker) pump(jb *job, r io.Reader) {
+ buf := make([]byte, chunkBytes-utf8.UTFMax) // room for a rune carried over from the last read
+ var carry []byte
+ var total int64
+ var seq uint64
+ over := false
+ emit := func(b []byte) {
+ seq++
+ w.send(protocol.JobStdout{ID: jb.req.ID, ChunkSeq: seq, Chunk: strings.ToValidUTF8(string(b), "�")})
+ }
+ for {
+ n, err := r.Read(buf)
+ if n > 0 && !over {
+ b := buf[:n]
+ if left := w.caps.MaxOutputBytes - total; int64(n) > left {
+ b, over = b[:left], true
+ }
+ total += int64(len(b))
+ b = append(carry, b...)
+ cut := completeRunes(b)
+ if over {
+ cut = len(b)
+ }
+ if cut > 0 {
+ emit(b[:cut])
+ }
+ carry = append([]byte(nil), b[cut:]...)
+ if over {
+ w.kill(jb, protocol.ExitLimit)
+ }
+ }
+ if err != nil {
+ if len(carry) > 0 {
+ emit(carry)
+ }
+ return
+ }
+ }
+}
+
+// completeRunes is how many leading bytes of b end on a rune boundary: a rune cut by the end of a read waits for
+// the next read. Invalid bytes count as complete (they become U+FFFD).
+func completeRunes(b []byte) int {
+ for i := len(b) - 1; i >= 0 && i >= len(b)-utf8.UTFMax; i-- {
+ if utf8.RuneStart(b[i]) {
+ if utf8.FullRune(b[i:]) {
+ return len(b)
+ }
+ return i
+ }
+ }
+ return len(b)
+}
+
+// readResult reads what the function writes to fd 3, its result; more than resultBytes kills the job (limit).
+func (w *Worker) readResult(jb *job, r io.Reader) []byte {
+ b, _ := io.ReadAll(io.LimitReader(r, resultBytes+1))
+ if len(b) > resultBytes {
+ w.kill(jb, protocol.ExitLimit)
+ _, _ = io.Copy(io.Discard, r)
+ return nil
+ }
+ return b
+}
+
+// jobEnv is the whole environment of a job: nothing is inherited from the Node (no credential, no secrets), only a
+// fixed PATH, the working directory as HOME and TMPDIR, the job id, and the function's own env from the config.
+func jobEnv(fn config.FunctionConfig, dir, id string) []string {
+ env := map[string]string{"PATH": "/usr/local/bin:/usr/bin:/bin", "HOME": dir, "TMPDIR": dir, "LANG": "C.UTF-8",
+ "OPENVIBE_JOB_ID": id}
+ for k, v := range fn.Env {
+ env[k] = v
+ }
+ out := make([]string, 0, len(env))
+ for k, v := range env {
+ out = append(out, k+"="+v)
+ }
+ sort.Strings(out)
+ return out
+}
+
+func ms(v int64) time.Duration { return time.Duration(v) * time.Millisecond }
new file mode 100644
new file mode 100644
@@ -0,0 +1,189 @@
@@ -0,0 +1,189 @@
+//go:build linux
+
+package worker
+
+import (
+ "errors"
+ "fmt"
+ "os"
+ "os/exec"
+ "strconv"
+ "strings"
+ "syscall"
+ "unsafe"
+
+ "github.com/OpenVibers/OpenVibe.Node/internal/config"
+)
+
+// clkTck is USER_HZ, the unit of /proc/<pid>/stat times: 100 on every architecture Go runs Linux on.
+const clkTck = 100
+
+var pageSize = int64(os.Getpagesize())
+
+// isolate starts cmd in its own process group and in new user, network and PID namespaces: no network but a loopback
+// that is down, and when the process exits every process it left behind is killed with it. It runs as worker.run_as
+// (with no supplementary groups; the Node must be root) or, with allow_same_user, as the Node's own non-root user.
+func isolate(cmd *exec.Cmd, cfg config.WorkerConfig) error {
+ uid, gid := os.Geteuid(), os.Getegid()
+ attr := &syscall.SysProcAttr{Setpgid: true, Pdeathsig: syscall.SIGKILL,
+ Cloneflags: syscall.CLONE_NEWUSER | syscall.CLONE_NEWNET | syscall.CLONE_NEWPID}
+ switch {
+ case cfg.RunAs != nil:
+ if uid != 0 {
+ return errors.New("worker.run_as needs the Node to run as root")
+ }
+ r := cfg.RunAs
+ attr.UidMappings = []syscall.SysProcIDMap{{ContainerID: int(r.UID), HostID: int(r.UID), Size: 1}}
+ attr.GidMappings = []syscall.SysProcIDMap{{ContainerID: int(r.GID), HostID: int(r.GID), Size: 1}}
+ attr.GidMappingsEnableSetgroups = true
+ attr.Credential = &syscall.Credential{Uid: r.UID, Gid: r.GID} // no Groups: setgroups(0) drops root's groups
+ case cfg.AllowSameUser:
+ if uid == 0 {
+ return errors.New("the Node runs as root: set worker.run_as instead of allow_same_user")
+ }
+ attr.UidMappings = []syscall.SysProcIDMap{{ContainerID: uid, HostID: uid, Size: 1}}
+ attr.GidMappings = []syscall.SysProcIDMap{{ContainerID: gid, HostID: gid, Size: 1}}
+ default:
+ return errors.New("set worker.run_as (or allow_same_user, for development) to run jobs")
+ }
+ cmd.SysProcAttr = attr
+ return nil
+}
+
+// probe starts /bin/sh the way a job is started and checks that it got user, network and PID namespaces of its own,
+// that its CPU limit can be set and that its usage can be read.
+func probe(cfg config.WorkerConfig, isolate func(*exec.Cmd, config.WorkerConfig) error) error {
+ cmd := exec.Command("/bin/sh", "-c", "read x")
+ cmd.Env, cmd.Dir = []string{"PATH=/usr/bin:/bin"}, "/"
+ if err := isolate(cmd, cfg); err != nil {
+ return err
+ }
+ in, err := cmd.StdinPipe()
+ if err != nil {
+ return err
+ }
+ if err := cmd.Start(); err != nil {
+ in.Close()
+ return fmt.Errorf("starting a process in new user, network and PID namespaces: %w", err)
+ }
+ defer func() { killGroup(cmd.Process); in.Close(); _ = cmd.Wait() }()
+ pid := cmd.Process.Pid
+ for _, ns := range []string{"user", "net", "pid"} {
+ mine, err1 := os.Readlink("/proc/self/ns/" + ns)
+ its, err2 := os.Readlink("/proc/" + strconv.Itoa(pid) + "/ns/" + ns)
+ if err1 != nil || err2 != nil || mine == its {
+ return fmt.Errorf("the probe process did not get a %s namespace of its own (%v, %v)", ns, err1, err2)
+ }
+ }
+ if err := limitCPU(pid, 1000); err != nil {
+ return err
+ }
+ if _, _, err := statOf(pid); err != nil {
+ return fmt.Errorf("reading the probe's CPU and memory: %w", err)
+ }
+ return nil
+}
+
+// killGroup SIGKILLs the job's process group while its leader lives, then the leader; the leader is PID 1 of the
+// job's PID namespace, so its death also kills anything that left the group.
+func killGroup(p *os.Process) {
+ if p.Signal(syscall.Signal(0)) == nil {
+ _ = syscall.Kill(-p.Pid, syscall.SIGKILL)
+ }
+ _ = p.Kill()
+}
+
+// limitCPU sets the process's RLIMIT_CPU (SIGXCPU at cpuMS rounded up to a second, SIGKILL a second later; every
+// process it starts inherits it) and RLIMIT_CORE 0: the kernel's backstop behind the sampled cpu_ms check.
+func limitCPU(pid int, cpuMS int64) error {
+ sec := uint64((cpuMS + 999) / 1000)
+ limits := []struct {
+ res int
+ lim syscall.Rlimit
+ }{{syscall.RLIMIT_CPU, syscall.Rlimit{Cur: sec, Max: sec + 1}}, {syscall.RLIMIT_CORE, syscall.Rlimit{}}}
+ for _, l := range limits {
+ lim := l.lim
+ if _, _, e := syscall.RawSyscall6(syscall.SYS_PRLIMIT64, uintptr(pid), uintptr(l.res), uintptr(unsafe.Pointer(&lim)), 0, 0, 0); e != 0 {
+ return fmt.Errorf("prlimit: %w", e)
+ }
+ }
+ return nil
+}
+
+func pidNS(pid int) (string, error) { return os.Readlink("/proc/" + strconv.Itoa(pid) + "/ns/pid") }
+
+// sampleProcs sums the CPU time and resident memory of every process in the job's PID namespace ns (the leader alone
+// while ns is unknown), so a process that left the process group is still counted.
+func sampleProcs(pid int, ns string) (cpuMS, rssBytes int64, err error) {
+ if ns == "" {
+ return statOf(pid)
+ }
+ ents, err := os.ReadDir("/proc")
+ if err != nil {
+ return 0, 0, err
+ }
+ for _, e := range ents {
+ p, err := strconv.Atoi(e.Name())
+ if err != nil {
+ continue
+ }
+ if l, err := os.Readlink("/proc/" + e.Name() + "/ns/pid"); err != nil || l != ns {
+ continue
+ }
+ if c, r, err := statOf(p); err == nil {
+ cpuMS, rssBytes = cpuMS+c, rssBytes+r
+ }
+ }
+ return cpuMS, rssBytes, nil
+}
+
+// statOf reads one process's CPU time (its own and its reaped children's) and resident memory from /proc.
+func statOf(pid int) (cpuMS, rssBytes int64, err error) {
+ b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
+ if err != nil {
+ return 0, 0, err
+ }
+ s := string(b)
+ i := strings.LastIndexByte(s, ')')
+ if i < 0 {
+ return 0, 0, errors.New("malformed /proc stat")
+ }
+ f := strings.Fields(s[i+1:]) // f[0] is field 3, state
+ if len(f) < 22 {
+ return 0, 0, errors.New("short /proc stat")
+ }
+ var ticks int64
+ for _, k := range []int{11, 12, 13, 14} { // utime, stime, cutime, cstime
+ v, err := strconv.ParseInt(f[k], 10, 64)
+ if err != nil {
+ return 0, 0, err
+ }
+ ticks += v
+ }
+ rss, err := strconv.ParseInt(f[21], 10, 64)
+ if err != nil {
+ return 0, 0, err
+ }
+ return ticks * 1000 / clkTck, rss * pageSize, nil
+}
+
+// exitStatus is the process's exit code (nil when a signal ended it) and whether RLIMIT_CPU ended it.
+func exitStatus(ps *os.ProcessState) (code *int, cpuLimit bool) {
+ ws, ok := ps.Sys().(syscall.WaitStatus)
+ if !ok {
+ return nil, false
+ }
+ if ws.Exited() {
+ c := ws.ExitStatus()
+ return &c, false
+ }
+ return nil, ws.Signaled() && ws.Signal() == syscall.SIGXCPU
+}
+
+// maxRSS is the peak resident memory the kernel reports for the process and the children it waited for.
+func maxRSS(ps *os.ProcessState) int64 {
+ if r, ok := ps.SysUsage().(*syscall.Rusage); ok {
+ return int64(r.Maxrss) * 1024
+ }
+ return 0
+}
new file mode 100644
new file mode 100644
@@ -0,0 +1,35 @@
@@ -0,0 +1,35 @@
+//go:build !linux
+
+package worker
+
+import (
+ "errors"
+ "os"
+ "os/exec"
+
+ "github.com/OpenVibers/OpenVibe.Node/internal/config"
+)
+
+// errNotLinux: the namespaces the worker isolates jobs with exist on Linux only, so elsewhere no job ever runs.
+var errNotLinux = errors.New("the job worker runs on Linux only")
+
+func isolate(*exec.Cmd, config.WorkerConfig) error { return errNotLinux }
+
+func probe(config.WorkerConfig, func(*exec.Cmd, config.WorkerConfig) error) error { return errNotLinux }
+
+func killGroup(p *os.Process) { _ = p.Kill() }
+
+func limitCPU(int, int64) error { return errNotLinux }
+
+func pidNS(int) (string, error) { return "", errNotLinux }
+
+func sampleProcs(int, string) (int64, int64, error) { return 0, 0, errNotLinux }
+
+func exitStatus(ps *os.ProcessState) (*int, bool) {
+ if c := ps.ExitCode(); c >= 0 {
+ return &c, false
+ }
+ return nil, false
+}
+
+func maxRSS(*os.ProcessState) int64 { return 0 }
new file mode 100644
new file mode 100644
@@ -0,0 +1,187 @@
@@ -0,0 +1,187 @@
+package worker
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "log/slog"
+ "os"
+ "os/exec"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/OpenVibers/OpenVibe.Node/internal/config"
+ "github.com/OpenVibers/OpenVibe.Node/internal/protocol"
+)
+
+// sink records every frame the worker sends.
+type sink struct {
+ mu sync.Mutex
+ frames []protocol.Message
+}
+
+func (s *sink) send(m protocol.Message) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ s.frames = append(s.frames, m)
+}
+
+func (s *sink) all() []protocol.Message {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return append([]protocol.Message(nil), s.frames...)
+}
+
+func frameID(m protocol.Message) string {
+ switch f := m.(type) {
+ case protocol.JobStarted:
+ return f.ID
+ case protocol.JobStdout:
+ return f.ID
+ case protocol.JobUsage:
+ return f.ID
+ case protocol.JobExit:
+ return f.ID
+ }
+ return ""
+}
+
+// count is how many frames of type typ for job id were sent.
+func (s *sink) count(typ, id string) int {
+ n := 0
+ for _, m := range s.all() {
+ if m.MessageType() == typ && frameID(m) == id {
+ n++
+ }
+ }
+ return n
+}
+
+// wait returns the n-th (from 1) frame of type typ for job id, waiting up to timeout for it.
+func (s *sink) wait(t *testing.T, typ, id string, n int, timeout time.Duration) protocol.Message {
+ t.Helper()
+ end := time.Now().Add(timeout)
+ for {
+ seen := 0
+ for _, m := range s.all() {
+ if m.MessageType() == typ && frameID(m) == id {
+ if seen++; seen == n {
+ return m
+ }
+ }
+ }
+ if time.Now().After(end) {
+ t.Fatalf("no %s #%d for %s within %s", typ, n, id, timeout)
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+}
+
+func (s *sink) exit(t *testing.T, id string) protocol.JobExit {
+ t.Helper()
+ return s.wait(t, protocol.TypeJobExit, id, 1, 20*time.Second).(protocol.JobExit)
+}
+
+func testJob(n int, name string) protocol.Job {
+ return protocol.Job{ID: fmt.Sprintf("job_01JAB2C3D4E5F6G7H8J9K%05d", n), Class: protocol.ClassFunction,
+ Artifact: &protocol.Artifact{Name: name, Version: "1.0.0"}, Args: json.RawMessage(`{"n":7}`), TTLMS: 60000,
+ Limits: protocol.JobLimits{WallMS: 60000, CPUMS: 60000, MemBytes: 4 << 30}}
+}
+
+// helperConfig declares one function per helper mode, each running this test binary's TestHelperProcess.
+func helperConfig(t *testing.T, caps config.WorkerCaps, modes ...string) config.WorkerConfig {
+ t.Helper()
+ exe, err := os.Executable()
+ if err != nil {
+ t.Fatal(err)
+ }
+ cfg := config.WorkerConfig{Enabled: true, AllowSameUser: true, Caps: caps}
+ for _, m := range modes {
+ cfg.Functions = append(cfg.Functions, config.FunctionConfig{Name: m, Version: "1.0.0",
+ Command: []string{exe, "-test.run=^TestHelperProcess$", "--", m}})
+ }
+ return cfg
+}
+
+func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
+
+// TestAdmitRefusals: only a declared function at its exact version is admitted, never another class; max_jobs bounds
+// the jobs held at once; while the stop latch is set every job is refused with its fault; nothing ever starts.
+func TestAdmitRefusals(t *testing.T) {
+ s := &sink{}
+ w := New(helperConfig(t, config.WorkerCaps{MaxJobs: 1}, "sleep"), s.send, quiet())
+ w.isolate = func(*exec.Cmd, config.WorkerConfig) error { return errors.New("not in this test") }
+
+ wrongVersion := testJob(2, "sleep")
+ wrongVersion.Artifact.Version = "1.0.1"
+ code := testJob(3, "sleep")
+ code.Class = protocol.ClassCode
+ for _, c := range []struct {
+ j protocol.Job
+ fault, reason string
+ }{
+ {testJob(1, "nope"), protocol.FaultUnsupported, protocol.JobUnknownArtifact},
+ {wrongVersion, protocol.FaultUnsupported, protocol.JobUnknownArtifact},
+ {code, protocol.FaultUnsupported, protocol.JobNotAvailable},
+ } {
+ if f, r := w.Admit(c.j); f != c.fault || r != c.reason {
+ t.Errorf("%s %+v: %q %q, want %q %q", c.j.Class, c.j.Artifact, f, r, c.fault, c.reason)
+ }
+ }
+
+ held := testJob(4, "sleep")
+ if f, r := w.Admit(held); f != "" {
+ t.Fatalf("%s %s", f, r)
+ }
+ if f, r := w.Admit(held); f != "" {
+ t.Fatalf("a held id re-admitted: %s %s", f, r)
+ }
+ if f, r := w.Admit(testJob(5, "sleep")); f != protocol.FaultNotReady || r != protocol.JobBusy {
+ t.Fatalf("over max_jobs: %q %q", f, r)
+ }
+ w.Cancel(held.ID)
+ w.Launch(held.ID)
+ if ex := s.exit(t, held.ID); ex.Reason != protocol.ExitCancelled {
+ t.Fatalf("%+v", ex)
+ }
+
+ w.SetStopped(protocol.FaultLocalStop)
+ if f, r := w.Admit(testJob(6, "sleep")); f != protocol.FaultLocalStop || r != protocol.JobStopped {
+ t.Fatalf("latched: %q %q", f, r)
+ }
+ w.SetStopped("")
+ if f, r := w.Admit(testJob(6, "sleep")); f != "" {
+ t.Fatalf("after the latch cleared: %q %q", f, r)
+ }
+ for _, m := range s.all() {
+ if _, ok := m.(protocol.JobStarted); ok {
+ t.Fatalf("a refused job started: %+v", m)
+ }
+ }
+}
+
+// TestIsolationUnavailableRefused: when the namespaces cannot be created the probe fails (the class is not
+// advertised), and a job whose isolation fails ends `failed` without its process ever starting.
+func TestIsolationUnavailableRefused(t *testing.T) {
+ s := &sink{}
+ w := New(helperConfig(t, config.WorkerCaps{}, "sleep"), s.send, quiet())
+ w.isolate = func(*exec.Cmd, config.WorkerConfig) error { return errors.New("no user namespaces") }
+ t.Cleanup(w.Close)
+ if err := w.Probe(); err == nil {
+ t.Fatal("the probe passed without namespaces")
+ }
+ j := testJob(1, "sleep")
+ if f, r := w.Admit(j); f != "" {
+ t.Fatalf("%s %s", f, r)
+ }
+ w.Launch(j.ID)
+ ex := s.exit(t, j.ID)
+ if ex.Reason != protocol.ExitFailed || ex.Code != nil || ex.Usage.WallMS != 0 || ex.Usage.StartedMS != nil || string(ex.Result) != "null" {
+ t.Fatalf("%+v", ex)
+ }
+ if n := s.count(protocol.TypeJobStarted, j.ID); n != 0 {
+ t.Fatalf("job_started sent %d times for a job that could not be isolated", n)
+ }
+}