openvibers/openvibe.link Commits
v0.16.1: the runtime holds no motion command past its deadline; the five contract answers in plugins.md (#62)
Changes
@@@ -5,8 +5,8 @@"wave": "T15","plan": "Plan track T15 (Bot complete) and ADR-043 (Bot: devices, pairing, control and safety); the computer plugin and the job worker serve OpenVibe.Actor and the Codes job runner (tracks T14/T17)","kind": "client",-- "version": "0.16.0",-- "release": "v0.16.0, published 2026-10-27 as a GitHub release (openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS",++ "version": "0.16.1",++ "release": "v0.16.1, published 2026-10-27 as a GitHub release (the Python runtime refuses a motion command whose deadline passed while it waited behind a slow handler, so a driver never moves on a stale command; plugins.md answers where deadlines come from, re-stamping, order across a change of driver, stop versus estop, and that nothing is held past its validity; v0.16.0: openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS","runtime": "go 1.24 (one static binary per platform); plugins in Python 3; an ESP32 Arduino library","database": "none","deployed": false,@@@ -120,6 +120,75 @@ So a driver needs no growing replay cache. Refuse a command if any of these is tThe runtime does all three. A driver that wants more can also refuse an `issued_at_ms` older than the newest it hashandled by more than `max_command_ms`.++#### Deadlines, turns and stops: the platform's answers++++Asked by the Generic LOOI driver's authors (2026-10-27). Each answer is what Bot (`server/realtime.js`,++`server/domain`), this core (`internal/node`, `internal/plugins`) and the Python runtime do:++++1. **Where a deadline comes from.** Bot's control gate sets it when it decides a command:++ - The deadline is Bot's clock now plus the robot's `max_command_ms`: 300 ms unless the owner set less.++ - An operator may ask for less with `ms`, never more.++ - Only `drive`, `actuator`, `ptz` and a held button's `down` get one.++ - The command is sent in the same step. Its id (`cmd_<ULID>`) and `issued_at_ms` are minted then, after the gate's++ last await.++ - The core turns the absolute instant into what is left of it. It compares the instant with its estimate of Bot's++ clock, which is the local monotonic clock plus the largest `ts − arrival` seen on the connection. So time a frame++ spent queued on the way counts as lateness, and a delayed run of frames can never make a deadline look open again.++ - It caps what is left at the robot's `max_command_ms`.++ - The plugin gets that remainder as `deadline_ms`, and `expires_at_ms` in `provenance` unchanged.++ - Neither a viewer nor Live can choose a later deadline.++2. **Nothing re-stamps a command.**++ - Bot sends each device command once and never again, under any id. One with no answer by its deadline plus 1.5 s++ is recorded `expired` and dropped.++ - The core passes `provenance` through unchanged.++ - An operator who repeats their own id gets the first result, and no second command reaches the robot.++ - A retry under a new id, or a held control re-sent every 150 ms, is a **new** command. It passes the gate again at++ that moment: whose turn it is, the epoch, the e-stop and the limits. Its id, `seq`, `issued_at_ms`, deadline and++ `turn_epoch` are all new.++ - So a stamp is always a fresh decision, never an old command made current.++3. **Order across a change of driver.** A queue turn can end, its holder can leave, or the owner can take it over. Then:++ - Bot ends the turn, moves the robot's epoch and sends its `halt` in one synchronous step.++ - A command decided under the old turn but not yet sent is refused (`bot.not_your_turn`). There is no await between++ that check and the send.++ - On the one device connection, everything from turn N therefore precedes N's halt. The halt carries the new epoch,++ and every later command carries it or a newer one.++ - The core and the runtime refuse an older epoch after a halt (`expired`).++ - On a takeover, the halt goes first and the owner's command follows with the new epoch.++ - A waiting viewer who leaves causes no halt.++ - Owners and operators of a robot without a queue have no turns. Their commands reach the driver in the order Bot++ sent them, and the newest wins.++ - Acks may come back in any order. Commands never reach a plugin out of order.++4. **`stop` versus `estop`.**++ - `stop` stops every actuator **now and latches nothing**. The next valid command moves the robot again. It comes++ from:++ - a `halt` (a change of driver, with its `turn_epoch`, or an operator's halt);++ - the link to Bot dropping;++ - the core shutting down (then stdin closes);++ - `openvibe-node quit`;++ - a plugin missing its reply window.++ - `estop` stops every actuator **and latches**. The core refuses `drive`, `ptz` and `actuator` itself (`estopped`,++ `local_stop`), and so does the runtime, until `resume`. The latch:++ - is set by Bot's e-stop (an owner or operator presses it), by `openvibe-node stop`, or by a driver's `local_stop`;++ - is written to disk, survives a Node restart, and is sent again at once to a plugin that restarts;++ - is reapplied from Bot's config on every reconnect.++ - Only the owner clears Bot's e-stop. A local stop is cleared only by `openvibe-node resume` on the machine.++ - A `halt` still passes while latched.++ - The driver's own stops (a motion deadline passing, a second without a heartbeat, stdin closing) are `stop`s.++ Heartbeat loss also refuses motion until heartbeats return.++5. **No command is held past its validity.**++ - Bot never queues: an offline device is refused (`bot.device_offline`), and nothing is replayed after a reconnect.++ - The core decides at arrival:++ - it refuses a command whose deadline has passed;++ - it refuses a plugin that is down or not ready at once (`plugin_down`, `not_ready`) rather than holding the command;++ - it hands the rest straight to the plugin's stdin. A plugin that stops reading is killed, so no line waits there.++ - A plugin that does not answer within max(deadline, 1 s) is told to `stop`, and the command fails with++ `plugin_timeout`.++ - The Python runtime counts each deadline from the moment its line arrived. It refuses (`expired`) a motion command++ whose deadline passed while it waited behind a slow handler, so the driver never sees it. Before Node 0.16.1 such a++ command still reached `handle` and was stopped on the next tick.++ - A driver without the runtime should do the same: note the time each line is read (on a reader thread) and refuse++ motion that is already past its `deadline_ms` when its turn comes.++### Plugin → core| op | fields |@@@ -151,7 +220,8 @@ installed, otherwise with its built-in encoder at up to 5 frames per second. TheA plugin must stop every actuator itself — the core dying must never leave motors running:- when its **stdin closes** (EOF), then exit;--- when a **motion command's deadline passes** without a newer command (`deadline_ms` after receipt);++- when a **motion command's deadline passes** without a newer command (`deadline_ms` after receipt), and refuse one++ whose deadline passed before its turn to run (`expired`);- when it has **not heard a heartbeat for 1 s** (and refuse motion with `no_heartbeat` until heartbeats return);- on `stop`, `estop` and a `halt` command;- when its own sensors say so (Cozmo: cliff, pick-up).@@@ -653,8 +653,6 @@ func (n *Node) computer(c protocol.Command, entry *safety.Entry, sentMS int64, r}()}--// serverNow is the server's clock when a frame stamped sentMS was received: the link's estimate, else the frame's--// own ts.// noteHaltEpoch records a halt's turn_epoch for its robot (the newest wins); haltEpoch reads it (0: none seen).func (n *Node) noteHaltEpoch(robotID string, epoch int64) {if epoch <= 0 {@@@ -676,6 +674,8 @@ func (n *Node) haltEpoch(robotID string) int64 {return n.haltEpochs[robotID]}++// serverNow is the server's clock when a frame stamped sentMS was received: the link's estimate, else the frame's++// own ts.func (n *Node) serverNow(sentMS int64, received time.Time) int64 {if n.link != nil {if ms, ok := n.link.ServerNow(received); ok {@@@ -414,6 +414,13 @@ class Runtime:"message": "a replay: seq %d is not above %d on this session" % (prov["seq"], self.last_seq)})returnself.last_session, self.last_seq = prov["session"], prov["seq"]++ # A motion command whose time ran out while it waited behind a slow handler (its deadline counts from when its++ # line arrived) never reaches the driver: the runtime holds nothing past its deadline.++ if moves and self.clock() >= cmd.deadline:++ self.send({"op": "nack", "id": cid, "fault_code": "expired",++ "message": "its deadline passed while it waited behind earlier input (%d ms after it arrived)"++ % round((self.clock() - now) * 1000)})++ returnif not self.ready:self.send({"op": "nack", "id": cid, "fault_code": "not_ready"})return@@@ -195,17 +195,37 @@ class RuntimeTest(unittest.TestCase):self.assertFalse(p.moving)def test_backlog_counts_from_arrival(self):-- # A command that waited in the inbox while a handler blocked keeps its arrival-based deadline.++ # A command that waited in the inbox while a handler blocked keeps its arrival-based deadline, and one whose++ # deadline passed while it waited is refused without reaching the driver: it never moves the robot at all.rt, p, out, clock = make()self.hello(rt)arrived = clock.tclock.t += 0.5 # the handler was stuck for 500 msrt.feed_line(json.dumps({"op": "heartbeat", "t": 1}), arrived)rt.feed_line(json.dumps({"op": "command", "id": "old", "kind": "drive", "deadline_ms": 300}), arrived)++ last = lines(out)[-1]++ self.assertEqual((last["op"], last["id"], last["fault_code"]), ("nack", "old", "expired"))++ self.assertIn("500 ms after it arrived", last["message"])++ self.assertFalse(p.moving)++ # One that waited less than its deadline still runs, with only the time it has left.++ rt.feed_line(json.dumps({"op": "command", "id": "fresh", "kind": "drive", "deadline_ms": 300}), clock.t - 0.1)++ self.assertEqual(lines(out)[-1], {"op": "ack", "id": "fresh"})++ self.assertTrue(p.moving)++ clock.t += 0.21rt.tick()self.assertFalse(p.moving)self.assertIn("deadline", rt.stop_reasons)++ def test_only_motion_expires_in_the_backlog(self):++ # say has no motion deadline to miss (its deadline is a reply window): a late one still runs.++ rt, p, out, clock = make()++ self.hello(rt)++ arrived = clock.t++ clock.t += 0.5++ rt.feed_line(json.dumps({"op": "heartbeat", "t": 1}), arrived)++ rt.feed_line(json.dumps({"op": "command", "id": "s1", "kind": "say", "deadline_ms": 300}), arrived)++ self.assertEqual(lines(out)[-1], {"op": "ack", "id": "s1"})++def test_heartbeat_loss_stops_and_refuses(self):rt, p, out, clock = make()self.hello(rt)
@@ -5,8 +5,8 @@
@@ -5,8 +5,8 @@
"wave": "T15",
"wave": "T15",
"plan": "Plan track T15 (Bot complete) and ADR-043 (Bot: devices, pairing, control and safety); the computer plugin and the job worker serve OpenVibe.Actor and the Codes job runner (tracks T14/T17)",
"plan": "Plan track T15 (Bot complete) and ADR-043 (Bot: devices, pairing, control and safety); the computer plugin and the job worker serve OpenVibe.Actor and the Codes job runner (tracks T14/T17)",
"kind": "client",
"kind": "client",
- "version": "0.16.0",
- "release": "v0.16.0, published 2026-10-27 as a GitHub release (openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS",
+ "version": "0.16.1",
+ "release": "v0.16.1, published 2026-10-27 as a GitHub release (the Python runtime refuses a motion command whose deadline passed while it waited behind a slow handler, so a driver never moves on a stale command; plugins.md answers where deadlines come from, re-stamping, order across a change of driver, stop versus estop, and that nothing is held past its validity; v0.16.0: openvibe-node quit ends a running node cleanly with nothing latched, for launchers; each driver command carries its provenance (link session, seq, Bot send time and deadline) and a halt's stop its turn_epoch, and the Python runtime refuses a replay on a session and a command from an ended turn; the no-redelivery contract written down in plugins.md): binaries for linux amd64/arm64/armv7, darwin amd64/arm64 and windows amd64, the plugins tarball, install.sh and SHA256SUMS",
"runtime": "go 1.24 (one static binary per platform); plugins in Python 3; an ESP32 Arduino library",
"runtime": "go 1.24 (one static binary per platform); plugins in Python 3; an ESP32 Arduino library",
"database": "none",
"database": "none",
"deployed": false,
"deployed": false,
@@ -120,6 +120,75 @@ So a driver needs no growing replay cache. Refuse a command if any of these is t
@@ -120,6 +120,75 @@ So a driver needs no growing replay cache. Refuse a command if any of these is t
The runtime does all three. A driver that wants more can also refuse an `issued_at_ms` older than the newest it has
The runtime does all three. A driver that wants more can also refuse an `issued_at_ms` older than the newest it has
handled by more than `max_command_ms`.
handled by more than `max_command_ms`.
+#### Deadlines, turns and stops: the platform's answers
+
+Asked by the Generic LOOI driver's authors (2026-10-27). Each answer is what Bot (`server/realtime.js`,
+`server/domain`), this core (`internal/node`, `internal/plugins`) and the Python runtime do:
+
+1. **Where a deadline comes from.** Bot's control gate sets it when it decides a command:
+ - The deadline is Bot's clock now plus the robot's `max_command_ms`: 300 ms unless the owner set less.
+ - An operator may ask for less with `ms`, never more.
+ - Only `drive`, `actuator`, `ptz` and a held button's `down` get one.
+ - The command is sent in the same step. Its id (`cmd_<ULID>`) and `issued_at_ms` are minted then, after the gate's
+ last await.
+ - The core turns the absolute instant into what is left of it. It compares the instant with its estimate of Bot's
+ clock, which is the local monotonic clock plus the largest `ts − arrival` seen on the connection. So time a frame
+ spent queued on the way counts as lateness, and a delayed run of frames can never make a deadline look open again.
+ - It caps what is left at the robot's `max_command_ms`.
+ - The plugin gets that remainder as `deadline_ms`, and `expires_at_ms` in `provenance` unchanged.
+ - Neither a viewer nor Live can choose a later deadline.
+2. **Nothing re-stamps a command.**
+ - Bot sends each device command once and never again, under any id. One with no answer by its deadline plus 1.5 s
+ is recorded `expired` and dropped.
+ - The core passes `provenance` through unchanged.
+ - An operator who repeats their own id gets the first result, and no second command reaches the robot.
+ - A retry under a new id, or a held control re-sent every 150 ms, is a **new** command. It passes the gate again at
+ that moment: whose turn it is, the epoch, the e-stop and the limits. Its id, `seq`, `issued_at_ms`, deadline and
+ `turn_epoch` are all new.
+ - So a stamp is always a fresh decision, never an old command made current.
+3. **Order across a change of driver.** A queue turn can end, its holder can leave, or the owner can take it over. Then:
+ - Bot ends the turn, moves the robot's epoch and sends its `halt` in one synchronous step.
+ - A command decided under the old turn but not yet sent is refused (`bot.not_your_turn`). There is no await between
+ that check and the send.
+ - On the one device connection, everything from turn N therefore precedes N's halt. The halt carries the new epoch,
+ and every later command carries it or a newer one.
+ - The core and the runtime refuse an older epoch after a halt (`expired`).
+ - On a takeover, the halt goes first and the owner's command follows with the new epoch.
+ - A waiting viewer who leaves causes no halt.
+ - Owners and operators of a robot without a queue have no turns. Their commands reach the driver in the order Bot
+ sent them, and the newest wins.
+ - Acks may come back in any order. Commands never reach a plugin out of order.
+4. **`stop` versus `estop`.**
+ - `stop` stops every actuator **now and latches nothing**. The next valid command moves the robot again. It comes
+ from:
+ - a `halt` (a change of driver, with its `turn_epoch`, or an operator's halt);
+ - the link to Bot dropping;
+ - the core shutting down (then stdin closes);
+ - `openvibe-node quit`;
+ - a plugin missing its reply window.
+ - `estop` stops every actuator **and latches**. The core refuses `drive`, `ptz` and `actuator` itself (`estopped`,
+ `local_stop`), and so does the runtime, until `resume`. The latch:
+ - is set by Bot's e-stop (an owner or operator presses it), by `openvibe-node stop`, or by a driver's `local_stop`;
+ - is written to disk, survives a Node restart, and is sent again at once to a plugin that restarts;
+ - is reapplied from Bot's config on every reconnect.
+ - Only the owner clears Bot's e-stop. A local stop is cleared only by `openvibe-node resume` on the machine.
+ - A `halt` still passes while latched.
+ - The driver's own stops (a motion deadline passing, a second without a heartbeat, stdin closing) are `stop`s.
+ Heartbeat loss also refuses motion until heartbeats return.
+5. **No command is held past its validity.**
+ - Bot never queues: an offline device is refused (`bot.device_offline`), and nothing is replayed after a reconnect.
+ - The core decides at arrival:
+ - it refuses a command whose deadline has passed;
+ - it refuses a plugin that is down or not ready at once (`plugin_down`, `not_ready`) rather than holding the command;
+ - it hands the rest straight to the plugin's stdin. A plugin that stops reading is killed, so no line waits there.
+ - A plugin that does not answer within max(deadline, 1 s) is told to `stop`, and the command fails with
+ `plugin_timeout`.
+ - The Python runtime counts each deadline from the moment its line arrived. It refuses (`expired`) a motion command
+ whose deadline passed while it waited behind a slow handler, so the driver never sees it. Before Node 0.16.1 such a
+ command still reached `handle` and was stopped on the next tick.
+ - A driver without the runtime should do the same: note the time each line is read (on a reader thread) and refuse
+ motion that is already past its `deadline_ms` when its turn comes.
+
### Plugin → core
### Plugin → core
| op | fields |
| op | fields |
@@ -151,7 +220,8 @@ installed, otherwise with its built-in encoder at up to 5 frames per second. The
@@ -151,7 +220,8 @@ installed, otherwise with its built-in encoder at up to 5 frames per second. The
A plugin must stop every actuator itself — the core dying must never leave motors running:
A plugin must stop every actuator itself — the core dying must never leave motors running:
- when its **stdin closes** (EOF), then exit;
- when its **stdin closes** (EOF), then exit;
-- when a **motion command's deadline passes** without a newer command (`deadline_ms` after receipt);
+- when a **motion command's deadline passes** without a newer command (`deadline_ms` after receipt), and refuse one
+ whose deadline passed before its turn to run (`expired`);
- when it has **not heard a heartbeat for 1 s** (and refuse motion with `no_heartbeat` until heartbeats return);
- when it has **not heard a heartbeat for 1 s** (and refuse motion with `no_heartbeat` until heartbeats return);
- on `stop`, `estop` and a `halt` command;
- on `stop`, `estop` and a `halt` command;
- when its own sensors say so (Cozmo: cliff, pick-up).
- when its own sensors say so (Cozmo: cliff, pick-up).
@@ -653,8 +653,6 @@ func (n *Node) computer(c protocol.Command, entry *safety.Entry, sentMS int64, r
@@ -653,8 +653,6 @@ func (n *Node) computer(c protocol.Command, entry *safety.Entry, sentMS int64, r
}()
}()
}
}
-// serverNow is the server's clock when a frame stamped sentMS was received: the link's estimate, else the frame's
-// own ts.
// noteHaltEpoch records a halt's turn_epoch for its robot (the newest wins); haltEpoch reads it (0: none seen).
// noteHaltEpoch records a halt's turn_epoch for its robot (the newest wins); haltEpoch reads it (0: none seen).
func (n *Node) noteHaltEpoch(robotID string, epoch int64) {
func (n *Node) noteHaltEpoch(robotID string, epoch int64) {
if epoch <= 0 {
if epoch <= 0 {
@@ -676,6 +674,8 @@ func (n *Node) haltEpoch(robotID string) int64 {
@@ -676,6 +674,8 @@ func (n *Node) haltEpoch(robotID string) int64 {
return n.haltEpochs[robotID]
return n.haltEpochs[robotID]
}
}
+// serverNow is the server's clock when a frame stamped sentMS was received: the link's estimate, else the frame's
+// own ts.
func (n *Node) serverNow(sentMS int64, received time.Time) int64 {
func (n *Node) serverNow(sentMS int64, received time.Time) int64 {
if n.link != nil {
if n.link != nil {
if ms, ok := n.link.ServerNow(received); ok {
if ms, ok := n.link.ServerNow(received); ok {
@@ -414,6 +414,13 @@ class Runtime:
@@ -414,6 +414,13 @@ class Runtime:
"message": "a replay: seq %d is not above %d on this session" % (prov["seq"], self.last_seq)})
"message": "a replay: seq %d is not above %d on this session" % (prov["seq"], self.last_seq)})
return
return
self.last_session, self.last_seq = prov["session"], prov["seq"]
self.last_session, self.last_seq = prov["session"], prov["seq"]
+ # A motion command whose time ran out while it waited behind a slow handler (its deadline counts from when its
+ # line arrived) never reaches the driver: the runtime holds nothing past its deadline.
+ if moves and self.clock() >= cmd.deadline:
+ self.send({"op": "nack", "id": cid, "fault_code": "expired",
+ "message": "its deadline passed while it waited behind earlier input (%d ms after it arrived)"
+ % round((self.clock() - now) * 1000)})
+ return
if not self.ready:
if not self.ready:
self.send({"op": "nack", "id": cid, "fault_code": "not_ready"})
self.send({"op": "nack", "id": cid, "fault_code": "not_ready"})
return
return
@@ -195,17 +195,37 @@ class RuntimeTest(unittest.TestCase):
@@ -195,17 +195,37 @@ class RuntimeTest(unittest.TestCase):
self.assertFalse(p.moving)
self.assertFalse(p.moving)
def test_backlog_counts_from_arrival(self):
def test_backlog_counts_from_arrival(self):
- # A command that waited in the inbox while a handler blocked keeps its arrival-based deadline.
+ # A command that waited in the inbox while a handler blocked keeps its arrival-based deadline, and one whose
+ # deadline passed while it waited is refused without reaching the driver: it never moves the robot at all.
rt, p, out, clock = make()
rt, p, out, clock = make()
self.hello(rt)
self.hello(rt)
arrived = clock.t
arrived = clock.t
clock.t += 0.5 # the handler was stuck for 500 ms
clock.t += 0.5 # the handler was stuck for 500 ms
rt.feed_line(json.dumps({"op": "heartbeat", "t": 1}), arrived)
rt.feed_line(json.dumps({"op": "heartbeat", "t": 1}), arrived)
rt.feed_line(json.dumps({"op": "command", "id": "old", "kind": "drive", "deadline_ms": 300}), arrived)
rt.feed_line(json.dumps({"op": "command", "id": "old", "kind": "drive", "deadline_ms": 300}), arrived)
+ last = lines(out)[-1]
+ self.assertEqual((last["op"], last["id"], last["fault_code"]), ("nack", "old", "expired"))
+ self.assertIn("500 ms after it arrived", last["message"])
+ self.assertFalse(p.moving)
+ # One that waited less than its deadline still runs, with only the time it has left.
+ rt.feed_line(json.dumps({"op": "command", "id": "fresh", "kind": "drive", "deadline_ms": 300}), clock.t - 0.1)
+ self.assertEqual(lines(out)[-1], {"op": "ack", "id": "fresh"})
+ self.assertTrue(p.moving)
+ clock.t += 0.21
rt.tick()
rt.tick()
self.assertFalse(p.moving)
self.assertFalse(p.moving)
self.assertIn("deadline", rt.stop_reasons)
self.assertIn("deadline", rt.stop_reasons)
+ def test_only_motion_expires_in_the_backlog(self):
+ # say has no motion deadline to miss (its deadline is a reply window): a late one still runs.
+ rt, p, out, clock = make()
+ self.hello(rt)
+ arrived = clock.t
+ clock.t += 0.5
+ rt.feed_line(json.dumps({"op": "heartbeat", "t": 1}), arrived)
+ rt.feed_line(json.dumps({"op": "command", "id": "s1", "kind": "say", "deadline_ms": 300}), arrived)
+ self.assertEqual(lines(out)[-1], {"op": "ack", "id": "s1"})
+
def test_heartbeat_loss_stops_and_refuses(self):
def test_heartbeat_loss_stops_and_refuses(self):
rt, p, out, clock = make()
rt, p, out, clock = make()
self.hello(rt)
self.hello(rt)