Skip to content

Commit 22fc771

Browse files
kerthcetclaude
andcommitted
fix: don't reconnect after a command longer than the ack timeout
The daemon handles each message inline, so while a command runs its serve loop reads nothing and the controller's heartbeat acks sit unread in the socket. When the command finished, the next ack_check saw no ack for longer than ack_timeout (30s by default) and reconnected, so the daemon vanished from the registry for its 5s backoff right after answering. This broke the E2E run on #44: `microdnf install -y curl` on rocky took the full 60s timeout, daemon-rocky-1 reconnected the same instant, and the following tests failed with "Daemon daemon-rocky-1 not found". Restart the ack deadline after each inline message. The buffered acks are read on the next iterations; a controller that really stopped acking is still caught one ack_timeout later. Introduced in #40. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kerthcet <kerthcet@gmail.com>
1 parent c400a41 commit 22fc771

2 files changed

Lines changed: 26 additions & 0 deletions

File tree

‎python/tests/test_e2e.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,23 @@ def test_daemon_restart(self, server):
278278
assert result.success
279279
assert "after" in result.stdout
280280

281+
def test_daemon_stays_connected_after_long_command(self, server):
282+
"""A command longer than the daemon's ack timeout (30s) must not make it reconnect
283+
284+
The daemon handles a command inline, so heartbeat acks go unread while it runs.
285+
It used to read that as an unresponsive controller and reconnect right after
286+
answering, so the next call failed with "daemon not found".
287+
"""
288+
result = server.exec("daemon-debian-2", "sleep 35 && echo done", timeout=60)
289+
assert result.success
290+
assert "done" in result.stdout
291+
292+
# No wait: a reconnecting daemon is missing from the registry for its 5s backoff.
293+
time.sleep(2)
294+
result = server.exec("daemon-debian-2", "echo still-here", timeout=5)
295+
assert result.success
296+
assert "still-here" in result.stdout
297+
281298

282299
class TestE2EStats:
283300
"""Test statistics with Docker daemons"""

‎sandd/src/main.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -620,6 +620,15 @@ where
620620
{
621621
error!("Error handling message: {}", e);
622622
}
623+
624+
// While the message was handled inline (an exec can run for minutes)
625+
// this loop read nothing, so the controller's acks sat unread in the
626+
// socket. That is OUR silence, not the controller's: without this, any
627+
// command longer than ack_timeout made the next ack_check reconnect,
628+
// dropping the daemon from the registry right after it answered. The
629+
// buffered acks are read on the next iterations; a controller that
630+
// really stopped acking is still caught one ack_timeout from now.
631+
last_ack = Instant::now();
623632
}
624633
}
625634
};

0 commit comments

Comments
 (0)