Make process lifecycle robust: escalating stop, restart backoff, non-fatal registration

[sequoiap]

Depends on #92: merge #92 first. This branch is built on #92's branch, so until #92 is merged, this PR's diff and commit list also include #92's commit (7e976cd). Once #92 is in master, only this PR's own commit (1a9e17a) will remain. Both PRs target master.

Makes the process manager reliable when processes hang, crash, or can't reach a nameserver. Closes #50, #51, #52, #53, #59.

Stopping (#50)

• Escalation: the manager sends the stop message, waits 10 s for a clean exit, then calls terminate(), waits 5 s, then calls kill().
• Entries are forgotten only once the process is dead. If a process somehow survives kill(), it stays in the table and stop reports failure.
• Deregistration on its behalf: a daemon that had to be killed can't deregister itself, so the manager removes its nameserver entries.
• Parallel stops: several entities are asked to stop at once and waited for together (_stop_many), so down and reload take one grace period, not one per entity.
• Waits happen outside the manager lock, so ps keeps responding while something is shutting down.
• pyrolab down now gives up after --timeout (60 s by default) instead of looping forever.

Crash recovery (#51)

• Backoff: a crashed process is restarted after 5, 15, 60, then 300 s.
• Giving up: after 5 crashes in a row, the entity is marked failed and left alone until it's started by hand or pyrolab reload runs. Its nameserver entries are removed, so clients don't get sent to a dead URI.
• Resetting the count: a process that stayed up 10 minutes before dying starts a fresh count, so a weekly crash never accumulates into "failed".
• Error reporting: runners report their exception to the manager through a shared dict, and ps has a new LAST ERROR column. Example:

DAEMON    CREATED              STATUS                  URI    LAST ERROR
crashy    2026-10-04 14:39:28  Restarting in 4s (1/5)         RuntimeError: instrument not connected

• Faster detection: checkup() now runs every second instead of every 30, so crashes are noticed promptly and the 5-second backoff means something.

Nameserver registration (#52, #53)

• Non-fatal and retried: registration happens per nameserver in a background thread, so the daemon starts serving immediately. A nameserver that can't be reached is logged with one warning and retried every 30 s. It no longer kills the whole daemon and every service in it.
• Timeout: every nameserver call goes through nameserver_proxy(), which sets a 5 s timeout. Pyro's default is no timeout, so a host that silently drops packets could block for minutes.
• No leaked proxies: each attempt opens one proxy per nameserver and always releases it, and the ns variable that was reused for both a config name and a proxy is gone.
• host: public nameservers: these are now resolved to this machine's IP before connecting. Before, the literal hostname "public" was passed to the connection.

Results that mean something (#59)

• Launching can wait: launch_*(wait=True) waits for the runner to report ready, or to die, and returns {"status", "error"}.

• pyrolab start uses that and reports exactly what happened:

  • Daemon 'x' is running.
  • failed to start: <reason>
  • did not report ready in time
  • There is no daemon named 'x' in the configuration.

  It also no longer starts a second process for something that's already running. Before, the first one was silently orphaned. Autolaunch and automatic restarts don't wait.

• reload returns False unless everything stopped cleanly and every relaunched entity reported ready.

• LockableDaemon._lock returns False when another client holds the lock.

Also

• Daemon output goes to a file: the background daemon's stdout/stderr go to logs/pyrolabd_output.log. Before, on Linux, a crashing child's traceback printed into whichever terminal had run pyrolab up. logs export skips that file, since it has no timestamps (pyrolab logs export crashes on any log entry without a parseable timestamp #71 still covers the general case).
• start_ns_loop has a new optional on_ready callback, used by the nameserver runner.

Checks run

• Suite: 350 passed, 9 skipped. pre-commit passes on every file.

• tests/test_lifecycle.py (new): fake runners that behave like real ones, plus a fake clock. It covers:

  • stop escalation through terminate and kill, an unkillable process, and parallel stops taking one grace period;
  • deregistration after a kill;
  • the exact backoff schedule [5, 15, 60, 300], then failed;
  • the stable-uptime reset, and revival by start and by reload;
  • launch-and-wait outcomes, and no duplicate spawn;
  • status text.

• tests/test_registration.py (new): real nameservers in threads. It covers:

  • one reachable and one unreachable nameserver;
  • a nameserver that appears later;
  • the timeout, against a socket that accepts connections but never replies;
  • one warning instead of one per retry;
  • every proxy being released.

  It also runs the real DaemonRunner.run() and NameServerRunner.run() in a thread: serving while a nameserver is unreachable, registering, deregistering on exit, and reporting errors.

• The timing-sensitive tests passed repeated runs.

• End-to-end run with real child processes, from a scratch copy with isolated runtime files:

  • A crashing daemon: start reported failed to start: RuntimeError: instrument not connected, and the log showed restarts after 5, 15 and 60 s.
  • An unreachable nameserver: the service kept running, and registered with that nameserver about 18 s after it was started.
  • A daemon whose request loop ignores the stop signal: stop escalated to terminate after the 10 s grace period and reported success.
  • down: took 3 s and left no processes behind.

🤖 Generated with Claude Code