Symptom
A freshly-spawned worker can be visible in who and yet silently lose messages sent to it shortly after spawn. The send tool returns "Message sent" but the recipient never sees the message — not delayed, lost.
Discovered while debugging an unrelated worker-spawn config issue; the spawn issue was the proximate cause in that case, but this race is independently real and would bite anyone doing rapid spawn-then-send.
Root cause
Three things happen in order during a worker's intercom MCP boot (intercom/server.py:_main):
_acquire_lock() — worker now appears in who
ServerSession enters context, _session is set
_watch_event_bus task spawned via tg.start_soon(...)
Two problems:
(a) Visible-but-not-listening window between (1) and (3). A sender that polls who, sees the worker, and immediately sends will append to events.jsonl before the worker is tailing it.
(b) No replay. When _watch_event_bus finally starts, it initializes:
# server.py:244-248
try:
pos = _EVENTS_FILE.stat().st_size
except OSError:
pos = 0
i.e., it starts at the current end of the file. Anything appended during window (a) is permanently lost from that worker's perspective — there's no per-recipient inbox, just a shared events.jsonl tailed from end-of-file.
Bonus: _handle_send returns "Message sent to: <name>" (server.py:392-395) immediately after _append_event writes the line. That's a bus-append confirmation, not a delivery confirmation. Callers reasonably interpret it as the latter.
Possible fixes (in increasing order of work)
who reports per-agent ready state. Surface whether the agent has completed all three boot steps, so senders can wait for "actually listening" before sending. Cheap, doesn't change protocol.
- Replay-from-position per agent. Each worker remembers (or rediscovers) its last-seen offset on startup and replays anything since. Solves the race cleanly; requires per-agent state file.
- Per-recipient inbox files with ack-based delivery. Bigger restructure but solves the race and lets
send return a real delivery confirmation.
I'd lean toward (1) as a quick mitigation and (2) as the real fix.
Workaround for callers today
After spawn, send a ping and treat the worker as ready only after it replies via intercom (not just appearance in who). Retry until acked.
Symptom
A freshly-spawned worker can be visible in
whoand yet silently lose messages sent to it shortly after spawn. Thesendtool returns "Message sent" but the recipient never sees the message — not delayed, lost.Discovered while debugging an unrelated worker-spawn config issue; the spawn issue was the proximate cause in that case, but this race is independently real and would bite anyone doing rapid spawn-then-send.
Root cause
Three things happen in order during a worker's intercom MCP boot (
intercom/server.py:_main):_acquire_lock()— worker now appears inwhoServerSessionenters context,_sessionis set_watch_event_bustask spawned viatg.start_soon(...)Two problems:
(a) Visible-but-not-listening window between (1) and (3). A sender that polls
who, sees the worker, and immediately sends will append toevents.jsonlbefore the worker is tailing it.(b) No replay. When
_watch_event_busfinally starts, it initializes:i.e., it starts at the current end of the file. Anything appended during window (a) is permanently lost from that worker's perspective — there's no per-recipient inbox, just a shared
events.jsonltailed from end-of-file.Bonus:
_handle_sendreturns"Message sent to: <name>"(server.py:392-395) immediately after_append_eventwrites the line. That's a bus-append confirmation, not a delivery confirmation. Callers reasonably interpret it as the latter.Possible fixes (in increasing order of work)
whoreports per-agent ready state. Surface whether the agent has completed all three boot steps, so senders can wait for "actually listening" before sending. Cheap, doesn't change protocol.sendreturn a real delivery confirmation.I'd lean toward (1) as a quick mitigation and (2) as the real fix.
Workaround for callers today
After spawn, send a
pingand treat the worker as ready only after it replies via intercom (not just appearance inwho). Retry until acked.