Symptom
execute_command intermittently fails with:
[Error] execute_command: spawn error for '<command>': spawn /bin/bash ENOENT
ENOENT normally means the executable doesn't exist — but /bin/bash demonstrably exists and is executable on every machine this has been observed on. Something about the spawn() call itself is failing in a way libuv/Node reports as ENOENT, not a genuinely missing binary.
A second, related shape (found live 2026-08-27, see src/tools/executor.ts's child.on("close", ...) handler): the same underlying failure can surface with no error event at all — just close firing with a negative code (-2 = -ENOENT, Node's posix_spawn fast path reporting a negated errno directly). A retry path that only listens for the error event misses this shape entirely.
Current mitigations (already in src/tools/executor.ts, dated 2026-08-27)
isBashSpawnEnoent() detects the ENOENT-on-bash shape from the error event.- The
close handler separately detects the negative-code shape.
- Both retry the exact same spawn up to
MAX_SPAWN_ATTEMPTS (8) with escalating backoff (500ms * attempt).
- A double-fire bug (the same failure triggering both
error and close, each independently retrying and compounding) was found and fixed live 2026-08-27 — each attempt now settles exactly once.
Despite all of this, the failure still recurs often enough overnight (2026-08-28/29) to meaningfully slow down real headlesscode sessions — sometimes 50-90% of execute_command calls in a single session, sometimes closer to 0%.
What's confirmed as a real contributing factor (2026-08-28/29 investigation)
System memory commit pressure. cat /proc/meminfo | grep -i Committed_AS vs CommitLimit — when Committed_AS sits above roughly 95-98% of CommitLimit, spawn failure rates go up sharply; when it's brought back down to ~75-85%, failures drop a lot (not to zero).
Root-caused one specific driver of that pressure: the local llama-server (llama.cpp) process serving the code model was defaulting to --parallel auto-detecting 4 concurrent slots, each allocating a full -c 65536-token KV cache — 4x more memory than needed for a workload that only ever has one active session against it at a time. Pinning --parallel 1 dropped system Committed_AS from ~98% to ~80% immediately. llama-server's own RSS also grows slowly over hours of continuous use (observed climbing back toward 90%+ over several hours even at --parallel 1), so this needs periodic restarts, not just a one-time fix.
What's NOT explained
- Even at comfortable memory levels (~75-85%
Committed_AS), spawn failures still occur occasionally — just far less often. Memory pressure looks like a strong contributing factor, not the sole cause.
- The leading theory for the residual failures (per the 2026-08-27 comments already in the code) is a transient Node.js/libuv hiccup under sustained process load — plausible given this only ever happens inside a long-running headlesscode process, and a live standalone repro of the same command outside that process was observed spawning cleanly on the first try. This is not proven, just the best working theory so far.
- No correlation has yet been checked against: open file descriptor count, zombie/defunct process accumulation,
ulimit -u (max user processes) headroom, or GC pause timing in the long-running Node process. Worth checking if this recurs with memory pressure ruled out.
Suggested next steps
- Add lightweight diagnostic logging on a spawn failure: capture
Committed_AS/CommitLimit, open fd count for the process, and total system process count at the moment of failure, so the next occurrence can directly confirm or rule out memory pressure as the proximate cause each time (rather than inferring it after the fact from separate /proc/meminfo snapshots).
- Consider whether
execute_command should default to a non-shell spawn path (direct execFile where the command doesn't need shell features) to reduce the surface area, since the failure is specifically on spawning /bin/bash itself.
- If the residual (non-memory-pressure) failures keep recurring, capture strace/ltrace output on a live failure to see the actual syscall-level error, rather than continuing to infer from Node's own (possibly misleading) error reporting.
Evidence source
All of the above was observed directly during a long (~12+ hour) overnight headlesscode session running local Qwen3.5-9B+LoRA workers against joeos issue #26 real repo work — not synthetic testing. Specific measurements (Committed_AS percentages, the --parallel finding) are reproducible via the commands cited above.
Symptom
execute_commandintermittently fails with:ENOENTnormally means the executable doesn't exist — but/bin/bashdemonstrably exists and is executable on every machine this has been observed on. Something about thespawn()call itself is failing in a way libuv/Node reports as ENOENT, not a genuinely missing binary.A second, related shape (found live 2026-08-27, see
src/tools/executor.ts'schild.on("close", ...)handler): the same underlying failure can surface with noerrorevent at all — justclosefiring with a negativecode(-2=-ENOENT, Node's posix_spawn fast path reporting a negated errno directly). A retry path that only listens for theerrorevent misses this shape entirely.Current mitigations (already in
src/tools/executor.ts, dated 2026-08-27)isBashSpawnEnoent()detects the ENOENT-on-bash shape from theerrorevent.closehandler separately detects the negative-codeshape.MAX_SPAWN_ATTEMPTS(8) with escalating backoff (500ms * attempt).errorandclose, each independently retrying and compounding) was found and fixed live 2026-08-27 — each attempt now settles exactly once.Despite all of this, the failure still recurs often enough overnight (2026-08-28/29) to meaningfully slow down real headlesscode sessions — sometimes 50-90% of
execute_commandcalls in a single session, sometimes closer to 0%.What's confirmed as a real contributing factor (2026-08-28/29 investigation)
System memory commit pressure.
cat /proc/meminfo | grep -i Committed_ASvsCommitLimit— whenCommitted_ASsits above roughly 95-98% ofCommitLimit, spawn failure rates go up sharply; when it's brought back down to ~75-85%, failures drop a lot (not to zero).Root-caused one specific driver of that pressure: the local
llama-server(llama.cpp) process serving the code model was defaulting to--parallelauto-detecting 4 concurrent slots, each allocating a full-c 65536-token KV cache — 4x more memory than needed for a workload that only ever has one active session against it at a time. Pinning--parallel 1dropped systemCommitted_ASfrom ~98% to ~80% immediately.llama-server's own RSS also grows slowly over hours of continuous use (observed climbing back toward 90%+ over several hours even at--parallel 1), so this needs periodic restarts, not just a one-time fix.What's NOT explained
Committed_AS), spawn failures still occur occasionally — just far less often. Memory pressure looks like a strong contributing factor, not the sole cause.ulimit -u(max user processes) headroom, or GC pause timing in the long-running Node process. Worth checking if this recurs with memory pressure ruled out.Suggested next steps
Committed_AS/CommitLimit, open fd count for the process, and total system process count at the moment of failure, so the next occurrence can directly confirm or rule out memory pressure as the proximate cause each time (rather than inferring it after the fact from separate/proc/meminfosnapshots).execute_commandshould default to a non-shell spawn path (directexecFilewhere the command doesn't need shell features) to reduce the surface area, since the failure is specifically on spawning/bin/bashitself.Evidence source
All of the above was observed directly during a long (~12+ hour) overnight headlesscode session running local Qwen3.5-9B+LoRA workers against
joeosissue #26 real repo work — not synthetic testing. Specific measurements (Committed_AS percentages, the--parallelfinding) are reproducible via the commands cited above.