fix(mcp): bound the machine-global smart_state lock, time the edit phases - #10
Merged
Merged
Conversation
…ases Edits that wrote to disk and then never returned were observed several times across sessions -- the file lands, the response does not, and the host eventually times out. The call keeps running: 17 of 1284 logged edits exceeded 60s, every one finishing `ok` long after its caller gave up. The shape says lock, not slowness. Two of them, started five minutes apart, completed one second apart (11:16:44 / 11:16:45 after 45.4 and 39.8 minutes) -- waiters released together, not work finishing. `_acquire_smart_state_flock` took `fcntl.flock(LOCK_EX)` with no timeout on a lock file under the LemonCrow root, so it is held across every MCP process and daemon on the machine (5+ running here). One stalled holder parks every tool call everywhere, with no error and no deadline -- and the call being parked has already done its real work; it is queued only to record an advisory counter. Now it polls LOCK_NB to a deadline (5s, LEMONCROW_SMART_STATE_LOCK_TIMEOUT) and gives up, degrading to the same unsynchronised path non-POSIX platforms already take. Worst case is a lost counter update, which is cheaper than an unbounded stall. The post-write phases are also now measured -- write / hooks / contract -- and reported as `timing_ms` past a floor (2s, LEMONCROW_EDIT_TIMING_MS). The event log records one duration per call with no phase breakdown, so diagnosing the last occurrence meant reconstructing it from completion timestamps; the next one says which phase held it. Below the floor nothing is emitted: every edit result is re-billed on each later round-trip, so a healthy edit stays silent. A slow edit keeps the block even on the silent success path -- dropping it there would hide the one signal worth having. Ruled out while narrowing this, each by measurement rather than reading: post-edit hooks are bounded (a `jj` shim sleeping 25s under total_timeout_s=2 returned at 2048ms), CodeContextEngine construction is lazy and costs 0.01s, the reindex runs off-thread, and the single worst outlier (4.9h) spans a machine sleep window in pmset and is a wall-clock artifact, not a hang. Co-Authored-By: lemoncrow <302591943+lemoncrow-agent[bot]@users.noreply.github.com>
This was referenced Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes from the RCA into edits that write to disk and then never return. The file lands, the response doesn't, and the host times out ~30 min later while the call keeps running.
Evidence
~/.lemoncrow/live_savings_events.jsonllogs every tool call with a duration. Of 12,650 calls (1,284 edits), 17 edits exceeded 60s — every one finishingstatus: ok, long after its caller gave up.The distribution says lock, not slowness:
Started five minutes apart, finished one second apart. Waiters released together. The same cluster holds six overlapping slow edits in one session between 10:09 and 14:21, plus one in a second session.
1.
_acquire_smart_state_flockhad no deadlineThe lock file lives under the LemonCrow root, so it is held across every MCP process and daemon on the machine — 5+ running on this box. One stalled holder parks every tool call everywhere, with no error and no bound. The parked call has already done its real work; it is queued only to record an advisory counter.
Now polls
LOCK_NBto a deadline (5s,LEMONCROW_SMART_STATE_LOCK_TIMEOUT) and gives up, degrading to the same unsynchronised path non-POSIX platforms already take. Worst case is a lost counter update — cheaper than an unbounded stall.2. Edits now report where the time went
The event log records one duration per call with no phase breakdown, so diagnosing the last occurrence meant reconstructing it from completion timestamps.
timing_msnow carrieswrite/hooks/contract, emitted only past a floor (2s,LEMONCROW_EDIT_TIMING_MS; 0 = always).Below the floor nothing ships — every edit result is re-billed on each later round-trip, so a healthy edit stays silent. Above it, the block survives the silent-success path too; dropping it there would hide the one signal worth having.
Ruled out while narrowing this
Each by measurement, not by reading the code:
jj/gitstep)jjshim sleeping 25s,total_timeout_s=2total_ms 2048— boundedCodeContextEngine(...)pmset -g logStill open and not addressed here:
_attach_contract_literal_reviewruns four engine-backed analyses with no deadline, and theworkflowhandler holds_STATE_LOCKacross an entire workflow run including agent execution (mcp_server.py:1984). Both are real; neither matches the synchronized-release signature, and the timing block will identify them if they are involved next time.Verification
Baseline on the parent commit with all four files reverted: 12 failed, 209 passed — identical failures, and the +3 are the new tests.
ruff checkclean;mypy --strictclean onsmart_state.py.Upstream-worthy: both defects are
lemoncrow-labcode, not fork artifacts.