Repository navigation
V3: Recover interrupted runs when the built server starts - #683
Merged
Merged
Conversation
The node-server and docker targets now call agent.recover() once the server listens, so a run or session turn a crash left in-progress is finished from its last checkpoint without a client resending it. Runs paused on an approval are left alone; leased runs are skipped and retried. LOUSHO_RECOVER_ON_START=0 turns it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LinuxDevil
force-pushed
the
v3-recover-on-start
branch
from
October 19, 2026 22:08
03ecae3 to
b256b0c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes audit finding OPS-2 (Vercel competitor audit, 2026-10-19).
What was wrong
A server built with
lousho build --target=node-server(or docker) and run withLOUSHO_STORE=sqlite:never finished a turn that a kill interrupted mid-tool. After a restart,GET /chat/<id>returned{"messages":[],"pending":{"status":"in-progress"}}and nothing resumed on its own. Resending the POST without anIdempotency-Keyran the tool again and duplicated the turn. The generated server never calledagent.recover().What changed
createDeployedServer()(src/deploy/nodeServer.ts) startsagent.recover()once the server listens. It first logslousho server: resuming N interrupted run(s)when it finds any, then onerecovered interrupted run '<id>'line per finished run. Recovery errors are logged and never crash startup. A store with no listable checkpoints (none, or a custom store withoutlist()) is skipped quietly.in-progressruns are resumed. Runs paused on an approval or a question are left alone, and so are runs whoselastErroris not retryable. A run whose lease another process holds getsLOUSHO_SESSION_BUSYand is retried on a later sweep without an error, using the existingagent.recover()semantics.shutdown()and serverclosestop the recovery loop. Resumes still running are aborted, and their checkpoints stay for the next start.LOUSHO_RECOVER_ON_START=0(orfalse), or the internal optionrecovery: false. The option also takes{ intervalMs }, used by the tests.server.tsheader comment documents it.docs/deployment.mdhas a new "Crash recovery" section, andllms.txt/llms-full.txtare regenerated./chatstatus mapping is unchanged. In one process, a resend without a key that races the auto-resume runs after the resumed turn, as a new turn, because the session's turns already run one at a time. The interrupted tool does not run twice. A resend with the originalIdempotency-Keygets the resumed turn's reply (feat(server): Idempotency-Key on POST /chat #667 behaviour kept).Behavior change: the built node-server and docker servers now resume interrupted runs from the store at startup and re-sweep every 30 s. A tool call that was in flight at the crash runs again (at-least-once), so make such tools idempotent. Set
LOUSHO_RECOVER_ON_START=0to keep the old behaviour, for example when another process already runsagent.recover().Tests
New
src/deploy/nodeServerRecover.test.ts(8 tests, fileStore-backed). Two of them fail on origin/main:listen.chargeran 1×, the in-flightcrashran 2× (only the in-flight step re-ran), the transcript has one user turn ending in the resumed reply, and both log lines are printed.LOUSHO_RECOVER_ON_START=0, with=false, and withrecovery: false.Idempotency-Keygets the recovered reply, nothing runs again, and the transcript has one turn.Verification:
npx tsc --noEmit,npm run typecheck:tests,npm run lint,npm run api:check(no changes) anddocs:llms:check.npm run test:coveragepassed: 407 files, 6024 tests.npm run fallowexited 0.Changelog: The built node-server/docker server now resumes runs a crash left in-progress at startup (
agent.recover()), leaving approval pauses alone; opt out withLOUSHO_RECOVER_ON_START=0.🤖 Generated with Claude Code