Skip to content

V3: Recover interrupted runs when the built server starts - #683

Merged
LinuxDevil merged 1 commit into
mainfrom
v3-recover-on-start
Oct 19, 2026
Merged

LinuxDevil merged 1 commit into
mainfrom
v3-recover-on-start

Conversation

@LinuxDevil

Copy link
Copy Markdown
Owner

Fixes audit finding OPS-2 (Vercel competitor audit, 2026-10-19).

What was wrong

A server built with lousho build --target=node-server (or docker) and run with LOUSHO_STORE=sqlite: never finished a turn that a kill interrupted mid-tool. After a restart, GET /chat/<id> returned {"messages":[],"pending":{"status":"in-progress"}} and nothing resumed on its own. Resending the POST without an Idempotency-Key ran the tool again and duplicated the turn. The generated server never called agent.recover().

What changed

  • createDeployedServer() (src/deploy/nodeServer.ts) starts agent.recover() once the server listens. It first logs lousho server: resuming N interrupted run(s) when it finds any, then one recovered interrupted run '<id>' line per finished run. Recovery errors are logged and never crash startup. A store with no listable checkpoints (none, or a custom store without list()) is skipped quietly.
  • Only in-progress runs are resumed. Runs paused on an approval or a question are left alone, and so are runs whose lastError is not retryable. A run whose lease another process holds gets LOUSHO_SESSION_BUSY and is retried on a later sweep without an error, using the existing agent.recover() semantics.
  • shutdown() and server close stop the recovery loop. Resumes still running are aborted, and their checkpoints stay for the next start.
  • Opt-out: LOUSHO_RECOVER_ON_START=0 (or false), or the internal option recovery: false. The option also takes { intervalMs }, used by the tests.
  • The generated server.ts header comment documents it. docs/deployment.md has a new "Crash recovery" section, and llms.txt / llms-full.txt are regenerated.
  • /chat status mapping is unchanged. In one process, a resend without a key that races the auto-resume runs after the resumed turn, as a new turn, because the session's turns already run one at a time. The interrupted tool does not run twice. A resend with the original Idempotency-Key gets the resumed turn's reply (feat(server): Idempotency-Key on POST /chat #667 behaviour kept).

Behavior change: the built node-server and docker servers now resume interrupted runs from the store at startup and re-sweep every 30 s. A tool call that was in flight at the crash runs again (at-least-once), so make such tools idempotent. Set LOUSHO_RECOVER_ON_START=0 to keep the old behaviour, for example when another process already runs agent.recover().

Tests

New src/deploy/nodeServerRecover.test.ts (8 tests, fileStore-backed). Two of them fail on origin/main:

  • A session turn interrupted mid-tool is resumed after listen. charge ran 1×, the in-flight crash ran 2× (only the in-flight step re-ran), the transcript has one user turn ending in the resumed reply, and both log lines are printed.
  • An awaiting-approval turn is left alone: the tool is not run and it stays pending.
  • The opt-out works with LOUSHO_RECOVER_ON_START=0, with =false, and with recovery: false.
  • Startup is quiet with an empty memory store and with no checkpoint store at all.
  • A racing resend without a key runs after the recovered turn, and no tool runs twice.
  • A racing resend with the same Idempotency-Key gets the recovered reply, nothing runs again, and the transcript has one turn.

Verification:

  • Passed: npx tsc --noEmit, npm run typecheck:tests, npm run lint, npm run api:check (no changes) and docs:llms:check.
  • npm run test:coverage passed: 407 files, 6024 tests.
  • npm run fallow exited 0.
  • Focused runs of the nodeServer and node-server adapter suites passed.

Changelog: The built node-server/docker server now resumes runs a crash left in-progress at startup (agent.recover()), leaving approval pauses alone; opt out with LOUSHO_RECOVER_ON_START=0.

🤖 Generated with Claude Code

The node-server and docker targets now call agent.recover() once the
server listens, so a run or session turn a crash left in-progress is
finished from its last checkpoint without a client resending it. Runs
paused on an approval are left alone; leased runs are skipped and
retried. LOUSHO_RECOVER_ON_START=0 turns it off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LinuxDevil
LinuxDevil merged commit 8c538e8 into main Oct 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant