[orchestrator-v2] fix(orchestrator): Surface Claude background wake turns as continuation runs#3752
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
vouch:unvouched
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| ordinal: projection.messages.length + 1, | ||
| }); | ||
| const commandId = CommandId.make(`provider-continuation:${messageId}`); | ||
| yield* threads.dispatch({ |
There was a problem hiding this comment.
🟡 Medium orchestration-v2/ProviderContinuationService.ts:42
dispatchContinuation builds the message.dispatch without passing request.providerThreadId or request.driver, so the orchestrator resolves the target provider thread from the thread's current active/model state. If the app thread has switched providers or has no active provider thread when the wake arrives, the continuation run starts on the wrong (or empty) native thread and the buffered wake messages are drained from an unrelated buffer — the wake turn silently vanishes. Consider propagating providerThreadId and driver into the dispatch payload so the run targets the originating native thread.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/ProviderContinuationService.ts around line 42:
`dispatchContinuation` builds the `message.dispatch` without passing `request.providerThreadId` or `request.driver`, so the orchestrator resolves the target provider thread from the thread's current active/model state. If the app thread has switched providers or has no active provider thread when the wake arrives, the continuation run starts on the wrong (or empty) native thread and the buffered wake messages are drained from an unrelated buffer — the wake turn silently vanishes. Consider propagating `providerThreadId` and `driver` into the dispatch payload so the run targets the originating native thread.
mwolson
force-pushed
the
fix/claude-wake-turns
branch
from
7174931 to
5ebd8b9
Compare
| updated.set(key, { ...latestEntry, pinnedSinceMs }); | ||
| return updated; | ||
| }); | ||
| yield* scheduleIdleReleaseInternal(input.providerSessionId); |
There was a problem hiding this comment.
🟠 High orchestration-v2/ProviderSessionManager.ts:588
The pin-deferral path in releaseIfStillIdle deadlocks: when hasPendingBackgroundWork is true and the pin hasn't expired, it calls scheduleIdleReleaseInternal, which calls cancelIdleFiber(entry.idleFiber). That idleFiber is the fiber currently executing releaseIfStillIdle, so Fiber.interrupt interrupts the current fiber and then awaits it — a self-interrupt/self-await that never resolves. The replacement idle timer is never scheduled, so a session with pending background work stays pinned forever after the first idle timeout, bypassing maxIdlePinMs and leaking the provider process until shutdown.
The re-arm should fork the new timer first and update entry.idleFiber before cancelling the old fiber, or the cancellation should skip the current fiber.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/ProviderSessionManager.ts around line 588:
The pin-deferral path in `releaseIfStillIdle` deadlocks: when `hasPendingBackgroundWork` is true and the pin hasn't expired, it calls `scheduleIdleReleaseInternal`, which calls `cancelIdleFiber(entry.idleFiber)`. That `idleFiber` is the fiber currently executing `releaseIfStillIdle`, so `Fiber.interrupt` interrupts the current fiber and then awaits it — a self-interrupt/self-await that never resolves. The replacement idle timer is never scheduled, so a session with pending background work stays pinned forever after the first idle timeout, bypassing `maxIdlePinMs` and leaking the provider process until shutdown.
The re-arm should fork the new timer first and update `entry.idleFiber` before cancelling the old fiber, or the cancellation should skip the current fiber.
5ebd8b9 to
6054588
Compare
There was a problem hiding this comment.
🟡 Medium
makeDefaultClaudeAdapterV2 reads ProviderContinuationRequests from the context, but since it is a Context.Reference with a no-op default (offer: () => Effect.void), any caller that provides ClaudeAdapterV2.layer without also supplying the live continuation queue captures the no-op. In that configuration Claude background wake turns are silently dropped, defeating the continuation behavior this PR introduces. Consider adding ProviderContinuationRequests to layer's environment type so the live queue must be provided, or falling back to the live implementation inside the adapter when the default is detected.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts around line 3848:
`makeDefaultClaudeAdapterV2` reads `ProviderContinuationRequests` from the context, but since it is a `Context.Reference` with a no-op default (`offer: () => Effect.void`), any caller that provides `ClaudeAdapterV2.layer` without also supplying the live continuation queue captures the no-op. In that configuration Claude background wake turns are silently dropped, defeating the continuation behavior this PR introduces. Consider adding `ProviderContinuationRequests` to `layer`'s environment type so the live queue must be provided, or falling back to the live implementation inside the adapter when the default is detected.
There was a problem hiding this comment.
🟡 Medium
When hasPendingBackgroundWork returns true and the pin cap has not expired, releaseIfStillIdle calls scheduleIdleReleaseInternal, which sleeps for the full idleTimeoutMs before checking again. With idleTimeoutMs=30m and maxIdlePinMs=5m, the session can stay pinned up to ~60 minutes instead of the intended ~35-minute ceiling — the maxIdlePinMs cap is only evaluated once per idle interval rather than as a hard deadline. Consider scheduling the re-check with min(idleTimeoutMs, maxIdlePinMs - elapsed) so the pin cap acts as a true upper bound on deferral time.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/ProviderSessionManager.ts around line 630:
When `hasPendingBackgroundWork` returns true and the pin cap has not expired, `releaseIfStillIdle` calls `scheduleIdleReleaseInternal`, which sleeps for the full `idleTimeoutMs` before checking again. With `idleTimeoutMs=30m` and `maxIdlePinMs=5m`, the session can stay pinned up to ~60 minutes instead of the intended ~35-minute ceiling — the `maxIdlePinMs` cap is only evaluated once per idle interval rather than as a hard deadline. Consider scheduling the re-check with `min(idleTimeoutMs, maxIdlePinMs - elapsed)` so the pin cap acts as a true upper bound on deferral time.
| const hasPendingWork = | ||
| entry.runtime.hasPendingBackgroundWork === undefined | ||
| ? false | ||
| : yield* entry.runtime.hasPendingBackgroundWork.pipe( |
There was a problem hiding this comment.
🟡 Medium orchestration-v2/ProviderSessionManager.ts:568
releaseIfStillIdle swallows all failures from entry.runtime.hasPendingBackgroundWork with Effect.catchCause(() => Effect.succeed(false)), so any adapter defect or unexpected cancellation — not just the benign idle-fiber interrupt race — causes the manager to treat pending background work as absent and proceed to releaseEntry, closing the session and killing the very background task this feature is meant to preserve. Consider narrowing the catch to only the interrupt-race cause (or distinguishing Interrupt from other failures) so genuine probe failures do not trigger premature session release.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/ProviderSessionManager.ts around line 568:
`releaseIfStillIdle` swallows all failures from `entry.runtime.hasPendingBackgroundWork` with `Effect.catchCause(() => Effect.succeed(false))`, so any adapter defect or unexpected cancellation — not just the benign idle-fiber interrupt race — causes the manager to treat pending background work as absent and proceed to `releaseEntry`, closing the session and killing the very background task this feature is meant to preserve. Consider narrowing the catch to only the interrupt-race cause (or distinguishing `Interrupt` from other failures) so genuine probe failures do not trigger premature session release.
mwolson
force-pushed
the
fix/claude-wake-turns
branch
from
6054588 to
9bc390d
Compare
| providerSessionId: input.providerSessionId, | ||
| pinnedForMs: now - pinnedSinceMs, | ||
| }); | ||
| yield* Ref.update(sessions, (latest) => { |
There was a problem hiding this comment.
🟠 High orchestration-v2/ProviderSessionManager.ts:579
When hasPendingBackgroundWork is slow, the deferred-pin branch in releaseIfStillIdle writes pinnedSinceMs back into the session entry without revalidating idleGeneration. If the session goes busy and returns to idle during that check, markBusy clears pinnedSinceMs for the new turn, but this stale callback overwrites it with the old timestamp. The new idle period then inherits the previous pin age, so maxIdlePinMs is reached prematurely and the session is released while background work is still running. Consider capturing the generation and skipping the pinnedSinceMs write when the entry's idleGeneration no longer matches.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/ProviderSessionManager.ts around line 579:
When `hasPendingBackgroundWork` is slow, the deferred-pin branch in `releaseIfStillIdle` writes `pinnedSinceMs` back into the session entry without revalidating `idleGeneration`. If the session goes busy and returns to idle during that check, `markBusy` clears `pinnedSinceMs` for the new turn, but this stale callback overwrites it with the old timestamp. The new idle period then inherits the previous pin age, so `maxIdlePinMs` is reached prematurely and the session is released while background work is still running. Consider capturing the generation and skipping the `pinnedSinceMs` write when the entry's `idleGeneration` no longer matches.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| providerThreadId: route.providerThreadId, | ||
| driver: CLAUDE_PROVIDER, | ||
| detail, | ||
| }); |
There was a problem hiding this comment.
Dedup blocks wake after dispatch fails
Medium Severity
When a continuation request fails to dispatch (e.g., for an archived thread), the requestedContinuations flag isn't cleared, and wakeBuffers remain undrained. This prevents future wake events on that native thread from triggering new continuations, effectively stranding buffered wake output.
Reviewed by Cursor Bugbot for commit 9bc390d. Configure here.
| // dropped as before instead of triggering a spurious continuation. | ||
| const isPendingTaskNotification = | ||
| isNotification && (yield* Ref.get(pendingBackgroundTaskIds)).has(message.task_id); | ||
| const isWakeEvidence = |
There was a problem hiding this comment.
Pending task id survives result-only drain
Medium Severity
pendingBackgroundTaskIds is populated on task_started for local_bash work but is only removed when a task_notification is handled in an active turn (including replay). A continuation that drains a wake buffer containing only a result (no buffered notification) can complete successfully while the task id remains in the set, so hasPendingBackgroundWork may stay true and idle release stays deferred after the wake was already ingested.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9bc390d. Configure here.
ApprovabilityVerdict: Needs human review 6 blocking correctness issues found. This PR introduces new capability for Claude background wake turns with new services, state management, and idle release deferral logic. Multiple unresolved review comments identify potential deadlocks and race conditions in the session manager and adapter code. You can customize Macroscope's approvability policy. Learn more. |
Introduce a provider-agnostic continuation request queue, worker, and optional hasPendingBackgroundWork idle pin so adapters can surface post-settle native wake traffic as first-class runs. No adapter offers continuations yet; behavior is unchanged until Claude or Grok specialty commits wire attach mode.
…on runs When a background task (run_in_background Bash) settles, the Claude CLI streams a whole new turn over the still-open SDK query. Buffer null-activeTurn wake messages, request one internal continuation run per wake, and drain the buffer in attach mode without re-prompting the CLI. Report pending background work so idle release stays pinned until the wake is ingested.
mwolson
force-pushed
the
fix/claude-wake-turns
branch
from
9bc390d to
10bd126
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
There are 5 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 10bd126. Configure here.
| const message = input.message; | ||
| const context = yield* Ref.get(activeTurn); | ||
| if (context === null) { | ||
| yield* bufferWakeMessage({ nativeThreadId: liveQuery.nativeThreadId, message }); |
There was a problem hiding this comment.
Wake task_started messages ignored
Medium Severity
With no active provider turn, every SDK frame goes through bufferWakeMessage, which only keeps assistant, user, result, or tracked task_notification frames. task_started for another local_bash job during that wake window is dropped, so the session never registers the task and later notifications are not treated as wake evidence.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 10bd126. Configure here.
| ? false | ||
| : yield* entry.runtime.hasPendingBackgroundWork.pipe( | ||
| Effect.catchCause(() => Effect.succeed(false)), | ||
| ); |
There was a problem hiding this comment.
Pending work check hides failures
Medium Severity
During idle release, hasPendingBackgroundWork is wrapped in Effect.catchCause that returns false on any failure. A defect or error in the adapter check is treated as “no pending work,” so the manager can idle-release the provider session while wake buffers or background tasks are still outstanding.
Reviewed by Cursor Bugbot for commit 10bd126. Configure here.
| dispatchMode: { type: "queue_after_active" }, | ||
| createdBy: "agent", | ||
| creationSource: "provider", | ||
| }); |
There was a problem hiding this comment.
Continuation ignores wake provider thread
Medium Severity
ProviderContinuationService records providerThreadId on the wake request but dispatches only threadId. Queued continuation runs bind to the thread’s active provider thread, not the route stored when the wake was buffered, so a provider switch before dispatch can start a continuation on a different session with an empty wake buffer.
Reviewed by Cursor Bugbot for commit 10bd126. Configure here.
1 participant
Summary
Background tasks started inside a Claude turn (
run_in_backgroundBash)currently vanish from t3code when they settle: the CLI's post-settle wake
turn is silently dropped, so the displayed thread diverges from the real
native transcript, and the 30 minute idle release kills the CLI (and the
pending task) before it can even wake. This PR ingests wake turns as
first-class continuation runs and keeps the provider session pinned while
background work is pending.
Stacked on #3750 (
fix/claude-session-resume); one commit onfix/claude-wake-turnsabove it.Problem and Fix
ClaudeAdapterV2.handleSdkMessageearly-returns whenactiveTurnis null, so every message of the wake turn is dropped.activeTurnmessages now buffer per native thread when they are wake evidence (assistant/user/result, or a task notification for a tracked pending task, with or without a summary). A pending-task notification or result offers one continuation request (deduped per wake) to a newProviderContinuationRequestsqueue; a worker (ProviderContinuationService) dispatches an internalmessage.dispatch(createdBy: "agent",creationSource: "provider",queue_after_active), so the wake flows through the normal run pipeline, starting immediately on an idle thread and queueing behind any active run. The continuation turn sends no prompt to the CLI and instead drains the buffered wake messages into itself (results replayed last; a result finalizes the turn).ProviderSessionManageridle-releases the provider session after 30 minutes. Closing the SDK query kills the CLI child, and any pending background task dies with it (observed: a task notification flushed to the transcript 13ms afterquery.close, with no response).hasPendingBackgroundWorkruntime capability (pendinglocal_bashtasks, non-empty wake buffer, or outstanding continuation request).releaseIfStillIdledefers theidle_timeoutrelease while it reports true, re-arming the timer, with cumulative deferral capped by a newmaxIdlePinMsoption (default 4h) so a hung task cannot pin a session forever. Other release reasons are unchanged.Design notes:
hasPendingBackgroundWorkyields to the adapter between the idle check and the release, so
releaseEntrynow takesonlyIfIdleGenerationand revalidatesbusyCount/idleGeneration inside its atomic entry removal, making a release
of a session that turned busy during the check structurally impossible
(the existing interrupt-on-activity path already covered every
deterministic interleaving; this closes the preemption-timing residue).
continuation queue behind the user run (
queue_after_active; Claude setssupportsQueuedMessages); the user turn leaves the wake buffer intact andthe continuation drains it afterwards with correct attribution. A
continuation turn that finds an empty buffer settles immediately as a
completed run with no items.
local_bashtasks no longer render as subagent nodes; theirlifetime is tracked from
task_startedto terminaltask_notificationin a session-scoped registry (the durable cross-turn version of the
previous per-turn
ignoredTaskIds).creationSource: "provider"just produces a continuation turn thatattaches to an empty buffer and settles instantly.
continuation sink are optional with no-op defaults.
Validation
vp check: passvp run typecheck: pass (full monorepo)vp test apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts:19/19, including 5 new wake-turn tests (buffer + single deduped
continuation request with correct routing and detail; drain into a
continuation turn with no CLI prompt and the wake result surfacing as
assistant output; racing user turn leaving the buffer for the queued
continuation; null-summary notification still clearing the pending-task
registry; spurious continuation settling immediately)
all Medium findings addressed (null-summary notifications, queued-run
race, user-drain attribution), remainder documented below
vp test apps/server/src/orchestration-v2/ProviderSessionManager.test.ts:18/18, including 3 new tests (idle release deferred while background work
is pending, pinned session released once
maxIdlePinMsexpires, and aturn starting during the pending-work check never losing the session)
ClaudeReplayFixtures.integration.test.tsandOrchestratorReplayFixtures.integration.test.ts(claude fixtures): passKnown limitations
the server and the wake is lost (out of scope; nothing the orchestrator
can do server-side).
active turn, exactly as today (the over-settle family tracked around
[orchestrator-v2] fix(orchestrator): Harden Grok v2 settlement and steer message visibility #3578); this PR only changes the previously-dropped null-
activeTurnpath.
today and that wake is lost (same outcome as before, minus up to 4 hours).
dispatch failure) is logged but not retried; the wake buffer then sits
until a later turn drains it or the pin cap releases the session.
hasPendingBackgroundWorkchecks the pending-task registry and wakebuffer but not the outstanding-continuation flag; the dispatch window is
covered by the run's own busy pin.
dedicated unit tests; its behavior is covered indirectly by the adapter
tests plus the trivial dispatch body.
Note
Medium Risk
Changes orchestration turn lifecycle, session idle release timing, and Claude SDK message handling; incorrect buffering or release could drop wake output or keep sessions pinned longer than intended.
Overview
Claude background wake turns are no longer dropped when the SDK streams messages with no active provider turn.
ClaudeAdapterV2buffers wake evidence per native thread, emits one deduped continuation request, and provider-synthesized runs (createdBy: "agent",creationSource: "provider") drain that buffer without sending a prompt to the CLI.local_bashbackground tasks are tracked session-wide so they do not render as subagent nodes, and native queries resume whenproviderTurnOrdinal > 1so idle-released sessions do not reopen with a conflicting session id.A new
ProviderContinuationRequestsqueue andProviderContinuationServiceworker dispatch internalmessage.dispatchwithqueue_after_active, wired through productionruntimeLayerand shared adapter infrastructure.ProviderSessionManagerdefersidle_timeoutrelease while optionalhasPendingBackgroundWorkis true, withmaxIdlePinMs(default 4h) and anonlyIfIdleGenerationguard so release cannot race a session that became busy during the check.Reviewed by Cursor Bugbot for commit 10bd126. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Surface Claude background wake turns as continuation runs in orchestrator-v2
ProviderContinuationRequestsqueue and a background worker inProviderContinuationServicethat drains continuation requests and dispatches internalmessage.dispatchevents to enqueue continuation runs withcreatedBy: 'agent'andcreationSource: 'provider'.ClaudeAdapterV2to buffer SDK wake messages when no active turn exists, emit a single continuation request per wake, drain buffered messages into a provider-synthesized continuation turn, and resume native sessions when a prior provider turn exists.ProviderSessionManagerto defer idle session release whilehasPendingBackgroundWorkis true, bounded by amaxIdlePinMscap, with a generation guard to prevent stale releases.runtimeLayer.tsandProviderOrchestrationAdapterInfrastructure.idleTimeoutMswhen background work is pending; the pin cap (DEFAULT_MAX_IDLE_PIN_MS) bounds the maximum deferral window.Macroscope summarized 10bd126.