fix(watcher): reap finished issue-processing task handles [DISCORD-15363517]

[claudear]

Problem

appwrite/claudear gets OOM-killed intermittently. The daemon leaks memory in proportion to the number of issues it processes.

Watcher.spawn_handles is a Vec<JoinHandle<()>> that is push-only in production. dispatch_lane pushes a handle for every spawned process_issue task, and the only code that removes entries is drain_spawned_tasks(), whose call sites were all inside #[cfg(test)] mod tests. Nothing in the run loop, the housekeeping loop, or the stop path ever reaped them.

Tokio frees a task's allocation only once both the scheduler and the last JoinHandle are dropped, so every completed issue-processing task stayed resident for the process lifetime. That allocation is not a 24-byte handle: process_issue awaited IssueProcessor::run inline with no boxing, so each retained handle pinned the entire inlined pipeline state machine.

RSS therefore climbed monotonically across days of polling until the kernel OOM-killed the container, which then restarted with a fresh (empty) Vec — matching the reported "sometimes gets OOM killed" pattern.

Agent spawning itself is not out of bounds: the per-source/per-lane gating in dispatch_lane is correct, ProcessingState::remove decrements properly, and the intent-classification path is explicitly sequential.

Fix

Bound the handle list (the leak)

• Reap finished handles once per poll cycle (top of poll_source, before the rate-limit early return so an idle or paused watcher still frees them) and on every dispatch, so the list is bounded by concurrency rather than by issues-processed-ever.
• Box::pin the processor.run(...) await so each spawned task allocation carries a pointer instead of the fully inlined pipeline state machine, shrinking the per-task footprint while tasks are running, not just after they finish.

Make shutdown drain correctly (the drain path)

• Drain spawned tasks in stop_and_drain so shutdown waits for their teardown instead of letting runtime teardown abort them mid-operation and misreport a graceful stop.
• Close the shutdown race that recorded spawned tasks: dispatch re-checks is_running under the same lock the drain takes, then spawns and records atomically, so a dispatch either records its handle before the drain's mem::take (and is joined) or sees the stop and never spawns.
• Make the drain cancel-safe: handles are joined via &mut, so a timeout drops only the borrow — the owned handles survive for the abort pass and are never detached.
• Bound the drain: give in-flight tasks a budget to finish, then abort stragglers explicitly and log it, so a task wedged in an external git/LLM call can't stall a non-interactive shutdown (e.g. a redeploy's SIGTERM).
• Bound the post-abort join too. abort() only takes effect at a task's next await, so a task parked in synchronous blocking work (block_in_place bridging an LLM call, e.g. the agent repo classifier) never observes it until that call returns — awaiting it to completion re-introduced the unbounded wait the budget exists to prevent. The post-abort join now runs under a short grace; any handle still unfinished is dropped and logged.
• Bound runtime teardown. Detaching an unfinished handle does not stop its non-preemptible blocking work, and dropping the multi-thread runtime by default waits indefinitely for that work to return — so an orphaned block_in_place op could still stall process exit after the drain returned. main now owns the runtime and calls Runtime::shutdown_timeout(5s) after block_on, capping teardown and detaching whatever remains for process exit to reclaim (the sentry and logging flush guards stay intact).
• Don't re-poll completed handles. The graceful pass joins via &mut and can poll an early handle to completion before it times out on a later wedged one; the post-abort pass then iterated all handles again and awaited the finished one, and awaiting a JoinHandle after it returned Ready panics ("JoinHandle polled after completion"), crashing shutdown. The post-abort join now skips already-finished handles.

Track retry/review processing in spawn_handles (this commit)

Retries and review feedback reach process_issue through trigger_issue_with_feedback, which awaited it inline on the housekeeping task. That execution was never recorded in spawn_handles, so it escaped the drain entirely:

• The tokio::select! in start() completes as soon as the poll branch returns on shutdown and drops the housekeeping future, cancelling an in-flight inline process_issue at its next await.
• drain_or_abort only joins spawn_handles, so it reported a clean stop while the inline retry/review run was still mutating tracker, notifier, or agent state — exactly the failure mode the drain was built to prevent for the spawned path.

Route that path through the same spawn-and-track mechanism as the poll loop:

• trigger_issue_with_feedback now spawns process_issue into spawn_handles under the same lock the drain takes, with the identical is_running re-check, so the work survives the housekeeping future being dropped and is joined by the graceful drain.
• A oneshot channel carries the started bool back and fires at the end of process_issue, preserving the synchronous contract: the call still blocks until processing completes and still returns "already being processed" when it did not start, so per-source retry concurrency and the review-wait loop are unchanged.
• A trigger arriving after stop() is refused instead of leaking a handle past the drain's mem::take.
• Propagates self: &Arc<Self> through trigger_issue, trigger_issue_with_feedback, process_review_action, check_reviews, process_ready_retries, and run_housekeeping_cycle. The daemon paths already hold Arc<Watcher>; the one-shot retry and Trigger CLI commands build a never-started watcher, so they now mark it running for the duration of the trigger (a trigger is refused once the watcher is stopping).

Test

• test_watcher_reaps_finished_spawn_handles seeds spawn_handles with the handles of 64 completed tasks, waits for them to finish, then runs a normal poll cycle and asserts nothing is retained. Fails on main with retained 64 handles, passes with this change.
• test_stop_and_drain_joins_recorded_task / test_stop_and_drain_joins_task_to_completion assert the drain waits for a recorded task's teardown rather than reporting a clean stop while it runs.
• test_drain_or_abort_bounds_wedged_task asserts a wedged task can't stall shutdown past its budget.
• test_drain_or_abort_bounds_post_abort_join_for_blocking_task asserts a task parked in non-preemptible blocking work can't stall shutdown past the abort grace.
• test_drain_or_abort_no_double_poll_of_completed_handle asserts a completed-then-wedged handle ordering doesn't panic the drain (panics without the fix).
• test_trigger_records_spawn_handle_for_drain asserts a retry/review trigger records a drain-tracked handle in spawn_handles.
• test_trigger_refuses_once_stopped asserts a trigger after stop() spawns no handle.

Full suite: 1201 passed, 0 failed (cargo test -p claudear-engine --lib --features sqlite). cargo fmt and cargo clippy --features sqlite --lib are clean.

Reported in Discord.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

The PR bounds watcher task-handle retention and makes issue-processing shutdown tracking race-safe.

• Reaps completed issue-processing handles during polling and dispatch.
• Tracks retry and review-feedback processing through the same task registry as normal dispatch.
• Adds bounded graceful drain, abort grace, and Tokio runtime teardown.
• Adds regression coverage for handle retention, shutdown races, timeout behavior, and trigger tracking.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the current code synchronizes spawning with draining, tracks retry and review tasks, bounds both post-abort joining and runtime teardown, and avoids polling completed handles twice.

Important Files Changed

Filename	Overview
crates/claudear-engine/src/watcher.rs	Adds task-handle reaping, synchronized spawn-and-record behavior, tracked retry/review processing, and bounded shutdown draining; the previously reported shutdown defects are addressed.
src/main.rs	Replaces the runtime macro path with explicit runtime ownership so teardown is bounded after asynchronous shutdown completes.
crates/claudear-engine/src/processing.rs	Preserves the invariant that Sentry issues always enter the fix pipeline and adds focused routing coverage.
scripts/monitor_claudear.sh	Adds an operational resource-monitoring helper without affecting watcher runtime behavior.
tests/e2e_real_repo.rs	Updates test setup for the Arc-based watcher method contracts introduced by tracked trigger execution.

_{Reviews (18): Last reviewed commit: "added comments for running" | Re-trigger Greptile}

[claudear]

Fix Confidence: 88/100

High confidence the leak is real and fixed: the push-only Vec was verified directly in code (grep over crates/ and src/ returns only the field decl, init, the test-only drain, and the push), the new test fails on main and passes with the change, and all 1186 engine unit tests plus all 16 PR checks are green. Deducted for two things I could not verify end-to-end: (1) I did not measure actual RSS of a running daemon before/after, so the OOM kills could have a second contributing cause beyond this leak; (2) the stop_and_drain addition and the Box::pin change are covered only by the existing suite, not by targeted new assertions — both are low-risk (the drain is bounded by the existing 30s budget and the box is semantics-preserving), but neither is directly proven by a new test.

[ArnabChatterjee20k]

@claudear
This is a comment left during a code review.
Path: crates/claudear-engine/src/watcher.rs
Line: 1185-1188

Comment:
Shutdown misses concurrently recorded tasks

When shutdown overlaps a dispatch that has passed its running check but has not yet recorded its spawned task, drain_spawned_tasks takes the current handle vector before the new handle is inserted. The one-shot drain then returns without joining that task, causing stop_and_drain to report a graceful stop while issue processing continues to mutate tracker, notifier, or agent state.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Start a greploop in Claude Code and it will work through the open comments and keep going until this PR reviews clean.