[bug]: session-processor cancel/retry tests intermittently never see the canceled status on CI #9948

Is there an existing issue for this problem?

• I have searched the existing issues

Install method

Manual (dev checkout): the python tests CI workflow

Operating system

Linux and Windows (GitHub Actions py3.12: linux-cpu and py3.12: windows-cpu)

GPU vendor

None (CPU)

GPU model

No response

GPU VRAM

No response

Version number

main at e8db86fc71 and later

Browser

No response

System Information

GitHub-hosted runners, Python 3.12, CPU torch wheels, pytest -n logical.

What happened

Session-processor tests that cancel a queue item from an on_after_run_node callback intermittently never observe the canceled status. They fail at:

assert registered_event_bus.wait_for_status(item_id, "canceled")
AssertionError: assert False

A different subset fails on each run, including two runs of the same commit. The tests seen failing:

• tests/app/services/session_processor/test_for_loop_processor_sqlite.py
  • test_processor_sqlite_flat_for_cancel_retry_isolates_execution_state[False|True]
  • test_processor_sqlite_nested_iterate_for_cancel_retry_reloads_fresh_stream_state
  • test_processor_sqlite_three_level_nested_for_cancel_restart_retry_isolates_execution_state
  • test_processor_sqlite_four_level_nested_for_cancel_restart_retry_isolates_execution_state
  • test_processor_sqlite_two_sibling_nested_for_failure_cleans_runtime
  • test_processor_sqlite_three_level_nested_for_failure_cleans_runtime
• tests/app/services/session_processor/test_if_processor_sqlite.py
  • test_if_retry_starts_fresh_and_preserves_selected_output_without_stale_activation[...] (several parametrizations)
  • test_if_early_cancellation_before_resolution_does_not_run_any_branch_continuation[False-...]

Runs where this happened:

The same suites pass on the other runners of these runs, and on reruns of other commits.

What you expected to happen

A cancel issued from on_after_run_node is reported as canceled every time, and the tests are deterministic.

How to reproduce the problem

Run the python tests workflow. On main, 3 of the 12 runs between 871143c0a1 and 2aa9fc95e4 (2026-10-19) hit it. I have not reproduced it locally.

Additional context

The event never arrives; it is not just late. The If tests wait in the test thread with _PROCESSOR_TIMEOUT = 30 s (test_if_processor_sqlite.py:27) and still fail. A longer timeout alone, as #9930 did for the nested For cleanup waits, will not fix these.

Where the wait runs:

• In test_processor_sqlite_flat_for_cancel_retry_isolates_execution_state, wait_for_status (default 5 s, test_for_loop_processor_sqlite.py:247) runs inside the on_after_run_node callback, on the processor thread, right after queue.cancel_queue_item(...).
• The If tests call cancel_queue_item from the callback and wait in the test thread (test_if_processor_sqlite.py:250, :326).

Where to look:

• whether cancel_queue_item from inside the runner's callback can race the runner's own status transition for that item, so that the canceled QueueItemStatusChangedEvent is never emitted or is overwritten;
• whether _RecordingRegisteredEventService can miss an event it is handed while the processor thread is inside the callback.