Skip to content

Retry ambiguous task starts for active worker polls - #11734

Open
ali-khokhar-nvidia wants to merge 3 commits into
temporalio:mainfrom
ali-khokhar-nvidia:ali/recover-ambiguous-task-starts
Open

Retry ambiguous task starts for active worker polls#11734
ali-khokhar-nvidia wants to merge 3 commits into
temporalio:mainfrom
ali-khokhar-nvidia:ali/recover-ambiguous-task-starts

Conversation

@ali-khokhar-nvidia

@ali-khokhar-nvidia ali-khokhar-nvidia commented Aug 23, 2026

Copy link
Copy Markdown

What changed?

  • Keep one RecordWorkflowTaskStarted or RecordActivityTaskStarted request, including its request ID, across attempt-local deadlines while the original worker poll remains active.
  • After such a deadline makes the result ambiguous, keep a subsequent BusyWorkflow inside the same-ID recovery path. History's existing workflow-lock wait paces these attempts while the original operation finishes.
  • Return the complete embedded-activity start response when History recognizes a same-ID retry.

An initial BusyWorkflow, queue redelivery, other ResourceExhausted causes, and all definitive errors preserve their existing behavior. A different worker poll still receives a new request ID.

Why?

History can commit a task start while Matching observes its shorter child-context deadline before receiving the response. If the original operation is still finishing, the same-ID recovery attempt can first receive BusyWorkflow. Leaving that recovery path lets a later dispatch create a new request ID; History then returns TaskAlreadyStarted, and no worker receives the committed start response.

The safe recovery boundary is the existing parent worker poll. The Poll*TaskQueue handler creates the record-start request once, and the retry closure reuses that request while each attempt context remains a child of the poll context. When that poll context ends, recovery stops. A later poll enters a new handler invocation and creates a new request ID, so the recovered response can return only through the original poll and cannot be transferred to another worker. Fixes #11733.

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)
go test -tags disable_grpc_modules,test_dep ./service/matching \
  -run 'TestMatchingEngine_(Classic|Pri|Fair)_Suite/.*Record.*TaskStarted.*|TestRecordTaskStartedWithRetryStopsAtExistingBoundaries' \
  -count=1

go test -tags disable_grpc_modules,test_dep \
  ./service/history/api/recordactivitytaskstarted \
  -run 'TestRecordActivityTaskStarted_DuplicateRequest' \
  -count=1

go vet -tags disable_grpc_modules,test_dep \
  ./service/matching ./service/history/api/recordactivitytaskstarted

@ali-khokhar-nvidia
ali-khokhar-nvidia requested a review from a team August 23, 2026 00:28
@ali-khokhar-nvidia
ali-khokhar-nvidia requested review from a team as code owners August 23, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Task start can remain undelivered after an ambiguous History timeout

1 participant