MPP task can remain alive after cancel and abort return

[King-Dylan]

Problem

An MPP task can remain alive for hours even after TiFlash receives cancel and MPPTask::abort() returns. Repeated queries then keep adding retained tasks until the TiFlash process is restarted.

Incident trigger and impact

On 2026-08-26, all three TiFlash compute instances entered RSS memory protection: RSS was about 47 GiB, above the 45.61 GiB data-compute limit. Two instances recovered, but tikc2v continued accumulating MPP tasks.

Its last long-lived-task monitor entry at 15:07:05 PDT contained:

• 5,794 live tasks across 3,286 unique queries;
• task ID 3: 2,945 entries;
• task ID 6: 2,158 entries;
• 5,094 tasks aged at least 10 hours.

The 1,000 rows returned by several memory-protection searches were the Loki UI limit, not a query count.

Confirmed retained-task lifecycle

Representative task:

• query_ts=1787741933891977310
• local_query_id=13747
• start_ts=468645821513728101
• task ID: 3
• instance: db-13dd8e6a-compute-tiflash-tikc2v
• SQL digest: 2e9084c507b31179f6f27687f0c04726a6cb05a979d963d8dd22c2c60c0c3752
• plan digest: a6820fa8b4e72e1dd94ead2567b8779d5b82db50276568a351ca594fd8bab4eb
• stage: Columnar table scan -> projection -> hash aggregation -> exchange sender

Observed timeline (PDT):

Time	Event
03:58:54.590	TiFlash receives the TiDB cancel request.
03:58:54.591	Begin abort task and Finish abort task from running are logged.
same lifecycle	Tunnel 3+4 establish completes; total cost is 608 ms.
afterward	No mpp task finish execute, task statistics/end timestamp, destructor, or unregister is found.
15:07:05	TMTContext still reports the exact task alive for 40,091.56 seconds.

This sample proves that cancel was delivered and the synchronous abort() path returned, but the task did not finish execution and unregister.

The older IDs previously listed as dispatch-only orphans are not evidence for this issue: full-namespace searches show establish, FINISHED, and nonzero task-end timestamps for them. Also, the absence of an establish RPC for every non-root task is expected and is not an orphan signal.

Source assessment

Confirmed deployed build:

• TiFlash version: v26.3.10
• TiFlash commit: d89f347e5331ad47abe277764f67ff21f6a6976c
• CSE commit: 219b17ddf78dae2b9ae41afeca38dd33f08b9c0c

The cancellation chain is:

MPPTask::abort() -> QueryExecutor::cancel() -> PipelineExecutorContext::cancel()

Relevant behavior at the deployed TiFlash commit:

• MPPTask::abort() requests executor cancellation and explicitly leaves the remaining work to the running runImpl().
• MPPTask::runImpl() can record task end and unregister only after query_executor_holder->execute() returns.
• Production PipelineExecutor::wait() has no timeout.
• PipelineExecutorContext::wait() waits for active_ref_count == 0; cancel() marks the context cancelled and asks the scheduler to cancel the query.
• Scheduler cancellation moves queued tasks to cancellation queues, but cannot preempt a Task already owned by a CPU/IO worker. A Task decrements active_ref_count only when its destructor runs.

The representative task is a Columnar scan stage. The deployed Columnar path contains several non-cancel-aware blocking sections:

• buildColumnarReadTaskWithBackoff() and createColumnarReaderWithBackoff() retry without a query-cancel check;
• getOrCreateReader() waits on RNColumnarReaderWork::cv without cancellation in the predicate;
• RNColumnarSourceOp::executeIOImpl() enters current_input_stream->read() after only an outer entry check;
• RNColumnarInputStream::readImpl() calls the CSE FFI fn_read_block() without a cancellation token/check inside the call.

The deployed and current master blobs for StorageDisaggregatedColumnar.cpp are identical as of 2026-08-26, so no upstream fix was found in that path.

Working failure mechanism

Cancel reaches TiFlash -> synchronous abort returns -> an already-running pipeline/preprocess operation does not observe cancellation or return -> its Task remains alive and active_ref_count stays nonzero -> PipelineExecutorContext::wait() never returns -> MPPTask::runImpl() never records task end, unregisters, or destructs the MPP task.

Proof boundary

Confirmed:

• The retained entries are real MPPTask objects in TiFlash's in-process monitor.
• Cancel reached TiFlash for the representative task.
• The synchronous abort path returned, but execution completion/unregister did not occur.
• The source allows an already-running Task to keep active_ref_count nonzero after cancel.
• The Columnar path has concrete cancellation gaps capable of producing this state.

Not confirmed because native stacks were not captured before restart:

• The exact blocked frame for task 3: preprocess/backoff, reader materialization wait, fn_read_block, deserialization, tunnel send, or another operator.
• Whether all 5,794 retained tasks share the same blocked primitive; the large task-6 population suggests receiver/intermediate stages also need inspection.

Requested change

• Make Columnar build/materialization/read waits cancellation-aware, including backoff loops, RNColumnarReaderWork waits, and the CSE FFI read boundary.
• When executor drain after cancel exceeds a bounded interval, log active_ref_count and the active task/status/operator information; provide an on-demand stack/task dump for diagnosis.
• Add a regression test that cancels while a task is already executing/blocking in Columnar IO and asserts executor drain, task-end statistics, unregister, and TMT monitor removal.
• Evaluate a safe bounded cleanup/watchdog path for cancelled tasks whose executor does not drain. Do not reclaim an MPPTask while a worker can still reference it.
• Improve TiDB/TiFlash per-address cancel-result logging and idempotent retry as defense in depth.

Related TiDB hardening issue: pingcap/tidb#70685. TiDB cancellation reliability does not fix this confirmed cancel-delivered TiFlash stall.

The restart-time HandleHttpRequestReadyz SIGSEGV is a separate recovery-stage defect and is intentionally excluded from this MPP retention RCA.

[King-Dylan]

Evidence correction: the five previously listed no-establish/no-cancel samples completed normally when searched across the full namespace. The issue body now uses two validated queries whose tasks remained alive on tikc2v after the TiDB-side query was cancelled.

[King-Dylan]

Evidence update: query_ts 1787741933891977310 task 3 received cancel and completed MPPTask::abort logging at 03:58:54, but was still reported alive 40,091.56 seconds later. The body now focuses on the executor/runImpl cancellation gap and records the separate restart-time readiness SIGSEGV.