multiprocessing.Pool terminate shuts down workers too early #96995

Activity

1. mtrofin

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 21, 2022
2. AlexWaygood

   added

   topic-multiprocessing

   on Sep 21, 2022
3. AlexWaygood

   added this to Multiprocessing issueson Sep 21, 2022
4. lorenzph commented on Nov 7, 2024

   

   lorenzph

   on Nov 7, 2024

   More actions

   I have experienced the same issue and the following snippet triggers this deadlock quite reliably on my machine (Python 3.12.7 on Linux):

   import multiprocessing
   import os
   import faulthandler
   
   def func(value):
       if value >= 32:
           raise Exception
       return None
   
   def run():
       print("Start")
       faulthandler.dump_traceback_later(10)
       try:
           with multiprocessing.Pool(processes=16) as pool:
               result = list(pool.imap(func, range(0, 100)))
       except Exception as e:
           pass
       print("Done")
   
   while True:
       print("Next run")
       run()
   

   Switching imap to map mitigates the issue as in this particular case map doesn't seem to raise the exception until the workers are finished (hence avoiding the issue that the lock is held by one when the pool is torn down).
5. ondrejch commented on Oct 31, 2026

   

   ondrejch

   on Oct 31, 2026

   More actions

   We hit what looks like a second mechanism behind this hang, in a batch job on Linux x86-64 workers (Python 3.12.15, also reproduced on 3.12.3). The reordering proposed here is not sufficient alone.

   Mechanism. Connection._send_bytes sends a message larger than 16384 bytes as two writes, a 4-byte header and then the payload (connection.py, 3.12: lines 417-421). SimpleQueue.put holds outqueue._wlock across both. If terminate() SIGTERMs a worker between the two writes, the worker dies holding _wlock and leaves a truncated message in the result pipe. The parent keeps the write end open, so there is no EOF:

   • the result handler waits in its exit drain for the rest of the payload: pool.py:626 get() -> connection.py:250 recv -> 437 _recv_bytes -> 395 _recv (in the field case, read(fd, 19770) with 0 bytes unread);
   • the task handler, if it has not yet sent its sentinel, blocks at pool.py:562 outqueue.put(None) -> queues.py:398 with self._wlock;
   • the caller of terminate() waits in _terminate_pool at pool.py:720 (task_handler.join()), or at 724 (result_handler.join()) if the task handler had already exited.

   The frames are identical on 3.12.3 and 3.12.15. The split send and SimpleQueue.put are unchanged in 3.13, 3.14, 3.15 and main; if n > 16384 is at connection.py line 417 on 3.12, 422 on 3.13 and 439 on 3.14, 3.15 and main. Those pool.py lines stay at 562, 626, 703, 720 and 724 through 3.15, and _terminate_pool is unchanged through 3.15. On main the sentinel put is at 728 and the joins at 745 and 749, and _terminate_pool also releases the task-queue buffer semaphores after task_handler._state = TERMINATE; that does not release _wlock. The same truncation is the subject of the open gh-159689 and the open, unmerged PR #159697 (checked 2026-10-31). That PR teaches the ProcessPoolExecutor result read to watch worker sentinels. Pool is unchanged there, and a recv() without sentinels still blocks.

   Reordering alone. Separately, a hang with no kill at all has the caller of terminate() at pool.py:703 (outqueue.put(None) -> queues.py:398, with self._wlock). One trial of a longer, instrumented version of the script below (8 workers, 64 tasks, and a 300 ms pause after the last input item) was snapshotted on unpatched 3.12.3 in that state: the task handler at pool.py:562 -> queues.py:398, the workers alive, 3687 of the 4096 bytes of the result pipe unread, and the _wlock semaphore value 0. The diff in this issue does not move line 703. It lies above the worker terminate() loop (lines 712-716) and the joins (720, 724), so a live holder of _wlock stops _terminate_pool there whether or not the loop is moved. I did not apply the diff and rerun; this follows from the unpatched snapshot and the line order.

   Reproducer (stdlib only, Linux). It widens the window with two tricks: the result pipe is shrunk to 4 KiB with F_SETPIPE_SZ, and unpickling a result in the parent sleeps 20 ms. Neither patches multiprocessing.

   import faulthandler, fcntl, multiprocessing as mp, os, sys, threading, time
   
   def rebuild(payload, delay):
       time.sleep(delay)
       return payload
   
   class Slow:
       def __init__(self, payload): self.payload = payload
       def __reduce__(self): return (rebuild, (self.payload, 0.02))
   
   def work(_):
       time.sleep(0.03)
       return Slow(b"x" * 20000)       # pickled size > 16384
   
   if __name__ == "__main__":
       ctx = mp.get_context("spawn")
       for trial in range(1, 31):
           pool = ctx.Pool(4)
           fcntl.fcntl(pool._outqueue._reader.fileno(), fcntl.F_SETPIPE_SZ, 4096)
           it = pool.imap_unordered(work, range(32))
           for _ in range(4):
               next(it)
           t = threading.Thread(target=pool.terminate, daemon=True)
           t.start(); t.join(5)
           if t.is_alive():
               print("HANG in trial", trial, file=sys.stderr)
               faulthandler.dump_traceback()
               os._exit(3)
           pool.join()
       print("no hang in 30 trials")

   On one host (3.12.3) this exits 3 with the stacks above in the first 30 trials in 3 of 3 invocations (trials 6, 5 and 8). A longer, instrumented version of the same script hung 3 times in 22 trials across four invocations (the last invocation stopped after 2 trials without a hang); the task and result handlers were stuck as above in one, the result handler only in two. These numbers are from one machine.

   This looks like the same root cause as gh-79448 (the _wlock held by a killed worker) and gh-78178 (the result handler stuck in the drain). A fix probably has to stop the workers cooperatively, or make the two-part send safe against a kill, rather than only reorder the joins.
6. picnixz

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Nov 1, 2026