[davidackerman]

cellmap-analyze 0.6 hung and crashed on /nrs in salamander's and c-elegans' runs. Every run of three salamander tasks hung, and all five c-elegans fill-holes runs did. They used worldarray 0.4.0 on tensorstore 0.1.75.

Root cause. tensorstore strands a queued task in every release up to at least 0.1.85. A pool worker that has been idle for 20 ms exits, and a task queued as it exits gets no thread, because the pool still counts itself full. The future then waits forever with no tensorstore thread alive.

• Upstream fixed this in google/tensorstore ed51718085 ("Fix lost wakeups in TaskGroup when workers block or exit", 2026-09-30). No release contains it yet.
• Pools capped at 1 meet it most, and cellmap-analyze's capped_context(concurrency=1) is exactly that.
• It is most likely also tensorstore#206 (Stuart Berg, 2024), which was closed as a fork problem. cellmap-analyze's read retries from June 2025 ("it sometimes hangs") were working around it.
• 0.6.0 hung because its writes moved from zarr-python to worldarray, which had no write timeout. Its read retries crashed on the empty TimeoutError message.
• It was proven on live stuck workers: a one-voxel read on the same context released the stuck write within seconds, while un-kicked workers stayed stuck. It is not the file lock: the hang reproduced with locking off.

Changes

• Watchdog. Every context the package makes or opens a store on is held weakly by one daemon thread. Once a second, that thread gives the context's decoding pool and its file I/O pool one no-op task each: an in-memory copy and a local read of a missing key, with no store touched. A stranded task then runs within about a second, at any concurrency, whoever waits on it: worldarray, vol.ts or await. Spec open question 14 says when it goes.
• Write retries. write_index, write and write_chunk time out and retry as reads do, through a new retries= argument (Behaviour 6).
• Warning fix. The retry warning names the error's type when its message is empty; it used to raise IndexError instead of retrying.
• Thread budget. The shared context's data_copy and file_io pools take wa.thread_budget(): OMP_NUM_THREADS (dask sets 1 in each worker), else the job's slots (LSB_DJOB_NUMPROC, SLURM_CPUS_PER_TASK, NSLOTS), else the CPUs allowed. tensorstore alone used every core on the node whatever bsub -n said. Pinned to one CPU, tensorstore's default context started 43–45 pool threads; this one starts 4–5.

Reproduced on purpose. The opt-in test tests/test_lost_wakeup.py (run with WORLDARRAY_STRESS=1) aims tasks at the moment a one-thread pool's worker exits.

	tensorstore 0.1.75	0.1.78	0.1.85
Plain context: tasks stranded for good	40–43 per run	13	44
Watched context: tasks stranded	0	0	0
Watched context: slowest task	0.98 s	0.94 s	0.98 s

The end-to-end run is cellmap-analyze's block loop, a chunk read then a whole-chunk write_index, on its 0.6.1 context, with 16 workers for 90 s:

	worldarray 0.4.0	this branch
IndexErrors	29	0
Writes stuck past 20 s	55	0
Slowest operation	20.5 s	0.98 s
Read-and-write pairs completed	7,800	31,900

Speed was measured against cellmap-analyze 0.5.0's own block I/O, on NFS, with 128³ uint64 label blocks of one chunk each. From one process at one thread per pool, cellmap-analyze 0.6.1 on this branch reads a block in 26 ms instead of 40 and writes it in 45 ms instead of 62. From 8 processes it is about the same for reads and about 1.15x faster for writes. An earlier "2.3–5x" figure counted import zarr as write time and was wrong.

No placement change and no unit table change. This is a minor release: writes behave differently, and thread_budget is new public API.

https://claude.ai/code/session_0141wgF8mkGbjGfBFKfKhp64