Repository navigation
MPI: plain Dagger.@spawn tasks all run on rank 0, thread 1 #760
Description
Related to #757 and #759. With accelerate!(:mpi), tasks spawned with plain Dagger.@spawn (outside spawn_datadeps) all seem to run on rank 0, thread 1, one after the other, also with several ranks and threads:
using Dagger, MPI
Dagger.accelerate!(:mpi)
work(n) = (s = 0.0; for i in 1:n; x = (i - 0.5) / n; s += 4.0 / (1.0 + x * x); end; s / n)
ts = [Dagger.@spawn work(1 << 27) for _ in 1:16]
foreach(fetch, ts)| time | where the 16 tasks ran | |
|---|---|---|
| MPI, 1 rank × 4 threads | 1.12 s | all on (MPI: 0, W: 1, TID: 1), one at a time |
| MPI, 2 ranks × 2 threads | 1.16 s | all on (MPI: 0, W: 1, TID: 1), rank 1 idle |
MPI, 2 ranks × 2 threads, scope=Dagger.scope(mpi_ranks=:) |
1.16 s | same |
| plain Dagger (no MPI), 4 threads | 0.37 s | TID 1–4, four at a time |
Tested with Dagger 0.22.6, MPI.jl 0.20.27 (MPICH_jll), Julia 1.13.0 on macOS.
We guess this is the scheduler's rule for uniform execution (in schedule_one!, "every rank must take the first usable processor in the deterministic order"), which #757 now bypasses for spawn_datadeps, but not for plain @spawn.
We saw that the docs recommend datadeps over @spawn/fetch chains under MPI. Still, plain @spawn is a common pattern, so maybe it could be spread across ranks and threads too, or the docs could mention that it currently runs serially?
Thanks a lot for looking into this!
Full script with timing
using Dagger, MPI
const USE_MPI = get(ENV, "NO_MPI", "0") != "1"
USE_MPI ? Dagger.accelerate!(:mpi) : MPI.Init()
const rank = MPI.Comm_rank(MPI.COMM_WORLD)
const LOG = Tuple{Float64,Float64,String}[]; const LOG_LOCK = ReentrantLock()
function work(n)
t1 = time()
s = 0.0; for i in 1:n; x = (i - 0.5) / n; s += 4.0 / (1.0 + x * x); end
p = Dagger.short_name(Dagger.task_processor())
lock(() -> push!(LOG, (t1, time(), p)), LOG_LOCK)
s / n
end
function max_concurrent(iv)
ev = sort!(vcat([(s, 1) for (s, _, _) in iv], [(e, -1) for (_, e, _) in iv])); c = m = 0
for (_, d) in ev; c += d; m = max(m, c); end; m
end
for rep in 1:2
empty!(LOG)
t = @elapsed begin
ts = [Dagger.@spawn work(1 << 27) for _ in 1:16]
foreach(fetch, ts)
end
if rep == 2
USE_MPI && MPI.Barrier(MPI.COMM_WORLD)
procs = Dict{String,Int}(); foreach(l -> procs[l[3]] = get(procs, l[3], 0) + 1, LOG)
print("rank $rank: $(round(t; digits=3)) s, $(length(LOG)) tasks ran here, max $(max_concurrent(LOG)) at a time, procs $(sort(collect(procs)))\n")
end
endmpiexec -n 2 julia --project=. --threads=2 script.jl
NO_MPI=1 julia --project=. --threads=4 script.jlOutput (MPI, 2 ranks × 2 threads):
rank 0: 1.162 s, 16 tasks ran here, max 1 at a time, procs ["(MPI: 0, W: 1, TID: 1)" => 16]
rank 1: 1.162 s, 0 tasks ran here, max 0 at a time, procs Pair{String, Int64}[]
Activity
AnnaLena77 commentedon Oct 10, 2026
Related consequence we measured (Dagger 0.22.6, MPI backend): since plain @Spawn doesn't spread, we pin tasks with ExactScope. Per-rank service tasks go to the rank's first processor, and long compute-bound tasks (seconds, no yield) go to every thread. When one of those lands on the first thread, the service tasks, and the other ranks waiting for them, queue behind it.
4 ranks × 16 threads on one node, same code:
all 16 threads run compute tasks: steps alternate between 7.3 s and 13.4 s
first thread left free: steady 7.5 s
So under MPI one thread per rank has to be reserved. Spreading per-rank tasks over the rank's threads would avoid that.
Spread MPI eager tasks without rank-local scheduling state
jpsamaroo commentedon Oct 12, 2026
Thanks for the report! #771 should temporarily resolve this issue until we figure out how to relax certain uniformity constraints that the MPI backend imposes on us.