MPI: spread plain @spawn tasks across ranks and threads by jpsamaroo · Pull Request #771 · JuliaParallel/Dagger.jl

jpsamaroo · 2026-10-12T18:49:06Z

Fixes #760.

Plain Dagger.@spawn tasks under MPI always selected the first compatible processor, putting independent work on rank 0's first thread. Rotate the deterministic candidate list by the uniform thunk ID so tasks spread across ranks and threads without depending on rank-local load or readiness order. For tasks with input-owner affinity, rotate within the owner's rank; scopes still restrict placement, and calls without a task retain the canonical order used by Datadeps and processor grids.

On Julia 1.12.4 with MPI.jl 0.20.27, the issue's 16-task CPU-bound reproduction changed as follows:

Configuration	Before	After	Placement after
2 ranks × 2 threads	4.77 s	1.28 s	Four tasks per processor
1 rank × 4 threads	4.74 s	1.29 s	Four tasks per thread

Added regression coverage for default placement, rank and thread scopes, Chunk/DTask owner affinity, scope overrides, and uniformity checks enabled and disabled. The tests verify actual rank/thread execution and result values as well as uniform processor metadata.

Validation:

New placement tests fail on the original code (14 failures per rank); final 2-rank × 2-thread placement checks pass (476 assertions per rank).
Existing Datadeps thread-placement checks and 1-rank × 4-thread placement checks pass.
test/allocations.jl and return-type inference checks pass; allocation bounds are unchanged.
After 10 warmups and GC.gc(), minimum GC deltas over five runs of 1,000 selections with identical four-processor input lists were unchanged without affinity: 323,000 allocations / 13,840,000 bytes. With owner affinity, they changed from 323,000 / 13,968,000 to 324,000 / 13,984,000: one additional 16-byte backend box per task, with a function barrier avoiding per-candidate boxing.
The broader MPI CPU suite exceeded its 10-minute limit while compiling high-dimensional stencil tests. No test failure was reported before the timeout; the full suite is not verified green.

Written by OpenAI Codex

github-actions · 2026-10-12T19:24:01Z

Dagger benchmarks: `dirty` vs `master`

Counts are benchmark metrics (time, allocations, bytes). Timing changes require non-overlapping median ± IQR bands; both revisions need at least five timed samples, and timing regressions need an independent confirmation run.

Backend	Regressions	Improvements	Within noise	Inconclusive time	Suites
Threads	1	3	12	1	4/4
Distributed	0	4	3	0	4/4
Distributed+Threads	2	0	0	4	4/4
MPI	0	0	0	0	4/4
MPI+Threads	1	2	16	1	4/4
Total	4	9	31	6	20/20

Regressions across all backends

Backend	Benchmark	Metric	Change
Distributed+Threads	`array/dagger/N=1024 (block 512)/alloc (rand)`	time	+152.4%
Threads	`linalg/dagger/N=1024 (block 512)/matvec (A*x)`	time	+94.5%
Distributed+Threads	`stencil/dagger/N=1024 (block 512)/neighbors (Reflect)`	time	+72.8%
MPI+Threads	`linalg/dagger/N=256 (block 256)/qr`	time	+55.2%

Improvements across all backends

Backend	Benchmark	Metric	Change
MPI+Threads	`stencil/dagger/N=256 (block 256)/assign (const)`	time	-71.1%
Threads	`array/dagger/N=256 (block 128)/add (X + X)`	time	-69.7%
Distributed	`sparse/dagger/N=256 (block 16)/cg solve (laplacian)`	memory	-57.1%
Distributed	`linalg/dagger/N=1024 (block 128)/matmul (A*A)`	time	-56.6%
Distributed	`sparse/dagger/N=256 (block 16)/cg solve (laplacian)`	allocs	-52.6%
MPI+Threads	`array/dagger/N=1024 (block 128)/norm`	time	-41.9%
Threads	`array/dagger/N=256 (block 256)/add (X + X)`	time	-37.5%
Threads	`array/dagger/N=1024 (block 128)/transpose (permutedims)`	time	-35.0%
Distributed	`array/dagger/N=1024 (block 128)/add (X + X)`	memory	-29.9%

Threads (1 process × 4 threads) — 1 regressions, 3 improvements

Suite	Regressions	Improvements	Within noise	Inconclusive time
array	0	3	4	0
linalg	1	0	3	1
sparse	0	0	0	0
stencil	0	0	5	0

Regressions

Benchmark	Metric	Change
`linalg/dagger/N=1024 (block 512)/matvec (A*x)`	time	+94.5%

Improvements

Benchmark	Metric	Change
`array/dagger/N=256 (block 128)/add (X + X)`	time	-69.7%
`array/dagger/N=256 (block 256)/add (X + X)`	time	-37.5%
`array/dagger/N=1024 (block 128)/transpose (permutedims)`	time	-35.0%

Within noise

Benchmark	Metric	Change
`stencil/dagger/N=256 (block 128)/multi-expr`	time	+109.9%
`array/dagger/N=1024 (block 512)/add (X + X)`	time	+76.3%
`stencil/dagger/N=256 (block 128)/neighbors (Clamp)`	time	+62.7%
`linalg/dagger/N=256 (block 128)/syrk (A'*A)`	time	+59.4%
`stencil/dagger/N=1024 (block 512)/update (+)`	time	+53.5%
`linalg/dagger/N=256 (block 128)/matvec (A*x)`	time	+38.2%
`array/dagger/N=256 (block 256)/broadcast (X .+ 1)`	time	+28.1%
`stencil/dagger/N=256 (block 128)/neighbors (Wrap)`	time	+25.4%
`linalg/dagger/N=256 (block 128)/cholesky`	time	-27.5%
`array/dagger/N=1024 (block 512)/reduce (sum)`	time	-32.7%
`stencil/dagger/N=1024 (block 512)/assign (const)`	time	-35.4%
`array/dagger/N=1024 (block 512)/broadcast (X .+ 1)`	time	-47.1%

Inconclusive timing changes

Benchmark	Metric	Change
`linalg/dagger/N=256 (block 256)/lu`	time	+36.1%

Distributed (4 processes × 1 thread) — 0 regressions, 4 improvements

Suite	Improvements	Within noise
array	1	1
linalg	1	2
sparse	2	0
stencil	0	0

Improvements

Benchmark	Metric	Change
`sparse/dagger/N=256 (block 16)/cg solve (laplacian)`	memory	-57.1%
`linalg/dagger/N=1024 (block 128)/matmul (A*A)`	time	-56.6%
`sparse/dagger/N=256 (block 16)/cg solve (laplacian)`	allocs	-52.6%
`array/dagger/N=1024 (block 128)/add (X + X)`	memory	-29.9%

Within noise

Benchmark	Metric	Change
`linalg/dagger/N=1024 (block 128)/solve (A\b via lu)`	time	+138.0%
`array/dagger/N=1024 (block 512)/add (X + X)`	time	+38.6%
`linalg/dagger/N=1024 (block 128)/lu`	time	-63.0%

Distributed+Threads (2 processes × 2 threads) — 2 regressions, 0 improvements

Suite	Regressions	Inconclusive time
array	1	1
linalg	0	0
sparse	0	0
stencil	1	3

Regressions

Benchmark	Metric	Change
`array/dagger/N=1024 (block 512)/alloc (rand)`	time	+152.4%
`stencil/dagger/N=1024 (block 512)/neighbors (Reflect)`	time	+72.8%

Inconclusive timing changes

Benchmark	Metric	Change
`stencil/dagger/N=1024 (block 512)/neighbors (Wrap)`	time	+87.5%
`stencil/dagger/N=1024 (block 512)/neighbors (Clamp)`	time	+85.5%
`stencil/dagger/N=1024 (block 512)/neighbors (Pad)`	time	+77.3%
`array/dagger/N=1024 (block 512)/transpose (permutedims)`	time	+42.3%

MPI (4 ranks × 1 thread) — 0 regressions, 0 improvements

Suite	Regressions	Improvements	Within noise	Inconclusive time
array	0	0	0	0
linalg	0	0	0	0
sparse	0	0	0	0
stencil	0	0	0	0

MPI+Threads (2 ranks × 2 threads) — 1 regressions, 2 improvements

Suite	Regressions	Improvements	Within noise	Inconclusive time
array	0	1	8	0
linalg	1	0	4	1
sparse	0	0	0	0
stencil	0	1	4	0

Regressions

Benchmark	Metric	Change
`linalg/dagger/N=256 (block 256)/qr`	time	+55.2%

Improvements

Benchmark	Metric	Change
`stencil/dagger/N=256 (block 256)/assign (const)`	time	-71.1%
`array/dagger/N=1024 (block 128)/norm`	time	-41.9%

Within noise

Benchmark	Metric	Change
`stencil/dagger/N=256 (block 128)/multi-expr`	time	+156.8%
`array/dagger/N=256 (block 128)/norm`	time	+137.8%
`array/dagger/N=256 (block 128)/add (X + X)`	time	+103.9%
`linalg/dagger/N=256 (block 256)/syrk (A'*A)`	time	+99.3%
`array/dagger/N=1024 (block 512)/alloc (rand)`	time	+97.9%
`array/dagger/N=256 (block 128)/transpose (permutedims)`	time	+89.9%
`array/dagger/N=1024 (block 512)/norm`	time	+70.4%
`linalg/dagger/N=256 (block 128)/cholesky`	time	+69.6%
`array/dagger/N=1024 (block 512)/map (sin.(X))`	time	+36.7%
`linalg/dagger/N=256 (block 128)/solve (A\b via lu)`	time	+35.2%
`array/dagger/N=256 (block 256)/add (X + X)`	time	-36.9%
`array/dagger/N=256 (block 256)/map (sin.(X))`	time	-39.2%
`stencil/dagger/N=1024 (block 512)/multi-expr`	time	-43.7%
`stencil/dagger/N=256 (block 128)/update (+)`	time	-46.1%
`stencil/dagger/N=1024 (block 512)/update (+)`	time	-49.4%
`linalg/dagger/N=1024 (block 512)/matmul (A*A)`	time	-52.4%

Inconclusive timing changes

Benchmark	Metric	Change
`linalg/dagger/N=1024 (block 512)/cholesky`	time	+139.8%

Full results and plots (download the benchmark-results-* artifacts).

Uniform placement ignored local load and chose the same first processor for every plain spawn. Rotate the canonical candidates by the uniform thunk ID so placement is independent of each rank's readiness order. Keep input-owner affinity within its preferred rank, honor restricted scopes, and retain the canonical listing for Datadeps/proc-grid calls without a task. Reproduced #760 on Julia 1.12.4 / MPI 0.20.27: 16 CPU-bound tasks on 2 ranks x 2 threads went from 16 serial tasks on rank 0/thread 1 in 4.77s to four tasks on each processor in 1.28s. One rank x four threads likewise went from 4.74s to 1.29s, using all four threads. GC deltas after 10 warmups + GC.gc(), minima over five runs of 1,000 selections with identical reversed input lists (four processors): - No affinity: 323,000 allocations / 13,840,000 bytes before and after. - Owner affinity: 323,000 / 13,968,000 before; 324,000 / 13,984,000 after. The affinity scan adds one 16-byte backend box per task; an out-of-line pass-through barrier avoids boxing it separately for each candidate. Validation: new placement tests fail on the original code (14 failures per rank); final 2x2 placement tests pass (476 assertions per rank), as do Datadeps thread placement, 1x4 placement, return-type inference checks, and test/allocations.jl without changing bounds. The broader MPI CPU suite exceeded its 10-minute limit compiling high-dimensional stencils; no test failure was reported before the timeout. Record placement, boxing, and MPI timeout cleanup lessons in AGENTS.md.

jpsamaroo added bug performance mpi scheduler labels Oct 12, 2026

jpsamaroo mentioned this pull request Oct 12, 2026

Fix GPU CI startup and move CPU jobs to GitHub Actions #772

Merged

jpsamaroo force-pushed the fix/mpi-eager-placement branch from b3d7407 to 70067db Compare October 12, 2026 20:33

jpsamaroo merged commit 43e9906 into master Oct 12, 2026
35 of 54 checks passed

jpsamaroo deleted the fix/mpi-eager-placement branch October 12, 2026 20:36

jpsamaroo mentioned this pull request Oct 12, 2026

MPI: plain Dagger.@spawn tasks all run on rank 0, thread 1 #760

Closed

Navigation Menu

Repository navigation

MPI: spread plain @spawn tasks across ranks and threads - #771

MPI: spread plain @spawn tasks across ranks and threads#771
jpsamaroo merged 1 commit into
masterfrom
fix/mpi-eager-placement

jpsamaroo commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Regressions

Improvements

Within noise

Inconclusive timing changes

Improvements

Within noise

Regressions

Inconclusive timing changes

Regressions

Improvements

Within noise

Inconclusive timing changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jpsamaroo commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dagger benchmarks: dirty vs master

Regressions across all backends

Improvements across all backends

Regressions

Improvements

Within noise

Inconclusive timing changes

Improvements

Within noise

Regressions

Inconclusive timing changes

Regressions

Improvements

Within noise

Inconclusive timing changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jpsamaroo commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 12, 2026 •

edited

Loading

Dagger benchmarks: `dirty` vs `master`