Expose independent value producers in hierarchical Datadeps planning by jpsamaroo · Pull Request #773 · JuliaParallel/Dagger.jl

jpsamaroo · 2026-10-12T23:16:59Z

Hierarchical Datadeps currently walks a reduction tree depth-first. Planning a result consumer fetches its producers, so that fetch can stall submission before independent branches have been launched. This affects both scalar reductions and the recursively emitted collect(::DArray) concatenation tree.

Use a deterministic FIFO topological frontier for regions with value dependencies, exposing independent producers before newly unlocked consumers. Existing runtime dependencies still enforce completion before consumption. Buffer-only regions retain DFS.

Preserve possible RAW/WAR/WAW conflicts for unresolved mutable results, using only an isbits return type as proof that a result cannot alias storage. Add same-region explicit synchronization dependencies to the planning DAG, since their predecessors must already exist when the eager scheduler resolves their IDs. Cyclic plans now fail rather than falling back to unchecked submission order. Indegrees and dependency bookkeeping reuse typed scratch.

The PR adds regressions for independent and recursively emitted branches, mutable/unknown result aliases, explicit synchronization, and cycles, plus a standalone MPI benchmark harness. The MPI Cholesky smoke test now gathers its tiled parent before wrapping it in UpperTriangular, avoiding per-element remote fetches while preserving the numerical check.

Local performance

Julia 1.12.4, two local MPI ranks with two threads each, uniformity checks disabled; medians of seven samples after ten warmups. Baseline: 02381759.

Workload	Before	After	Speedup
Compute-heavy reduction tree	355.59 ms	163.09 ms	2.18×
Tiny-task reduction tree	25.99 ms	14.96 ms	1.74×
512² tiled gather	31.90 ms	24.71 ms	1.29×

The buffer-only control was 8.24 → 8.94 ms in the final comparison; repeated runs showed no consistent improvement. Planned copy counts stayed unchanged. On a 127-vertex tree, the ordering step fell from 13 allocations / 5,352 bytes to 2 / 1,120. Whole-region scalar-tree bytes remained within 1% of baseline; allocation-suite bounds are unchanged.

Reproduce with the same script on both revisions:

julia --project=/path/to/env test/mpi/run.jl 2 2 benchmark/launch_order.jl

Validation

Full Distributed Datadeps: 2,166 passing assertions.
Allocation suite: 24 passing assertions, unchanged bounds.
Full MPI CPU: 656 passing assertions per rank.
Focused launch-order regressions: 27 passing assertions, including one-rank/one-thread coverage. Rerun on the rebased PR head with two ranks × two threads: 27 assertions per rank plus a tiled-gather check.
Compatibility checked separately against aot-schedulers-rebased at 04a8ec2, including EFT launch positions and Greedy/IG/SA replay. This PR targets master and contains no AOT scheduling changes.

Addresses #759.

Written by OpenAI Codex

DFS visits a reduction's consumers before the remaining independent leaves have been submitted. A deterministic FIFO topological frontier exposes those leaves first, including recursively emitted DArray gather trees. Keep DFS for buffer-only regions and retain runtime result fetching, alias analysis, and batch flushes. Unresolved mutable results can alias earlier buffers, so preserve possible RAW/WAR/WAW conflicts until they are resolved. Only isbits result metadata proves no mutable alias; Any remains conservative. Explicit syncdep IDs also constrain submission and must be in the planning DAG. Reuse typed scratch for indegrees, unknown accesses, and syncdep UID lookup. Memory: after 10 warmups and GC.gc(), minimum of 7 Base.gc_num() deltas on a 127-vertex tree: DFS 13 allocations / 5,352 bytes; FIFO 2 / 1,120. In the quiet 2-rank x 2-thread tree benchmark, baseline minima were 11,271/11,273 allocations and 456,160/456,208 bytes per rank; final FIFO minima were 11,197/11,202 and 457,376/459,024 (less than 1% byte overhead). Allocation-suite bounds remain unchanged. Validated full Distributed Datadeps (2,166 assertions), allocations (24), and MPI CPU (656 per rank), plus 27 focused checks on 1 rank x 1 thread. The AOT-compatible port is checked at aot-schedulers-rebased 04a8ec2, including release-position gaps and Greedy/IG/SA replay consistency.

Generic collect of an UpperTriangular DArray scalar-indexes the distributed storage, making the production-mode smoke spend minutes fetching individual elements. Collect the parent through its concatenation tree and re-wrap it, preserving the numerical assertion. The complete MPI CPU suite passes 656 assertions per rank.

Provide one shared script for scalar reduction trees, tiny-task trees, tiled DArray gather, and a buffer-only control. Await fixtures, warm ten iterations, and collect seven samples with cooperative barriers and GC outside timing. Report maximum wall time across ranks and minimum GC counts/bytes separately per rank; optional untimed logging counts moves. Reject invalid case selections rather than emitting an empty comparison. Julia 1.12.4, two local MPI ranks with two threads each, uniformity checks off, baseline 0238175 versus the final implementation: scalar_tree: 355.591 -> 163.087 ms median (2.18x) fast_tree: 25.992 -> 14.964 ms median (1.74x) collect: 31.903 -> 24.714 ms median (1.29x) buffer_region: 8.243 -> 8.941 ms median (0.92x) Repeated buffer-control runs show no consistent improvement. Planned moves remain unchanged: zero for both trees and buffer control, 31 for the gather. Scalar-tree minimum allocations/bytes per rank: baseline: 11271 / 456160; 11273 / 456208 final: 11197 / 457376; 11202 / 459024 The traversal improves time without claiming lower whole-region bytes. Reproduce using the same script and Julia environment on both revisions: julia --project=/path/to/env test/mpi/run.jl 2 2 benchmark/launch_order.jl The full script ran on both revisions; final selector guard syntax checked.

github-actions · 2026-10-12T23:43:34Z

Dagger benchmarks: `dirty` vs `master`

Counts are benchmark metrics (time, allocations, bytes). Timing changes require non-overlapping median ± IQR bands; both revisions need at least five timed samples, and timing regressions need an independent confirmation run.

Backend	Improvements	Within noise	Inconclusive time	Suites
Threads	2	7	0	4/4
Distributed	3	2	1	4/4
Distributed+Threads	0	2	0	4/4
MPI	0	0	0	4/4
MPI+Threads	3	14	0	4/4
Total	8	25	1	20/20

Regressions across all backends

None in the available results.

Improvements across all backends

Backend	Benchmark	Metric	Change
MPI+Threads	`stencil/dagger/N=256 (block 128)/update (+)`	time	-68.8%
Threads	`linalg/dagger/N=1024 (block 512)/matvec (A*x)`	time	-53.4%
MPI+Threads	`array/dagger/N=256 (block 128)/alloc (rand)`	time	-48.0%
Distributed	`linalg/dagger/N=1024 (block 128)/lu`	memory	-37.2%
MPI+Threads	`linalg/dagger/N=256 (block 256)/syrk (A'*A)`	time	-37.1%
Distributed	`linalg/dagger/N=1024 (block 128)/lu`	allocs	-36.6%
Distributed	`array/dagger/N=1024 (block 128)/add (X + X)`	memory	-33.5%
Threads	`stencil/dagger/N=256 (block 128)/neighbors (Pad)`	time	-28.1%

Threads (1 process × 4 threads) — 0 regressions, 2 improvements

Suite	Improvements	Within noise
array	0	3
linalg	1	4
sparse	0	0
stencil	1	0

Improvements

Benchmark	Metric	Change
`linalg/dagger/N=1024 (block 512)/matvec (A*x)`	time	-53.4%
`stencil/dagger/N=256 (block 128)/neighbors (Pad)`	time	-28.1%

Within noise

Benchmark	Metric	Change
`array/dagger/N=256 (block 128)/reduce (sum)`	time	+85.0%
`linalg/dagger/N=256 (block 128)/matvec (A*x)`	time	+53.2%
`array/dagger/N=1024 (block 128)/broadcast (X .+ 1)`	time	+29.8%
`linalg/dagger/N=256 (block 256)/solve (A\b via lu)`	time	+26.3%
`linalg/dagger/N=256 (block 128)/syrk (A'*A)`	time	-27.6%
`array/dagger/N=256 (block 128)/norm`	time	-38.2%
`linalg/dagger/N=256 (block 128)/cholesky`	time	-40.6%

Distributed (4 processes × 1 thread) — 0 regressions, 3 improvements

Suite	Improvements	Within noise	Inconclusive time
array	1	0	1
linalg	2	2	0
sparse	0	0	0
stencil	0	0	0

Improvements

Benchmark	Metric	Change
`linalg/dagger/N=1024 (block 128)/lu`	memory	-37.2%
`linalg/dagger/N=1024 (block 128)/lu`	allocs	-36.6%
`array/dagger/N=1024 (block 128)/add (X + X)`	memory	-33.5%

Within noise

Benchmark	Metric	Change
`linalg/dagger/N=1024 (block 128)/qr`	time	-56.5%
`linalg/dagger/N=1024 (block 128)/lu`	time	-58.6%

Inconclusive timing changes

Benchmark	Metric	Change
`array/dagger/N=1024 (block 512)/alloc (rand)`	time	+61.3%

Distributed+Threads (2 processes × 2 threads) — 0 regressions, 0 improvements

Suite	Regressions	Improvements	Within noise	Inconclusive time
array	0	0	0	0
linalg	0	0	0	0
sparse	0	0	0	0
stencil	0	0	2	0

Within noise

Benchmark	Metric	Change
`stencil/dagger/N=1024 (block 128)/assign (const)`	time	+56.0%
`stencil/dagger/N=1024 (block 128)/neighbors (Wrap)`	time	+39.7%

MPI (4 ranks × 1 thread) — 0 regressions, 0 improvements

Suite	Regressions	Improvements	Within noise	Inconclusive time
array	0	0	0	0
linalg	0	0	0	0
sparse	0	0	0	0
stencil	0	0	0	0

MPI+Threads (2 ranks × 2 threads) — 0 regressions, 3 improvements

Suite	Improvements	Within noise
array	1	5
linalg	1	2
sparse	0	0
stencil	1	7

Improvements

Benchmark	Metric	Change
`stencil/dagger/N=256 (block 128)/update (+)`	time	-68.8%
`array/dagger/N=256 (block 128)/alloc (rand)`	time	-48.0%
`linalg/dagger/N=256 (block 256)/syrk (A'*A)`	time	-37.1%

Within noise

Benchmark	Metric	Change
`array/dagger/N=1024 (block 512)/reduce (sum)`	time	+287.0%
`stencil/dagger/N=256 (block 256)/assign (const)`	time	+95.3%
`stencil/dagger/N=256 (block 128)/assign (const)`	time	+65.1%
`stencil/dagger/N=256 (block 128)/neighbors (Clamp)`	time	+56.6%
`array/dagger/N=1024 (block 128)/alloc (rand)`	time	+52.2%
`linalg/dagger/N=256 (block 128)/matvec (A*x)`	time	+48.6%
`stencil/dagger/N=256 (block 128)/neighbors (Pad)`	time	+39.3%
`array/dagger/N=256 (block 128)/norm`	time	+37.5%
`array/dagger/N=256 (block 128)/broadcast (X .+ 1)`	time	-35.5%
`linalg/dagger/N=256 (block 256)/cholesky`	time	-38.7%
`stencil/dagger/N=256 (block 256)/neighbors (Clamp)`	time	-41.7%
`stencil/dagger/N=256 (block 256)/update (+)`	time	-52.6%
`array/dagger/N=256 (block 128)/map (sin.(X))`	time	-65.9%
`stencil/dagger/N=256 (block 128)/multi-expr`	time	-71.0%

Full results and plots (download the benchmark-results-* artifacts).

jpsamaroo added 4 commits October 12, 2026 16:15

Document dependency constraints behind Datadeps tree serialization

35b1aaa

jpsamaroo mentioned this pull request Oct 13, 2026

MPI: reduction over task results limits datadeps concurrency to 2 tasks in total (hierarchical planner) #759

Open

Navigation Menu

Repository navigation

Expose independent value producers in hierarchical Datadeps planning - #773

Expose independent value producers in hierarchical Datadeps planning#773
jpsamaroo wants to merge 4 commits into
masterfrom
fix/datadeps-value-launch-order

jpsamaroo commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 12, 2026

Improvements

Within noise

Improvements

Within noise

Inconclusive timing changes

Within noise

Improvements

Within noise

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jpsamaroo commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Local performance

Validation

Uh oh!

github-actions Bot commented Oct 12, 2026

Dagger benchmarks: dirty vs master

Regressions across all backends

Improvements across all backends

Improvements

Within noise

Improvements

Within noise

Inconclusive timing changes

Within noise

Improvements

Within noise

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jpsamaroo commented Oct 12, 2026 •

edited

Loading

Dagger benchmarks: `dirty` vs `master`