Repository navigation
Conversation
MultiEventLog aggregators run at collection, so emitting one task never started the renderer and curl failed to connect. Enable the logging runtime, collect the batch, and assert the actual HTTP status and HTML response. Keep the five-second window semantics and disable logging afterward. Validation: DaggerWebDash Basics passes all four assertions on Julia 1.12.4 with two threads.
LogWindow used the first retained index as a deletion count, dropping an additional valid event from mixed-age batches. Remove only the preceding entries and preserve alignment across consumer columns. Validation: all 77 TimespanLogging assertions pass on Julia 1.12.4 with four threads, including mixed, entirely retained, and entirely expired batches.
Use DistributedNext for the Julia 1.10/1.11 multiprocess CI jobs, where stdlib Distributed raises lock-must-be-held concurrency errors. Configure all three transport preferences before imports, use the same transport for test macros and worker management, and preserve it after preference tests. Workers created inside scheduler/processor tests now inherit the active test project. Validation: Julia 1.10.12, two driver threads and three workers with two threads each: thunk, scheduler, processors, options, task queues, datadeps (2143 passed), reuse (12615 passed), and preferences all pass. Existing broken tests remain reported as broken. Newer Julia jobs retain stdlib transport coverage.
LinearAlgebra now passes a Factorization to _zeros instead of its column count. Forward that signature to Dagger existing allocator so distributed RHS solves keep their storage and dispatch. Cover vectors and matrices, numerical results, RHS preservation, and allocation via the new protocol on every supported Julia. Validation: on Julia 1.14.0-DEV.3709 the Cholesky reproducer changes from Vector to DVector; all 25 allocation-suite checks pass with existing bounds. Python integration also passes all six checks on this nightly; the independent Pkg CommandSpec precompile failure is fixed upstream in JuliaLang/Pkg.jl#4863.
NoPivot allocated an automatically partitioned identity pivot vector, while row-swap kernels assume its blocks match the factors. This corrupted solves depending on processor count. Rectangular tiles also left pivoted LU pivots in the temporary square-panel layout. Allocate or repartition pivots to the returned factors row blocking. Validation: Julia 1.11.9 with four threads passes 136 LU and 44 solve assertions, including real/complex, square/rectangular tile layouts, partial edge tiles, and vector/matrix RHS. A deterministic old-layout reproducer has solution error 0.5249 versus 2.09e-16 after alignment. Memory: Julia 1.11.9, identical dependencies, pinned 16x16 fixture with 8x8 tiles, 10 warmups then GC and minima over five runs. NoPivot: 9464 allocations / 441440 bytes before, 9663 / 448496 after (two required pivot tiles instead of the incorrect single automatic tile). RowMaximum: 10937 / 503104 before, 10985 / 504384 after. Existing allocation bounds are unchanged.
Sparse inputs still used arbitrary placement, allowing driver-local allocation reports to change with ownership. Use the existing CPU Distributed assignment helper and seeded input generators in setup and capability probes. MPI/GPU keep arbitrary placement; operations, result placement, samples, and regression thresholds are unchanged. Validation: Julia 1.12.4 with three one-thread workers runs SpMV, CG, and SpGEMM benchmark leaves with three samples each at N=64 and block size 16. Repeated seeded matrices collect identically and have the same balanced 4/4/4/4 tile-owner layout. Smoke allocation minima: SpMV 54387 / 2454224 bytes, CG 623058 / 27935464, SpGEMM 69602 / 3083344; these remain process-local measurements.
Out-of-place Finch GEMM can start with a host CSC destination from similar. The fallback sent Finch tensors through generic collect and failed because Tensor has no length. Build the Finch result and replace the sparse wrapper storage; materialize an existing non-Finch destination before nonzero-beta accumulation. Remove the obsolete test LOAD_PATH injection since workers already inherit the active project. Validation: Julia 1.11.9, one driver thread and three one-thread workers: all 32 GEMM and 16 SpMV checks pass, including new mixed-storage beta checks. A separate two-thread run exposes Finch 1.6.0 global LogicExecutor.codes Dict races; no serialization workaround is introduced. Memory: native 2x2 Finch GEMM with nonzero beta, Julia 1.11.9, 10 warmups, GC, minimum over five measurements: 1530 allocations / 65472 bytes before and after. The previously failing CSC-destination path has no valid baseline allocation measurement.
The harness removed worker sockets one second after requesting scheduler shutdown, racing cleanup RPCs and logging late transport errors. Wait on the existing scheduler-exit condition instead of sleeping. Validation: Julia 1.11.9 with a one-thread driver and three two-thread workers passes 131 thunk and 30 processor assertions and exits without the previously observed unknown-socket error. Julia 1.10.12 with four threads and no initial workers passes 120 thunk assertions and exits cleanly. Existing broken checks remain reported.
The cold 4D case generates very large planning signatures and leaves peer ranks waiting while LLVM compiles. Disable only this case for CPU and GPU MPI entry points, retaining the existing backend-specific high-dimensional skips and all non-MPI coverage. Validation: Julia 1.10.12, four MPI ranks with two threads each: all 52 remaining stencil assertions pass on every rank (211 seconds).
Cancelled jobs previously stopped at a single combined stencil-suite message, hiding whether CPU execution, GPU execution, or high-dimensional compilation was active. Emit progress at backend and high-dimensional case boundaries, and record the local PoCL profiling evidence. Validation: the shared suite passes 53 assertions on PoCL, including 3D and 4D; this change only adds test progress messages.
A 4D halo unrolls 80 regions. Inlining host view construction and kernel-launch bodies into every tuple element gives LLVM a huge function to optimize. Keep the typed tuple but put construction behind a specialized non-inlined per-region barrier. Julia 1.11.9 PoCL: isolated cold 4D stencil 114.75s -> 47.35s; full shared GPU suite 283.31s -> 215.98s, all 53 assertions pass. Four Julia 1.10.12 MPI ranks also pass all 52 remaining assertions each. Allocation profiling: 10 warmups, GC, minimum of 5 gc_num deltas. CPU allocations/bytes unchanged: 2D 11/1536, 3D 29/6176, 4D 86/29264. PoCL unchanged: 2D 533/63712, 3D 1247/150704, 4D 2696/337968. Inferred return types are unchanged for all six fixtures.
With DistributedNext selected, direct stdlib Distributed imports report myid() == 1 on every worker. OpenCL workers 2 and 3 consequently registered GPU owner 1. Import the existing functions from Dagger in GPU and Python extensions so registration and transfers use the same registry as the scheduler. Validation: Julia 1.11.9, DistributedNext, two two-thread workers: OpenCL ownership and distributed GPU dataflow 51 assertions pass; Python ownership 6 and Python numerical integration 6 assertions pass. The OpenCL probe logged a late transport socket error during teardown after numerical checks passed; this change addresses ownership and RPC selection.
Local DTask vs remote Chunk neighbors (including Chunk processor/scope
parameters) specialized each submission on placement. Wrap neighbor
metadata in concrete Read{Any}; execution still specializes on actual
arrays. Normalize received CPU Wrap regions to full-range views so local
and remote regions also share one concrete halo tuple/kernel signature.
The original Julia 1.11 three-worker CPU 4D case took 720 seconds; the
final DistributedNext run took 49 seconds. Both CPU and PoCL shared suites
pass (126 assertions, 344 seconds total). Four Julia 1.10 MPI ranks each
pass all 62 assertions through 3D. All 24 allocation bounds pass unchanged.
A late DistributedNext unknown-socket teardown diagnostic remains separate
from the passing numerical suites.
After 10 warmups and GC, minimum of five Base.gc_num deltas on Julia 1.12:
2D: 7,321 allocations / 374,880 bytes -> 7,104 / 362,320.
4D: 217,684 allocations / 15,401,096 bytes -> 202,918 / 13,964,416.
A mixed local/received halo fixture also becomes concretely inferred and
checks value correctness, homogeneous representation, and retained aliasing.
GPU extensions now honor the selected transport. The three-worker CPU and PoCL stencil suites pass 126 assertions with DistributedNext, and the separate functional GPU probe passes 51 assertions. Keep the Julia 1.11 OpenCL job consistent with the CPU job transport choice; newer Julia jobs retain stdlib Distributed coverage.
Contributor
Dagger benchmarks:
|
| Backend | Regressions | Improvements | Within noise | Inconclusive time | Suites |
|---|---|---|---|---|---|
| Threads | 0 | 3 | 9 | 3 | 4/4 |
| Distributed | 3 | 0 | 3 | 0 | 4/4 |
| Distributed+Threads | 0 | 1 | 0 | 4 | 4/4 |
| MPI | 0 | 1 | 0 | 0 | 4/4 |
| MPI+Threads | 1 | 1 | 15 | 3 | 4/4 |
| Total | 4 | 6 | 27 | 10 | 20/20 |
Regressions across all backends
| Backend | Benchmark | Metric | Change |
|---|---|---|---|
| Distributed | linalg/dagger/N=1024 (block 128)/syrk (A'*A) |
allocs | +121.8% |
| Distributed | linalg/dagger/N=1024 (block 128)/syrk (A'*A) |
memory | +82.7% |
| MPI+Threads | linalg/dagger/N=1024 (block 512)/solve (A\b via lu) |
time | +57.8% |
| Distributed | array/dagger/N=1024 (block 128)/add (X + X) |
memory | +27.6% |
Improvements across all backends
| Backend | Benchmark | Metric | Change |
|---|---|---|---|
| Threads | array/dagger/N=256 (block 128)/broadcast (X .+ 1) |
time | -82.4% |
| MPI+Threads | linalg/dagger/N=256 (block 128)/matmul (A*A) |
time | -61.1% |
| MPI | array/dagger/N=1024 (block 512)/norm |
time | -47.4% |
| Threads | stencil/dagger/N=256 (block 256)/multi-expr |
time | -42.9% |
| Distributed+Threads | stencil/dagger/N=1024 (block 512)/neighbors (Clamp) |
time | -41.5% |
| Threads | array/dagger/N=256 (block 256)/broadcast (X .+ 1) |
time | -25.9% |
Threads (1 process × 4 threads) — 0 regressions, 3 improvements
| Suite | Regressions | Improvements | Within noise | Inconclusive time |
|---|---|---|---|---|
| array | 0 | 2 | 5 | 0 |
| linalg | 0 | 0 | 3 | 0 |
| sparse | 0 | 0 | 0 | 0 |
| stencil | 0 | 1 | 1 | 3 |
Improvements
| Benchmark | Metric | Change |
|---|---|---|
array/dagger/N=256 (block 128)/broadcast (X .+ 1) |
time | -82.4% |
stencil/dagger/N=256 (block 256)/multi-expr |
time | -42.9% |
array/dagger/N=256 (block 256)/broadcast (X .+ 1) |
time | -25.9% |
Within noise
| Benchmark | Metric | Change |
|---|---|---|
array/dagger/N=1024 (block 512)/add (X + X) |
time | +50.0% |
linalg/dagger/N=256 (block 128)/matvec (A*x) |
time | +40.8% |
array/dagger/N=256 (block 128)/reduce (sum) |
time | +30.3% |
stencil/dagger/N=256 (block 128)/multi-expr |
time | +26.5% |
linalg/dagger/N=1024 (block 512)/matvec (A*x) |
time | -32.1% |
array/dagger/N=1024 (block 512)/reduce (sum) |
time | -33.1% |
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) |
time | -33.5% |
linalg/dagger/N=256 (block 128)/matmul (A*A) |
time | -43.6% |
array/dagger/N=256 (block 128)/map (sin.(X)) |
time | -67.5% |
Inconclusive timing changes
| Benchmark | Metric | Change |
|---|---|---|
stencil/dagger/N=1024 (block 512)/update (+) |
time | +43.1% |
stencil/dagger/N=1024 (block 128)/multi-expr |
time | +42.9% |
stencil/dagger/N=1024 (block 512)/multi-expr |
time | +33.9% |
Distributed (4 processes × 1 thread) — 3 regressions, 0 improvements
| Suite | Regressions | Improvements | Within noise | Inconclusive time |
|---|---|---|---|---|
| array | 1 | 0 | 0 | 0 |
| linalg | 2 | 0 | 2 | 0 |
| sparse | 0 | 0 | 1 | 0 |
| stencil | 0 | 0 | 0 | 0 |
Regressions
| Benchmark | Metric | Change |
|---|---|---|
linalg/dagger/N=1024 (block 128)/syrk (A'*A) |
allocs | +121.8% |
linalg/dagger/N=1024 (block 128)/syrk (A'*A) |
memory | +82.7% |
array/dagger/N=1024 (block 128)/add (X + X) |
memory | +27.6% |
Within noise
| Benchmark | Metric | Change |
|---|---|---|
linalg/dagger/N=1024 (block 128)/lu |
time | +124.9% |
linalg/dagger/N=1024 (block 128)/syrk (A'*A) |
time | +62.4% |
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) |
time | +50.8% |
Distributed+Threads (2 processes × 2 threads) — 0 regressions, 1 improvements
| Suite | Regressions | Improvements | Within noise | Inconclusive time |
|---|---|---|---|---|
| array | 0 | 0 | 0 | 2 |
| linalg | 0 | 0 | 0 | 0 |
| sparse | 0 | 0 | 0 | 0 |
| stencil | 0 | 1 | 0 | 2 |
Improvements
| Benchmark | Metric | Change |
|---|---|---|
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) |
time | -41.5% |
Inconclusive timing changes
| Benchmark | Metric | Change |
|---|---|---|
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) |
time | +71.0% |
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) |
time | +65.1% |
array/dagger/N=1024 (block 512)/transpose (permutedims) |
time | +52.3% |
stencil/dagger/N=1024 (block 512)/neighbors (Pad) |
time | +49.3% |
MPI (4 ranks × 1 thread) — 0 regressions, 1 improvements
| Suite | Regressions | Improvements | Within noise | Inconclusive time |
|---|---|---|---|---|
| array | 0 | 1 | 0 | 0 |
| linalg | 0 | 0 | 0 | 0 |
| sparse | 0 | 0 | 0 | 0 |
| stencil | 0 | 0 | 0 | 0 |
Improvements
| Benchmark | Metric | Change |
|---|---|---|
array/dagger/N=1024 (block 512)/norm |
time | -47.4% |
MPI+Threads (2 ranks × 2 threads) — 1 regressions, 1 improvements
| Suite | Regressions | Improvements | Within noise | Inconclusive time |
|---|---|---|---|---|
| array | 0 | 0 | 8 | 1 |
| linalg | 1 | 1 | 3 | 2 |
| sparse | 0 | 0 | 0 | 0 |
| stencil | 0 | 0 | 4 | 0 |
Regressions
| Benchmark | Metric | Change |
|---|---|---|
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) |
time | +57.8% |
Improvements
| Benchmark | Metric | Change |
|---|---|---|
linalg/dagger/N=256 (block 128)/matmul (A*A) |
time | -61.1% |
Within noise
| Benchmark | Metric | Change |
|---|---|---|
stencil/dagger/N=1024 (block 512)/assign (const) |
time | +228.6% |
array/dagger/N=256 (block 128)/norm |
time | +212.9% |
array/dagger/N=256 (block 128)/broadcast (X .+ 1) |
time | +101.5% |
array/dagger/N=1024 (block 512)/norm |
time | +84.0% |
stencil/dagger/N=256 (block 256)/neighbors (Reflect) |
time | +61.8% |
array/dagger/N=256 (block 128)/map (sin.(X)) |
time | -35.2% |
linalg/dagger/N=256 (block 128)/syrk (A'*A) |
time | -37.3% |
linalg/dagger/N=1024 (block 128)/cholesky |
time | -37.5% |
linalg/dagger/N=256 (block 128)/matvec (A*x) |
time | -39.4% |
array/dagger/N=1024 (block 512)/reduce (sum) |
time | -41.2% |
stencil/dagger/N=256 (block 128)/neighbors (Wrap) |
time | -41.7% |
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) |
time | -44.4% |
array/dagger/N=256 (block 128)/transpose (permutedims) |
time | -51.0% |
array/dagger/N=1024 (block 512)/alloc (rand) |
time | -57.9% |
stencil/dagger/N=256 (block 256)/assign (const) |
time | -62.8% |
Inconclusive timing changes
| Benchmark | Metric | Change |
|---|---|---|
linalg/dagger/N=1024 (block 512)/matvec (A*x) |
time | +313.3% |
array/dagger/N=256 (block 128)/reduce (sum) |
time | +132.2% |
linalg/dagger/N=1024 (block 512)/cholesky |
time | +47.9% |
Full results and plots (download the benchmark-results-* artifacts).
DistributedNext starts workers with using DistributedNext in Main. Its presence as a transitive Dagger dependency is insufficient there, so direct runtests.jl invocations failed before any tests ran. Add it to test/Project.toml. With global environments hidden, removing this declaration reproduces the exact GHA startup error; the actual processor and preference test bodies pass 32 assertions each on Julia 1.10 and 1.11 with three two-thread workers.
A fresh GHA runner's time_ns counter was only 104 seconds old when the regression fixture subtracted 120 seconds. The wrapped timestamp looked newer than every real event. An absolute cutoff also underflows whenever uptime is shorter than the retention window. Compare elapsed UInt64 ages and use the retained-prefix count to return when nothing expired. The full TimespanLogging suite passes all 79 assertions on Julia 1.13, including a deterministic wrap fixture that the previous method retains incorrectly. After 10 warmups and GC, minimum of five 1,000-call gc_num deltas on Julia 1.12.4: 9,000 allocations / 528,000 bytes before versus 6,000 / 496,000 after (9 -> 6 allocations per retained-event update).
SparseArrays now has a vararg getindex(::AbstractSparseMatrixCSC, I...) fallback. Dagger's AbstractArray x ArrayDomain method constrains the other argument, making distribute(sparse_matrix, Blocks(...)) ambiguous. Add the intersection method in the sparse extension and retain native CSC slicing. Current Julia 1.14.0-DEV.3754 passes 17 domain assertions and all 981 array allocation assertions, including the sparse distribute case that failed in both nightly GHA jobs. The new test checks inferred sparse results, values, and empty-domain behavior. With 10 warmups plus GC and minimum of five 1,000-call gc_num deltas on Julia 1.12.4, the old generic indexing body and the new specialization both use 6,000 allocations / 1,504,000 bytes.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI failures include incorrect LU solve results, factorization solves falling back to local arrays on Julia 1.14, Finch multiplication with CSC destinations, and logging test failures. CPU/OpenCL stencil tests also spend minutes compiling new variants for worker-placement patterns, contributing to job cancellations.
This PR fixes those source/test defects and makes the selected Distributed transport consistent across the harness and extensions. The changes remain separated into 19 commits for review.
Validation performed locally:
On the M2 host, the cold CPU stencil suite fell from 310s to 117s despite ten added regression assertions; 4D fell from 200s to 31s. In the Julia 1.11 three-worker topology, CPU 4D fell from approximately 720s to 49s. These are end-to-end cold test timings; the original M2 4D run overlapped part of a separate LU check. The OpenCL GPU and stencil test bodies above ran in separate fresh processes, so their combined duration excludes dependency installation and is not an end-to-end CI measurement. Local PoCL used POCL_MAX_PTHREAD_COUNT=4 to bound host resource use; that setting was not added to CI.
Allocation measurements used Julia 1.12.4, fully awaited/pinned fixtures, ten warmups followed by GC, and the minimum of five Base.gc_num() deltas:
No CI timeouts or allocation bounds were increased.
Remaining limitations: Finch 1.6.0 has a separate unsynchronized global compilation-cache Dict; a Finch-only concurrent reproducer fails while its serial control passes. No Dagger-side locking workaround is included. A sporadic DistributedNext unknown-socket diagnostic still occurs during worker shutdown after passing assertions, and one stdlib Distributed run lost a worker without a useful crash traceback; neither is claimed fixed. The broader macOS CI slowdown is also unresolved: original and fixed iterative-solver controls both finish in about two minutes on the M2 host, versus 33 minutes in the GitHub job. A complete GitHub-hosted macOS job has not yet been rerun with these changes.
Three earlier benchmark jobs also reported regressions independently of package loading: Distributed matmul allocations, Distributed+Threads Clamp stencil timing, and MPI+Threads LU solve timing. The matmul allocation discrepancy was approximately 123% in the first run but approximately 1.5% in its confirmation run; the two timing comparisons did repeat in CI. These are not claimed fixed, and no benchmark thresholds were changed.