Fix CI correctness defects and excessive stencil compilation

.github/workflows/CI.yml

@@ -23,6 +23,10 @@ jobs:
     if: ${{ !contains(github.event.head_commit.message, '[skip tests]') }}
     env:
       JULIA_NUM_THREADS: '1'
+      # Distributed's older releases race when used from multiple threads.
+      # Keep stdlib coverage on newer Julia while testing the supported
+      # DistributedNext transport on 1.10/1.11.
+      CI_DISTRIBUTED_PACKAGE: ${{ (matrix.version == '1.10' || matrix.version == '1.11') && 'DistributedNext' || 'Distributed' }}
     strategy:
       fail-fast: false
       matrix:
@@ -128,6 +132,7 @@ jobs:
     env:
       JULIA_NUM_THREADS: '1'
       CI_USE_OPENCL: '1'
+      CI_DISTRIBUTED_PACKAGE: DistributedNext
     steps:
       - uses: actions/checkout@v7
       - uses: julia-actions/setup-julia@v3

AGENTS.md

@@ -859,3 +859,158 @@ lesson.
    the failing native call, not its root cause. Keep Intel MPI functional tests
    enabled separately from the benchmark jobs; a green small matmul test is
    insufficient evidence to enable the larger tiled benchmark matrix.
+
+82. **Dashboard aggregators run when logs are collected.** A `D3Renderer`
+    attached to a `MultiEventLog` does not start its HTTP listener when a task
+    emits an event. Call `get_logs!` to run the consumers and aggregators before
+    requesting its page. A short `LogWindow` may legitimately discard every
+    event emitted before a long first compilation; test the HTTP response
+    independently of whether that batch still contains recent events.
+
+83. **A log-window cutoff is a retained index, not a removal count.**
+    `findfirst(timestamp >= cutoff)` names the first event that must survive;
+    deleting through that index discards one valid event whenever a batch mixes
+    old and recent records. Convert it to the preceding index, and use the
+    full batch length only when every record expired. Apply the same removal
+    count to every consumer column and deletion hook.
+
+84. **Transport selection must cover the test harness and every extra worker.**
+    Dagger, MemPool, and TimespanLogging read their transport preference before
+    loading; setting only Dagger's preference or continuing to import stdlib
+    Distributed in tests creates separate process registries. Import the module
+    that owns `Dagger.myid` throughout the harness. Even workers launched inside
+    individual tests must inherit `Base.active_project()`; bare `--project`
+    selects the checkout instead and can lose DistributedNext and test-only
+    dependencies. Preference tests must restore the incoming transport.
+
+85. **Track LinearAlgebra's factorization allocation protocol across Julia versions.**
+    Dagger's `_zeros(T, rhs, n::Integer)` hook keeps factorization solves over
+    distributed right-hand sides distributed. Julia 1.14 now passes the
+    factorization itself as the third argument. Without the matching method,
+    Cholesky/QR solves silently allocate a local Array and bypass Dagger's
+    distributed solve path. Forward the new protocol to the existing column-count
+    method, and check both result type and values in solve tests.
+
+86. **LU pivot chunks must match the returned factors' row blocks.**
+    `_dagger_apply_ipiv_rows!` interprets every pivot chunk using the factors'
+    row block size. AutoBlocks for NoPivot can therefore permute an identity
+    permutation incorrectly, or run its unchecked swap kernel out of bounds.
+    Pivoted LU also needs to re-tile its pivots after square-panel buffering of
+    a non-square tile layout. Read the returned matrix's partitioning at that
+    point: the buffering closure reassigns the captured `mb`/`nb` locals to the
+    temporary square layout, so those locals no longer describe the original.
+
+87. **Sparse benchmark fixtures need fixed placement and fixed nonzeros.**
+    The sparse suite used arbitrary placement after the other suites adopted
+    `benchmark_assignment()`. Driver-only BenchmarkTools allocation counts then
+    changed with the driver's share of sparse tiles, even across the same
+    revision. Seed sparse matrices and right-hand sides in untimed setup and
+    use the shared assignment helper for both operands and capability probes.
+    Keep result placement and regression thresholds unchanged; MPI/GPU fixtures
+    must still use the helper's scope-aware arbitrary placement.
+
+88. **A Finch multiply may receive a CSC destination from `similar`.**
+    Dagger's generic sparse allocator does not retain Finch's tensor format.
+    Forwarding such a destination to generic matmul also forwards raw Finch
+    tensors to `collect`, which requires an iteration/length interface they do
+    not implement. Compute the Finch result and replace the DSparseArray's
+    storage; for nonzero beta, materialize the previous destination into Finch
+    before accumulation. Native Finch output still uses the same path. Finch
+    1.6.0 also has an independent thread-unsafe global `LogicExecutor.codes`
+    Dict: cold two-thread GEMM can raise a concurrent-write assertion even when
+    the one-thread-per-worker CI topology passes.
+
+89. **Wait for scheduler teardown before removing test workers.**
+    Notifying `state.halt` starts cleanup RPCs; it does not finish them. A
+    one-second sleep can expire while a worker is still compiling its cleanup
+    method, and removing its socket then logs a transport error after passing
+    assertions. Wait on `Sch.EAGER_STATE_LOCK` until `EAGER_INIT` clears, which
+    happens after scheduler cleanup, before calling `rmprocs`.
+
+90. **Inspect peer compilation before diagnosing an MPI receive timeout.**
+    The four-rank Julia 1.10 CPU suite can fail in the 4D stencil and cascade
+    into later uniformity checks (reproduced as 1,433 passes and seven errors).
+    Peer stack snapshots showed LLVM's `LiveDebugValues`/PHI-placement passes
+    compiling planning calls while other ranks waited on uniformity or result
+    broadcasts. The receive stack alone therefore does not establish a message
+    ordering defect. Isolate the cold 4D stencil and inspect all ranks; changing
+    planner specialization alone is not yet a validated fix, and increasing a
+    timeout would not fix the underlying compilation behavior.
+
+91. **A slow OpenCL stencil may be compiling host halo assembly.** A local
+    Julia 1.11/PoCL run passes all 53 shared stencil assertions, including 4D,
+    but takes 283 seconds cold: the first simple kernel takes 86 seconds and
+    the 4D case another 80. A stack snapshot in 4D showed host LLVM instruction
+    scheduling below `build_fused_halo`, before the device kernel runs; the
+    first simple kernel instead showed GPUCompiler inference/inlining. Time
+    the backend and dimension separately and inspect the active stack before
+    treating silence in the combined CPU/GPU suite as a device deadlock. Also
+    reproduce the CI worker topology: a single-process device pass does not
+    establish that the Distributed CPU half of the same job completes.
+
+92. **Unroll the halo tuple, not each region's construction body.**
+    `_build_fused_halos` uses `ntuple(Val(NH))` to retain heterogeneous device
+    view types, but inlining the view/kernel-launch body into every element
+    duplicates it 80 times in 4D. A specialized, non-inlined per-region helper
+    reduced an isolated cold PoCL 4D stencil from 115 to 47 seconds. Deep-warmed
+    CPU and OpenCL fixtures in 2D/3D/4D retained exactly the same allocation
+    counts, bytes, and inferred return types. Keep the barrier on the host
+    construction path; erasing halo types or moving dynamic dispatch into the
+    device sweep would break the reason for the typed tuple in the first place.
+
+93. **Extensions must use Dagger's selected transport too.** Importing
+    `Distributed.myid` or `remotecall_fetch` directly in an extension bypasses
+    the transport preference. Under DistributedNext, stdlib Distributed has
+    no workers, so OpenCL workers 2 and 3 both reported stdlib ID 1 and
+    registered their GPU processors with owner 1. Import these functions from
+    Dagger, which already selected the transport. The same rule applies to
+    Python processor registration and to every GPU extension's transfer RPCs.
+
+94. **Stencil placement must not become a kernel or planner type parameter.**
+    Local halo dependencies are `DTask`s while remote ones are `Chunk`s whose
+    parameters also encode processor/scope provenance. Wrapping each neighbor
+    in `Read{Any}` keeps the dependency wrapper concrete without specializing
+    submission on every placement pattern; execution still specializes on the
+    actual arrays. Separately, local CPU Wrap halos are views but received
+    regions are Arrays. Wrap received Arrays in full-range views too, or every
+    local/remote mask creates another heterogeneous 80-region tuple and unrolled
+    `halo_load` kernel in 4D. These are independent sources of compilation
+    growth. On Julia 1.11 with three workers, CPU 4D fell from 720 to 49 seconds;
+    warmed 4D allocations also fell from 217,684 / 15,401,096 bytes to
+    202,918 / 13,964,416. Check both dependency metadata types and unwrapped
+    value types before adding a dynamic-dispatch threshold to submission.
+    Apple M2/macOS 26.7 with Julia 1.13.2 reproduces the same problem with
+    CI's coverage flag and three two-thread workers: 4D fell from 200 to 31
+    seconds, and the full CPU stencil suite from 310 to 117 seconds despite
+    ten additional regression assertions. Native samples of the original
+    driver showed LLVM dead-store elimination and alias analysis, rather than
+    a blocked transport. `addprocs` inherits the coverage flag, so retain it
+    when reproducing the CI compilation workload.
+
+95. **The test project must directly declare its worker transport.**
+    DistributedNext launches each worker with `using DistributedNext` in Main.
+    Dagger can import a transitive dependency from its own module, but that
+    does not make the package visible to Main in the active test project.
+    Declaring it only in the root Project.toml therefore lets the driver load
+    and every worker fail before printing its host/port. Validate with a copy
+    of test/Project.toml and `JULIA_LOAD_PATH=@:@stdlib`; an ad-hoc environment
+    that explicitly adds DistributedNext hides this defect.
+
+96. **Log windows need elapsed ages, not an absolute unsigned cutoff.**
+    `time_ns()` and event timestamps are UInt64 counters. On a freshly booted
+    runner, subtracting a long window can underflow; constructing an older
+    fixture can wrap too (GHA's counter was about 104 seconds when the fixture
+    subtracted 120 seconds). Numeric timestamp ordering then retains expired
+    records or drops recent ones. Compare `now - timestamp <= window_length`
+    instead, and test a timestamp across UInt64 wrap explicitly so validation
+    does not depend on the developer machine's uptime.
+
+97. **A sparse indexing fallback can make ArrayDomain indexing ambiguous.**
+    Julia 1.14's SparseArrays adds `getindex(::AbstractSparseMatrixCSC, I...)`.
+    It is more specific in the array argument while Dagger's
+    `getindex(::AbstractArray, ::ArrayDomain)` is more specific in the index,
+    so ordinary `distribute(::SparseMatrixCSC, ::Blocks)` fails while slicing
+    its first tile. Put the intersection method in SparseArraysExt and expand
+    the domain's indices there. Test both direct sparse-domain slicing and the
+    existing allocation/distribute suite on the current nightly; an older
+    nightly without the fallback does not exercise the conflict.

benchmark/suites/sparse.jl

@@ -35,6 +35,9 @@ function sparse_suite(ctx; method, accels)
     @assert accel == "cpu" || haskey(GPU_BACKENDS, accel) "Unknown backend"
 
     T = benchmark_eltype()
+    # Driver-local allocation counts require the same tile placement in both
+    # revisions. Seed the untimed fixtures too, so sparse structure is fixed.
+    fixture_assignment = benchmark_assignment()
     atol, rtol = T(1e-8), T(1e-6)
     # Target a fixed number of nonzeros per row, so density shrinks with N and
     # the nonzero count stays ~O(N).
@@ -48,17 +51,17 @@ function sparse_suite(ctx; method, accels)
     # operation present on one revision but not the other simply won't be
     # compared. This self-heals once the baseline gains the feature.
     spmv_ok = supported("sparse/spmv (S*x)") do
-        S = distribute(sprand(T, 8, 8, 0.5), Blocks(4, 4))
-        x = distribute(rand(T, 8), Blocks(4))
+        S = distribute(sprand(Random.Xoshiro(1234), T, 8, 8, 0.5), Blocks(4, 4), fixture_assignment)
+        x = distribute(rand(Random.Xoshiro(4321), T, 8), Blocks(4), fixture_assignment)
         wait(S * x)
     end
     spgemm_ok = supported("sparse/spgemm (S*S)") do
-        S = distribute(sprand(T, 8, 8, 0.5), Blocks(4, 4))
+        S = distribute(sprand(Random.Xoshiro(1234), T, 8, 8, 0.5), Blocks(4, 4), fixture_assignment)
         wait(S * S)
     end
     cg_ok = KRYLOV_AVAILABLE && isdefined(Dagger, :cg) && supported("sparse/cg solve") do
-        A = distribute(laplacian_1d(T, 8), Blocks(4, 4))
-        rhs = distribute(rand(T, 8), Blocks(4))
+        A = distribute(laplacian_1d(T, 8), Blocks(4, 4), fixture_assignment)
+        rhs = distribute(rand(Random.Xoshiro(4321), T, 8), Blocks(4), fixture_assignment)
         wait(first(Dagger.cg(A, rhs; atol=atol, rtol=rtol, itmax=50)))
     end
 
@@ -72,23 +75,23 @@ function sparse_suite(ctx; method, accels)
         # SpMV: sparse matrix tiles, dense vectors.
         if spmv_ok && fits_budget(sparse_bytes(N; nmats=2, density=density, T=T))
             sub["spmv (S*x)"] = @benchmarkable(wait(S * x),
-                setup = (S = distribute(sprand($T, $N, $N, $density), Blocks($b, $b));
-                         x = distribute(rand($T, $N), Blocks($b)); wait(S); wait(x)),
+                setup = (S = distribute(sprand(Random.Xoshiro(1234), $T, $N, $N, $density), Blocks($b, $b), $fixture_assignment);
+                         x = distribute(rand(Random.Xoshiro(4321), $T, $N), Blocks($b), $fixture_assignment); wait(S); wait(x)),
                 teardown = (S = nothing; x = nothing; @everywhere GC.gc()))
         end
 
         # SpGEMM: the result fills in, so budget generously.
         if spgemm_ok && fits_budget(sparse_bytes(N; nmats=6, density=density, T=T))
             sub["spgemm (S*S)"] = @benchmarkable(wait(S * S),
-                setup = (S = distribute(sprand($T, $N, $N, $density), Blocks($b, $b)); wait(S)),
+                setup = (S = distribute(sprand(Random.Xoshiro(1234), $T, $N, $N, $density), Blocks($b, $b), $fixture_assignment); wait(S)),
                 teardown = (S = nothing; @everywhere GC.gc()))
         end
 
         # Iterative solve of an SPD system via conjugate gradients.
         if cg_ok && fits_budget(sparse_bytes(N; nmats=2, density=3 / N, T=T))
             sub["cg solve (laplacian)"] = @benchmarkable(wait(first(Dagger.cg(A, rhs; atol=$atol, rtol=$rtol, itmax=200))),
-                setup = (A = distribute(laplacian_1d($T, $N), Blocks($b, $b));
-                         rhs = distribute(rand($T, $N), Blocks($b)); wait(A); wait(rhs)),
+                setup = (A = distribute(laplacian_1d($T, $N), Blocks($b, $b), $fixture_assignment);
+                         rhs = distribute(rand(Random.Xoshiro(4321), $T, $N), Blocks($b), $fixture_assignment); wait(A); wait(rhs)),
                 teardown = (A = nothing; rhs = nothing; @everywhere GC.gc()))
         end
 

ext/CUDAExt.jl

@@ -5,7 +5,7 @@ export CuArrayDeviceProc
 import Dagger, MemPool
 import Dagger: CPURAMMemorySpace, Chunk, unwrap, CuArrayDeviceProc
 import MemPool: DRef, poolget
-import Distributed: myid, remotecall_fetch
+import Dagger: myid, remotecall_fetch
 import LinearAlgebra
 using KernelAbstractions, Adapt
 

ext/FinchExt.jl

@@ -116,10 +116,6 @@ function Dagger.matmatmul!(
     alpha,
     beta
 )
-    if !isa(C.mat, Finch.Tensor)
-        # Not supported here, forward to generic matmatmul!
-        return Dagger.matmatmul!(C.mat, transA, transB, A, B, alpha, beta)
-    end
     transA in ('N', 'T', 'C') || throw(ArgumentError("Invalid transA: $transA"))
     transB in ('N', 'T', 'C') || throw(ArgumentError("Invalid transB: $transB"))
 
@@ -136,7 +132,11 @@ function Dagger.matmatmul!(
     # sparse `Tensor` so accumulation and later iterations stay well-typed.
     C_out = _finch_materialize(C_out, eltype(Cm), size(Cm))
     if beta != 0
-        C_out = C_out + beta * Cm
+        # `similar(::DArray)` may initially allocate host CSC tiles. The
+        # wrapper permits replacing their storage with the Finch result, but
+        # accumulation must first bring the previous values into Finch too.
+        C_prev = Cm isa Finch.Tensor ? Cm : _finch_materialize(Cm, eltype(Cm), size(Cm))
+        C_out = C_out + beta * C_prev
     end
     C.mat = C_out
 
@@ -185,4 +185,4 @@ function Dagger.transpose_tile(B::Finch.Tensor, uplo=nothing)
     return _finch_materialize(C, eltype(_B), size(C))
 end
 
-end # module FinchExt
+end # module FinchExt

ext/IntelExt.jl

@@ -5,7 +5,7 @@ export oneArrayDeviceProc
 import Dagger, MemPool
 import Dagger: CPURAMMemorySpace, Chunk, unwrap, oneArrayDeviceProc
 import MemPool: DRef, poolget
-import Distributed: myid, remotecall_fetch
+import Dagger: myid, remotecall_fetch
 import LinearAlgebra
 using KernelAbstractions, Adapt
 

ext/MetalExt.jl

@@ -5,7 +5,7 @@ export MtlArrayDeviceProc
 import Dagger, MemPool
 import Dagger: CPURAMMemorySpace, Chunk, unwrap, MtlArrayDeviceProc
 import MemPool: DRef, poolget
-import Distributed: myid, remotecall_fetch
+import Dagger: myid, remotecall_fetch
 import LinearAlgebra
 using KernelAbstractions, Adapt
 

ext/OpenCLExt.jl

@@ -5,7 +5,7 @@ export CLArrayDeviceProc
 import Dagger, MemPool
 import Dagger: CPURAMMemorySpace, Chunk, unwrap, CLArrayDeviceProc
 import MemPool: DRef, poolget
-import Distributed: myid, remotecall_fetch
+import Dagger: myid, remotecall_fetch
 import LinearAlgebra
 using KernelAbstractions, Adapt
 

ext/PythonExt.jl

@@ -8,7 +8,7 @@ end
 
 import Dagger
 import Dagger: Processor, OSProc, ThreadProc, Chunk
-import Distributed: myid
+import Dagger: myid
 
 const CPUProc = Union{OSProc, ThreadProc}
 

ext/ROCExt.jl

@@ -5,7 +5,7 @@ export ROCArrayDeviceProc
 import Dagger, MemPool
 import Dagger: CPURAMMemorySpace, Chunk, unwrap, ROCArrayDeviceProc
 import MemPool: DRef, poolget
-import Distributed: myid, remotecall_fetch
+import Dagger: myid, remotecall_fetch
 import LinearAlgebra
 using KernelAbstractions, Adapt
 

ext/SparseArraysExt.jl

@@ -6,6 +6,11 @@ import LinearAlgebra
 import Dagger
 import Dagger: Blocks, AutoBlocks, BlocksOrAuto, AssignmentType, DSparseArray, DSparseMatrix
 
+# SparseArrays' vararg CSC indexing and Dagger's AbstractArray domain indexing
+# each constrain a different argument. Resolve their intersection explicitly.
+Base.getindex(A::SparseArrays.AbstractSparseMatrixCSC, d::Dagger.ArrayDomain) =
+    A[Dagger.indexes(d)...]
+
 # Keep tiles sparse through `collect`/`cat`; the outer `collect` densifies.
 Dagger._sparse_collect(M::SparseMatrixCSC) = copy(M)
 

lib/DaggerWebDash/test/runtests.jl

@@ -1,5 +1,5 @@
 using DaggerWebDash
-import DaggerWebDash: LinePlot, GanttPlot
+import DaggerWebDash: LinePlot, GanttPlot, HTTP
 using TimespanLogging
 using Dagger
 using Test
@@ -33,6 +33,18 @@ using Test
     ml.aggregators[:logwindow] = lw
     ctx = Context(; log_sink=ml)
     Dagger.Sch.EAGER_CONTEXT[] = ctx
-    fetch(Dagger.@spawn 1+1)
-    run(pipeline(`curl -s localhost:8080/index.html`; stdout=devnull))
+    TimespanLogging.enable!()
+    try
+        @test fetch(Dagger.@spawn 1+1) == 2
+        # Consumers and aggregators run when logs are collected, rather than
+        # during event emission. Collection starts the renderer's HTTP server.
+        logs = TimespanLogging.get_logs!(ml)
+        # The five-second window may prune events emitted before compilation.
+        @test haskey(logs[1], :core)
+        response = HTTP.get("http://localhost:8080/index.html")
+        @test response.status == 200
+        @test occursin("<html", String(response.body))
+    finally
+        Dagger.disable_logging!()
+    end
 end