Repository navigation
Reduce CI pressure on the JuliaGPU cluster - #761
Merged
Merged
Conversation
…nsistently Both have failed on every build since at least 2026-09-16, including on master, while taking up the single oneAPI runner and the shared Metal runners. Comment them out, along with the gpu-mpi benchmarks that depend on them, until test/mpi/oneapi.jl and test/mpi/metal.jl pass.
The GPU benchmarks ran on every pull request build, as about 40 jobs across the CUDA, ROCm, oneAPI and Metal queues of the shared JuliaGPU cluster, which is heavily backlogged. Run them only when the pull request is labeled `benchmarks`, or when the `run benchmarks` tag is in the commit message.
maleadt
changed the title
maleadt
force-pushed
the
ci-disable-failing
branch
from
October 7, 2026 07:21
18d0a58 to
326a47b
Compare
Member
|
Thanks, and sorry for the build pressure! Will try to run more of these locally for now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The JuliaGPU Buildkite cluster, which Dagger's GPU CI runs on, is heavily backlogged (the CUDA queue is saturated, and the oneAPI and ROCm queues only have one and two runners). This PR reduces what Dagger runs there:
Disable the oneAPI and Metal MPI steps, which have failed on every build since at least 2026-09-16, including on master. For example, in https://buildkite.com/julialang/dagger-dot-jl/builds/2254:
ZE_RESULT_ERROR_INVALID_ARGUMENTand "Non-uniform value" errors in the dense matvec tests;commit command buffer with uncommitted encoderwhile running test/mpi/metal.jl.The oneAPI and Metal gpu-mpi benchmark steps that depend on them are commented out too, and dropped from the summary steps'
depends_on. The summaries useallow_dependency_failure, but it's worth checking that the report scripts cope with the missing gpu-mpi results for these two backends.Only run the GPU benchmarks when requested: they ran on every pull request build, as about 40 jobs across all four GPU queues. They now run when the PR is labeled
benchmarksor the commit message contains therun benchmarkstag (which already worked for non-PR builds). Buildkite does not start a build when a label is added, so a push is needed after labeling. This is the most opinionated change here; feel free to drop that commit.Assisted by: Opus 5.5.
This PR hasn't been reviewed in detail. I'm opening these based on Claude's analysis of the Buildkite queue history in an attempt to reduce CI pressure.