Add opt-in support for Intel's LTS driver stack (ONEAPI_LTS)

[michel2323]

Intel's LTS driver releases, as deployed on Aurora (NEO 25.18 / 2523.40) and other data center deployments, differ enough from the rolling stack that oneAPI.jl currently fails to compile kernels, silently corrupts reductions, and gets contexts banned on them. This PR adds the workarounds needed to run there, each gated behind a master switch.

The switch

ONEAPI_LTS (oneL0.LTS[]), resolved from the environment at the top of oneL0.__init__, defaults to off. With it off every code path below is skipped and the package behaves exactly as on main, so this PR is a no-op for anyone on the rolling stack. An LTS deployment opts in with ONEAPI_LTS=1.

A second, independent switch, ONEAPI_SYNC_EACH_SUBMISSION (also default off), covers one workaround with a large throughput cost; see below.

Both are parsed by a shared parse_env_bool accepting 1/true/yes/on and 0/false/no/off, which warns rather than silently falling back to the default on an unrecognized value.

What is gated on ONEAPI_LTS

Area	Problem on the LTS stack	Workaround
SPIR-V codegen (src/compiler/compilation.jl)	NEO/IGC only accepts SPIR-V from the Khronos translator, and needs extensions declared explicitly	select backend = :khronos and declare SPV_EXT_relaxed_printf_string_address_space / SPV_EXT_shader_atomic_float_add; the rolling stack keeps :llvm
BFloat16	the translator cannot codegen native bfloat in generic kernels (clamp! → InvalidIRError), and declaring SPV_KHR_bfloat16 segfaults NEO	force supports_bfloat16 = false; the test suite and examples/bfloat16.jl skip to match
Reductions (src/mapreduce.jl)	IGC miscompiles strided (non-coalesced) global reads in the reduction kernel — silently wrong results for sum(A; dims=2), a == transpose(b), ishermitian	materialize strided inputs to dense, and route dim-1-kept reductions to a new coalesced one-work-item-per-slice kernel
Buffer free (src/pool.jl, src/context.jl)	NEO advertises ZE_extension_memory_free_policies but ignores BLOCKING_FREE, so a GC-driven free of a buffer with work in flight pagefaults and bans the context	a queue registry plus synchronize_all_queues before free
Queue destruction (lib/level-zero/cmdqueue.jl)	zeCommandQueueDestroy does not wait for in-flight work; the same fault-and-ban follows	drain in the finalizer with a 10 s bound, then leak the queue rather than hang GC and process exit

ONEAPI_SYNC_EACH_SUBMISSION

Under heavy multi-process oversubscription of a single tile, a whole-queue zeCommandQueueSynchronize does not reliably retire the tail of an earlier, separately submitted command list: the last work-items of a kernel or the last elements of a copy silently go missing. Synchronizing after every submission eliminates it at roughly 3× throughput cost, so it is a separate opt-in rather than folded into ONEAPI_LTS. oneL0.sync_each_submission(f, enable) scopes it temporarily, which is needed for submit-then-signal patterns that would otherwise deadlock.

Not gated (applies to both stacks)

• src/oneAPI.jl: dlopen the NEO driver by full path at init, so that libsycl's bundled Level Zero loader resolves it by soname later. Required when no system NEO is installed. The adjacent LD_LIBRARY_PATH extension only ever affected child processes.
• src/compiler/precompile.jl: the precompile workload mirrors the runtime backend choice, so precompilation warms the pipeline that will actually be used.
• SPIRV_LLVM_Translator_jll becomes a dependency — GPUCompiler resolves the tool from Base.loaded_modules, so both back-ends must be loaded for either path to work, and is reported by versioninfo.

CI

The self-hosted Aurora runner sets ONEAPI_LTS=1 and ONEAPI_SYNC_EACH_SUBMISSION=1, builds liboneapi_support.so from deps/src (and verifies the preference actually took effect, rather than silently testing the registered JLL), and disables AVX512-FP16 host codegen via -C native,-avx512fp16: under concurrent oneMKL load the native FP16 path miscomputes host Float16 reference values on these Sapphire Rapids nodes, failing tests whose GPU result is correct.

The buildkite runner is unchanged and gets the defaults, so it keeps exercising the non-LTS :llvm back-end path.

Tests

test/array.jl gains a "strided mixed reductions" regression test using exact Int32 sums (immune to Float32 accumulation-order rounding) across several shapes and dims combinations, covering both the reductions routed to the coalesced kernel and the mixed ones that deliberately stay on the workgroup-per-slice kernel.

[github-actions]

Your PR requires formatting changes to meet the project's style guidelines.
Please consider running Runic (git runic main) to apply these changes.

Click here to view the suggested changes.

diff --git a/test/level-zero.jl b/test/level-zero.jl
index bbf8f46..101b7c6 100644
--- a/test/level-zero.jl
+++ b/test/level-zero.jl
@@ -210,21 +210,21 @@ pool = ZeEventPool(ctx, 2)
 signal_event = pool[1]
 wait_event = pool[2]
 
-# This is a submit-then-signal pattern: the kernel is gated on `wait_event`, which is
-# only signaled *after* submission. The ONEAPI_SYNC_EACH_SUBMISSION=1 workaround (Aurora
-# LTS) makes `execute!` block in zeCommandQueueSynchronize right after submitting, which
-# would deadlock here since the kernel cannot retire before `wait_event` is signaled. No
-# production code path submits event-gated work, so disable the workaround just here.
-oneL0.sync_each_submission(false) do
-    execute!(queue) do list
-        append_launch!(list, kernel, 1, signal_event, wait_event)
-    end
-    @test !Base.isdone(signal_event)
-
-    signal(wait_event)
-    synchronize(queue)
-    @test Base.isdone(signal_event)
-end
+        # This is a submit-then-signal pattern: the kernel is gated on `wait_event`, which is
+        # only signaled *after* submission. The ONEAPI_SYNC_EACH_SUBMISSION=1 workaround (Aurora
+        # LTS) makes `execute!` block in zeCommandQueueSynchronize right after submitting, which
+        # would deadlock here since the kernel cannot retire before `wait_event` is signaled. No
+        # production code path submits event-gated work, so disable the workaround just here.
+        oneL0.sync_each_submission(false) do
+            execute!(queue) do list
+                append_launch!(list, kernel, 1, signal_event, wait_event)
+            end
+            @test !Base.isdone(signal_event)
+
+            signal(wait_event)
+            synchronize(queue)
+            @test Base.isdone(signal_event)
+        end
 
 end
 

[codecov]

Codecov Report

❌ Patch coverage is 90.38462% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.24%. Comparing base (0a71d96) to head (45f89ca).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/mapreduce.jl	75.00%	4 Missing ⚠️
lib/level-zero/oneL0.jl	70.00%	3 Missing ⚠️
lib/level-zero/cmdqueue.jl	66.66%	2 Missing ⚠️
src/compiler/compilation.jl	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #574      +/-   ##
==========================================
+ Coverage   79.76%   80.24%   +0.47%     
==========================================
  Files          50       50              
  Lines        3391     3487      +96     
==========================================
+ Hits         2705     2798      +93     
- Misses        686      689       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.