Add opt-in support for Intel's LTS driver stack (ONEAPI_LTS)

.github/workflows/ci.yml

@@ -15,9 +15,21 @@ permissions:
 
 jobs:
   self-runner:
-    continue-on-error: true
+    # Spelled out so the PR checks box says which driver stack this exercises: this job is
+    # the LTS stack (ONEAPI_LTS=1, self-hosted Aurora runner), while buildkite covers the
+    # rolling stack. Without an explicit name GitHub derives one from the matrix
+    # ("self-runner (ubuntu-latest, 1, x64)"), which says nothing about the stack and
+    # misreports the runner as ubuntu-latest — `os` is unused, `runs-on` is self-hosted.
+    name: LTS stack (self-hosted, Julia ${{ matrix.julia-version }})
     env:
       GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      # Pin each parallel test worker to a distinct GPU tile instead of
+      # oversubscribing device 0 (see test/runtests.jl).
+      ONEAPI_TEST_SPREAD_GPUS: '1'
+      ONEAPI_LTS: '1'
+      # Synchronize after every command-list submission to work around the
+      # Aurora LTS NEO dropped-tail corruption (see lib/level-zero/cmdlist.jl).
+      ONEAPI_SYNC_EACH_SUBMISSION: '1'
     runs-on: [self-hosted, linux, X64]
     strategy:
       matrix:
@@ -36,7 +48,50 @@ jobs:
         with:
           version: ${{ matrix.julia-version }}
       - uses: julia-actions/cache@v3
+      # Build liboneapi_support.so from deps/src rather than using the registered
+      # oneAPI_Support_jll artifact, which lags behind the wrappers in this branch.
+      # Writes LocalPreferences.toml (plus a copy in test/, for Pkg.jl#2500) pointing
+      # oneAPI_Support_jll at the locally-built library. Not continue-on-error: if this
+      # fails we would silently fall back to the JLL and test the wrong library.
+      - name: Build the oneAPI support library
+        timeout-minutes: 90
+        run: julia --color=yes --project=deps deps/build_local.jl
       - uses: julia-actions/julia-buildpkg@latest
         continue-on-error: true
-      - uses: julia-actions/julia-runtest@latest
-        continue-on-error: true
+      # Must run after julia-buildpkg: Manifest.toml is gitignored, so the top-level
+      # project is not instantiated until then and `using oneAPI_Support_jll` would
+      # fail with "required but does not seem to be installed".
+      - name: Check the local support library is picked up
+        run: |
+          julia --color=yes --project=. -e '
+              using TOML, oneAPI_Support_jll
+              want = TOML.parsefile("LocalPreferences.toml")["oneAPI_Support_jll"]["liboneapi_support_path"]
+              got = oneAPI_Support_jll.liboneapi_support_path
+              @info "support library" want got
+              want == got || error("oneAPI_Support_jll is not using the locally-built library")'
+      # Disable AVX512-FP16 host codegen on the Aurora Sapphire Rapids nodes. Under concurrent
+      # oneMKL load the native AVX512-FP16 path silently miscomputes *host* Float16 (e.g. the
+      # GPUArrays `A .* B .+ c` broadcast reference), failing tests even though the GPU result
+      # is correct (single-process clean; MXCSR clean; only the native-FP16 path, not Float32).
+      # `-C native,-avx512fp16` routes Float16 through Float32 and propagates to the Pkg.test
+      # subprocess and its parallel workers via Base.julia_cmd(). `julia-runtest` cannot pass a
+      # cpu-target, so invoke Pkg.test() directly.
+      - name: Run tests (AVX512-FP16 disabled)
+        run: julia -C "native,-avx512fp16" --color=yes --project=. -e 'import Pkg; Pkg.test(coverage=true)'
+      # Report coverage from this job too, not just from buildkite. Every LTS-gated branch
+      # (the queue registry, the coalesced reduction path, the :khronos codegen selection,
+      # the drain-before-free) is unreachable on the rolling stack, so without this upload
+      # Codecov counts all of it as uncovered. `--code-coverage` propagates to the Pkg.test
+      # subprocess and its parallel workers through Base.julia_cmd(), as the cpu-target
+      # above does. Directories match the buildkite julia-coverage plugin's.
+      - uses: julia-actions/julia-processcoverage@v1
+        with:
+          directories: src,lib,examples
+      - uses: codecov/codecov-action@v5
+        with:
+          files: lcov.info
+          token: ${{ secrets.CODECOV_TOKEN }}
+          # Coverage reporting must not be able to fail the build: this job is the only
+          # signal for the LTS stack, and a Codecov outage or a missing token is not a
+          # test failure.
+          fail_ci_if_error: false

Project.toml

@@ -1,7 +1,7 @@
 name = "oneAPI"
 uuid = "8f75cd03-7ff8-4ecb-9b8f-daf728133b1b"
 authors = ["Tim Besard <tim.besard@gmail.com>", "Alexis Montoison", "Michel Schanen <michel.schanen@gmail.com>"]
-version = "2.7.3"
+version = "2.8.0"
 
 [deps]
 AbstractFFTs = "621f4979-c628-5d54-868e-fcf4e3e8185c"
@@ -23,6 +23,7 @@ Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 SPIRVIntrinsics = "71d1d633-e7e8-4a92-83a1-de8814b09ba8"
 SPIRV_LLVM_Backend_jll = "4376b9bf-cff8-51b6-bb48-39421dff0d0c"
+SPIRV_LLVM_Translator_jll = "4a5d46fc-d8cf-5151-a261-86b458210efb"
 SPIRV_Tools_jll = "6ac6d60f-d740-5983-97d7-a4482c0689f4"
 SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
 SpecialFunctions = "276daf66-3868-5448-9aa4-cd146d93841b"
@@ -47,6 +48,7 @@ PrecompileTools = "1"
 Preferences = "1"
 SPIRVIntrinsics = "1"
 SPIRV_LLVM_Backend_jll = "22"
+SPIRV_LLVM_Translator_jll = "21"
 SPIRV_Tools_jll = "2025.4.0"
 SpecialFunctions = "1.3, 2"
 StaticArrays = "1"

README.md

@@ -73,6 +73,7 @@ Binary dependencies:
 - libigc: 2.34.4+1
 - gmmlib: 22.10.0+0
 - SPIRV_LLVM_Backend: 22.1.8+1
+- SPIRV_LLVM_Translator: 21.1.1+0
 - SPIRV_Tools: 2025.4.0+0
 - oneAPI_Support: 0.10.0+0 (oneMKL v2025.3.0)
 
@@ -154,7 +155,10 @@ julia> execute!(queue) do list
 Built on top of that, are kernel programming capabilities for executing Julia code on oneAPI
 accelerators. Device-side intrinsics are provided by
 [SPIRVIntrinsics.jl](https://github.com/JuliaGPU/SPIRVIntrinsics.jl), and code is compiled to
-SPIR-V using [LLVM's SPIR-V back-end](https://llvm.org/docs/SPIRVUsage.html):
+SPIR-V using [LLVM's SPIR-V back-end](https://llvm.org/docs/SPIRVUsage.html) — or, on Intel's
+LTS driver stack, the [Khronos SPIR-V
+translator](https://github.com/KhronosGroup/SPIRV-LLVM-Translator) (see [Intel's LTS driver
+stack](#intels-lts-driver-stack) below):
 
 ```julia
 julia> function kernel()
@@ -292,6 +296,38 @@ julia> oneArray([1.]) .+ 1
 │ error: Double type is not supported on this platform.
 ```
 
+### Intel's LTS driver stack
+
+Intel ships the Compute Runtime both as rolling releases and as a long-term-servicing (LTS)
+branch that large deployments — Aurora, for instance — stay on for years. oneAPI.jl targets
+the rolling stack by default. The LTS branch predates a number of driver and compiler fixes,
+some of which silently corrupt results rather than raise an error, so the package carries a
+set of workarounds behind an opt-in switch:
+
+```bash
+export ONEAPI_LTS=1
+```
+
+With it enabled, oneAPI.jl:
+
+- compiles kernels to SPIR-V with the [Khronos SPIR-V
+  translator](https://github.com/KhronosGroup/SPIRV-LLVM-Translator) instead of LLVM's
+  SPIR-V back-end, which the LTS NEO/IGC runtime does not accept;
+- disables `BFloat16`, which that SPIR-V stack cannot translate in generic kernels;
+- materializes strided inputs and uses a coalesced reduction kernel, working around an IGC
+  miscompile of non-coalesced reads that silently corrupts e.g. `sum(transpose(x))`;
+- drains command queues before freeing their buffers, since LTS NEO ignores the Level Zero
+  `BLOCKING_FREE` policy and would otherwise fault and ban the context.
+
+A separate switch, `ONEAPI_SYNC_EACH_SUBMISSION=1`, additionally synchronizes after every
+command-list submission. That works around a "dropped tail" corruption seen when several
+processes oversubscribe a single tile, at roughly a 3× throughput cost.
+
+Neither switch installs an LTS driver — they tell oneAPI.jl how to behave against one, and
+are typically combined with the local-toolchain configuration described below. Leave both
+unset on a rolling-release driver: the workarounds trade performance for correctness, and
+none of them are needed there. See the
+[documentation](https://juliagpu.github.io/oneAPI.jl/dev/lts/) for the full details.
 
 
 ## Development

docs/make.jl

@@ -27,6 +27,7 @@ makedocs(
             "Memory Management" => "memory.md",
             "Device Intrinsics" => "device.md",
             "Performance Guide" => "usage/performance.md",
+            "Intel LTS Driver Stack" => "lts.md",
         ],
         "API Reference" => [
             "Overview" => "api.md",

docs/src/index.md

@@ -17,7 +17,8 @@ Direct use of `oneAPI`-specific macros (like `@oneapi`) and types (like `oneArra
 
 - **High-level Array Abstractions**: `oneArray` type fully implementing the `GPUArrays.jl` interface.
 - **Kernel Programming**: Execute custom kernels written in Julia on Intel GPUs, compiled to
-  SPIR-V through LLVM's SPIR-V back-end.
+  SPIR-V through LLVM's SPIR-V back-end (or the Khronos SPIR-V translator on Intel's
+  [LTS driver stack](lts.md)).
 - **KernelAbstractions.jl Backend**: The `oneAPIBackend` back-end runs portable kernels on
   Intel GPUs unmodified.
 - **Level Zero Integration**: Low-level access to the Level Zero API via the `oneL0` submodule.

docs/src/installation.md

@@ -45,6 +45,21 @@ julia> devices()
 julia> device!(1) # Select the first available device
 ```
 
+## Intel LTS Driver Stack
+
+Large deployments (for example Aurora) run Intel's long-term-servicing branch of the
+Compute Runtime rather than a rolling release. That branch needs a number of workarounds,
+which oneAPI.jl provides behind an opt-in switch:
+
+```bash
+export ONEAPI_LTS=1
+```
+
+Among other things this compiles kernels with the Khronos SPIR-V translator instead of
+LLVM's SPIR-V back-end. See [Intel LTS Driver Stack](lts.md) for what else changes and what
+it costs. The switch does not install an LTS driver — combine it with the system-library
+configuration below.
+
 ## Using System Libraries (Advanced)
 
 !!! warning

docs/src/lts.md

@@ -0,0 +1,155 @@
+# Intel LTS driver stack
+
+Intel ships the Compute Runtime in two lines: frequent *rolling* releases, and a
+long-term-servicing (LTS) branch that large deployments stay on for years — for example
+the NEO/IGC LTS stack on [Aurora](https://www.alcf.anl.gov/aurora). oneAPI.jl targets the
+rolling stack by default.
+
+The LTS branch predates a number of driver and compiler fixes that landed in rolling. Left
+alone, some of those defects are not merely inconvenient: they silently corrupt results, or
+get the Level Zero context banned so that every later submission fails. oneAPI.jl therefore
+carries a set of workarounds, kept behind a single opt-in switch so that the default
+(rolling) path is completely unaffected.
+
+## Enabling
+
+Set the environment variable before loading oneAPI.jl:
+
+```bash
+export ONEAPI_LTS=1
+```
+
+`1`, `true`, `yes` and `on` enable it; `0`, `false`, `no` and `off` disable it. An
+unrecognized value warns and falls back to the default (disabled), rather than silently
+reading as off.
+
+You can check which mode is active from Julia:
+
+```julia
+julia> using oneAPI
+
+julia> oneAPI.oneL0.LTS[]
+true
+```
+
+!!! note
+    This flag does not install or select an LTS driver. oneAPI.jl ships pinned rolling
+    NEO artifacts; on an LTS system you point the package at the system libraries instead
+    (see the "Using System Libraries" section of [Installation](installation.md)).
+    `ONEAPI_LTS=1` tells oneAPI.jl to *compile and behave* for that driver.
+
+## What changes
+
+### SPIR-V code generation
+
+Julia kernels are compiled to SPIR-V. By default oneAPI.jl uses [LLVM's SPIR-V
+back-end](https://llvm.org/docs/SPIRVUsage.html) (`SPIRV_LLVM_Backend_jll`). The LTS
+NEO/IGC runtime does not accept that output, so with `ONEAPI_LTS=1` the package switches to
+the [Khronos SPIR-V translator](https://github.com/KhronosGroup/SPIRV-LLVM-Translator)
+(`SPIRV_LLVM_Translator_jll`).
+
+Both tools are dependencies of oneAPI.jl and are resolved lazily by GPUCompiler, so the
+choice costs nothing on either path and needs no reconfiguration of your environment.
+`oneAPI.versioninfo()` reports both, and `@device_code_spirv` will show which one produced a
+given module in its `Generator:` line.
+
+### BFloat16 is unavailable
+
+The LTS SPIR-V stack cannot translate the LLVM `bfloat` type in generic kernels: a kernel
+that merely keeps a `BFloat16` value (`clamp!`, say) fails with an `InvalidIRError`, and
+declaring the `SPV_KHR_bfloat16` extension crashes the LTS runtime outright. On LTS the
+compiler is therefore configured with `supports_bfloat16 = false`, regardless of what the
+device reports — `oneAPI._device_supports_bfloat16()` is a *hardware* capability check and
+does not capture this limitation.
+
+Consequently `BFloat16` is dropped from the element types exercised by the test suite, and
+`examples/bfloat16.jl` exits early with a message. Other floating-point types are unaffected.
+
+### Reductions over strided inputs
+
+The LTS IGC miscompiles non-coalesced (strided) global reads inside the reduction kernel,
+silently producing wrong results — no error, just bad numbers. Reading an array along a
+non-contiguous axis is enough to trigger it, so `sum(transpose(x))`, `a == transpose(b)` and
+`ishermitian(x)` are all affected. Elementwise copies are not.
+
+Two mitigations apply on LTS:
+
+- An input that is not densely laid out (a transposed, permuted or otherwise strided view,
+  or a broadcast containing one) is materialized into a dense `oneArray` before the
+  reduction runs. This costs an extra allocation and copy.
+- A reduction that keeps the contiguous leading dimension (`sum(A; dims=2)` and friends) is
+  routed to a coalesced kernel that assigns one work-item per output slice, so neighbouring
+  lanes read neighbouring memory. Reductions with few output slices get less parallelism
+  than the default workgroup-per-slice kernel, but stay correct.
+
+Reductions that include dimension 1 (`dims=(1,3)`, or a full reduction) keep a contiguous
+innermost axis and use the normal kernel.
+
+### Frees are synchronized against in-flight work
+
+NEO LTS advertises `ZE_extension_memory_free_policies` but does not honor its
+`BLOCKING_FREE` policy: it unmaps an allocation immediately, even with work in flight that
+references it. A garbage-collected free of a dead array whose last kernel has not retired
+then faults on the GPU, which bans the kernel context — after which *every* later submission
+fails with `ZE_RESULT_ERROR_UNKNOWN`.
+
+On LTS, oneAPI.jl keeps a registry of the command queues in use and drains those that could
+reference a buffer before freeing it. Command queues are likewise drained before being
+destroyed; a queue still busy after 10 s is deliberately leaked, since destroying it would
+trigger the very fault the drain prevents. The visible cost is that a GC-driven free can
+block until outstanding work completes.
+
+### Optional: synchronize after every submission
+
+Under heavy multi-process oversubscription of a single tile, a whole-queue
+`zeCommandQueueSynchronize` on the LTS stack does not reliably retire the tail of an
+earlier, separately submitted command list. The result is a silent *dropped tail*: the last
+work-items of a kernel, or the last elements of a copy, never land.
+
+Synchronizing after every submission eliminates it, at roughly a 3× throughput cost. It is
+off by default and enabled with:
+
+```bash
+export ONEAPI_SYNC_EACH_SUBMISSION=1
+```
+
+It can also be controlled at runtime:
+
+```julia
+oneL0.sync_each_submission()            # query
+oneL0.sync_each_submission!(true)       # set, returns the previous value
+
+oneL0.sync_each_submission(false) do    # scoped, restores afterwards
+    # ...
+end
+```
+
+!!! warning
+    A *submit-then-signal* pattern — work submitted with a wait event that is only signaled
+    after submission returns — deadlocks with this enabled, because `execute!` blocks in the
+    synchronize before the gating event can be signaled. No high-level oneAPI.jl code path
+    submits event-gated work, but hand-written Level Zero code may; wrap those regions in
+    `oneL0.sync_each_submission(false) do ... end`.
+
+This workaround is independent of `ONEAPI_LTS` and can be enabled on its own.
+
+## Caveats
+
+- **Precompilation.** The precompilation workload warms up whichever SPIR-V tool
+  `ONEAPI_LTS` selects *at precompile time*, and Julia does not invalidate the cache when an
+  environment variable changes. After flipping the variable, run `Pkg.precompile()` to
+  refresh it. This only affects first-call latency, never correctness.
+- **No effect on the rolling stack.** Every workaround above is gated on the switch, so with
+  `ONEAPI_LTS` unset oneAPI.jl behaves exactly as it does upstream. The workarounds trade
+  performance for correctness and should not be enabled on a driver that does not need them.
+
+## Running the test suite
+
+The test suite reads the same switch and skips the cases the LTS stack cannot support (the
+`BFloat16` element type, the BFloat16 example). On a multi-tile node, pinning each test
+worker to its own tile avoids the oversubscription that provokes the dropped-tail
+corruption:
+
+```bash
+ONEAPI_LTS=1 ONEAPI_TEST_SPREAD_GPUS=1 julia --project=. -e 'import Pkg; Pkg.test()'
+```

docs/src/troubleshooting.md

@@ -36,6 +36,36 @@ Some Intel GPUs (especially integrated graphics) lack native hardware support fo
 - When several processes share one GPU, give each a budget with `ONEAPI_MEMORY_LIMIT`; see
   [Memory Management](memory.md).
 
+### Silently wrong results, or every submission failing
+
+**Symptom**: on a system running Intel's long-term-servicing (LTS) Compute Runtime — such
+as Aurora — reductions over transposed or otherwise strided arrays return wrong values
+without raising an error, kernels drop their last work-items, or every submission starts
+failing with `ZE_RESULT_ERROR_UNKNOWN` after a garbage collection.
+
+**Solution**:
+Enable the LTS workarounds and restart Julia:
+
+```bash
+export ONEAPI_LTS=1
+```
+
+See [Intel LTS Driver Stack](lts.md) for the individual defects, the additional
+`ONEAPI_SYNC_EACH_SUBMISSION=1` switch for oversubscribed tiles, and the performance
+trade-offs involved. Do not enable these on a rolling-release driver, which does not need
+them.
+
+### "InvalidIRError" for a BFloat16 kernel
+
+**Symptom**: a kernel using `BFloat16` fails to compile even though the device reports
+BFloat16 support.
+
+**Solution**:
+The LTS SPIR-V stack cannot translate the LLVM `bfloat` type in generic kernels; the
+device-level check (`oneAPI._device_supports_bfloat16()`) reports hardware capability and
+does not capture this. Use `Float16` or `Float32` on that stack — see
+[Intel LTS Driver Stack](lts.md).
+
 ## Debugging
 
 ### Validation Layer