Intel GPU (Level Zero) engine and Nemotron 3.5 Lightning on Intel Arc (experimental, CLI)

[BHCC2025]

Part 1 of 3. Opened after the issue that proposes the backend and the checks below (#522).

What this adds

An experimental Linux build, zig build -Dxpu, that runs Nemotron 3.5 Lightning 30B-A3B (MLX affine 4-bit, group 64) on an
Intel Arc Pro B70 (Battlemage, 32 GB). OpenCL C kernels are compiled to SPIR-V with ocloc at build time and run through
Level Zero, using the matrix engine for prompt GEMMs, prompt attention and decode attention. It has no CUDA or Metal code in its path.

• zig/src/xpu/: the runtime (driver, context, memory, modules, streams, loader for the checkpoint).
• zig/src/families/nemotron/xpu*.zig: the Nemotron operators and model for this runtime; the family keeps its weights and
  operators, no scheduler is added.
• zig/kernels/xpu/*.cl: the kernels (Mamba2, MoE, attention, 4-bit matvec, GEMMs).
• zig/src/cli/xpu_main.zig: tensorfold-xpu (devices, run on token ids) and the runner tf-xpu-test.
• zig/tests/xpu/: op and model tests (xpu-tests step), tools/zig/xpu_*: forced-token comparison against the reference,
  the long-context needle, the one-process-at-a-time GPU guard, fixture generators and the small shipped reference files.
• docs/recipes/xpu-arc.md, THIRD_PARTY_NOTICES.md and LICENSES/: recipe and notices.

Not in this PR: the native server backend (part 2) and Qwen, EXL3 and GGUF on Intel (part 3). Not supported anywhere: NVFP4 and
FP8-Mamba checkpoints (refused at load with a message), MTP or copy drafts, grammars, prompt cache, other cards.

Gates (CONTRIBUTING.md)

Item	Result
Lean checker	python3 tools/zig/lean_check.py: 512 problems, the same as upstream 1.0.2 (f8fe17d); none added
Build and host tests	zig build -Dxpu, xpu-tests, zig build test: 85 of 86 pass. The one failure, direct_io: direct reads return the same bytes as buffered reads, fails the same way on an unmodified upstream 1.0.2 checkout on the test machine
test-golden	Not applicable: the step exists only in the Metal build graph
Third-party	THIRD_PARTY_NOTICES.md section for the Intel GPU engine; MLX license text in LICENSES/
Exported kernel sources	Measurement comments were removed from the .cl files; the compiled SPIR-V is byte-identical to the commented sources

Exactness

Run with tf-xpu-nem_rows-test, nem_prefill, nem_slot, nem_seam, nem_fp64, nem_attn_dec, nem_attn_pf (see the recipe):

• Windows of 2, 3, 4, 8 and 16 rows give logits bit-identical to one-row windows (40 tokens, including pending-then-commit).
• Prompt chunk sizes 32 to 512 give bit-identical last-token logits; Mamba conv state, SSM state and KV bytes below the prompt are
  byte-identical for chunks of 64, 128 and 512 (640 tokens, real text and random ids).
• Not identical by design: the prefill GEMM path and the single-row path accumulate in a different order; the size of the
  difference is reported in the recipe (worst conv difference 0.68 against magnitude 33, next 8 decode steps up to 0.94 logit,
  same top-1 on real text).
• Independent FP64 reference per projection class (decode, 8-row window and 64-row prefill kernels separately, worst column
  reported): all classes within the test bounds; table in the recipe.
• Fidelity: teacher-forced against the CUDA reference of the same checkpoint, 15 prompts, 1,440 positions: top-1 1,421 (98.7%),
  mean absolute log-probability difference 0.0233; every top-1 miss is listed with both margins. KL divergence against BF16 over
  the first 64 chunks: 0.0864 mean, same top-1 87.24%.
• Generation parity against TensorFold 0.6.4 on CUDA, 41 prompts: 14 identical, 25 of 27 first divergences are bf16 ties or
  within 0.25 logit. Our engine is deterministic (41 of 41 reruns identical).

Receipts

Same machine (Intel Arc Pro B70, Linux 7.0.0 xe, Zig 0.17.0), checkpoint TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
revision d9d758fb83953437f7263256b0d96157e2a348b8, public prompt token ids shipped under tools/zig/prompts/.

Measurement	Result
Decode, 128 greedy tokens, 5-token prompt	5.72 ms/token, 174.8 tok/s (174.2 with 512-row prefill); token digest af29e8d9691a in plain and 512-row prefill
Prefill, CLI, 8,192 tokens	51.4 s token by token, 7.5 s in 128-row windows, 2.65 s in 512-row windows (identical tokens)
Prefill 32,768 tokens	11.3 s in 512-row windows
Decode at 8,192 / 131,072 tokens of context (needle runs)	171.1 / 137.8 tok/s (146.6 / 82.6 with the earlier per-row decode attention, NEM_OLD_DEC=1)
Needle at 50% depth, 8,192 and 131,072 tokens	found; 8,192: prefill 3,103 tok/s, decode 171.1 tok/s; 131,072: prefill 2,285 tok/s, 62 s wall, decode 137.8 tok/s
Device memory	17.85 GB weights, 18.05 GB at 4k, 18.56 GB after 8k, 18.71 GB after 32k, 19.37 GB after 128k
Before / after	Prompt attention on the matrix engine (CLI, 512-row windows, identical tokens afterwards; NEM_OLD_ATTN=1 selects the earlier per-row kernels): 8,192 tokens 3.45 s to 2.62 s, 32,768 tokens 26.6 s to 11.3 s, 131,000 tokens 389.5 s to 57.5 s

Reproduce

zig build -Dxpu && zig build -Dxpu xpu-tests -j4 && zig build test
export TF_FIXTURES_DIR=/path/to/op-fixtures        # generated by tools/zig/xpu_fixtures_*.py; tests skip cleanly without them
TF_GPU_LOCK=/tmp/b70.lock python3 tools/zig/xpu_compare_ref.py      # takes the guard per program itself
tools/zig/xpu_guard.sh 3600 zig-out/bin/tf-xpu-test all
tools/zig/xpu_guard.sh 3600 zig-out/bin/tensorfold-xpu run MODEL --tokens-file tools/zig/prompts/p0.ids --max-tokens 128 --ignore-eos --no-drafts
python3 tools/zig/xpu_nem_needle.py --bin zig-out/bin --depths 50 8192 131072      # takes the guard itself

zig build defaults to -Doptimize=fast (ReleaseFast); every number in this description was measured with it, and a debug build is much slower.

The guard takes the GPU lock, waits while other processes hold over 8 GB of device memory and stops a command only with SIGINT.
Do not wrap a script that takes the guard itself (xpu_nem_needle.py, xpu_needle.py, xpu_spec_identity.sh, and the comparison
scripts when TF_GPU_LOCK is set): the inner guard would wait forever on the lock the outer one holds.

The fixtures contain measurements of TensorFold 0.6.4 on CUDA and slices of public text (wikitext-2, CC BY-SA 3.0); see
tools/zig/fixtures/README.md. No weight slices are shipped.

Out of scope and open questions

• Raising the exact lane width above 1 and copy or MTP drafts on this backend: no claim is made.
• No claim for cards other than the B70, or for 16 GB cards (the weights alone are 17.3 GiB).

[ashhart]

Thank you, @BHCC2025. We'll land this behind -Dxpu as an experimental source build. Nothing here can run an Intel GPU, so our gates cover the host side only, and two things need to change first, on your side where you can rerun your checks:

• The switches that pick other paths or print debug output go: NEM_LAYERS (xpu_win.zig:259), NEM_OLD_ATTN, NEM_SCALAR and NEM_TRACE. The three ARC_* memory settings should become one documented admission rule, like the CUDA server's reserve, rather than switches of their own.
• Decode runs through the window path your exactness checks cover. xpu_engine.zig:80 decodes one row through Model.forward, a second path beside the windows. A width-1 window should be the one path.

The .cl kernels also have about 33 comments that run over several lines, and the lean rule is one line each. Once these are in and the host build passes, we'll merge it. #547 and #550 come after it.

[BHCC2025]

Thank you, @ashhart. Pushed 2d27b63 on top of c69d53f:

• Switches. NEM_LAYERS, NEM_OLD_ATTN, NEM_SCALAR and NEM_TRACE are gone. So is NEM_OLD_DEC, which picked the earlier decode attention in the model loader. NEM_SCALAR selected a second row-kernel path through the windows, so that path and its seven kernels in nem_rows.cl are deleted, along with the --tiny mode of the rows test that existed for NEM_LAYERS. The recipe no longer names any of them.
• Memory. The ARC_* memory settings are one fixed rule: a program does not start while other processes hold more than 8 GB of device memory, with no override. It is stated in the recipe next to the guard script. ARC_STOP_FILE stays, since it is the stop request and not a memory setting.
• Comments. Every comment in the .cl kernels is now one line of at most 120 columns. Only comment lines changed.

Checked on the Arc Pro B70 against c69d53f: tensorfold-xpu run gives the same tokens (128 greedy tokens from a 5-token prompt, sha 127e1d364b02; 3,000-token prompts with 512-row and 128-row prefill, shas 5faacaccc199 and 54ee4e3ead99) at the same speed (175 tok/s). tf-xpu-nem_rows-test has 0 rows differing bitwise from windows of 2 to 16 rows, in place and with commit. tf-xpu-nem_seam-test finds chunk sizes bitwise equal. zig build -Dxpu and xpu-tests compile, and lean_check.py stays at the pinned 512.

One item is not done: decode through a window of one row. I tried it (Engine.feed as a width-1 window, argmax and logits read from the window) and it does not hold up:

• Decode drops from 175 to 78.8 tok/s (12.7 ms/token against 5.7), because a window of one row uses the general matrix-engine GEMMs where the single-row path uses dedicated matvec kernels.
• The head also sums in a different order, so the greedy tokens diverge from the single-row path after about 25 (sha 43008ac20766 against 127e1d364b02). Both are valid roundings of the same logits; the existing rows test measures the gap at 4.3e-2 of the largest logit.

Two ways forward, and I would rather you choose than guess. Either (a) I give the window path width-1 specialisations of the projection and head kernels, so a window of one row is the only decode path and keeps its speed, or (b) you accept the width-1 window as it is, at the lower decode speed. I would do (a), as a separate commit on this PR with the same checks.

[ashhart]

Hello @BHCC2025,

Thank you for the opt-in Intel CLI: the OpenCL kernels compiled to SPIR-V at build time, and the Level Zero loader opened at run time.

Intel Arc is now accepted as a backend through #802's server, landing on the author's receipts plus our host checks. Your CLI waits on one thing. A prompt chunk above 16 rows takes the prefill GEMMs (xpu_win.zig:16), which sum in a different order from the one-row path, so the same prompt cut at a different place can give different bits. When every prompt uses the one-row sum, or the GEMM matches it bit for bit, Nemotron on Arc can come in, ideally on #802's Level Zero runtime rather than a second one.

Thank you again.

[BHCC2025]

@ashhart, done. Every prompt now takes the one-row sum, and the CLI runs on #802's Level Zero runtime. The branch is rebased onto #802 (its head, 25ef56c5):

• One runtime (c936acf). The Nemotron runtime's own loader and ABI declarations are gone. It resolves ze.Api through ze.resolve and keeps its device as a ze.Gpu. ze.zig becomes a build module that the Qwen27 runner and Nemotron share, which is a one-line import change in five of XPU backend: Qwen3.8-27B on Intel Arc Pro B70 (Xe2) over Level Zero #802's files. zig/build/xpu.zig holds both sets of steps. The CLI gives the same tokens as before the move (128 greedy tokens from a 5-token prompt, 2fb8107dc0ce; a 3,000-token prompt with 512- and 128-row prefill, b0d8c69d8fa3) at the same speed.

• Windows take the decode step's arithmetic (5dc655b). The 4-bit projections of a window of up to 16 rows run one row-kernel body with the decode kernels' per-output order:

  • a lane takes the 8-input words l, l+16, …;
  • per word, the dot and the sum of inputs are taken in the decode's order;
  • one sub-group reduction per row;
  • each weight word is read once for all the rows.

  A one-row window equals the decode step bit for bit (tf-xpu-nem_rows-test: max |diff| 0). Windows of 2 to 16 rows are identical, in place and pending-then-committed. The split-K matrix-engine path is gone rather than kept behind a switch. This also covers your earlier point about decode and the width-1 window being two paths: they are now the same arithmetic. Decode keeps the single-row kernels because they are faster (5.54 ms against 6.47 ms for a one-row window) and give the same bits.

• Prompts take the same arithmetic (cd41379). Prompt chunks above 16 rows run nem_prow.cl:

  • every output sums exactly as the decode kernels do;
  • a sub-group holds 8 rows and 8 outputs, so each weight word is unpacked once for 8 rows and each input word once for 8 outputs;
  • routed experts take their slots per expert in the same order;
  • prompt attention runs the decode kernel per row, 16 rows per launch.

  The prefill GEMMs (xpu_pf.zig, nem_pf.cl, nem_attn_pfs.cl) and their test are removed.

Checks on the B70:

• tf-xpu-nem_seam-test. 640 tokens cut into chunks of 16, 17, 100, 128 and 512 rows. The Mamba conv and SSM states and the KV bytes are byte-identical to the row-by-row state, and the next 8 decode logits are identical. Before this change, 128 and 512 differed from rows in 85–95% of elements, with top-1 matching in 6 of 8.
• tf-xpu-nem_prefill-test. Chunks of 32, 100 and 512 give the row-by-row last-token logits.
• Teacher-forced compare against the CUDA reference. 15 prompts, 1,440 positions. The plain run, 512-row prompt windows and 128-row prompt windows all give the same figures: top-1 1,421, mean |d| 0.0233, max 0.514.
• tf-xpu-nem_fp64-test. All classes are within bounds, and the window and prompt kernels give the decode kernels' figures.
• KL against BF16. The first 64 chunks give 0.0866 ± 0.0011, same top-1 87.41%.
• zig build test passes, and the lean count is unchanged from XPU backend: Qwen3.8-27B on Intel Arc Pro B70 (Xe2) over Level Zero #802's head.

The cost is prompt speed. The decode's fp32 arithmetic runs on the vector units instead of the matrix engine:

Prompt	This change	Earlier prefill GEMMs
2,048 tokens	851 tok/s	3,108 tok/s
32,768 tokens	755 tok/s	2,888 tok/s
131,072 tokens	511 tok/s	2,281 tok/s

For reference, the same prompts run at about 215 tok/s in 16-row windows and 167 tok/s one token at a time. A 32,768-token prompt gives the same tokens afterwards as it did with the GEMMs.

[ashhart]

Thank you, @BHCC2025. Nemotron on Intel Arc over #802's Level Zero table rather than a second loader, with prompt rows that sum as the decode step does so a prompt's bits do not depend on where it is cut, is the shape we hoped for.

Your seven commits ship in TensorFold 1.0.12 as they are, merged in ba0499eab on #802 as it landed, so this pull request shows as merged. One commit of ours, 54d3cb9b7, runs zig fmt 0.17 on six of your files, which also renames @intFromEnum to @backingInt in ten calls. It is a preview you build from source; the release archives do not carry it.

We have no Intel GPU, so this lands on your Arc Pro B70 receipts and our host checks: with stand-in tools the XPU builds pass for x86_64 Linux (the server and runner, and every -Dxpu program), the host tests pass, and a scan of the new code against the open kernels we keep found no copied code.

Posted by Ash's AI Assistant Synthena