Native server: --backend xpu for Nemotron on Intel Arc

[BHCC2025]

Part 2 of 3, stacked on part 1 (#523: the Intel GPU runtime and the Nemotron engine). Issue: #522. Part 1 is not merged yet, so this branch contains its commit; review the second commit only (5ef2493, 9 files). Opened as a draft until part 1 is settled.

What this adds

The Intel GPU engine as a lane backend of tensorfold-native serve, so serve MODEL --backend xpu answers OpenAI-style
requests. It is registration plus one backend; no scheduler is added beside the engine.

• zig/src/native/xpu.zig: registers the family nemotron_h / format mlx-q4g64 and the chip class intel-<PCI id>, reads
  device memory for admission, builds the lane engine and api.LaneHost (one lane, one stream, exact width 1).
• zig/src/native/xpu_lane.zig: the backend behind lanes.backend.Backend, generic over a family spec (prefill in 512-row windows, first,
  read, verify as one-row windows, keep, release); zig/src/native/xpu_nemotron.zig is the Nemotron spec. Greedy uses the device argmax. For temperature above 0 the step's logits are
  drawn on the host with the lane core's keyed sampler (temperature, top_k, top_p, min_p, seed; keyed by seed,
  absolute token position and candidate id). Presence and repetition penalties are not implemented.
• zig/src/server/cli.zig, main.zig: the xpu backend choice and the usage text; README.md: one line about the backend.
• Nothing under zig/src/lanes/ or in the other families changes.

Requests queue and run one at a time. Not supported: drafts, tool-call gates, grammars, structured output, prompt cache.

Gates (CONTRIBUTING.md)

Item	Result
Lean checker	512 problems, the same as upstream 1.0.2; none added
Build and host tests	zig build -Dxpu native-xpu, xpu-tests, zig build test: 88 of 89 pass; the failure is upstream's direct_io test (same on an unmodified 1.0.2 checkout on this machine)
Concurrent reply == solo run	bench_concurrent.py --levels 1,4 --alone --serial: 12 of 12 four-stream replies equal their solo runs, 3 of 3 for one stream, 0 failed (code and chat prompts)
Resumed == fresh, cancel leaves the next request valid	A client that disconnects mid-stream cancels its request; the next greedy request gives the same digest as before
Seeded sampling	Same seed gives the same tokens (digest 7a7c26a1ac4e twice); another seed differs (8650291a63e2)
Shutdown	SIGINT during a generation: the stream ends after its current token with an error event and [DONE], a queued request is answered with the server is shutting down, exit status 0 in 0.11 s, 0 MB of device memory left
test-golden	Not applicable (Metal build graph only)

Receipts

Same machine and checkpoint as part 1 (Intel Arc Pro B70, Linux 7.0.0, Zig 0.17.0, revision d9d758fb83953437f7263256b0d96157e2a348b8),
greedy, 256 tokens, tools/bench_openai.py; cold prefill through /v1/completions with exact token-id arrays, --prompt-cache-gib 0,
cached_tokens 0, 8 tokens generated.

Measurement	Result
Decode through the server	171.8 tok/s (raw code prompt), 171.9 tok/s (chat prose, thinking off); first token after 0.09 to 0.10 s. CLI for comparison: 174.8 tok/s
Concurrency, 1 stream / 4 streams	170.9 and 171.1 tok/s (code, chat) / 163.3 and 163.4 aggregate; the last of four waited about 4.9 s for its first token
Greedy digest	ca2e7e412e5d for the test prompt, identical alone, three at once, after a disconnect
Cold prefill 2k / 8k / 32k / 64k / 128k	0.66 / 2.65 / 11.35 / 24.7 / 57.6 s (3,108 down to 2,276 tok/s); decode after it 173 / 168 / 160 / 152 / 137 tok/s
Native window (262,080-token prompt)	147.7 s prefill (1,774 tok/s), decode 113.6 tok/s
Device memory	17.85 GB weights, 18.53 GB at --context 4096, 19.37 GB at 131,200, 20.25 GB at 262,144
Host memory	338 MB resident after load, 355 MB peak after the 262,080-token prompt (62.2 GiB RAM host, 24 threads); load ready in 7 s
Standard server (llama.cpp Vulkan, build 9c2e0e491, Mesa 25.2.8, Q4_0 GGUF)	cold prefill 1.57 / 6.93 / 41.8 / 126.5 / 432.2 s; decode 44.5 / 40.5 / 31.0 / 23.4 / 15.6 tok/s at the same lengths; a different quantization, full table in the recipe
Previous release	None exists for this backend

Reproduce

zig build -Dxpu native-xpu
tools/zig/xpu_guard.sh 7200 zig-out/native-xpu/bin/tensorfold-native serve MODEL --backend xpu --context 131200 --prompt-cache-gib 0 --port 18393 &
python3 tools/bench_openai.py http://127.0.0.1:18393 MODEL_NAME --tokens 256 --reps 3 --temperatures 0
python3 tools/bench_concurrent.py http://127.0.0.1:18393 MODEL_NAME --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial
curl -s http://127.0.0.1:18393/v1/completions -H 'content-type: application/json' --data-binary @cold_65536.json   # exact token-id arrays
curl -s http://127.0.0.1:18393/metrics | grep device_memory

zig build defaults to -Doptimize=fast (ReleaseFast); every number in this description was measured with it, and a debug build is much slower. (The tensorfold-native binary keeps its HTTP side at ReleaseSafe, as upstream does.)

xpu_token_arrays.py writes the cold-prefill arrays (the first L tokens of a public text under the model's tokenizer).
tensorfold-native capabilities --json reports "backends": ["xpu"], chip intel-e223, family nemotron_h, format mlx-q4g64.

What moved (a reviewer's guide)

The lane backend that lived in zig/src/families/nemotron/xpu_lanes.zig is now zig/src/native/xpu_lane.zig, generic over a family spec (its Engine type and the rows of one prompt window): the sampler, the round logic and the two sampler
tests are the same code. zig/src/native/xpu_nemotron.zig is the Nemotron spec (family table row, loader with the NVFP4 message, 512-row windows) and zig/src/native/xpu.zig dispatches on the model type over a list of specs.
This keeps a second family (part 3) to one added file. The family's own xpu.zig no longer exports the lane. Behaviour for Nemotron is unchanged: the server gates above were rerun on this layout.

Digests

The CLI prints the SHA-256 of the generated ids written as a JSON list ([1, 2, 3]) and --report stores it; the server's token_sha is the SHA-256 of the ids joined by commas (1,2,3). The same tokens give different digests.
tools/zig/xpu_token_sha.py REPORT.json prints both for one token list. The CLI digest quoted in part 1 (af29e8d9691a) and the server digest above (ca2e7e412e5d) belong to different prompts and are not to be compared.

Notes for review

• The running stream's error event at shutdown carries the core's raw name ShuttingDown; the queued request gets prose. Left as is
  because the text comes from lane_host.zig, which this PR does not touch.
• Exact width is 1 and one stream is served at a time; no claim is made for larger widths.

[ashhart]

Thank you, @BHCC2025. This comes after #523. Before it can land, the server path needs to be lane-native like every other backend's: Nemotron's drafts (its MTP head and copies) through the lane core, and concurrent requests sharing rounds, with drafted equal to "draft": false and concurrent equal to solo. One request at a time with no drafts is the serial path our rules keep out of the server. Once #523 is in, we'll bring this up to date with main ourselves.

[ashhart]

Hello @BHCC2025,

Thank you for the Arc server entry and the B70 receipts, including the four-stream queue and the seeded draws.

The server path still decodes one stream, one row, with drafts off. xpu_lane.zig:77 sets mtp false, speculate false, exact_width 1, and max_streams 1. draft refuses (xpu_lane.zig:164). A prompt longer than 16 tokens is prefilled in windows (xpu_lane.zig:115), and each decode round is one row. There is no prompt-cache restore, so a second turn does not keep the first prompt.

Please serve through the lane core the way the other backends do: Nemotron's MTP head and copy drafts on, concurrent requests in one round, a resumed turn equal to a fresh prompt, and drafted output equal to a run with drafts off. A push with those is the update we need. The B70 is the machine for the receipts.

Thank you again.

[BHCC2025]

@ashhart, here is the update, measured on the Arc Pro B70 with nemotron-3.5-lightning-mlx4.

What changed underneath. The branch is rebased onto #802 (its head, 25ef56c5) and onto #523 as updated today. Two consequences:

• One runtime and one server.
  • Nemotron now runs on XPU backend: Qwen3.8-27B on Intel Arc Pro B70 (Xe2) over Level Zero #802's Level Zero table instead of a second runtime.
  • XPU backend: Qwen3.8-27B on Intel Arc Pro B70 (Xe2) over Level Zero #802's native/xpu.zig lists nemotron_h (format mlx-q4g64) beside qwen3_5 and opens it on its own lane host in native/xpu_nem.zig.
  • Nothing in the core or the server CLI changes. Use --backend auto, as with XPU backend: Qwen3.8-27B on Intel Arc Pro B70 (Xe2) over Level Zero #802.
• Windows and prompt rows take the decode step's arithmetic (Intel GPU (Level Zero) engine and Nemotron 3.5 Lightning on Intel Arc (experimental, CLI) #523).
  • A one-row window equals the decode step bit for bit.
  • A prompt gives the same bits however it is cut.

The three things you asked for, all through that server:

1. Drafts go through the lane core. With the checkpoint's mtp-4bit.safetensors loaded, the host serves windows of up to 16 rows:

   • verify runs the pending token and its drafts as one window;
   • keep commits the path;
   • draft absorbs the kept rows into the head and holds its chain.

   Results:

   • Identity: every drafted reply's token_sha equals its "draft": false run, 8 of 8 (256 tokens; greedy and seeded at T 0.8 / seed 7; code, prose, list, copy). A server started with --no-drafts gives the same seeded token_shas, 4 of 4.
   • Plain speed: a one-row round is the decode step itself (5.54 ms, against 6.47 ms as a window), so plain replies with the head loaded run at 173.0–174.3 tok/s.
   • Drafted speed: on this MoE each extra row costs about 3.7 ms, because more distinct experts are read. Drafted greedy runs at 0.90–0.99× plain. plain_guard keeps plain replies at decode speed. A real gain needs expert reads shared across rows, which is follow-up work.

2. Concurrent requests share rounds. Each stream has its own Mamba, KV and head state.

   • Identity: four concurrent requests (greedy and seeded) return what each returns alone, 4 of 4 in each of two rounds (tools/zig/xpu_concurrent_receipt.py).
   • Speed: 72.0–73.6 tok/s aggregate, against 83.8 run one after another. Each stream's window is still its own forward pass.

3. Prompt cache. Prompts are planned in 512-row chunks. A state is kept at each chunk end and restored by device copy. The budget is --prompt-cache-gib, else 4 GiB within the free memory reported by ze.Gpu.memory, less 2 GiB.

   • Identity: in a two-turn chat over a 6,685-token document, turn 2's token_sha equals the same turn on a server with --prompt-cache-gib 0, greedy and seeded (tools/zig/xpu_resume_receipt.py).
   • Speed: turn 2 resumes 6,656 tokens and prefills in 0.21 s instead of 8.67 s.

zig build xpu-native builds the server with both families. zig build test passes, and the lean count is unchanged from #802's head.

[ashhart]

Hello @BHCC2025,

Thank you for the native server on the Intel engine, and for the fast fixes on the Qwen pull.

Intel Arc is accepted through #802's server. One Level Zero runtime, and one 27B forward. This host is one stream, one row, no draft head, and no prompt cache (xpu_lane.zig:77, xpu_lane.zig:164). #802's server sizes streams from the card's free memory, restores a kept prompt, and runs DFlash2 when a drafter is attached.

Nemotron waits until a prompt wider than 16 rows uses the same sum as one row. Today the wider projection takes the dense GEMM (xpu_win.zig:324). It comes onto #802's runtime with that. EXL3, GGUF, and the MTP head from the Qwen pull come onto that runtime too.

Thank you again.

[ashhart]

Thank you, @BHCC2025. Serving Nemotron on Arc through the lane core, with MTP and copy drafts, shared rounds and a prompt cache whose resumed turn equals a cold one, turns the Arc path into a real server.

Your commits ship in TensorFold 1.0.12 as they are, merged in 4b5cad41c after #523, so this pull request shows as merged. One commit of ours, d3d1dc875, updates two comments that still named the prefill GEMMs #523 removed. Like #523, it is a preview you build from source.

We have no Intel GPU, so this lands on your receipts and our host checks, as #523 does.

One gap we noticed: the two sampling tests in native/xpu_lane.zig sit in a module no test step builds, so they never run. A host test step for that module would let them.

Posted by Ash's AI Assistant Synthena