XPU backend: Qwen3.8-27B on Intel Arc Pro B70 (Xe2) over Level Zero

[Mirrdhyn]

What this changes

A native Intel XPU backend (Arc Pro B70, Xe2, Level Zero) serving Qwen3.8-27B through the lane host, new files only and no core changes. Proposal and questions: #801.

Receipt

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: TensorFold 1.0.9 (c36b536d) + this commit; no MLX or PyTorch at run time; Intel Arc Pro B70 32 GB (power-capped at 230 W), Ubuntu 26.04.1, kernel 7.0.0-38, compute runtime 26.35.39758.10, Level Zero loader 1.32.0, xgrammar v0.2.8, Zig 0.17.0; SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 at 9d189a60, drafter z-lab/Qwen3.8-27B-DFlash2.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: --levels 1,2,4,8, temperatures 0 and 1, four configurations and --parallel 1: 80/80 series equal alone and serial; runner contracts (drafted == plain for chains and trees, 8 slots in shared rounds == solo) 30/30; identical token_sha and output hashes before and after the rebase at 2k to 64k.
• Decode speed before and after, from tools/bench_openai.py: no previous release has this backend; the same port on 1.0.7 → this commit: fibonacci-raw 142.1 → 142.2 tok/s, gpu-chat-no-think 90.3 → 90.3 tok/s. vLLM 0.31.0 XPU on the same card: 78.6 tok/s for one request.
• Prompt speed before and after, from tools/prefill_cold.py: that script is not in the repository, so tools/bench_cache.py --require-cold with exact token arrays and --prompt-cache-gib 0: 2k / 8k / 32k / 64k 1,907 / 1,916 / 1,687 / 1,453 → 2,049 / 1,963 / 1,713 / 1,470 tok/s (vLLM 0.31.0: 1,785 at 1k, 1,730 at 8k).
• Tests run, with pass, fail and skip counts: zig build test 700 passed, 0 failed, 5 skipped (46/46 steps); tools/zig/lean_check.py 538, the 1.0.9 baseline; zig fmt --check clean.
• Not run, and why: Metal and CUDA (no change to their code paths, no hardware here); 128k and the native window (one B70 holds a 69,632-token context at the default prompt-cache budget).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why. (The default path keeps the checkpoint's precision; the W4A8 and W4A4 prompt paths are opt-in and documented with their perplexity cost.)
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (zig build test runs the stream admission rule (xpu_admit.zig) on the host. zig build xpu-tests builds tf-xpu-qwen27-means-test (group means against FP64; fails before 913cf15, passes after) and tf-xpu-qwen27-seam-test MODEL (a prompt cut at 64, 1024 or whole gives the same logits; fails before d1906c3). Both print skipped: and exit 0 without a Level Zero device or a model.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

---

Summary

A native xpu backend for the Zig engine. It drives an Intel Xe2 GPU (Arc Pro B70, 32 GB) directly through Level
Zero, with no PyTorch, SYCL or Python at run time, and serves Qwen3.8-27B (qwen3_5) through tensorfold-native serve
with the DFlash2 drafter. Two checkpoint formats: symmetric GPTQ 4/128 and MLX affine 4/64.

• Opt-in build steps only: zig build xpu-native xpu-qwen27 -Dngen-include=<oneDNN nGEN headers>. Nothing changes
  for Metal, CUDA or HIP builds.
• No core changes: the backend uses the lane host and the lone driver (Lone.resumes, hooks.resumed) as they
  are in 1.0.9; build.zig gains two lines. Every --parallel value, 1 included, goes through the lane host.
• Kernels: OpenCL C compiled by ocloc for the device, plus the int4 prompt GEMMs generated in ISA by nGEN at
  build time (zig/tools/xpu/w4_ngen.cpp); everything is embedded in the binary.
• Serving features: shared rounds across streams with DFlash2 chains or trees per stream, DFlash2 trees for a lone
  stream, prompt cache (--prompt-cache-gib), structured output (xgrammar masks, argmax on the device).
• Optional prompt paths (off by default): W4A8 (XPU_PREFILL_INT8=1, int8 activations on int8 DPAS) and, on
  Hadamard-rotated weights (zig/tools/xpu/rotate_w4.py), W4A4 (XPU_PREFILL_INT4=1).

Docs: docs/xpu/README.md (options, numbers, what is verified), docs/xpu/NAVIGATION.md (code map, request path,
contracts) and docs/xpu/RECIPE-qwen38-27b-b70.md (how the served setup is built).

Contracts

Verified at the same checkpoint, backend, runtime and settings, for the four configurations below (GPTQ, GPTQ + W4A8,
MLX, rotated + W4A4) and for --parallel 1:

• Drafted output equals "draft": false output, token for token: chains and trees, perfect and wrong drafts, greedy
  and sampled (XPU_VERIFY_TEST, and bench_concurrent.py --serial end to end).
• Each concurrent reply equals its solo run: 8 slots in shared rounds, chain windows and tree windows
  (XPU_MULTI_TEST, and bench_concurrent.py --alone at 1, 2, 4 and 8 streams, temperatures 0 and 1).
• A prompt-cache resume gives the same reply bits as an empty cache (same request, next turn, shared system prefix).
• The greedy tokens of the first 32 steps equal an fp32 torch reference of the same checkpoint.
• Host suite: zig build test 46/46 steps, 700/705 tests passed, 5 skipped (the suite's own skips).
• python3 tools/zig/lean_check.py: 538 problems, the same count as the 1.0.9 baseline (0 added); every XPU file is
  at most 600 lines.

The int4 GEMM's reduction order differs from Metal and CUDA (DPAS partial sums per group, then the scale, then the
bias by DPAS); the decode, verify and prompt kernels share one order, which is what makes drafted == plain hold. I have
not built a per-projection FP64 reference yet: tell me which form you want and I will attach it.

Receipts

Machine: Intel Arc Pro B70 (32 GB, power-capped at 230 W), Ubuntu 26.04.1 LTS, kernel 7.0.0-38, Intel
compute runtime 26.35.39758.10 (libze-intel-gpu1, intel-opencl-icd, intel-ocloc), Level Zero loader 1.32.0,
xgrammar v0.2.8, Zig 0.17.0, Python 3.14 for the clients. Checkpoint:
SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 at 9d189a60e4c0ad7f9f47cd94bfa393ca10b3924e (Apache-2.0), served
with its own tokenizer; drafter: z-lab/Qwen3.8-27B-DFlash2 (a mirror of incoai/Qwen3.8-27B-DFlash2), the repository the other backends use. One model process on the machine.

Cold prefill, one stream, --prompt-cache-gib 0, fresh state, 16 output tokens, exact token arrays (the first
2,048 / 8,192 / 32,768 / 65,536 tokens of LICENSE, CONTRIBUTING.md and zig/src/core concatenated, encoded with the
checkpoint's tokenizer), cached_tokens 0 on every request. tok/s = prompt tokens / time to first token.

Configuration	2k	8k	32k	64k
GPTQ, default	2,049	1,963	1,713	1,470
GPTQ + W4A8 (option)	2,757	2,612	2,181	1,792
Rotated weights + W4A4 (option)	3,022	2,879	2,361	1,920

The same port on 1.0.7 (before this rebase): 1,907 / 1,916 / 1,687 / 1,453, identical output hashes.

Served decode (bench_openai.py --tokens 256 --reps 3 --temperatures 0, --parallel 8, GPTQ default): 142.2
tok/s on fibonacci-raw, 90.3 tok/s on gpu-chat-no-think.

Concurrency (bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial):

	1 stream	4 streams (aggregate)	alone / serial
code	142.1 tok/s	320.9 tok/s	15/15 equal, 2/2 equal
chat	90.2 tok/s	228.5 tok/s	15/15 equal, 2/2 equal

Against vLLM 0.31.0 XPU on the same card and checkpoint (official image, MTP 4, fp8 KV, prefix cache off), same
clients:

	vLLM 0.31.0	This backend (GPTQ default)
Prefill 1k / 8k	1,785 / 1,730 tok/s	1,985 / 1,924 tok/s
Decode, 1 / 8 requests	78.6 / 464.2 tok/s	165.5 / 552.5 tok/s
Code, 1 / 4 / 8 streams (greedy)	62.4 / 199.0 / 195.2 tok/s	133.2 / 309.6 / 270.4 tok/s

No previous TensorFold release has an XPU backend, so the before/after comparison above is this port on 1.0.7 versus
1.0.9. Not run: 128k and the native window (the B70 holds a 69,632-token context with one stream at the default
prompt-cache budget); Metal, CUDA and HIP (no change to their code paths).

Notes

• New files only, under zig/src/xpu, zig/src/native/xpu*.zig, zig/kernels/xpu, zig/tools/xpu, zig/build/xpu.zig,
  tools/xpu and docs/xpu.
• zig/kernels/xpu/qwen27/triton/prompt_attention.spv is the Triton prompt-attention kernel of the Python port, kept as
  SPIR-V behind XPU_ATTN_TRITON=1 for comparison; the default is the backend's own DPAS kernel.
• The checkpoint, its tokenizer and the GPTQ recipe are by SergiioB; thanks to them.

[ashhart]

Hello @Mirrdhyn,

Thank you for the Qwen3.8-27B backend on Intel Arc Pro B70 (Xe2) over Level Zero. It serves qwen3_5 in MLX affine 4/64 and symmetric GPTQ 4/128, behind the lane host (xpu-native), with xpu-qwen27 as the check runner. A kept prompt prefix is restored in the prompt pass (xpu.zig:96, xpu_lanes.zig:188). The store uses planned marks on a grid of 64 (xpu.zig:146). The KV cache stays bf16 unless XPU_KV_INT8 is set (qwen27.zig:77). A lone stream with a drafter uses one path (xpu.zig:142). The prompt GEMMs are generated at build time by your nGEN tool (zig/build/xpu.zig:38). The change sits on v1.0.9, and the trial merge onto that base is clean. This pull request is not in 1.0.10. We have not built it. We do not have that card.

Two things have to change before this is ready.

1. Take zig/kernels/xpu/qwen27/triton/prompt_attention.spv out of the tree and out of the build. It is 56048 bytes, and the tree has no source for it. The docs call it the Python port's Triton kernel. The build compiles it on every XPU build (zig/build/xpu.zig:28) and embeds it (zig/build/xpu.zig:75). Runner.init loads that module on every start (qwen27.zig:183, the kernel at qwen27.zig:204). XPU_ATTN_TRITON only chooses the launch (forward.zig:296). Outside libraries stay at runtime. Nothing vendored or captured goes into the repo. Keep the OpenCL kernel attn_prompt (forward.zig:294).

2. slotsThatFit has to refuse a stream that does not fit. It uses a fixed card of 31 GiB (xpu.zig:112, 31 << 30) and returns at least one slot when the remainder is 0 (xpu.zig:116). openLanes uses that count (xpu.zig:121). Read the device's free memory, and return no slot when the stream does not fit.

Lines past 120 columns, the longest in docs/xpu/README.md:73, qwen27_main.zig:155, forward.zig:87, qwen27.zig:190, and xpu_lanes.zig:58, can be wrapped at landing. Runner.init opens device id 0xe223 (qwen27.zig:171). The comment on init says the first B70.

We cannot run this here. On an Intel Arc Pro B70, with Level Zero, ocloc for device bmg, and the nGEN headers, serving SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 on the bf16 cache, the receipts we can take are:

• a drafted reply equals "draft": false
• a resumed prompt equals a fresh one
• concurrent streams equal their solo runs
• speed against llama.cpp or IPEX-LLM on the same card

The prompt path includes the chunked GDN kernel (gdn_chunk.cl:1, C = 64). Those receipts cover it. With XPU_KV_INT8 the docs say the answers are not bit-identical to the bf16 cache. The three equalities above still have to hold in that mode. The default stays bf16. The numbers in docs/xpu/README.md are yours. We have not re-run them. The XPU steps are not on zig build test.

Thank you again.

[Mirrdhyn]

Thanks for the review. Both changes are in 25ef56c:

• zig/kernels/xpu/qwen27/triton/prompt_attention.spv is gone from the tree and the build, with its module, kernel and XPU_ATTN_TRITON; the prompt attention is attn_prompt only.
• slotsThatFit reads the device's memory through Sysman (Gpu.memory: zesMemoryGetState on the device's on-board modules, matched by UUID). The free memory it uses is the smaller of what Sysman reports and the module size less what the engine allocated: the driver backs an allocation when it is first written, so Sysman alone over-counts (a 262,144-token context opened on a card it could not hold). It returns 0 when the margin and the prompt cache do not fit, and open refuses with a message. On the B70 with the GPTQ checkpoint: 7 streams of 16,384 tokens, 2 of 65,536, and 196,608 tokens or a 40 GiB prompt cache are refused.
• The comment on Runner.init now says what it opens (the first Arc Pro B70, PCI id 0xe223). I left the long lines for landing, as you suggested; tell me if you'd rather have them wrapped in the branch.

Receipts on an Arc Pro B70 (power-capped at 230 W), Level Zero, ocloc for bmg, nGEN headers, SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 with its own tokenizer, this branch's binary:

	bf16 cache (default)	XPU_KV_INT8=1
Drafted reply equals "draft": false (perfect drafts, wrong drafts, DFlash2)	true (16 / 127 / 27 rounds)	true (16 / 127 / 26 rounds)
Concurrent streams equal their solo runs (bench_concurrent.py --alone --serial, 1/2/4/8 streams, code and chat, temperatures 0 and 1)	16/16	16/16
Resumed prompt equals a fresh one (kept prefix, later turn, shared system prompt)	equal hashes	equal hashes

Speed against llama.cpp on the same card (build 8c2134a, unsloth/Qwen3.8-27B-GGUF Q4_0, llama-bench -ngl 99 -fa 1 -r 3; ours: cold prompts with --prompt-cache-gib 0, decode without a drafter from xpu-qwen27, with DFlash2 from the server):

tok/s	llama.cpp Vulkan	llama.cpp SYCL	this backend
prompt 1k / 4k / 8k	694 / 590 / 486	814 / 805 / 789	1,988 / 1,966 / 1,936
decode, no drafter	17.0	27.6	34.9
decode with DFlash2 (server, one request)	–	–	165

IPEX-LLM is archived, so I did not run it.

[Mirrdhyn]

Comparison as you set it (Qwen3.8-27B, Arc Pro B70 power-capped at 230 W, PR head 25ef56c). llama.cpp 8c2134a: llama-bench -p 2048,8192,32768,65536 -n 128 -r 3 -fa 1 -ngl 99, Q4_0. TensorFold: cold prompts (--parallel 1 --prompt-cache-gib 0), decode bench_openai.py --tokens 128 --reps 3, greedy, median.

	2k	8k	32k	64k	decode 128
llama.cpp SYCL, Q4_0	788	787	696	600	27.7
llama.cpp Vulkan, Q4_0	645	484	aborted at 32k (Vulkan fence wait)	—	not reached
TensorFold XPU, GPTQ 4/128, no drafter	2,051	1,985	1,719	1,473	33.5
TensorFold XPU, DFlash2					122.9 (code) / 99.3 (chat)

The three decode runs give the same rate to 0.1 tok/s. Q4_K_M is not measured on the 27B. The FP64 reference for the int4 GEMM follows in a separate comment.

[ashhart]

Hello @Mirrdhyn,

Thank you for the Qwen3.8-27B server on the Arc Pro B70, and for taking the captured SPIR-V out of the tree and sizing streams from the device's free memory.

This is accepted. It lands on your B70 receipts plus our host checks: build, host tests, lean, privacy, and the copy scan. We wrap native/xpu.zig:179, docs/xpu/README.md:43, and docs/xpu/NAVIGATION.md:111 at landing. KV stays bf16 unless the int8 cache is asked for.

Your FP64 reference for the int4 prompt GEMM, per projection class, is welcome. That reduction order differs from Metal and CUDA.

Thank you again.

[Mirrdhyn]

One change since 25ef56c, in 913cf15: the int4 group means are stored ×256 so the low half of their fp16 hi+lo pair stays normal (it fell into subnormals on layers 0–1 of down/mixer_out, up to 1.6e-4 beyond the fp16 rounding, where an fp32 sequential sum gives 3.0e-5). repack and repack_gptq divide B by 256; no GEMM kernel changed. It does not hold for a group mean above 255 or an MLX bias below 2.4e-4 in magnitude; neither occurs in either checkpoint.

FP64 reference (XPU_FP64_REF=1, references from core/quant): the same 24 prompt rows, real activations, layers 0, 1, 31 and 63 and the head. Prompt and decode are compared separately on identical input rows. Every native path is within the fp16 rounding of the output (rel L2 2.1–2.9e-4 vs FP64, the same as fp16(FP64)). Beyond that rounding, no native error is worse than an fp32 sequential sum:

checkpoint	projection	beyond the fp16 rounding (rel L2 vs fp16(FP64)), worst layer: decode 1 row / decode 8 rows / prompt / fp32 sequential	worst column rel L2 vs FP64 on the prompt rows: native / fp32 sequential / fp16(FP64)	≤ 1 ulp, worst native path	max abs / rms
GPTQ 4/128	mixer_in	1.1e-05 / 1.1e-05 / 1.1e-05 / 2.7e-05	4.5e-04 / 4.5e-04 / 4.5e-04	99.9 %	8.9e-03
GPTQ 4/128	mixer_out	2.2e-05 / 1.8e-05 / 1.8e-05 / 2.9e-05	4.3e-04 / 4.3e-04 / 4.3e-04	99.9 %	3.3e-02
GPTQ 4/128	gate_up	2.2e-05 / 1.4e-05 / 1.3e-05 / 2.3e-05	4.5e-04 / 4.5e-04 / 4.5e-04	100.0 %	9.7e-03
GPTQ 4/128	down	1.4e-05 / 1.3e-05 / 1.3e-05 / 3.6e-05	4.2e-04 / 4.2e-04 / 4.2e-04	100.0 %	3.3e-02
GPTQ 4/128	head	7.7e-06 / 1.2e-05 / – / 2.0e-05	4.7e-04 / 3.8e-04 / 3.8e-04	100.0 %	1.7e-03
MLX 4/64	mixer_in	1.4e-05 / 1.2e-05 / 1.3e-05 / 2.3e-05	4.2e-04 / 4.2e-04 / 4.2e-04	100.0 %	9.4e-03
MLX 4/64	mixer_out	9.2e-06 / 1.5e-05 / 1.4e-05 / 1.1e-04	4.4e-04 / 4.4e-04 / 4.4e-04	100.0 %	3.6e-02
MLX 4/64	gate_up	2.0e-05 / 1.3e-05 / 1.3e-05 / 2.1e-05	4.3e-04 / 4.3e-04 / 4.3e-04	100.0 %	1.2e-02
MLX 4/64	down	1.4e-05 / 1.8e-05 / 1.6e-05 / 3.3e-05	4.3e-04 / 4.3e-04 / 4.3e-04	100.0 %	4.5e-02
MLX 4/64	head	7.5e-06 / 1.2e-05 / – / 1.9e-05	4.4e-04 / 3.7e-04 / 3.7e-04	100.0 %	2.3e-03

The head runs on decode only, so its worst column is over 8 rows, against 24 for the other references.

Teacher-forced fidelity (XPU_FIDELITY=1, 3 texts × 512 tokens, every top-token difference with its FP64 margin logged): top-1 differs from FP64 at 22 of 1,536 positions on the prompt path and 27 on the decode path. The FP64 margins at those positions run from 9e-4 to 1.5 logits, mostly below 0.1. The prompt and decode paths differ from each other about as often (31 positions), so neither path is the source. The patch leaves these counts within run-to-run spread.

Receipts on 913cf15 (Arc Pro B70 power-capped at 230 W, SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 with its own tokenizer):

	bf16 cache	XPU_KV_INT8=1
Drafted reply equals "draft": false (perfect drafts, wrong drafts, DFlash2)	true (16 / 127 / 27 rounds)	true
Concurrent streams equal their solo runs (bench_concurrent.py --alone --serial, 1/2/4/8 streams, code and chat, temperatures 0 and 1)	120/120	120/120
Resumed prompt equals a fresh one (kept prefix, other question, later turn, shared system prompt; full and empty cache)	equal hashes (5/5)	equal hashes (5/5)

MLX 4/64, W4A8 (XPU_PREFILL_INT8=1) and --parallel 1 are also green on all three. Speed is unchanged: cold prompt 2,056 / 1,992 / 1,722 / 1,473 tok/s at 2k / 8k / 32k / 64k; decode 33.5 without a drafter, and 122.9 code / 99.2 chat with DFlash2.

[Mirrdhyn]

Three commits since 913cf15.

d1906c3, prompt seams. The rule you set on #523 also applied here: the server cut prompts at message boundaries, and a chunk of 16 rows or fewer ran the decode kernels (serial GDN, decode attention and norms; the int4 GEMMs were already bit-equal between the two paths, checked on real weights). The same ids sent as a chat or through /v1/completions gave different logits on 4 of 8 prompts. Now the engine cuts every prompt on a 64-row grid from position 0, the prompt-cache marks sit on that grid, and every prompt chunk runs the prompt kernels at any length; a rest under 128 rows joins the last chunk. After it: chat equals /v1/completions with the same ids on 8 of 8, a prompt cut at 64, 1,024 or 8,192 gives the same logits, and the receipts hold (drafted equals "draft": false, concurrent equals solo 120/120 in bf16, XPU_KV_INT8, W4A8 and MLX, resumed equals fresh 5/5). Cold prompts are within 1 % (2,048 / 1,979 / 1,718 tok/s at 2k / 8k / 32k). One cost: a prompt a few tokens past a multiple of 512 pays one more pass over the weights (about 17 ms, −3 % at 1,032 tokens).

402d09a, dad16a9, tests. Stream admission is a pure function with host tests in zig build test. zig build xpu-tests adds two device tests that fail before their change and skip without a device or a model (see the checklist).

[Mirrdhyn]

Two commits since dad16a9.

Fidelity reference (618b196). To be precise about what the earlier fidelity counts measure: the XPU_FIDELITY reference replaces every int4 GEMM with fp16(FP64) of the same weights and rows; norms, GDN, attention and the online rotation are the device's. The counts measure the GEMMs' share of the drift, not the whole model's. The code comment that said otherwise is fixed.

GDN prompt path (489baa2). Measuring the state recurrence itself against FP64 (XPU_FP64_REF, new: the prompt's chunked GDN and the serial decode on the same real rows, layers 0, 1, 30 and 62, prompts of 512 and 4,096 rows) showed the chunked path was not as precise as the serial one it replaces: it rounded its fp32 operands (T·diag(βe^g), W, S, Uᵀ, P) to bf16 before the DPAS. Beyond the output's bf16 rounding, y was 2.1–3.0e-3 off (80–89 % within 1 bf16 ulp) against 0.9e-6–4e-5 for decode, and the final state 1.6–5.1e-3 against 4e-7–2e-6. Each fp32 operand now enters the bf16 DPAS as three terms (h + m + l, six products when both sides are split), with W, HS and VN stored as three planes. After it, y is 1.2e-6–1.4e-5 beyond rounding at 512 rows (up to 8.5e-5 on one layer at 4,096 rows), 100 % within 1 ulp, and the final state 2–9e-7, no worse than decode in any layer. A two-term split was not enough (0.7–1.5e-4).

The cost is prompt speed: cold 1,929 / 1,855 / 1,632 / 1,404 tok/s at 2k / 8k / 32k / 64k, 5–6 % below the previous head; decode is unchanged (33.6 tok/s, DFlash2 123.2 code / 99.1 chat). All receipts were rerun on 489baa2: drafted equals "draft": false, concurrent equals solo 120/120 in bf16, XPU_KV_INT8, W4A8 and MLX, resumed equals fresh 5/5, chat equals /v1/completions 8/8, both device tests pass, and teacher-forced top-1 differs from the reference at 22 of 1,536 positions on the prompt path and 20 on the decode path.

[Mirrdhyn]

GDN prompt path, cheaper form (17facba). The three bf16 terms of 489baa2 cost the 35B MoE 14–21 % of its prompt, where the GDN weighs more. Each pair of fp32 operands now enters fp16 DPAS as hi + lo terms after a power-of-two row scale (the row's largest magnitude in [2^14, 2^15), its exponent kept per row and undone on the fp32 result): three products instead of six for S·Wᵀ and P·U, two instead of three for Q·Sᵀ, with W, HS and VN stored as two fp16 planes; the products against bf16 k and v keep three bf16 terms.

Measured against FP64 on y before its bf16 rounding (the fp32 output of the chunked kernel and of the serial decode on the same rows; layers 0, 1, 30 and 62; 512 and 4,096 rows; GPTQ and MLX), the prompt path is at or below decode in every layer (1.0–4.6e-7 against 1.2–6.3e-7), with one cell above decode but below an fp32 sequential sum (3.2e-7 against 2.8e-7 decode and 3.4e-7 sequential). The final state is 2.2e-7–1.0e-6 against 3.8e-7–2.2e-6 for decode, and 100 % of the bf16 outputs are within 1 ulp.

Cold prompt is 1,937 / 1,878 / 1,643 / 1,416 tok/s at 2k / 8k / 32k / 64k (489baa2: 1,929 / 1,855 / 1,632 / 1,404; dad16a9, before the GDN fix: 2,048 / 1,979 / 1,718 at 2k–32k); decode is unchanged (33.6 tok/s, DFlash2 123.0 code / 102.3 chat). All receipts were rerun on 17facba: drafted equals "draft": false, concurrent equals solo 120/120 in bf16, XPU_KV_INT8, W4A8 and MLX, resumed equals fresh 5/5, chat equals /v1/completions 8/8, both device tests pass, and teacher-forced top-1 differs from the reference at 21 of 1,536 positions on the prompt path and 23 on the decode path.

[ashhart]

Thank you, @Mirrdhyn, for separating GEMM and recurrence fidelity and reporting the prompt-speed cost of the new precision path.

At 489baa2, the earlier safety gaps remain. zig/src/xpu/forward.zig:77 floors scratch sizing: XPU_CHUNK=66 with 193 prompt rows still pads beyond capacity. Context splits also round down. zig/src/xpu/slot_states.zig:111 can leave a half-created slot, and its snapshot error path frees memory before queued writes finish. Mapping bounds in zig/src/xpu/dev.zig:25 and runner/device cleanup in native close also remain open.

Please send a follow-up closing these allocation, input-validation and cleanup paths before we move this into landing checks.

Posted by Ash's AI Assistant Synthena