Qwen3.8-27B on Intel Arc: MLX 4-bit, EXL3 and GGUF (experimental)

[BHCC2025]

Part 3 of 3, stacked on part 1 (#523) and part 2 (#547). Issue: #522. Parts 1 and 2 are not merged yet, so this branch contains their commits; review the last commit only (91 files).

What this adds

Qwen3.8-27B (Gated DeltaNet + attention, 64 layers) on the Intel GPU engine from three weight formats behind one linear layer: MLX affine 4-bit,
EXL3 (trellis) and GGUF (ggml block quantization: Q2_K, Q4_K, Q5_K, Q6_K, Q8_0, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL).

• Decode on all three; windowed decode and speculative decoding (an MTP head from the EXL3 or GGUF pack, copy drafts, and an auto drafter that picks between them; there is no automatic fallback to plain decoding); the KV cache in bf16, q8 or q4
  (q8 is the default); prompt windows on the matrix engine (--prefill ROWS|auto); the delta-rule window kernel with 2 or 4 value rows a sub-group.
  The copy drafter finds its match through an n-gram index, so the cost of a draft no longer grows with the history; it returns the same match as the full scan up to a 512-candidate chain bound (qwen_copyidx: 0 of 9,000 calls differ from the scan, 0.0001 ms a call against 0.72 ms for the scan at 123K tokens).
• Native server: tensorfold-native serve MODEL --backend xpu now also opens Qwen3.8-27B MLX 4-bit and EXL3 directories (one stream at a time, host sampling, no drafts), through the same
  lane backend as Nemotron. The server resolves directories, so GGUF files are served by the CLI only. The KV format is the environment variable TENSORFOLD_XPU_KV (bf16, q8, q4; default q8).
• Context: at most 131,072 positions. The model's configuration names 262,144; the engine does not hold that, and a larger --context is refused with a message that says so.
  No claim is made for the 262,144 window.
• Files: zig/src/families/qwen3_5/xpu*.zig, zig/src/xpu/exl3.zig, ggml.zig, zig/kernels/xpu/{qwen_*,exl3*,ggml_*}, zig/src/native/xpu_qwen.zig (the Qwen entry of the lane host),
  zig/tests/xpu/{qwen_*,exl3,ggml*}.zig, tools/zig/{xpu_needle.py,xpu_spec_identity.sh,xpu_fixtures_ggml.py,xpu_fixtures_exl3.py}, the Qwen section of docs/recipes/xpu-arc.md, and LICENSES/ plus the notices for ggml/llama.cpp,
  ExLlamaV3 and MLX.

Not in this PR: GGUF through the native server; drafts on the server; other Qwen3.5 sizes (refused before loading); 16 GB cards.

Gates (CONTRIBUTING.md)

Item	Result
Lean checker	513 problems against upstream's 512: the one added is zig/kernels/xpu/ggml_tables.h, 632 lines of a generated lookup table (generator tools/gen_ggml_tables.py). The exemption is the question in #522; a split of the table under 600 lines is ready if it is refused.
Build and host tests	zig build -Dxpu, native-xpu, xpu-tests, zig build test: 89 of 90 pass; the failure is upstream's direct_io test, which fails the same way on an unmodified 1.0.2 checkout on this machine.
drafted == plain	xpu_spec_identity.sh, token for token, 18 runs of the script, all PASS, on the exported tree. Short prompts (7 prompts, 299 tokens, k=4): EXL3 mtp, GGUF IQ3_S mtp, MLX mtp with the EXL3 head, MLX copy drafts, EXL3 auto. Long prompts (2048-row windows on both runs, k=3 and k=4, 80 tokens): copy, auto and mtp drafters on EXL3 and GGUF IQ3_S at 8,192 and 16,384 tokens (12 runs), and mtp k=4 (64 tokens) at 32,768 tokens on EXL3, GGUF IQ3_S and MLX with the EXL3 head. The copy-drafter index agrees with the full scan on 9,000 random calls (qwen_copyidx).
Window rows == one-row decode	qwen_rows: 2, 3, 4, 8, 16 rows plus pending-then-commit, 0 differing values, for MLX, EXL3 and Q5_K_XL. The delta-rule window kernel at 1, 2 and 4 rows a sub-group is bit-identical at 16 to 2048 rows (qwen_gdnrows).
Queued forwards == synced	qwen_slot: 0 of 248,320 logits differ.
Concurrent reply == solo run (server)	12 of 12 replies at four streams and 3 of 3 for one stream equal their solo runs, MLX and EXL3, 0 failed; a disconnect leaves the next request valid; seeded sampling repeats; SIGINT exits 0 in 0.1 s with 0 MB held.
Seams	prompt chunks of 64, 128 and 512 rows are byte-identical for MLX (state and decode logits). EXL3 has a named seam at 256 rows (below). Prefill against row by row is not bit-identical by design; sizes below.
FP64 reference per projection class	qwen_fp64, MLX 4-bit, real weights, decode / 8-row window / 64-row prefill kernels: all 90 rows within the bounds (history below). EXL3 and the GGUF types have their own oracle tests (below).
Fidelity	teacher-forced top-1 over 4 prompts x 48 steps against a reference made by another engine on the same checkpoint (not BF16): 189/192 MLX, 188/192 EXL3, 190/192 IQ3_S GGUF. The reference logs are not shipped, so these counts cannot be reproduced from the tree; how they were made is in the recipe (TensorFold 0.6.2 CUDA for MLX and EXL3, llama.cpp build 11427 for the GGUF). BF16 KL divergence (reference BF16 via llama.cpp, --prefill 512, bf16 KV, first 64 chunks only, not the full corpus, measured with a build bit-identical to this export: 16,320 of 16,320 rows identical to the prototype's, max log-prob difference 0, 0 same-top flips): recipe C 0.0437, MLX 4-bit 0.0450, EXL3 3.00 bpw 0.0436. Full-corpus figures from the engine's KL harness: recipe C 0.0497, MLX 4-bit 0.0536, EXL3 3.00 bpw 0.0645.
test-golden	Not applicable (Metal build graph only).
Third-party	THIRD_PARTY_NOTICES.md section for the Intel GPU engine; license texts for ggml, ExLlamaV3 and MLX in LICENSES/. The fixtures: see below.

Exactness details

• EXL3 at 256 rows. qwen_seam (640 tokens of real text): chunks of 64 and 128 are byte-identical; chunks of 512 differ from them. Below 256 rows the EXL3 projections use the decode kernels in 16-row steps; from 256 rows they use the matrix-engine
  prefill GEMM, and the fp16 gate projections switch kernels at the same size. In a scratch build with both removed, chunks of 64, 128 and 512 agree bit for bit, so these two switches are the whole cause. They are kept, because unifying them
  would change the bits of every prompt chunk above 16 rows. Size: decode logits after the prompt differ between chunks of 128 and 512 by at most 2.07e-1 (largest logit 22.8), top-1 equal in 7 of 8 steps. The test fails on any chunk-size
  difference unless told where a path change is expected (differ=256). It is not new: the earlier clean-clone runs show EXL3 chunks of 64 and 512 differing in the same way.
• Prefill vs one row at a time (640 tokens): 74% to 99% of the state elements differ at rounding scale (worst conv difference 0.31 against a maximum of 62); the next 8 decode steps differ by up to 0.39 in logit on MLX (top-1 equal in 8 of 8)
  and 0.27 on EXL3 (7 of 8).
• FP64 bound. The first run of qwen_fp64 failed four one-row decode rows (gdn.out, gdn.z, attn.q, mlp.down) under a bound that divided each column's error by that column's own maximum, which is too strict for a single-row check
  of a column whose exact value cancels to about 4e-5: its error of 2e-6 to 4e-6 is 3e-8 to 5e-8 of the sum of the absolute terms, below one fp32 rounding unit. The test now divides by the larger of the column's maximum and 1e-4 of the class maximum
  and still prints the old figure beside the new one. With that rule the aggregate error is 1.3e-3 to 2.0e-3 in all 90 rows; the worst column is 3.9e-3 for window and prefill kernels and 1.6e-2 for one-row decode, against a bound of 2e-2; error relative to the term sum is at most 3.5e-4.
  The Nemotron test of part 1 keeps the rule it was published with; its table was rerun and is unchanged.
• EXL3 and GGUF oracle tests. tf-xpu-exl3-test: 32 cases, 27 synthetic trellises and five rows of real tensors (fixtures from tools/zig/xpu_fixtures_exl3.py: synth needs no checkpoint, real reads TF_EXL3_DIR; nothing from weights is committed, and without the real slices the five real-tensor cases do not run), decode bit-exact where checked, worst relative error 5.7e-7, row-invariance differences 0.
  tf-xpu-ggml-test: all 14 block-type fixtures including the new Q8_0 and IQ4_NL: dequantization and quantized-activation differences against libggml 0; matvec within 3.4e-7 of float64 (fp32 output) and 3.5e-3 (bf16); window rows 0 differences (m = 2..8);
  chunked prefill 0 differences.
• Speculative decoding at long context. With drafts the prompt is prefilled in the same big windows and the MTP head then absorbs its rows one at a time, which costs more than a plain prefill: EXL3, 32,768 tokens, 29.4 s plain against 55.6 s with the head; 120,000 tokens, 137.7 s against 256.6 s. Drafted == plain holds at 32,768 tokens with 2048-row windows on both runs (64 tokens, k=4, token for token; EXL3, GGUF IQ3_S and MLX with the EXL3 head all PASS; EXL3 digest 4d63633f4fb0 for both), and in the mtp-test at 8,000 tokens (k=3, identical, 100% of the drafts accepted). The two runs must use the same prompt windows: a plain run that prefills in 16-row windows and a drafted run that prefills in 2048-row windows are different prefill paths (the prefill-GEMM seam above) and can differ at near-tie tokens. At 120,000 tokens on EXL3 the drafted run decoded 33.6 tok/s against 21.6 tok/s plain on this prompt (64 tokens, 40 of 86 drafts accepted, identical digest ded18707a820). On a GGUF mix (recipe D19pm, the same family as recipe C and also unpublished) with the prototype build (arc-engine main 1c7fec1, ReleaseFast, q8 KV, needle prompts at depth 50%, 80 tokens, k=3, plain and drafted runs from the same prefill path; not rerun on this tree), decode in tok/s for plain / MTP drafts / the auto drafter: 20.94 / 41.40 / 44.94 at 65,536 tokens; 19.93 / 37.01 / 38.40 at 98,304; 19.29 / 35.88 / 31.62 at 122,880. MTP drafts are 1.86x to 1.98x faster than plain at every depth measured there; the auto drafter's figure at 122,880 used the full-history copy scan that the copy-drafter index of this PR replaces (that run is being repeated). The drafted prefill costs more than the plain one there too (plain / drafted: 57.6 / 137.0 s at 65K, 95.6 / 222.1 s at 98K, 128.3 / 293.2 s at 123K). There is no automatic fallback to plain decoding; drafts only change speed, never the tokens. No claim is made that drafts win or lose at every long length, and nothing above 32K is gated for identity.

Digests

The CLI prints the SHA-256 of the generated ids written as a JSON list ([1, 2, 3]); the server's token_sha is the SHA-256 of the ids joined by commas (1,2,3). The same tokens give different digests.
tools/zig/xpu_token_sha.py REPORT.json (part 2) prints both. For the MLX p0 run the CLI digest is 67f96813a817 and the server's a4fda3139fa0; for EXL3 f579aeec19ba and 294fb1554dd4; the server's completions for the same prompt hash to
those values, so the two produce the same tokens (KV q8 and bf16 gave identical tokens for this prompt on both).

Receipts

Intel Arc Pro B70, Linux 7.0.0 (xe), Zig 0.17.0, host with 62.2 GiB RAM and 24 threads. Checkpoints: TensorFold/Qwen3.8-27B-MLX-4bit revision 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c;
turboderp/Qwen3.8-27B-exl3 branch 3.00bpw revision 6fe61ad620abfe97c5b49f9722c2bceeea4ccc28; Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf from ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF; Qwen3.8-27B-UD-Q5_K_XL.gguf, the Unsloth UD-Q5_K_XL GGUF as downloaded on 2026-09-12 (20,876,938,144 bytes, sha256 8601193d3d5760c37fb8ce1b43afebc69df5fb24e1fbc5a547c32e2200305276; repository and revision not recorded). KV q8 unless stated.

Recipe C, used for the KV and long-context rows, is not a published checkpoint: an importance-weighted llama-quantize mix of the BF16 GGUF (12.56 GB; importance matrix and per-tensor type overrides from a sensitivity sweep, described in the recipe). Its per-tensor type table is not shipped, so it cannot be rebuilt from this tree.

Measurement	MLX 4-bit	EXL3 3.00 bpw	GGUF Q5_K_XL
Device memory at load	15.7 GB	13.2 GB	21.1 GB
Decode, CLI, 128 tokens	30.8 tok/s	27.0 tok/s	23.8 tok/s
With MTP drafts, k=4 (identical tokens)	77.6 tok/s (EXL3 head)	73.7 tok/s	not measured
Decode through the server (code / chat)	30.4 / 30.4 tok/s	26.5 / 26.5 tok/s	not served
Four streams, aggregate	29.1 / 29.8 tok/s	25.5 / 26.1 tok/s
Cold prefill 2k / 8k / 32k / 64k / 131,008 (server)	1.46 / 5.86 / 25.7 / 57.4 / 139.0 s	1.72 / 6.87 / 29.6 / 65.3 / 154.8 s
Decode after those prompts, tok/s	29.4 / 29.3 / 28.1 / 26.3 / 23.4	25.4 / 25.1 / 24.5 / 23.4 / 21.0
Device memory after the 131,008-token prompt	21.17 GB	18.99 GB
Host memory (MLX)	415 MB resident after load, 1.80 GB peak at load

• Prompt attention old-vs-new (same 8,000 and 32,768 tokens, 1,024-row windows, bf16 KV, tokens identical in every row), prefill seconds, default / ARC_PF=old / ARC_PF=1: MLX 6.69 / 7.08 / 8.41 at 8k and 27.9 / 34.7 / 44.7 at 32k;
  EXL3 7.68 / 8.08 / 9.48 at 8k and 31.9 / not run / 48.8 at 32k. F16_NOPF=1 on EXL3: 7.94 s at 8k, 33.0 s at 32k.
• Needle (EXL3, q4 KV): found at 4,032 and 131,008 tokens, prefill 4.7 s and 199.6 s, peak 16.23 GB. A probe at three depths (54 cells: recipe C and MLX 4-bit, bf16/q8/q4 KV, 32K/64K/120K prompts) found the code every time. It is one needle at three depths, not a measure of retrieval quality.
  Timed cells of that probe (depth-50 prompts, 64 generated tokens, each run alone, measured with a prototype build whose optimize mode is being confirmed, not on this tree), prefill s / decode tok/s / peak GB, bf16 | q8 | q4: recipe C at 32K 25.6 / 30.45 / 15.98 | 25.5 / 32.07 / 15.09 | 25.5 / 32.92 / 14.55;
  at 120K 130.0 / 22.57 / 21.88 | 127.4 / 26.22 / 18.23 | 126.5 / 28.35 / 16.21; MLX 4-bit at 32K 25.7 / 26.88 / 18.64 | 25.4 / 28.63 / 17.75 | 25.4 / 29.33 / 17.21; at 120K 130.0 / 20.57 / 24.54 | 127.1 / 24.01 / 20.89 | 126.2 / 25.76 / 18.87.
  Prefill barely depends on the KV format; decode is faster with a smaller cache and more so at depth (recipe C at 120K: q8 +16%, q4 +26% over bf16). llama.cpp's Vulkan backend shows the opposite (q8_0 KV halves its decode at 32K depth),
  so this is a property of each engine's kernels, not of KV quantization itself.
• KV default. q8 is the default because a paired KL study on recipe C (mixed-document corpus; 16K and 32K chunks, 5 chunks at 32K; one quantization; no needle retrieval) gave -0.0002 for q8 against the bf16 cache (within one standard error) and saves 0.5 to 1.0 GB;
  q4 gives +0.0024 (+3.8%) at 16K and +0.0032 (+7.4%) at 32K, same-top-token 0.2 points lower, and saves 0.8 to 1.55 GB. q4 is an opt-in mode for pushing context, not a free saving.
• Against llama.cpp (Vulkan, build 9c2e0e491, Mesa 25.2.8, -ngl 99 -fa, -c 123392 -np 1, f16 KV, prompt cache off, needle-style prompts rather than the arrays above): on recipe C cold prefill 5.28 s at 2k, 60.75 s at 16k, 537.9 s at 64k, 1,654.8 s at 120k, decode after the prompt 13.56 to 7.18 tok/s; on the stock IQ3_S file
  5.25 s at 2k, 60.3 s at 16k, 536.1 s at 64k, 1,651.2 s at 120k, decode 11.39 to 6.48 tok/s. No engine number on prompts of those lengths has been measured on this tree (an earlier figure came from a prototype build whose optimize mode is not confirmed, and is not quoted).
  The full tables are in the recipe. This is an order-of-magnitude comparison, not a matched benchmark. There is no previous release for this backend.

Reproduce

zig build -Dxpu native-xpu && zig build -Dxpu xpu-tests -j4 && zig build test
export TF_FIXTURES_DIR=/path/to/fixtures        # python3 tools/zig/xpu_fixtures_ggml.py OUT_DIR (GGUF files and libggml-base) and xpu_fixtures_exl3.py OUT_DIR (TF_EXL3_DIR for the real slices)
TF_GPU_LOCK=/tmp/b70.lock python3 tools/zig/xpu_compare_ref.py --family qwen --model MODEL --ref ref_qwen27.json    # also ref_qwen27_exl3.json, ref_qwen27_gsq.json: not shipped
tools/zig/xpu_guard.sh 3600 zig-out/bin/tf-xpu-qwen_rows-test MODEL 40
tools/zig/xpu_guard.sh 3600 zig-out/bin/tf-xpu-qwen_seam-test MODEL 640 64 128 512 differ=256        # EXL3; MLX needs no differ=
tools/zig/xpu_guard.sh 3600 zig-out/bin/tf-xpu-qwen_fp64-test MLX_DIR
tools/zig/xpu_spec_identity.sh MODEL 299 4 mtp                    # takes the guard itself; IDS_FILE, PREFILL_ROWS and CONTEXT select a long prompt
tools/zig/xpu_guard.sh 7200 zig-out/native-xpu/bin/tensorfold-native serve MODEL_DIR --backend xpu --context 131072 --prompt-cache-gib 0 --port 18393 &
python3 tools/bench_openai.py http://127.0.0.1:18393 MODEL_NAME --tokens 256 --reps 3 --temperatures 0
python3 tools/bench_concurrent.py http://127.0.0.1:18393 MODEL_NAME --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial

zig build defaults to -Doptimize=fast (ReleaseFast); every number in this description was measured with it, and a debug build is much slower. (The tensorfold-native binary keeps its HTTP side at ReleaseSafe, as upstream does.)

tools/zig/xpu_token_arrays.py writes the cold-prefill arrays (use the Qwen checkpoint's tokenizer). Do not wrap a script that takes the guard itself (xpu_spec_identity.sh, xpu_needle.py, the comparison scripts when TF_GPU_LOCK is set) in xpu_guard.sh.

Fixtures: the ggml and EXL3 real-tensor fixtures hold raw quantized rows of real tensors and their libggml dequantization. They are generated locally by xpu_fixtures_ggml.py and are not in the tree; the tests skip cleanly without them.
The picks are rows of output.weight and blk.17.ffn_gate.weight of the Q5_K_XL file for Q8_0 and IQ4_NL and the equivalent tensors of the other files for the older types; the generator names each tensor and row range.

Open questions

• The exemption for ggml_tables.h (see Proposal: an Intel GPU (Level Zero) backend, starting with Nemotron 3.5 Lightning on Intel Arc #522).
• Whether the native server should also open a single .gguf file; today it resolves directories only.
• Speculative decoding is not offered by the server; the CLI path is described above.
• Whether to offer an official higher-bit recommendation for 32 GB cards is left out of this PR on purpose; accuracy-per-GB recipe data (the D14 to D19p mixes, Q5_K_XL, EXL3 at 4 and 5 bpw) is not claimed here.

[ashhart]

Thank you, @BHCC2025. This comes after #523 and #547, and two parts can't come in:

• The q8 KV default (xpu_qwen.zig:17 returns .q8 when TENSORFOLD_XPU_KV is unset). KV stays bf16, since we don't trade precision for memory.
• The GGUF tables generated from llama.cpp's source (ggml_tables.h) and the ggml block code in ggml_quant.cl. We don't take code or generated data from outside projects. The ExLlamaV3 and ggml licences the PR adds read as carrying their code, so EXL3 should build on the reference decoder from core: backend-neutral EXL3 trellis reference decoder with an oracle fixture #496 instead.

MLX 4-bit and EXL3 on #496's decoder can follow once #523 and #547 have landed, within the lean rules: files under 600 lines and one-line comments.

[ashhart]

Hello @BHCC2025,

Thank you for Qwen3.8-27B on the Arc, and for the row checks on MLX, EXL3, and Q5_K.

The KV cache defaults to q8. bf16 runs only when TENSORFOLD_XPU_KV=bf16. Please default to bf16, and make q8 and q4 opt-in. A reconstructed value that agrees within rounding is a different cache.

EXL3 projections use the decode kernels below 256 rows and the prefill GEMM at 256 (xpu_win.zig:346, xpu_win.zig:355). The compare already moved one top-1 in eight steps at that seam. Please use one EXL3 prompt kernel at every length.

The server facts are still exact_width 1, mtp false, and max_streams 1 (xpu_lane.zig:77). This stays blocked on the #547 host: one stream, one row, no draft head, no prompt cache.

zig/kernels/xpu/ggml_tables.h is 632 lines. Please split that generated table under 600.

A push with the bf16 default and the single EXL3 kernel is the update we need.

Thank you again.

[BHCC2025]

@ashhart, thank you. All three are in, on top of this branch's head:

KV cache defaults to bf16 (1213fd5). The CLI (--kv), the server (TENSORFOLD_XPU_KV) and the engine default to bf16; q8 and q4 are opt-in. The MTP head's cache now follows the target's format too (it used to drop to q4 past 16K with a bf16 target); MTP_KV still overrides it.

One prompt kernel at every length (06c33ff). The 256-row switch is gone: EXL3 and fp16 projections take their prefill GEMMs at every chunk size. Going further than the seam you named, a prompt's last piece of 16 rows or fewer used to run on the decode-window kernels, so a 648-token prompt cut 512+136 and one cut 10×64+8 still differed at the tail rows. Prompt rows now go through Model.promptRows, and every prompt piece, whatever its size, takes the prompt kernels (when the window has them, --prefill above 16); MLX and GGUF take theirs for any piece size as well. Decode and verify windows keep the decode kernels. qwen_seam no longer takes differ=, and it now reads .gguf files. On the B70, states, KV bytes and the decode logits after the prompt are byte-identical, top-1 8 of 8:

• EXL3: 640 tokens in chunks of 64, 128 and 512; 648 tokens (an 8-row last piece) in 64, 128 and 512; 600 tokens in 17, 64, 100, 256 and 512.
• MLX 4-bit, Q5_K_XL and IQ3_S: 648 tokens in 64, 128 and 512.

Unchanged: windows of 2-16 rows equal one-row decode (qwen_rows, EXL3/MLX/Q5_K_XL, 0 differ), the slot ring (0 of 248,320), qwen_gdnrows, tf-xpu-exl3-test (32 cases, 0 mismatches) and tf-xpu-ggml-test (prompt rows cut 13+27 equal one pass). Drafted == plain (xpu_spec_identity.sh) passes for EXL3 mtp, IQ3_S mtp, MLX copy and EXL3 auto at k=4, and for EXL3 mtp with 512-row prompt windows.

ggml_tables.h split (e76c56a): ggml_tables.h (375 lines) includes ggml_tables_iq1s.h (259). The generator the header names, tools/gen_ggml_tables.py, is now in the tree and writes both; the values are byte-identical to the old table. The lean checker drops from 513 to 512.

The server facts stay as they are until the #547 lane host is updated; that's next.

[BHCC2025]

@ashhart, closing this in favour of #802, which already covers Qwen3.8-27B on Arc with a better runner. Thanks for the reviews here.

Most of #550 is now in #802 or no longer needed. The one piece #802 lacks is EXL3 weights. On the B70 our EXL3 decode runs at 30.8 tok/s against #802's 4-bit at 34.9, and its only gain is about 4 GiB less VRAM. That isn't worth a separate path, so further Qwen-on-Arc work from us will go into #802's runner. The Nemotron side continues in #523 and #547, both now rebased onto #802.

[ashhart]

Hello @BHCC2025,

Thank you for the three fixes. The KV cache defaults to bf16, and q8 and q4 are opt-in (xpu_qwen.zig:17). Every prompt piece takes the prompt kernels (xpu_model.zig:139, xpu_win.zig:344). ggml_tables.h is 375 lines, and the rest is ggml_tables_iq1s.h.

Intel Arc is accepted through #802's server. One Level Zero runtime, and one 27B forward. EXL3 (xpu_qwen.zig:9), GGUF on the CLI (xpu_main.zig:209), and the MTP head (xpu_mtp.zig:158) are ahead of that server. It loads MLX affine 4/64 and GPTQ 4/128, and it drafts with DFlash2. Those three come onto #802's runtime. The copy index and the auto drafter (xpu_spec.zig:51, xpu_spec.zig:183) come with the MTP head. This server stays one stream, one row, and no draft head (xpu_lane.zig:77).

Thank you again.