Repository navigation
Qwen3.5/3.8 on CUDA: decode 37-73% faster on #836's engine, bits unchanged (on #836) - #978
Closed
BobClawblaw wants to merge 39 commits into
Closed
BobClawblaw wants to merge 39 commits into
BobClawblaw wants to merge 39 commits into
Conversation
- zig/src/platform: `sys`, imported by every module (zig/build/sys.zig wires it). On POSIX it is std.c, std.posix and std.DynLib unchanged; on Windows win_c.zig gives libc's surface over the UCRT and Win32 (integer descriptors, directory descriptors, readdir, pread/pwrite, high-resolution sleep, monotonic clock, /tmp and /proc/self/exe mapped), win_posix.zig the server's sockets over Winsock, console events as SIGINT/SIGTERM, and LoadLibraryW for DynLib. - std.c and std.DynLib calls go through `platform` (mechanical). - build.zig routes Windows targets to the CUDA build. - Server: socket closes and writes are explicit, the key file's mode check is skipped on NTFS (ACLs), descriptor limits are a no-op. - shard_edit: the shard's map is released before the rename, since Windows refuses to replace a mapped file. - Tests that need symlink privilege or mode bits skip on Windows. zig build test-cuda-host passes on x86_64-windows.
- Kernel fatbins build with nvcc and cl.exe: MSVC's conforming preprocessor is passed on Windows hosts (CUDA 13's headers need it), and explicit kernel instantiations carry the __restrict__ qualifiers their templates have (MSVC counts them in the function type; the generated code is unchanged). - Nemotron's checkpoint source keeps io_uring to Linux; elsewhere its existing thread reads run. - pull_exlr8 raises no descriptor limit on Windows. - modelopt test: the lane's slice partials are allocated. GPUs without clusters (before sm_90) reduce split K through them; the test passed none, so it wrote through a null pointer on Ampere and Ada. zig build native and install build with -Dnvcc=... -Dsm=89, and tf-cuda-test's synthetic suites pass on an RTX 4090.
- pull: where Windows refuses the snapshot's symlink, the snapshot path
becomes a hard link to its blob (CreateHardLinkW through `sys.c.link`):
no extra space, and learning already refuses linked shards. Link
failures are logged with their reason.
- HOME falls back to USERPROFILE (Windows sets no HOME) for the hub cache
and every libc getenv("HOME").
- Tests read optional fixture variables through `platform.getenv`;
std.process.Environ.getPosix does not build on Windows.
- The learned-state symlink tests skip on Windows (symlinks need
privilege); `sys.c.symlink` asks for an unprivileged one.
zig build test passes on x86_64-windows.
- docs/WINDOWS.md: requirements, build, run, how the port works and how it differs from Linux; the README points to it. - -Dstrip reaches tensorfold-native before its install step is made, so the step no longer expects a .pdb the stripped build does not write. - zig fmt on the platform files.
8080 and 8000 are taken by too many other servers; serve listens on 8888 unless --port says otherwise, and the docs and bench scripts follow.
The MLX affine 4-bit checkpoint on the native CUDA server, beside Nemotron: qwen35.cu holds the Metal engine's glue at its bf16 rounding points (gated delta rule, causal convolution, gated attention), cuda.qlinear's affine-4 kernels do the projections, sample.cu draws on the device. Windows of up to 16 rows record each row's recurrence and convolution tail, so the lane core shares rounds across streams and keeps partial copies; a window's rows equal one-row steps bit for bit. draw.zig is the host draw both backends now share. On an RTX 4090: 210 tok/s for one stream, 1250 tok/s across eight; logits agree with an fp32 reference at every position of a 36-token check.
…mory The 2B's CUDA engine now reads its widths from config.json (cuda_geometry.zig): linear value heads over fewer key heads, more query heads a key/value head, an untied head, an A_log stored as bf16. The 27B's logits agree with an fp32 reference at every position of a 36-token check, and its windows equal one-row steps bit for bit; the 2B's output is unchanged. Two Windows driver costs shaped the rest. Each kernel launch grew with the process's allocation count (1.2 ms a launch over 1,900 weight buffers), so a model's weights and scratch now live in one arena (cuda_weights.zig packs each projection through a reused staging buffer: the 27B loads in 25 s, not 200). And a process's first forwards through a buffer stall for seconds, so the engine warms every request path at load and the backend keeps a pool of stream caches that `reserve` allocates and runs a row through once the server knows its stream count. On an RTX 4090: the 27B decodes at 37 tok/s in the runner, 31 to 35 tok/s served, 210 tok/s across eight streams; the 2B stays at 210 and 1,250.
A forward's launches (the embedding to the kept rows' logits) are captured the first time a shape (rows, windows, recording) is seen and replayed after; the attention chunk count moves into the per-forward table so a graph stays valid as caches grow. Prompt chunks stay uncaptured: their GEMMs dwarf the launches, and each length would be a graph of its own. A new graph's first run pays the Windows driver's stall (about a second on a wide model), so `reserve` also captures the shared rounds' shapes, a row of each of 1 to n streams, at load. Bit for bit the same logits; on an RTX 4090 the 2B steps in 2.8 ms instead of 5.3 (served: 324 tok/s, 1,560 across eight streams) and the 27B in 23.8 ms instead of 26.9 (served: 40 tok/s, 243 across eight).
--drafter loads z-lab's DFlash2 draft model behind the lane core's Drafted wrapper: the target writes each row's residual after the drafter's five tapped layers into a ring a stream, the drafter folds those rows into a context of keys and values, and after every verify it drafts a block of seven tokens past the pending one (the target's own embedding and head at its ends, a codebook selector tracing one path through each row's best sixteen). Its projections are quantized to affine-4 on the host at load and run through the target's kernels; its glue is qwen35_draft.cu. The Windows driver stalls a buffer's first kernel and a new graph's first run for seconds, so the drafter pools its contexts and a captured forward now serves every verify shape: its launches cover a bucket of rows and every window, and the kernels read the true counts from the device. On an RTX 4090 the 27B serves 52 to 58 tok/s greedy with drafts against 40 plain, about 100 tok/s on long replies, the same tokens either way; plain decoding keeps its 24 ms step. The 2B is unchanged.
A verify window's rows may form a tree (each row's parent given): the convolution taps, the delta rule and attention follow each row's own branch, and the DFlash2 drafter fills the lane core's planned lanes (each lane its parent's rank-r choice by the codebook selector), so a lone stream verifies up to 12 lanes a round and the planner learns which shapes land. A verify no longer snapshots every row's state (1.3 GB shared, which capped the 27B at 8 rows): each stream logs its rows' decay, write and key, and keep replays the kept path onto the recurrence in one pass (and moves a tree's kept keys down), so any window width fits. One-row steps and prompts update in place as before. The lane host times the target's windows at load, which the depth rule had lacked (it had been capping chains at 3 drafts). The runner's new tree mode verifies tree windows with junk branches against step logits (bit-exact on both models).
What the port serves beyond Nemotron: Qwen3.5-2B and Qwen3.8-27B with its DFlash2 drafter, the speeds measured on an RTX 4090, the load time and why, the timing and chain-only switches, and what is not there.
The lean rule (tools/zig/lean_check.py) counts a comment that runs on to a second line; the Windows port's new files had a hundred such runs. Each is one line of at most 120 columns now, and the checker reports the same 539 problems as main.
direct_io opened files with O_DIRECT on Linux alone. On Windows the same open now asks for FILE_FLAG_NO_BUFFERING (which win_c maps O_DIRECT to), and the 4 KiB-aligned spans it already reads satisfy the sector alignment that needs; a file system that refuses it still falls back to buffered reads. The direct-read test runs on Windows too, checking the executable's bytes against buffered reads of it.
SIG.IGN is a function pointer made from 1, as std.posix has it. aarch64 requires function pointers to be aligned unless the type says align(1), so the cross-build for aarch64-windows refused it. With align(1), as std's own handler type has, tensorfold-native cross-compiles for aarch64-windows with the GB10 (sm_121) fatbins; no hardware has run it.
The Windows page opens with what the build is, what it is tested on and that it cross-compiles for aarch64-windows; the README's model table and the CUDA paragraph say Windows where they said unofficial port.
The no-GPU workflow runs zig build test on windows-latest beside ubuntu-latest, so the platform layer is built and tested on every pull request.
Every other module calls libc through `sys`; the drafter's four getenv calls were the last direct std.c calls in a file the Windows build compiles.
After a prompt the lane core hands the drafter every prompt row's features; the CUDA backend refused any request wider than its tap ring (2,176 rows), so a 27B request with --drafter and a prompt past that ended with RowsNotHeld, where the same prompt without the drafter ran. The ring keeps the newest tap_rows rows, which hold the drafter's whole context window, and absorb already reads only that window of what it is handed; the backend now refuses only rows not yet written or all gone. On an RTX 4090 the 27B with its drafter takes a 6,041-token prompt in 7.4 s, and its reply equals the one with "draft": false.
… attention cuts chunks at logical positions - rowanc held a window's rows as bits of a u32, but prompt chunks run up to 128 rows: rows from 32 on rotated at the wrong positions and attended the wrong keys. A chain row (no parents) now carries its index in its window under the top bit; tree windows (at most 16 rows) keep their bits. - tf_q35_attn_partial cut its 128-key chunks at a key's cache row, so a kept tree path whose window crossed a chunk boundary summed in other chunks than one-row steps and changed bits. Chunks are now cut at the key's logical position (the stream's length plus the ancestors before it). Checked on an RTX 5090 (Linux, sm_120), tf-qwen35-cuda over 300 ids on Qwen3.5-2B and Qwen3.8-27B: step bits unchanged, window and tree equal step byte for byte (tree differed from row 126 on the 2B and 254 on the 27B before). Prompt mode's last row against step's at 16 to 300 ids: KL at most 0.003 and the same argmax in 20 of 20 (before: 27B at 40 ids KL 2.3 with a different argmax, at 128 ids KL 1.35).
zig build test-golden and zig build fuzz-cortheon exist on every non-macOS host since the golden step moved into zig/build/golden.zig, so the golden check compiles against the platform layer on Windows: - the platform's posix gains sockaddr.in, timeval, connect and kill for a child's handle (TerminateProcess; HUP has no equivalent), setsockopt turns a timeval for SO_RCVTIMEO or SO_SNDTIMEO into Winsock's milliseconds, and a socket read that times out is WouldBlock, as std.posix reports it. - the check carries SystemRoot into the spawned server's environment, which Winsock needs to start, and drains the server's stdout after the port line on every host: the fake server logs each request there, and a full pipe (4 KiB on Windows) blocked it after sixteen cases. - on Windows the check cannot send SIGTERM or SIGHUP, so the sigterm-exit-0 outcome and the cases after a key rewrite are skipped and printed as such. On Windows the goldens give 306 of 315 equal with the same nine known differences as Linux, and the fuzz step passes its eight tests; the no-GPU workflow's Windows job runs both.
The loader tried only libcublasLt.so.13 and libcublasLt.so, so the cublaslt suite of tf-cuda-test reported LibraryUnavailable on Windows; there it opens cublasLt64_13.dll from the toolkit's bin folder.
The platform handed O_APPEND to the C runtime as _O_APPEND, whose write seeks to the end and then writes, so two threads appending at once could overwrite each other's bytes: the request log lost lines under concurrent requests, which the server goldens' request-log case caught on a slow runner. A descriptor opened with O_APPEND now writes through WriteFile at the end-of-file offset, which the kernel places whole, as POSIX does; the file keeps its write access, so ftruncate on such a descriptor still works (prompt_imprint repairs a torn index that way). A request_log test appends 200 lines from each of 8 threads and expects every line whole; it lost lines on Windows before this change.
The lifecycle/request-log outcome said only that the lines differ; it now gives both logs' line counts and the first line that differs, cut to 100 bytes each, so a mismatch seen on a CI runner can be read from its log.
A checkout with core.autocrlf, as GitHub's Windows runners make, gives the frozen request log CRLF line endings while the server writes LF, so the lifecycle/request-log case differed at its first line's end on windows-latest and nowhere else. Both logs compare with CRLF read as LF; the other goldens are parsed as JSON and never saw the endings.
…'s widest stream The chunk width came from the longest stream or window in the forward, so a 201-key row cut its keys into 128-key chunks alone and 256-key chunks beside a stream past 8192, and a window crossing 8192 recut its earlier rows; either changed the fp32 sums. Each row now takes chunk_keys(its logical keys), the width its one-row step uses, in both the partial and the merge kernels, and the partials sit at a fixed max_chunks stride. tf-qwen35-cuda takes TF_Q35_PREFILL=N (start after N cycled ids) and TF_Q35_BESIDE=N (step an N-token stream in each forward), which cover both cases: before this change, windows from 8151 keys kept 36 of 120 rows and trees 35, and a 151-key stream beside one from 8189 kept 4 to 11 of 120; after it all of them equal one-row steps on the 2B and 27B.
…fits its selector rank The integer reader accepted zero, so a conv_group_size or num_key_value_heads of 0 reached a remainder at admission; sizes and divisors are now positive (mask_token_id alone may be 0). The host staging area held 16 rows at rank 256, so a larger selector_rank overran it; it is now sized for the admitted rank. Config.parse takes the bytes, so a host test covers both.
The release's new modules called libc and POSIX directly, so zig build test and zig build native stopped building on Windows. They now go through sys, as the rest of the tree does: - std.c and std.posix calls become sys.c and sys.posix: recall's store, the rows store and its files, originals, the hip engine's timers and comm links, the XPU loader, tp_watch, the lanes' recorder, and their tests. The clock calls take CLOCK.MONOTONIC, a value Windows' clock_gettime accepts. - win_c gains flock (win_lock.zig: LockFileEx over a file's first byte, or over `.tensorfold-lock` inside a directory descriptor, AGAIN when held against) and _exit. win_posix gains IPPROTO, TCP.NODELAY and the keepalive options, SO.KEEPALIVE, SOCK.CLOEXEC (0), system.read and system.poll, and sockaddr.in's default family. - libjpeg-turbo, libtfvideo and RCCL load through sys.DynLib (turbojpeg.dll, tfvideo.dll on Windows); sys.libraryOf names a symbol's library where dladdr does elsewhere. - eaq_container's "inside the pack's folder" check takes either separator, so a Windows path's shards are not refused as outside. - The fake fabric's windows come from the page allocator rather than mmap. - nvcc gets -D_USE_MATH_DEFINES on Windows hosts, for vit.cu's M_SQRT2. - Refused by name on Windows: `tensorfold benchmark` (its serve child runs under POSIX signals and waits; its tests build elsewhere) and image URLs in media_fetch. - Tests that make symlinks skip without the privilege, and the owner-only mode checks skip, as Windows files have none. --drafter's Served also reserves the family's own streams, so native/cuda.zig stays within 600 lines.
…e on Windows, where an exclusive create follows a dangling one
…s and logits as tf_logprob_rows
…ng them for the projections after it
…un of rows a launch
…ries added in order
… with their parents and node chances
… children a node, chances from the selector's scores
…e computes; the norm writes four entries a store
…_tree.zig, comments one line within 120 columns
…wn trees' node chances from the nodes that landed (lanes: Absorb.offered)
8 tasks done
satindergrewal
added a commit
to satindergrewal/TensorFold
that referenced
this pull request
Oct 17, 2026
…ify window (TF_GLM_GREEDY_ROWS=0 keeps the per-row round trips). The count-span single all-gather variant corrupted picks nondeterministically - rank 0 send aliased the recv base while rank 1 did not, a mixed in-place pattern NCCL leaves undefined; the grouped count-2 gathers into a separate pairs_all buffer are bit-exact (sha c6b7f32e211c AB600 x2 default) and speed-neutral (98.9-103.4 vs 96.6-104.0 per-row). tf_glm_top1_rows kernel kept for the future single-launch pick. Credit: BobClawblaw PR ashhart#978 q35-speed launch surgery
satindergrewal
added a commit
to satindergrewal/TensorFold
that referenced
this pull request
Oct 17, 2026
…lves on the device (tf_glm_pick_greedy applies the host pick rule) straight into the MTP head ids slot, so no draft token crosses the host until one sync + one download at chain end - the profile showed the chain per-draft all-gather+synchronize round trips dominated non-GPU round time (5.7s of syncs, 77K eager launches). Bit-exact vs the host path (sha e6be52524388 AB600 and c803536e2eed LONG400 on both), sparse-phase LONG400 91.4 -> 94.6 tok/s (+3.3%), dense unchanged. Credit: Justin Casey decode-graph class + BobClawblaw ashhart#978 launch surgery
Contributor
|
@BobClawblaw, a GB10 run of this branch, the platform the description lists as not run, with #836 built the same way as the "before". Environment. One DGX Spark: NVIDIA GB10 (sm_121, 48 SMs, aarch64, 128 GB unified memory), Ubuntu 24.04, kernel 7.0.0-1019-nvidia, driver 580.178.04, CUDA 13.0 The 5090 table comes from the recipe harness's ruler at Decode, one stream (
|
| serve | build | code, T=1 | chat, T=1 | code, greedy | chat, greedy |
|---|---|---|---|---|---|
| DFlash2 drafts | #836 | 51.2 | 33.2 | 47.6 | 34.5 |
| DFlash2 drafts | this branch | 59.2 | 37.1 | 57.7 | 47.9 |
| change | +16% | +12% | +21% | +39% | |
plain (no --drafter) |
#836 | 13.4 | 13.4 | 13.3 | 13.2 |
plain (no --drafter) |
this branch | 14.2 | 14.1 | 14.2 | 14.1 |
| change | +6% | +5% | +7% | +7% |
- Most of the drafted gain comes from fewer rounds. Greedy replies of 128 tokens take 25 rounds instead of 29 on code, and 30 instead of 40 on chat (server log). A round got only a little shorter: about 92 ms before, 88 ms after. On the 5090 the launches this removes were 4.5 ms of a 27 ms round. Here they are a few ms of a round of about 90 ms, so the trees carry most of the gain.
tf-qwen35-cuda stepover 300 ids: 75.5 ms a token before, 70.9 ms after (13.2 to 14.1 tok/s).- For comparison, Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836's numbers here match your GB10 receipt on Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836 (
153d6b43, nvcc 13.3) within 4%: 50 / 32 / 46 / 34 drafted, 88 / 74 at four streams, 76.5 ms a step, 3.48 s cold.
Four streams (bench_concurrent.py --levels 1,4 --tokens 128 --reps 2, aggregate tok/s, median, mean of the two starts)
| serve | build | code, T=1 | chat, T=1 | code, greedy | chat, greedy |
|---|---|---|---|---|---|
| DFlash2 drafts | #836 | 86.7 | 65.8 | 89.5 | 75.0 |
| DFlash2 drafts | this branch | 84.2 | 71.7 | 95.3 | 76.5 |
| plain | #836 | 46.8 | 46.5 | 47.2 | 46.5 |
| plain | this branch | 49.3 | 48.9 | 50.0 | 49.4 |
At four drafted streams the gain is small, and sampled code is 3% slower. It was lower in both starts (84.0 and 84.3 against 87.7 and 85.6). The description lists shared rounds above two streams as not run, so these are the first numbers for them.
Prompt speed
bench_cache.py --require-cold, its 2,571-token fixture, --context 32768 --parallel 1 --drafter <DFlash2>, the first request's ttft_s, qualified both times: 3.41 s and 3.39 s before, 3.29 s and 3.28 s after.
Exactness
- Bits unchanged against Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836. Every reply's
token_shais equal between the two builds for the same request, in every run:- drafted,
bench_openai: 12 of 12 in each start pair (code and chat, T=1 seeded with three seeds, and greedy); - drafted,
bench_concurrent(alone, at four streams and"draft": false): 120 of 120 in each start pair; - plain: 12 of 12 and 120 of 120.
- drafted,
- Drafted equals plain. The drafted server's replies equal the plain server's for every request: 12 of 12 and 120 of 120, on both builds.
- Concurrent equals solo, drafted equals
"draft": false.bench_concurrent --levels 1,4 --alone --serial: 40 of 40 replies equal alone and 20 of 20 equal undrafted (10 distinct, printed once a level), 0 unequal, 0 failed. This holds for both builds, drafted and plain, and for every start. - Windows and trees.
tf-qwen35-cudaover 300 ids (the README's first 300 tokens):windowandtreeequalstepbyte for byte on both builds. This branch'sstep,window,treeandpromptoutputs are byte-identical to Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836's. As on Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836,prompt's rows are not byte-equal tostep's. - The 5090 receipt gives no
token_shavalues, so I could not compare replies across cards.
Tests on sm_121
tf-cuda-test:- the two new suites pass:
topk(1,008 rows equaltf_logprob_rows' k best) andq35-argmax(720 rows equal the host rule); sample(2,880 draws) andglue(1,293 of 1,293) pass;info,smoke,graph,launch-ex,ptx,symbols,cublaslt,norms,norm-rows,pocket,quant-ref,modeloptandexlr8also pass.
- the two new suites pass:
zig build test: 86/86 steps, 1,397 of 1,404 tests passed, 7 skipped, 0 failed.zig build kernel-contracton the GB10:SUMMARY 12 kernels: 12 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems. Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836's head prints the same line.
Not run
- The recipe harness ruler (prose / code / structured / list) and anything with qwen3_5 cuda: --kv-cache fp8, e4m3 attention caches with a scale a row #886's fp8 cache, so there are no 262k-context numbers.
--parallel 2serving andTF_NO_TREES=1.- The
test-golden,fuzz-cortheon,zig fmtand lean checks. - Qwen3.5-2B.
Nothing failed.
8 tasks done
Owner
|
Thanks @jschmied for the alternating GB10 runs and full-token comparisons. The one-stream gains are useful, and the four-stream sampled-code drop from 86.7 to 84.2 tok/s must stay beside them. This is not a uniform 37–73% gain on every platform or workload. We are carrying the focused increment onto the maintained Qwen engine, preserving image, cache and shared-draft contracts. Your results keep bf16 cache, context and compiler differences visible; they do not qualify the fp8/262K recipe or the unrun 2B and two-stream cells. The final merged candidate still needs its own full qualification and matched performance run. Posted by Ash's AI Assistant Synthena |
Open
Owner
|
Thank you, @BobClawblaw, for the CUDA decode increment and matched before/after measurements. This work moves with the CUDA lane changes to 1.1.1, the next patch release. The combined 64-row/shared-round build regresses lone-stream decode and startup, so it needs those costs resolved before release. We also need the combined build's numerical and resource receipts, drafted/plain and concurrent/solo comparisons, and cached resumes. Your branch's speed gains remain useful comparison points for that work. RTX PRO 6000 CUDA builds and synthetic checks passed, while the live contract stopped on a missing image binding and combined serving/resource reruns remain owed. Posted by Ash's AI Assistant Synthena |
ashhart
added a commit
that referenced
this pull request
Oct 19, 2026
Credit: TensorFold maintainers; inherited family 8 and #978 credits retained. Maintained release composition preserves the reviewed feature and its controls. Co-authored-by: BobClawblaw <288991054+BobClawblaw@users.noreply.github.com>
Owner
|
Thanks @BobClawblaw for #978. 1.1.1 includes the CUDA decode work with larger verification trees, shared stream rounds and batched DFlash proposals. Follow-ups size launches for live rows, avoid the redundant lone-stream gather and build straight-chain metadata directly. Lone-stream choices use their own complete-round measurements, with separate retry cooldowns for widths that did not repay their cost. Those scheduling repairs retain the kernel arithmetic. The earlier branch measurements remain separate from any new release speed result. Posted by Ash's AI Assistant Synthena |
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
Credit: TensorFold maintainers; inherited family 8 and ashhart#978 credits retained. Maintained release composition preserves the reviewed feature and its controls. Co-authored-by: BobClawblaw <288991054+BobClawblaw@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Decode speed for Qwen3.5/3.8 on #836's CUDA engine, with the model's bits unchanged: the drafter's top-k, the projections' group sums and the greedy argmax take far fewer and wider launches, and DFlash2 drafts best-first trees as the 27B engine of #608 did. Stacked on #836 (
26e9b686); only the last 10 commits are this pull request's.cuda/sample:topkRows, a row's k best over many blocks (4,096 entries a block, then a merge), with the same ids and logits astf_logprob_rows; the drafter uses it in place oflogprobRows, whose one block a row took 0.7 ms a round on the 27B's vocabulary.qwen3_5: a projection input's 64-group sums are made once (projAgainfor the projections that share an input), the norm writes them for the projections after it, andtf_q35_sumssums a group a thread in 16-byte loads, each group's entries added in order.qwen3_5: greedy rows' argmax in two stages (tf_q35_argmax_slices,tf_q35_argmax_merge), one pair of launches a run of greedy rows instead of one 1,024-thread block a row.lanes: a drafter may grow its own trees (Facts.tree_nodes), read back by the round loop with their parents and node chances (Drafter.tree,Drafter.chances);Absorb.offeredtells it how many nodes of its tree the verified window carried. Opt in: a drafter withouttree_nodesis unchanged.qwen3_5: DFlash2 grows best-first trees of up to 15 nodes (four children a node, the selector's parent-conditioned scores, softmax scale 1.5, as the 27B engine's draft policy), in 16-row blocks as the reference'sdflash2.block(the checkpoint'sblock_sizeis 8, which capped trees at depth 7). Node chances come from the nodes that landed, by own log-chance (greedy rounds', then every round's).TF_NO_TREES=1keeps chains and lane trees;TF_DRAFT_NODESsets the node cap. The chain, lane and tree selection now live incuda_draft_tree.zig.qwen3_5: the delta rule loads a row's inputs while the row before computes, and the norm reads and writes four entries a load and store.Receipt
Environment: Linux, RTX 5090 (32 GB, sm_120), driver 615.71, CUDA 13.3
nvcc, Zig 0.17.0,-Dsm=120. This branch at6f7acde6on Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836 at26e9b686.TensorFold/Qwen3.8-27B-MLX-4bitat22d8d538withz-lab/Qwen3.8-27B-DFlash2at50307d4c.Exactness:
token_shaequals Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836's unmodified build on the same prompts (4 prompts x 3 repeats), at bf16 and, merged with qwen3_5 cuda: --kv-cache fp8, e4m3 attention caches with a scale a row #886, at fp8.tf-qwen35-cudaover 300 ids:windowandtreeequalstepbyte for byte, andstepequals the unmodified build's bytes.bench_concurrent.py --levels 1,2 --tokens 128 --reps 2 --alone --serial,--parallel 2 --context 16384 --kv-cache fp8(with qwen3_5 cuda: --kv-cache fp8, e4m3 attention caches with a scale a row #886): 0 unequal, 0 failed, code and chat at T=1 seeded and T=0.tf-cuda-test topk(new): 1,008 rows equaltf_logprob_rows' k best, widths 10 to 248,320 with NaN and -inf rows.tf-cuda-test q35-argmax(new): 720 rows equal the host rule, ties, NaN, +inf and an all -inf row included.Decode speed, the recipe harness's ruler (one stream, 256 tokens, tok/s),
--context 262144 --parallel auto --kv-cache fp8on Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836 + qwen3_5 cuda: --kv-cache fp8, e4m3 attention caches with a scale a row #886, before (qwen3_5 cuda: --kv-cache fp8, e4m3 attention caches with a scale a row #886's head) and after (merged with this branch):Rounds for the ruler's four prompts fall from 54/74/49/40 to 49/66/35/31; the rest is shorter rounds (the top-k, sums and argmax took 4.5 ms of a 27 ms round).
Prompt speed:
bench_cache.py --require-cold, its 2,571-token fixture, coldttft_s2.26 s before and 2.22 s after, qualified,--context 32768 --parallel 1.Tests run:
zig build test(86/86 steps),test-kernel-contract,test-golden(309 of 319 equal, 0 unexpected),fuzz-cortheon,zig fmt --checkandtools/zig/lean_check.py(517 problems, as on Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836) all pass;tf-cuda-test topk,q35-argmax,sample,glue(1,293 of 1,293) andsmokepass.zig build kernel-contracton the 5090:SUMMARY 12 kernels: 12 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 6 limit problems, the same line on Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836's head (register records of kernels this does not touch); none of thetf_q35_*orsample.cukernels are in the CUDA registry yet, these included.Not run, and why: Windows and the other CUDA cards (sm_86, sm_89, sm_121); the 5090 on Linux only here. Metal: no Metal source changes. Qwen3.5-2B served: the 2B's engine path is the 27B's, and the kernel tests cover the shapes, but no 2B receipt. Shared rounds above two streams.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.SUMMARYline is in the receipt. (TheSUMMARYline is above; the new kernels have their own exactness tests but no contract records, as Windows: the native CUDA server, and Qwen3.5-2B and Qwen3.8-27B on CUDA #836's own kernels.)