Repository navigation
cuda: fix an experts-rows test race and measure the ModelOpt checkpoint on one GB10 - #1
Merged
akol1 merged 2 commits intoOct 18, 2026
Conversation
Rig.zeros cleared each buffer with cuMemsetD8 on the default stream, while the kernels run on a non-blocking stream that does not wait for it. On a GB10 (sm_121) the clear of the one-row reference's output sometimes landed after the decode kernel wrote it, and experts-rows failed with an all-zero or partly zero reference: 8 of 30 runs at beec1aa, 0 of 30 with this change, alternated on one machine. Clear on the rows' stream.
The recipe's measurements covered only the MLX affine checkpoint. Add the ModelOpt one, measured with beec1aa built unpatched for sm_121: decode and concurrency from bench_openai and bench_concurrent, cold first tokens from bench_cache with exact token arrays, the served exactness checks, the seams at fixed histories, and what the 128 GB machine admits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For ashhart#952 (ashhart#952). Two commits on top of
beec1aad6, independent of each other, so either can betaken alone.
What this changes
test(cuda): clear experts-rows buffers on the kernels' stream.Rig.zerosinzig/tests/cuda/experts_rows_tests.zigcleared each buffer withfill8(0, null), which iscuMemsetD8on thedefault stream. The kernels run on a non-blocking stream that does not wait for it, so on sm_121 the clear of the
one-row reference's output sometimes landed after the decode kernel wrote it, and
tf-cuda-test experts-rowsfailed:
FAIL experts_rows SwiGLU rows: first difference at byte 0, 76800 of 76800 words differ. The change clearson the rows' stream. One line, test only.
docs: measure the ModelOpt FP4-Local checkpoint on one GB10. A subsection, "The ModelOpt checkpoint on oneGB10", in
docs/recipes/flash-next-cuda.md: decode, concurrency, cold first tokens to 131,072 tokens, theexactness and seam checks, and what a 128 GB machine admits. All measured with
beec1aad6built unpatched. Docsonly.
Receipt
beec1aad6c967f6e32aa14a3fd21ca8358710279, built with-Dnvcc -Dsm=121 -Doptimize=fast, nvcc 13.3,zig 0.17.0, served with
TENSORFOLD_CUDA_KERNELS=native.nvidia/Qwen3.8-Flash-Next-FP4-Localrevisionb9ce18a6887e18c089fdea8509a33e63429a3300. No MLX or PyTorch.--prompt-cache-gib 0):bench_concurrent.py --alone --serial --levels 1,4,8 --tokens 256 --reps 3, three runs: concurrent equals solo78/78 and drafted equals
"draft": false6/6 in each.token_sha256. A second conversationturn resumed from the prompt cache (
--no-drafts) equals the same turn computed without it.flashnext-fidelity-singleagainst-serialat 7 fixed histories of 20 to 6,496 tokens: 7/7 byte-equal.40.7-41.1 tok/s (
fibonacci-raw/gpu-chat-no-think, one stream).2k / 8k / 32k / 64k / 128k: 1.83 / 7.11 / 29.2 / 60.1 / 131.7 s (
bench_cache.py --require-cold, exact arrays).tf-cuda-test experts-rows, thirty runs each, alternated, same build flags: before 8/30 failed, after 0/30.tf-cuda-test kernel-contracton sm_121:SUMMARY 32 kernels: 32 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems.zig build test104/104 steps, 1,483/1,490 passed, 7 skipped (libjpeg-turbo absent);test-kernel-contract,test-goldenandfuzz-cortheonpass.test-cuda-host test-flashnext-contract-hostfor sm_121: 204/204.lean_check.py: 0 new.zig fmt --check build.zig zig: clean.fill8(0, null)with non-blocking streams (exlr8_tests,bench,oracle_tests,exlr8_dense_tests,libs_tests,runtime_tests). I have not seen them fail and did not change or check them.Checklist
code changes.)
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.SUMMARYline is above.)
test: 8/30 failed before, 0/30 after.)